For each outcome, the control group (no letter) was compared with all of the intervention groups combined, with tailored and standard letters groups, and with mailed and electronic delivery methods. Comparisons by χ2 analysis were performed first. When P was <.05 by χ2 analysis, pairwise comparisons were performed.
aP < .05 for pairwise comparison with the no letter group, adjusted via the Holm method.
bP < .05 for pairwise comparison within the letter group, adjusted via the Holm method.
eFigure. Study Design
Protocol and Letter Copies
Data Sharing Statement
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Cappola AR, Schriver ER, Mowery DL, et al. Effect of Targeted Messaging on Return to In-Person Visits During the COVID-19 Pandemic: A Randomized Clinical Trial. JAMA Netw Open. 2021;4(6):e2115211. doi:10.1001/jamanetworkopen.2021.15211
During the COVID-19 pandemic, many patients have delayed care.1-4 We performed a randomized clinical trial to determine whether targeted messaging could improve the return to in-person visits.
A randomized, double-blind, clinical trial (1:9 no letter vs letter; 1:1 tailored vs standard letter, 1:1 electronic vs mail) was conducted at 3 hospitals in the University of Pennsylvania Health System (eFigure and eAppendix in the Supplement 1). The University of Pennsylvania institutional review board deemed the study exempt and waived the need for consent because they found minimal risk to patient privacy. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.
We identified adult patients aged 18 years or older who had canceled in-person appointments, procedures, or surgical procedures from March 9 through June 7, 2020, and had not rescheduled with clinicians in the Heart and Vascular Service Line and Penn Orthopaedics. Patients were randomized to groups that received no letter, a tailored letter, or a standard letter inviting them to reschedule canceled appointments. The tailored letter used an inoculation strategy in which patients’ perceived threats to safety were acknowledged and addressed early in the message,5,6 with age specificity for patients 65 years or older and informational and word count equivalence between letters. The protocol and letter copies are included in Supplement 2. Messages were sent to patients on June 12 through June 17, 2020, with a final follow-up on July 17, 2020.
Patients self-reported their race based on the following predefined categories: White, Black, Asian, Pacific Islander, American Indian, or other. Only data from White patients and Black patients were included because of the small number of patients who identified with the other categories listed for race. Race was included in the study because prior data suggested that there were differences in access to health care between Black individuals and White individuals during the COVID-19 pandemic.
The primary outcome was the percentage of patients who had an in-person visit, procedure, or surgery with a clinician in the service line within 1 month of sending the letter. Secondary outcomes included the percentage of patients who had a telemedicine visit within 1 month or who scheduled a visit (in-person or telemedicine) within 1 month for any future date.
We estimated a 90% power to detect a 2% difference between letter types and a 4% difference between delivery methods. The control group was compared with all groups combined, with letter types, and with delivery methods for each outcome. A χ2 analysis was performed, with pairwise comparisons adjusted by the Holm method. The percentage of patients who scheduled a future visit were compared by letter type and by delivery type overall and stratified by age, sex, race, median household income, insurance type, and service line, with testing for interactions. Analyses were conducted using R, version 3.63 (R Project for Statistical Computing). P values were 2-sided, and statistical significance set at P < .05.
Of 38 493 patients with cancellations, 27 373 rescheduled and 11 120 (28.9%) were randomized (Figure). The median age was 59.7 (range, 18-100) years, 6074 (55%) were women, and 6830 (61%) were White patients.
In-person return visits within 1 month were low and not significantly different between patients who did and did not receive a letter (503 [5.0%] vs 45 [4.1%]; P = .18) (Figure). Patients receiving any letter vs no letter had a higher percentage of telemedicine visits and future scheduled visits within 1 month (telemedicine visits: 132 [1.3%] vs 4 [0.4%]; P = .006; future visits: 1439 [14.4%] vs 130 [11.7%], P = .02).
Patients receiving the tailored letter vs standard letter scheduled more visits, achieving statistical significance overall (768 [15.3%] vs 671 [13.4%]; P = .006) and for individuals younger than 65 years, women, Black patients, White patients, patients with commercial insurance, patients with lower household income, and patients receiving orthopedic services (Table). For the subset randomized by delivery type, patients with a household income below the median were more likely to schedule a visit when mailed a letter vs receiving an electronic letter, with a statistically significant interaction by household income (230 [16.6%] vs 177 [13.0%]).
In this randomized clinical trial, we found that a large proportion of patients who canceled visits and procedures early in the COVID-19 pandemic did not reschedule once reopening occurred. A single message targeted directly to these patients did not affect the return to in-person visits within 1 month, but it resulted in a small increase in reengagement through telemedicine and rescheduling of future visits. Study limitations include limited outcomes and unknown generalizability to other locations and times. Strategies to improve patient engagement, with attention to message framing and delivery mode, are needed to encourage continuity of health care in the era of COVID-19.
Accepted for Publication: April 27, 2021.
Published: June 30, 2021. doi:10.1001/jamanetworkopen.2021.15211
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Cappola AR et al. JAMA Network Open.
Corresponding Author: Anne R. Cappola, MD, ScM, Division of Endocrinology, Diabetes, and Metabolism, Perelman School of Medicine at the University of Pennsylvania, 3400 Civic Center Blvd, Bldg 421, 12th Flr, Philadelphia, PA 19104-5160 (email@example.com).
Author Contributions: Dr Cappola had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Cappola, Ives, Gonzales, Cappella.
Acquisition, analysis, or interpretation of data: Cappola, Schriver, Mowery, Wollack.
Drafting of the manuscript: Cappola, Mowery, Cappella.
Critical revision of the manuscript for important intellectual content: Cappola, Schriver, Wollack, Ives, Gonzales, Cappella.
Statistical analysis: Schriver.
Obtained funding: Ives, Gonzales.
Administrative, technical, or material support: Mowery, Wollack, Ives, Gonzales.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study received funding from the University of Pennsylvania Health System.
Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Data Sharing Statement: See Supplement 3.
Additional Information: ClinicalTrials.gov Identifier: NCT04418947.
Create a personal account or sign in to: