Factors Associated With Increased Collection of Patient-Reported Outcomes Within a Large Health Care System

Key Points Question What are the characteristics associated with improving the collection of patient-reported outcomes (PROs)? Findings This cohort study of 205 clinics in Massachusetts that launched a PRO program found that clinician engagement, administrative engagement, and previous PRO collection experience were associated with increased collection rates. Having a clinical champion, payer incentives, and a higher collection rate during the initial month of the program were also associated with significantly increased collection rates. Meaning These findings suggest that health care systems that seek to implement PRO programs should focus on clinician and administrative engagement to maximize the potential for programmatic success.


Introduction
In recent years, there has been increasing interest in the collection of patient-reported outcomes (PROs), which may be the best measure of delivering value in health care, demonstrating that a patient's symptoms are improving or worsening through their interaction with the health care system. A focus on collection of PROs can address the discordance between the patient and the physician as to the gravity and severity of symptoms, [1][2][3] and use of PROs has led to improved overall survival in the setting of metastatic cancer. 4 For these reasons, payers and employers are interested in the collection of PROs to assess the value of care delivered.
Despite the benefits of PRO collection, the widespread implementation of routine collection has been limited owing to clinician, staff, and patient reluctance, inadequate resources to deal with positive or unexpected results, concerns for how the data will be used, and workflow and technology challenges. 5 While some organizations have published descriptions of their PRO programs in specific disciplines, there is a paucity of information on large-scale systemwide implementations that include diverse specialties and clinical settings. Even less is known about why individual clinics may succeed or fail in the collection of PROs and what resources must be in place to ensure successful collection.
The objective of this study was to prospectively assess the factors associated with success or failure of a clinic's PRO program within a diverse, integrated health care system.

Study Setting, Participants, and Implementation Model
This cohort study was conducted within the Partners Healthcare system, a large, nonprofit health care system in Massachusetts. Partners Healthcare comprises 2 academic medical centers (Brigham and Women's Hospital and Massachusetts General Hospital) as well as 7 community hospitals, 3 specialty hospitals, and a large network of community-based physician office practices. In 2014, Partners Healthcare deployed a central infrastructure to support the collection of PROs throughout the system. Collection of PROs was voluntary, and the implementation model allowed clinical specialties to choose whichever condition, PRO, and PRO measure (PROM) they deemed clinically appropriate. Standardized support included procurement of tablets, optimization of clinic wireless internet, building the PROMs in the electronic health record (EHR), assigning the PROMs to patients, and ensuring that the PROMs flowed into the enterprise data warehouse in a high-fidelity fashion.
Each clinic was assigned a project specialist who provided training to the front desk staff, clinic administrators, clinic nurses, medical assistants, and physicians on how to use PROMs and find them in the EHR and best practices (if they existed) for that particular PROM. It was the responsibility of the clinic to enforce that all clinicians and administrators received training. Weekly reporting on collection rates was centrally provided to clinics, but it was the responsibility of clinic administration to act on these results should collection rates decrease.
For our analysis, clinics were included if they launched a PRO program between March 15, 2014 (program inception), and December 31, 2018. To be included in the study, clinics had to have at least 6 months of collections data. This report follows Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies, and institutional review board approval and participant consent were waived per Partners Healthcare protocol for studies undertaken for the sake of quality improvement.

Data Sources
The study population included clinics that launched PRO collection programs between March 2014 and December 2018. Clinical characteristics were recorded by project specialists in project logs at the time of clinic launch and in an ongoing manner. Data detailing number and type of completed PROMs questionnaires, as well as collection rates, were obtained from our enterprise data warehouse. Each PROM was assigned in questionnaire sets assigned for a particular indication (eg, the PROM for

JAMA Network Open | Health Policy
Factors Associated With Increased Collection of Patient-Reported Outcomes orthopedics knee pain included Patient-Reported Outcomes Measurement Information System [PROMIS] Global-10, PROMIS Physical function, Knee Injury and Osteoarthritis Outcome, PROMIS Anxiety, and PROMIS Depression questionnaires). No more than 1 questionnaire set was administered per any given patient encounter. For this analysis, the unit of analysis was completion rate per assigned questionnaire set.

Outcome Measures
Our primary outcome was the mean PRO collection rate per clinic, defined as the clinic's mean collection rate during the most recent 6 months of collection. Clinics were excluded if they did not have at least 6 months of data, were missing more than 3 months of data within the most recent 6 months, or had incomplete assessment of the included clinical characteristics. A secondary outcome was the primary outcome dichotomized at a mean PRO collection rate of 50% or greater vs less. In this way, a clinic was designated as successful if its mean collection rate in the 6 months prior to January 2019 was 50% or greater.

Clinical Characteristics Potentially Associated With Collection Rates
To our knowledge, there are no previously published variables that are associated with the success of a PROs program. At time of program inception, 9 characteristics were believed to be relevant based on previous large-scale program implementations within Partners Healthcare and basic tenets of implementation science 6 and included in our model. To minimize bias, variables were recorded in a prospective fashion by project specialists who were not involved in the design of this study and stored in project logs.

Academic Medical Center
Partners Healthcare has clinics located in academic medical centers as well as freestanding community health clinics. Clinics were stratified by whether they were located in an academic medical center vs a community center.

Early Adoption
Within the early years of our program, there was a learning curve on how to implement a PROs program. Clinics were labeled as early adopters if they launched a PRO program in the first 2 years of our program, from March 2014 to December 2016.

Payer Incentive
Partners Healthcare includes PRO collection in several specialties in performance-based contracts with payers. If a clinic's collections were included in a payer contract with financial penalty at stake, we listed it as having a payer incentive.

Previous PRO Collection
Some clinics collected PROs on paper prior to our electronic platform. These clinics were categorized as having previous PRO collection.

Clinician Engagement
Our implementation model included standardized physician training on PROMs. The rate at which physicians participated in this training was overseen by each individual clinic. If more than 50% of clinicians in a clinic were trained, the clinic was listed as having high clinician engagement.

Leadership Mandate
Clinics were classified as having leadership mandate if the PRO program was launched at the request of the chair or senior medical executive of a department. In addition, clinics were included as having leadership mandate if departmental leadership required PROMs to be collected as part of a quality incentive for the department.

Clinical Champion
Clinics were considered to have a clinical champion if there was a physician, outside of the designated department leader and physically located within the clinic, who took responsibility as the clinical contact for the PRO program and was listed as the clinician responsible for performance.

Platform Transition
In the early years of the PRO program, PRO collection occurred electronically via a third-party vendor and was stored outside the EHR. By January 2016, the entire system had converted to a single EHR (Epic Systems) and our PRO collection platform was transitioned from the third-party vendor to the EHR solution. The EHR platform allowed for collection either in the clinic via tablet or kiosk or at home via the patient portal. During the transition, programs that had previously collected PROs using the third-party vendor might have PRO collection interrupted for months. If a clinic originally used a third-party non-EHR vendor, they were listed as undergoing platform transition.

Initial Collection Rate
In addition to the 9 clinic characteristics considered main variables, we considered initial collection rate for its potential association with long-term performance in the PRO program. Initial collection rate was defined as the collection rate during the first month after PRO program launch that a clinic's collection rate exceeded 10%. We selected this threshold because low collection rates can be achieved via the patient portal alone with no engagement of the clinic. A cutoff collection rate of greater than 10% ensures clinic engagement via distribution of tablets in the waiting room.

Statistical Analysis
Our primary analysis used a linear regression model with each clinic's mean collection rate as the dependent variable and 10 clinic characteristics as independent variables. Since our goal was to identify modifiable risk factors that could become interventions for clinics wishing to initiate similar PRO programs, our model did not include other clinic characteristics, such as specialty. Collinearity between the risk factors was assessed using the Belsey-Kuh-Welch diagnostic. Since the highest condition index was 4.1, well below the problem threshold of 30, we believe that the model with all risk factors simultaneously present does not mask any important effects. Model residuals were also tested and passed the Shapiro-Wilk test for normality (P = .18) Additionally, although incremental improvement of PRO collection is important, programs may desire to collect PROs at rates high enough to make valid inferences about the population being studied. To better understand the attributes of clinics that collected PROs at a high rate, we also ran a logistic regression model to identify clinic characteristics associated with success (ie, a collection rate of Ն50%).
Each 10% increase in collection rate in the initial month was associated with a 2.6% (95% CI, 1.2%-4.9%; P < .001) increase in the final collection rate. Presence in an academic medical center, leadership mandate, early adoption, and platform transition were not associated with increased rate of collection.

Discussion
This cohort study examined the factors associated with increased PROM collection rates and PRO program success in the largest PRO program reported in the literature, to our knowledge. After collecting more than 4 million questionnaires across hundreds of clinics, we found that several factors were associated with increasing a clinic's PRO collection rate. The strongest associations were with physician and administrative engagement, but previous collection of PROs, presence of a clinical champion, and inclusion in a payer incentive contract were also associated with increased performance. In addition, clinics that performed well in their initial month of collection also tended to perform better over time. When examining clinics based on whether they were successful (ie, Ն50% collection), all of these factors were still relevant with the exception of inclusion in a payer incentive contract.
A key takeaway from our findings is that health care systems interested in collecting PROs should focus primarily on engaging physicians in the value of the process and training them to be adept at the collection of PRO data. Additionally, administrators should not be ignored in the process, as their engagement and oversight of collection numbers was strongly associated with success. Another key finding is that mandating PRO collection, either by local leadership or payers, was not significantly associated with the creation of a successful program. Inclusion of a clinic's PROs in a payer contract was associated with an increase in PRO collections; however, it was not associated with clinics collecting successfully in a sustained fashion. This finding undermines the idea that payers could drive PRO collection to a meaningful level based on the implementation of PRO performance measures. 7 Instead, our data suggest that the most important factor associated with a clinic's success is physician engagement with the process.
An additional, pragmatic finding of our study is that improving a clinic's collections in the first month after launch was associated with improved collection over time. An additional pragmatic finding of our study is that improving a clinics collection in the first month after launch was associated with improved collection over time. This supports the importance of ensuring that local wireless internet, assignment logic of questionnaires, and training of physicians and administrators prior to launch. This clinical observation was supported by our models. Over time, we have begun to treat the month of launch as a critical junction in the lifecycle of a clinical PRO program and to focus exhaustively on ensuring that both technical and human errors are reduced to a minimum.
Finally, although the collection of PROs is an area of intense interest, detailed reports on implementation are generally lacking and either focus on implementation steps or single disease sites, such as PROs in patients at orthopedics clinics, or questionnaires assigned rather than questionnaires completed. 5,8,9 This study is an important contribution to the literature as it outlines some modifiable practices and characteristics that are associated with improving PRO collection.
Furthermore, it demonstrates that even with costly and intensive central support, programs are less likely to succeed without physician engagement. This finding can inform the strategy of other health care systems that wish to implement PRO programs and serves as a cautionary message to systems seeking to drive the collection of PRO data via mandate or financial penalty.

Strengths and Limitations
Our study has some strengths. One is that it is a prospective analysis of hundreds of medical and surgical clinics launching a PRO program within a standardized implementation model in an integrated delivery system. The standardized support model allowed us to make direct comparisons among attributes of the clinics, their collection rates, and whether they ultimately developed a successful program.
Our study also has some limitations. One is that it is impossible to collect all attributes that may affect an individual clinic's collection rate. For example, clinician engagement was the factor most strongly associated with improved collection of PROs; for the purposes of this study, we defined clinician engagement as more than half of the physicians in a clinic trained on PROMs. Yet this simple concept includes a tremendous amount of nuance. The Consolidated Framework for Advancing Implementation Science 6 would describe physician engagement as a complex concept at the nexus of outer and inner domains of a health care setting. Therefore, engagement is a reflection of how those physicians interact with external organizations, their individual patient panel, peer pressure from their own partners, competition with colleagues in other health care settings, external policies, structural characteristics of the organization, individual autonomy, and cultural attitudes about change. Defining engagement as attending a clinical training, by necessity, oversimplifies the concept. Furthermore, in a health care setting in which physicians have broad autonomy and freedom, such as Partners Healthcare, electively presenting for training inherently means a higher degree of engagement than in a health care system in which physicians are forced to undergo such a training. As such, caution must be taken when applying these results to other organizations. An additional limitation is that while the prospective collection of data minimizes the potential for bias, clinic attributes were assessed by project specialists and could be subject to some degree of bias in interpretation. Furthermore, some variables (eg, administrative surveillance) were reliant on clinic reporting.

JAMA Network Open | Health Policy
Factors Associated With Increased Collection of Patient-Reported Outcomes