EHR indicates electronic health record.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Pezold ML, Pusic AL, Cohen WA, et al. Defining a Research Agenda for Patient-Reported Outcomes in Surgery: Using a Delphi Survey of Stakeholders. JAMA Surg. 2016;151(10):930–936. doi:https://doi.org/10.1001/jamasurg.2016.1640
What are the primary areas that future patient-reported outcomes (PROs) research should focus on?
An iterative web-based interface was used to create a modified Delphi survey for use at the inaugural Patient-Reported Outcomes in Surgery (PROS) Conference to survey conference registrants including surgeons, PROs researchers, payers, and other stakeholders. The survey identified (1) PROs in the decision-making process, (2) challenges to integrating PROs into the electronic health record, and (3) PROs and the measurement of quality as the 3 most important areas for future research.
The research agenda generated from the PROS Conference will help researchers, surgeons, and funders identify crucial areas of future PROs research in surgery.
Identifying timely and important research questions using relevant patient-reported outcomes (PROs) in surgery remains paramount in the current medical climate. The inaugural Patient-Reported Outcomes in Surgery (PROS) Conference brought together stakeholders in PROs research in surgery with the aim of creating a research agenda to help determine future directions and advance cross-disciplinary collaboration.
To create a research agenda to help determine future directions and advance cross-disciplinary collaboration on the use of PROs in surgery.
Design, Setting, and Participants
An iterative web-based interface was used to create a conference-based, modified Delphi survey for registrants at the PROS Conference (January 29-30, 2015), including surgeons, PRO researchers, payers, and other stakeholders. In round 1, research items were generated from qualitative review of responses to open-ended prompts. In round 2, items were ranked using a 5-point Likert scale; attendees were also asked to submit any new items. In round 3, the top 30 items and newly submitted items were redistributed for final ranking using a 3-point Likert scale. The top 20 items by mean rating were selected for the research agenda.
Main Outcomes and Measures
An expert-generated research agenda on PROs in surgery.
Of the 143 people registered for the conference, 137 provided valid email addresses. There was a wide range of attendees, with the 3 most common groups being plastic surgeons (28 [19.6%]), general surgeons (19 [13.3%]), and researchers (25 [17.5%]). In round 1, participants submitted 459 items, which were reduced through qualitative review to 53 distinct items across 7 themes of PROs research. A research agenda was formulated after 2 successive rounds of ranking. The research agenda identified 3 themes important for future PROs research in surgery: (1) PROs in the decision-making process, (2) integrating PROs into the electronic health record, and (3) measuring quality in surgery with PROs.
Conclusions and Relevance
The PROS Conference research agenda was created using a modified Delphi survey of stakeholders that will help researchers, surgeons, and funders identify crucial areas of future PROs research in surgery.
Patient-reported outcomes (PROs), as defined by the US Food and Drug Administration, are “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else.”1(p 2) In practice, this process entails measuring a subjective outcome, such as function after knee arthroplasty, with a questionnaire. Publications2-4 on PROs in the surgical literature have tripled during the last decade. The increasing recognition among surgeons that many postoperative outcomes, such as functional improvement or symptom severity, are best measured by the patient and the increased availability of measures for historically subjective topics have undoubtedly contributed to this surge in publications. Currently, PROs play an important role in the planning of comparative effectiveness research and are used as primary outcomes in surgical trials.5 The Institute of Medicine and the National Quality Forum have long advocated that patient-centered care should be a requirement in modern health care, and this perspective has influenced health care deliberations during the last decade.6 Furthermore, the Patient Protection and Affordable Care Act has encouraged patient-reported data collection in surgery by creating new payment models through the Centers for Medicaid & Medicare Services (CMS) and by targeted funding of research from the Patient-Centered Outcomes Research Institute.7,8 These new models underscore the increasing importance of PROs in surgery.
Despite the potential value of PROs in surgical care, many methodologic and logistical concerns remain.9 Addressing these issues is difficult in a research climate of increasingly scarce funding.10 Thus, researchers and funding agencies need input and clarity from stakeholders to better prioritize research and improve collaboration among institutions. Furthermore, options for surgical techniques and technologies are rapidly expanding while health care resources are shrinking. To better understand the outcomes of surgery from the patient perspective, there is no better time than now for the surgical and research communities to come together and reach consensus regarding the most important and timely issues with PROs research in surgery.
To address these issues, the Patient-Reported Outcomes in Surgery (PROS) Conference was formed, with sponsorship from the Plastic Surgery Foundation and the Agency for Healthcare Research and Quality. The conference brought together stakeholders from diverse fields, including payers, patient advocates, surgeons, researchers, industry representatives, regulators, and health information technology vendors. The 2-day conference was held in Washington, DC, and included panel discussions on current PROs research in surgery within the areas of clinical care, comparative effectiveness, patient access, psychometric development, surgical trials, and quality, as well as mapping out the future directions for each field. The specific goals of the conference were to improve the accessibility and interpretability of PROs data for patients and clinicians, to develop a consensus on the methodologic issues of PROs measurement, and to develop a research agenda for PROs measurement in surgery. The formation of a research agenda should prioritize research questions deemed to be timely and important by stakeholders to guide future collaboration and funding. To meet this aim, the conference leadership conducted a Delphi survey of stakeholders to develop an agenda on future PROs research in surgery.
This study used a modified Delphi survey to achieve formal group consensus, maximizing dialogue through anonymous, structured feedback.11,12 To facilitate the use of the Delphi model, an expert panel was assembled before the opening of the conference. The panel, composed of conference leaders and methodologic experts, developed the study schema and consensus criteria on the basis of a 3-round, web-based survey (Figure 1). The study was approved and informed consent was waived by the Memorial Sloan Kettering Cancer Center Institutional Review Board.
Emails containing a link to a web-based survey were distributed to all conference registrants 1 week before the start of the conference (round 1), which was held January 29 to 30, 2015. Participants were asked to anonymously identify timely and important PROs research topics through 7 open-ended questions related to clinical care, comparative effectiveness, patient satisfaction, and quality metrics. There was no limit on the length of the responses. Item submissions were reviewed in parallel by two of us (M.L.P. and W.A.C.). This process involved separating compound responses into individual items, summarizing submissions into concise items, and categorizing items into generic themes. After all submissions were reviewed, identical items were deleted, and similar items with overlapping content were consolidated into broader concepts. The expert panel then evaluated the consistency of each researcher’s item reduction and endorsed the selection of research items for round 2.
All conference attendees were invited to participate in a web-based survey during the conference (round 2). Participants were given the final research items from round 1, in randomized order, and were asked to rank the items by research importance using a 5-point Likert scale, with high and low research priority as anchors. In addition, an open-ended question prompt at the conclusion of the survey allowed participants to submit additional research questions. Mirroring the process in round 1, all new research questions submitted in round 2 were subjected to qualitative review and consolidation into final research items. Round 2 concluded with the completion of the conference.
After the conference, all attendees were asked, via email, to participate in a web-based survey, regardless of whether they participated in previous rounds. Email reminders to complete the survey were distributed weekly, and each registrant was limited to 1 survey response each. The top 30 items by mean priority score from round 2, along with new item submissions, were distributed for final ranking by research priority to reach the a priori goal of a 20-item research agenda. To reduce the possibility of a ceiling effect, a 3-point Likert scale was used in round 3, and participants were encouraged to rate only 10 items as high research priority. As in round 2, item order was randomized. The top 20 items by mean Likert score were selected as the consensus research agenda.
Of the 143 people registered for the conference, 137 provided valid email addresses. Potential respondents were invited to participate during each round. There was a wide range of attendants, including individuals from several surgical subspecialties, payers, regulators, and patient advocates, with the 3 most common groups identified as being plastic surgeons (28 [19.6%]), general surgeons (19 [13.3%]), and researchers (25 [17.5%]). Table 1 presents the basic demographic characteristics of respondents by round. During item generation (rounds 1 and 2), participants were given the option to specify their occupation (Table 2). Specific occupation demographics were not collected during round 3 because there was a desire to keep the survey as short as possible to increase response rates. In round 1, a total of 83 participants (60.6% of conference registrants with email addresses) submitted 356 responses to open-ended questions. After review, a total of 459 research items were submitted, with a mean of 5.5 items submitted per participant.
After item reduction was performed and the expert panel reviewed the results, there were 53 items within 7 themes: clinical care, comparative effectiveness, data management, ethics, performance measurement, education, and other. In round 2, a total of 54 participants (39.4% of emailed registrants) responded to email invitations during the conference. Responders ranked the 53 items by research priority and submitted 9 new items, which were reviewed and consolidated to 3 new items. In round 3, a total of 59 participants (43.1% of emailed registrants) responded to survey invitations. The top 20 items from round 3 were selected as the consensus research agenda for future PROs research (Table 3). Ranking of item importance remained stable during the rounds, with only 2 items from the top 20 of round 2 failing to make the final research agenda. Stakeholder participation from rounds 2 and 3 was predominantly from experienced PROs researchers, with 81 participants (43.4%) reporting more than 7 years of experience.
Using a modified Delphi approach, we engaged an international stakeholder group of surgeons, researchers, patient advocates, funding agency representatives, health information technology vendors, and regulators to reach consensus on future PROs research priorities in surgery. The final results of the survey contain the top 20 items from more than 450 topics initially submitted by participants. Our results represent the first consensus-driven, surgery-focused PROs research agenda to date. Within the research agenda, 3 themes emerged as priorities for future research in surgery: (1) PROs in the surgical decision-making process, (2) challenges to integration of PROs in the electronic health record (EHR), and (3) PROs and the measurement of quality (Figure 2). Among all items, the decision-making process was rated of high importance, with “impact of PROs on patient and/or provider decision-making” the top-ranked item for both rounds.
It is not surprising that the use of PROs in the clinical decision-making process was the highest importance. Decision making in surgery has traditionally relied on surgeon experience and established objective measures, such as 30-day mortality and hospital length of stay. Although currently underused, PROs have proven to be effective at measuring subjective outcomes after surgery.13,14 In this respect, PROs data provide an added dimension to the evaluation of new surgical techniques and technology, which may enable surgeons to better understand subjective outcomes. For example, outcomes in randomized clinical trials comparing open and laparoscopic hernia repair techniques have previously focused on visual analog scale scores of pain, hernia recurrence, complications, and operative time.15,16 As the differences in emerging techniques become more nuanced (eg, robotic vs laparoscopic), traditional measures may be insensitive to improvements in patient disability and well-being, both of which are central to arguments for minimally invasive surgery. In addition, the inclusion of patients in shared decision making requires health care that aligns with patient preferences and values.17 Routine and accurate measurement of PROs in surgical trials and clinical care provides a valued outcome for surgeons and expands patient involvement in decision making.
Improving decision making in surgery will require more than just greater implementation of PROs in surgical trials and clinical care. This study identified problematic aspects within the theoretical framework for decision making, specifically, the association between decision making and patient expectations, education, and satisfaction. Understanding the effects of patient expectations on the decision-making process requires accurate measurement of expectations and honest assessment of the ability to recalibrate these expectations through preoperative education.18 A randomized clinical trial of hip and knee arthroplasties found that preoperative education can influence patient expectations of postoperative recovery.19 Furthermore, Ho et al20 reported that patient satisfaction with preoperative information was the strongest predictor of satisfaction with the overall outcome—stronger even than the method of surgery and whether complications occurred. However, thus far, patient expectations have inconsistently correlated with patient satisfaction after surgery, and there is no accepted method for capture of perioperative expectations.21 Studying the association between treatment decision making and patient education, expectations, and satisfaction has become paramount, given that patient satisfaction, which has been measured for the past decade, is now being used to calibrate surgeon and institutional reimbursement.22 Future research will need to further explore the theoretical framework for the decision-making process and identify measurable factors that surgeons and institutions can use to improve care.
Concerns remain regarding the integration of PROs into the EHR. Paper administration and processing of PROs can be time-consuming, costly, and too burdensome for a busy surgical practice. Integration of EHRs improves the logistics of administration, although it raises additional concerns related to the security of patient and clinician information. Existing research has focused primarily on PROs measurement rather than on the EHR interface with patients and clinicians.23 Effective systems must optimize the presentation of PROs data to enable surgeons to effectively interpret this information for decision making. Likewise, enhanced feedback to patients may help validate the time commitment required to complete patient-reported measures and may potentially improve patient response rates. Without significant collaborative efforts to develop and improve EHR platforms, the effective use of patient-reported data by clinicians is unlikely to increase.
The National Institutes of Health Patient Reported Outcomes Measurement Information System (PROMIS) may provide some insight for institutions collaborating on the electronic integration of PROs data collection and presentation. As a collective effort among institutions, PROMIS uses a centralized, web-based system for PROs data collection and features immediate, standardized scoring using a shared item bank.24,25 Despite the success of PROMIS with electronic administration and scoring across multiple institutions, the PROMIS measures are not uniformly calibrated or validated for measuring the effect of surgical procedures. Future systems that measure PROs in surgery should incorporate the advances made by PROMIS and address the issues identified by our Delphi survey, including presenting PROs data to patients that are responsive to educational level and language abilities, enabling easy interpretation by surgeons for immediate action, and improving the integration of PROs systems into the EHR.
Both PROs and the measurement of quality emerged as the final theme from the Delphi survey and raised concerns about the validity, efficacy, and risk adjustment of PROs measures in surgery. Foremost, the survey identified the validity of measuring quality with PROs vs traditional clinical outcome measures as a significant consideration for stakeholders. Moreover, the efficacy of PROs instruments to improve quality and lower costs has not been extensively studied to date. System-wide introduction of PROs instruments should follow thoughtful research initiatives that demonstrate their effectiveness. To address this need for validation, the CMS Innovation Center could be a potential resource in evaluating the efficacy of selected PROs measures to improve quality because it has already been successful in assessing new reimbursement strategies for accountable care organizations.22 Emerging evidence has begun to reveal a correlation between patient satisfaction and surgical outcomes; however, comparison of selected PROs measures among clinicians, surgical groups, and institutions will require thorough risk and case adjustment.26 Furthermore, to alleviate potential skepticism within the surgical community, the establishment of benchmarks will necessitate adequate transparency with regard to reasoning and methods. Successful use of PROs in performance measurement of surgery will require a thoughtful and open collaboration among stakeholders.
Measuring performance with PROs, however, is not a new concept. During the past decade, the UK National Health Service Patient Reported Outcome Measures initiative has collected health-related quality of life, patient satisfaction, and functional data after inguinal hernia repair, hip and knee arthroplasty, and varicose vein ablation.27 Internationally, the UK National Health Service Patient Reported Outcome Measures initiative is perhaps the most ambitious quality improvement project to date and has started to evaluate the changes in health-related quality of life and patient satisfaction after surgery at the clinician and institution level.28 In a comparable move, CMS plans to encourage clinicians and institutions to routinely collect PROs data through funding models, such as Meaningful Use and Physician Quality and Reporting System. The funding models initially incentivize PROs data collection through physician and institution reimbursements; however, after an introductory period, the models penalize participants who do not meet reporting and collection requirements. It remains to be determined whether routine measurement and comparison of PROs data will improve outcomes and lower costs in the long run.
This study has several limitations, many of which are inherent to the Delphi process. The qualitative round of the survey was susceptible to influence from the expert panel and reviewers. To address this, the independent reviewers worked separately, without interaction with the expert panel. Final review by the expert panel looked for differences among reviewers, which is an accepted method for item generation and review and has been implemented by other groups in creating a research agenda.11,29 In addition, the survey was susceptible to nonresponder bias. A previous study30 found that the demographic characteristics and survey results of responders are not equivalent to those of nonresponders. No qualitative differences in occupation were found between responders and nonresponders; however, subtle differences may have existed between these groups that were not measured. Despite this limitation, the survey received robust participation for all 3 rounds, and the sample size was comparable to or greater than that of similar conference-based surveys.31,32 Finally, the PROS Conference relied on patient advocacy groups as a surrogate for the patient’s voice. Ideally, one could envision incorporating nonadvocacy patients in a national research agenda; however, patient advocates represent the best practice currently to promote patient concerns in policy discussions, and this study remains one of the few Delphi surveys to incorporate the patient’s voice. Ongoing efforts to increase the patient’s voice in PROs research and implementation will be required for meaningful incorporation of PROs into surgical care, measurement, and research.
In an era of patient-centered care, PROs can serve as a useful complement to ongoing discussions on health care expenditures by including the patient’s voice, and they have considerable potential in the determination of quality in an evolving health care system. The incredible growth of PROs in clinical care and surgical trials has led to many potential research endeavors and collaborations. The PROS Conference developed a research agenda for researchers, surgeons, and funding agencies to help prioritize research on PROs measurement in surgery to direct funding and institutional collaboration. Future research initiatives should address PROs in the decision-making process, challenges to integrating PROs into the EHR, and PROs and the measurement of quality.
Accepted for Publication: April 2, 2016.
Corresponding Author: Larissa K. Temple, MD, Department of Surgery, Memorial Sloan Kettering Cancer Center, 1275 York Ave, Room C-1079, New York, NY 10065 (email@example.com).
Published Online: July 20, 2016. doi:10.1001/jamasurg.2016.1640
Author Contributions: Dr Temple had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Pezold, Pusic, Cohen, Hollenberg, Flum, Temple.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Pezold, Pusic, Cohen, Hollenberg, Flum, Temple.
Critical revision of the manuscript for important intellectual content: Pusic, Hollenberg, Butt, Flum, Temple.
Obtained funding: Pusic, Flum, Temple.
Administrative, technical, or material support: Pezold, Hollenberg, Flum, Temple.
Study supervision: Pusic, Hollenberg, Temple.
Conflict of Interest Disclosures: None reported.
Funding/Support: This project was supported by grants R13 HS023357-01 (Drs Pusic and Temple) and T32HS000066 (Dr Pezold) from the Agency for Healthcare Research and Quality and by the Plastic Surgery Foundation. Drs Pezold and Temple are supported in part by grant P30 CA008748 from the National Institutes of Health, National Cancer Institute Cancer Center.
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication. The Plastic Surgery Foundation distributed the Delphi survey through their conference listserv.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the Plastic Surgery Foundation or the Agency for Healthcare Research and Quality.
Additional Contributions: David Sewell, MA, MFA, Memorial Sloan Kettering Cancer Center, assisted in the preparation of this article and was compensated by Memorial Sloan Kettering Cancer Center.
Create a personal account or sign in to: