[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.
Flowchart for the 3 Rounds of the Modified Delphi Survey
Flowchart for the 3 Rounds of the Modified Delphi Survey
Figure 2.
Most Important Themes for Patient-Reported Outcome Research in Surgery Identified From the Delphi Survey
Most Important Themes for Patient-Reported Outcome Research in Surgery Identified From the Delphi Survey

EHR indicates electronic health record.

Table 1.  
Characteristics of the Survey Respondents by Rounda
Characteristics of the Survey Respondents by Rounda
Table 2.  
Occupations of Conference Registrants and Survey Respondentsa
Occupations of Conference Registrants and Survey Respondentsa
Table 3.  
Final Research Agenda From 459 Items Initially Submitted
Final Research Agenda From 459 Items Initially Submitted
1.
US Food and Drug Administration.  Guidance for industry: patient reported outcome measures: use in medical product development to support labeling claims.  Fed Regist. 2009;74(235):65132-65133.Google Scholar
2.
Basch  E, Jia  X, Heller  G,  et al.  Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes.  J Natl Cancer Inst. 2009;101(23):1624-1632.PubMedGoogle ScholarCrossref
3.
Möller  E, Weidenhielm  L, Werner  S.  Outcome and knee-related quality of life after anterior cruciate ligament reconstruction: a long-term follow-up.  Knee Surg Sports Traumatol Arthrosc. 2009;17(7):786-794.PubMedGoogle ScholarCrossref
4.
Pusic  AL, Klassen  AF, Scott  AM, Klok  JA, Cordeiro  PG, Cano  SJ.  Development of a new patient-reported outcome measure for breast surgery: the BREAST-Q.  Plast Reconstr Surg. 2009;124(2):345-353.PubMedGoogle ScholarCrossref
5.
Frobell  RB, Roos  HP, Roos  EM, Roemer  FW, Ranstam  J, Lohmander  LS.  Treatment for acute anterior cruciate ligament tear: five year outcome of randomised trial.  BMJ. 2013;346(7895):f232.PubMedGoogle ScholarCrossref
6.
Institute of Medicine.  Crossing the quality chasm: a new health system for the 21st century.  J Inst Med. March 2001:1-8.Google Scholar
7.
Centers for Medicare & Medicaid Services. Hospital value-based purchasing. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Hospital-Value-Based-Purchasing/#main_content. Accessed May 4, 2015.
8.
Selby  JV, Beal  AC, Frank  L.  The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda.  JAMA. 2012;307(15):1583-1584. doi:10.1001/jama.2012.500.PubMedGoogle ScholarCrossref
9.
Bilimoria  KY, Cella  D, Butt  Z.  Current challenges in using patient-reported outcomes for surgical care and performance measurement: everybody wants to hear from the patient, but are we ready to listen?  JAMA Surg. 2014;149(6):505-506.PubMedGoogle ScholarCrossref
10.
Johnson  JA.  Brief History of NIH Funding: Fact Sheet. Bethesda, MD: National Institutes of Health; 2014.
11.
Fink  A, Kosecoff  J, Chassin  M, Brook  RH.  Consensus methods: characteristics and guidelines for use.  Am J Public Health. 1984;74(9):979-983.PubMedGoogle ScholarCrossref
12.
Adler  M, Zigilo  E.  Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. Philadelphia, PA: Jessica Kingsley Publishers; 1996.
13.
Tosteson  ANA, Tosteson  TD, Lurie  JD,  et al.  Comparative effectiveness evidence from the spine patient outcomes research trial: surgical versus nonoperative care for spinal stenosis, degenerative spondylolisthesis, and intervertebral disc herniation.  Spine (Phila Pa 1976). 2011;36(24):2061-2068.PubMedGoogle ScholarCrossref
14.
McCarthy  CM, Klassen  AF, Cano  SJ,  et al.  Patient satisfaction with postmastectomy breast reconstruction: a comparison of saline and silicone implants.  Cancer. 2010;116(24):5584-5591.PubMedGoogle ScholarCrossref
15.
Sajid  MS, Bokhari  SA, Mallick  AS, Cheek  E, Baig  MK.  Laparoscopic vs open repair of incisional/ventral hernia: a meta-analysis.  Am J Surg. 2009;197(1):64-72.PubMedGoogle ScholarCrossref
16.
Karthikesalingam  A, Markar  SR, Holt  PJE, Praseedom  RK.  Meta-analysis of randomized controlled trials comparing laparoscopic with open mesh repair of recurrent inguinal hernia.  Br J Surg. 2010;97(1):4-11.PubMedGoogle ScholarCrossref
17.
Barry  MJ, Edgman-Levitan  S.  Shared decision making--pinnacle of patient-centered care.  N Engl J Med. 2012;366(9):780-781.PubMedGoogle ScholarCrossref
18.
Pusic  AL, Klassen  AF, Snell  L,  et al.  Measuring and managing patient expectations for breast reconstruction: impact on quality of life and patient satisfaction.  Expert Rev Pharmacoecon Outcomes Res. 2012;12(2):149-158.PubMedGoogle ScholarCrossref
19.
Mancuso  CA, Graziano  S, Briskie  LM,  et al.  Randomized trials to modify patients’ preoperative expectations of hip and knee arthroplasties.  Clin Orthop Relat Res. 2008;466(2):424-431.PubMedGoogle ScholarCrossref
20.
Ho  AL, Klassen  AF, Cano  S, Scott  AM, Pusic  AL.  Optimizing patient-centered care in breast reconstruction: the importance of preoperative information and patient-physician communication.  Plast Reconstr Surg. 2013;132(2):212e-220e.PubMedGoogle ScholarCrossref
21.
Waljee  J, McGlinn  EP, Sears  ED, Chung  KC.  Patient expectations and patient-reported outcomes in surgery: a systematic review.  Surgery. 2014;155(5):799-808.PubMedGoogle ScholarCrossref
22.
Centers for Medicare & Medicaid Services. Innovation Center models. https://innovation.cms.gov/initiatives/index.html#views=models. Accessed August 24, 2015.
23.
Hartzler  AL, Fey  BC, Flum  DR.  Integrating patient-reported outcomes into spine surgical care through visual dashboards: lessons learned from human-centered design.  EGEMS (Wash DC). 2015;3(2):1133. PubMedGoogle Scholar
24.
Cella  D, Yount  S, Rothrock  N,  et al; PROMIS Cooperative Group.  The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.  Med Care. 2007;45(5)(suppl 1):S3-S11.PubMedGoogle ScholarCrossref
25.
Assessment Center. What is the assessment center? http://www.assessmentcenter.net. Accessed August 24, 2015.
26.
Sacks  GD, Lawson  EH, Dawes  AJ,  et al.  Relationship between hospital performance on a patient satisfaction survey and surgical quality.  JAMA Surg. 2015;150(9):858-864.PubMedGoogle ScholarCrossref
27.
Health & Social Care Information Centre. Patient reported outcome measures. http://www.hscic.gov.uk/proms. Accessed April 29, 2015.
28.
Hamilton  DF, Lane  JV, Gaston  P,  et al.  What determines patient satisfaction with surgery? a prospective cohort study of 4709 patients following total joint replacement.  BMJ Open. 2013;3(4):e002525.PubMedGoogle ScholarCrossref
29.
Broder  MS, Landow  WJ, Goodwin  SC, Brook  RH, Sherbourne  CD, Harris  K.  An agenda for research into uterine artery embolization: results of an expert panel conference.  J Vasc Interv Radiol. 2000;11(4):509-515.PubMedGoogle ScholarCrossref
30.
Kotaniemi  JT, Hassi  J, Kataja  M,  et al.  Does non-responder bias have a significant effect on the results in a postal questionnaire study?  Eur J Epidemiol. 2001;17(9):809-817.PubMedGoogle ScholarCrossref
31.
Stefanidis  D, Montero  P, Urbach  DR,  et al.  SAGES research agenda in gastrointestinal and endoscopic surgery: updated results of a Delphi study.  Surg Endosc. 2014;28(10):2763-2771.PubMedGoogle ScholarCrossref
32.
Burt  CG, Cima  RR, Koltun  WA,  et al.  Developing a research agenda for the American Society of Colon and Rectal Surgeons: results of a Delphi approach.  Dis Colon Rectum. 2009;52(5):898-905.PubMedGoogle ScholarCrossref
Original Investigation
October 2016

Defining a Research Agenda for Patient-Reported Outcomes in SurgeryUsing a Delphi Survey of Stakeholders

Author Affiliations
  • 1Department of Surgery, Weill Cornell Medical Center–New York Presbyterian Hospital, New York
  • 2Division of Plastic and Reconstructive Surgery, Memorial Sloan Kettering Cancer Center, New York, New York
  • 3Department of Medicine, Weill Cornell Medical College, New York, New York
  • 4Departments of Medical Social Sciences, Surgery, and Psychiatry and Behavioral Sciences, Northwestern University Feinberg School of Medicine, Chicago, Illinois
  • 5Department of Surgery, University of Washington, Seattle
  • 6Colorectal Service, Department of Surgery, Memorial Sloan Kettering Cancer Center, New York, New York
 

Copyright 2016 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.

JAMA Surg. 2016;151(10):930-936. doi:10.1001/jamasurg.2016.1640
Key Points

Question  What are the primary areas that future patient-reported outcomes (PROs) research should focus on?

Findings  An iterative web-based interface was used to create a modified Delphi survey for use at the inaugural Patient-Reported Outcomes in Surgery (PROS) Conference to survey conference registrants including surgeons, PROs researchers, payers, and other stakeholders. The survey identified (1) PROs in the decision-making process, (2) challenges to integrating PROs into the electronic health record, and (3) PROs and the measurement of quality as the 3 most important areas for future research.

Meaning  The research agenda generated from the PROS Conference will help researchers, surgeons, and funders identify crucial areas of future PROs research in surgery.

Abstract

Importance  Identifying timely and important research questions using relevant patient-reported outcomes (PROs) in surgery remains paramount in the current medical climate. The inaugural Patient-Reported Outcomes in Surgery (PROS) Conference brought together stakeholders in PROs research in surgery with the aim of creating a research agenda to help determine future directions and advance cross-disciplinary collaboration.

Objective  To create a research agenda to help determine future directions and advance cross-disciplinary collaboration on the use of PROs in surgery.

Design, Setting, and Participants  An iterative web-based interface was used to create a conference-based, modified Delphi survey for registrants at the PROS Conference (January 29-30, 2015), including surgeons, PRO researchers, payers, and other stakeholders. In round 1, research items were generated from qualitative review of responses to open-ended prompts. In round 2, items were ranked using a 5-point Likert scale; attendees were also asked to submit any new items. In round 3, the top 30 items and newly submitted items were redistributed for final ranking using a 3-point Likert scale. The top 20 items by mean rating were selected for the research agenda.

Main Outcomes and Measures  An expert-generated research agenda on PROs in surgery.

Results  Of the 143 people registered for the conference, 137 provided valid email addresses. There was a wide range of attendees, with the 3 most common groups being plastic surgeons (28 [19.6%]), general surgeons (19 [13.3%]), and researchers (25 [17.5%]). In round 1, participants submitted 459 items, which were reduced through qualitative review to 53 distinct items across 7 themes of PROs research. A research agenda was formulated after 2 successive rounds of ranking. The research agenda identified 3 themes important for future PROs research in surgery: (1) PROs in the decision-making process, (2) integrating PROs into the electronic health record, and (3) measuring quality in surgery with PROs.

Conclusions and Relevance  The PROS Conference research agenda was created using a modified Delphi survey of stakeholders that will help researchers, surgeons, and funders identify crucial areas of future PROs research in surgery.

Introduction

Patient-reported outcomes (PROs), as defined by the US Food and Drug Administration, are “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else.”1(p 2) In practice, this process entails measuring a subjective outcome, such as function after knee arthroplasty, with a questionnaire. Publications2-4 on PROs in the surgical literature have tripled during the last decade. The increasing recognition among surgeons that many postoperative outcomes, such as functional improvement or symptom severity, are best measured by the patient and the increased availability of measures for historically subjective topics have undoubtedly contributed to this surge in publications. Currently, PROs play an important role in the planning of comparative effectiveness research and are used as primary outcomes in surgical trials.5 The Institute of Medicine and the National Quality Forum have long advocated that patient-centered care should be a requirement in modern health care, and this perspective has influenced health care deliberations during the last decade.6 Furthermore, the Patient Protection and Affordable Care Act has encouraged patient-reported data collection in surgery by creating new payment models through the Centers for Medicaid & Medicare Services (CMS) and by targeted funding of research from the Patient-Centered Outcomes Research Institute.7,8 These new models underscore the increasing importance of PROs in surgery.

Despite the potential value of PROs in surgical care, many methodologic and logistical concerns remain.9 Addressing these issues is difficult in a research climate of increasingly scarce funding.10 Thus, researchers and funding agencies need input and clarity from stakeholders to better prioritize research and improve collaboration among institutions. Furthermore, options for surgical techniques and technologies are rapidly expanding while health care resources are shrinking. To better understand the outcomes of surgery from the patient perspective, there is no better time than now for the surgical and research communities to come together and reach consensus regarding the most important and timely issues with PROs research in surgery.

To address these issues, the Patient-Reported Outcomes in Surgery (PROS) Conference was formed, with sponsorship from the Plastic Surgery Foundation and the Agency for Healthcare Research and Quality. The conference brought together stakeholders from diverse fields, including payers, patient advocates, surgeons, researchers, industry representatives, regulators, and health information technology vendors. The 2-day conference was held in Washington, DC, and included panel discussions on current PROs research in surgery within the areas of clinical care, comparative effectiveness, patient access, psychometric development, surgical trials, and quality, as well as mapping out the future directions for each field. The specific goals of the conference were to improve the accessibility and interpretability of PROs data for patients and clinicians, to develop a consensus on the methodologic issues of PROs measurement, and to develop a research agenda for PROs measurement in surgery. The formation of a research agenda should prioritize research questions deemed to be timely and important by stakeholders to guide future collaboration and funding. To meet this aim, the conference leadership conducted a Delphi survey of stakeholders to develop an agenda on future PROs research in surgery.

Methods

This study used a modified Delphi survey to achieve formal group consensus, maximizing dialogue through anonymous, structured feedback.11,12 To facilitate the use of the Delphi model, an expert panel was assembled before the opening of the conference. The panel, composed of conference leaders and methodologic experts, developed the study schema and consensus criteria on the basis of a 3-round, web-based survey (Figure 1). The study was approved and informed consent was waived by the Memorial Sloan Kettering Cancer Center Institutional Review Board.

Emails containing a link to a web-based survey were distributed to all conference registrants 1 week before the start of the conference (round 1), which was held January 29 to 30, 2015. Participants were asked to anonymously identify timely and important PROs research topics through 7 open-ended questions related to clinical care, comparative effectiveness, patient satisfaction, and quality metrics. There was no limit on the length of the responses. Item submissions were reviewed in parallel by two of us (M.L.P. and W.A.C.). This process involved separating compound responses into individual items, summarizing submissions into concise items, and categorizing items into generic themes. After all submissions were reviewed, identical items were deleted, and similar items with overlapping content were consolidated into broader concepts. The expert panel then evaluated the consistency of each researcher’s item reduction and endorsed the selection of research items for round 2.

All conference attendees were invited to participate in a web-based survey during the conference (round 2). Participants were given the final research items from round 1, in randomized order, and were asked to rank the items by research importance using a 5-point Likert scale, with high and low research priority as anchors. In addition, an open-ended question prompt at the conclusion of the survey allowed participants to submit additional research questions. Mirroring the process in round 1, all new research questions submitted in round 2 were subjected to qualitative review and consolidation into final research items. Round 2 concluded with the completion of the conference.

After the conference, all attendees were asked, via email, to participate in a web-based survey, regardless of whether they participated in previous rounds. Email reminders to complete the survey were distributed weekly, and each registrant was limited to 1 survey response each. The top 30 items by mean priority score from round 2, along with new item submissions, were distributed for final ranking by research priority to reach the a priori goal of a 20-item research agenda. To reduce the possibility of a ceiling effect, a 3-point Likert scale was used in round 3, and participants were encouraged to rate only 10 items as high research priority. As in round 2, item order was randomized. The top 20 items by mean Likert score were selected as the consensus research agenda.

Results

Of the 143 people registered for the conference, 137 provided valid email addresses. Potential respondents were invited to participate during each round. There was a wide range of attendants, including individuals from several surgical subspecialties, payers, regulators, and patient advocates, with the 3 most common groups identified as being plastic surgeons (28 [19.6%]), general surgeons (19 [13.3%]), and researchers (25 [17.5%]). Table 1 presents the basic demographic characteristics of respondents by round. During item generation (rounds 1 and 2), participants were given the option to specify their occupation (Table 2). Specific occupation demographics were not collected during round 3 because there was a desire to keep the survey as short as possible to increase response rates. In round 1, a total of 83 participants (60.6% of conference registrants with email addresses) submitted 356 responses to open-ended questions. After review, a total of 459 research items were submitted, with a mean of 5.5 items submitted per participant.

After item reduction was performed and the expert panel reviewed the results, there were 53 items within 7 themes: clinical care, comparative effectiveness, data management, ethics, performance measurement, education, and other. In round 2, a total of 54 participants (39.4% of emailed registrants) responded to email invitations during the conference. Responders ranked the 53 items by research priority and submitted 9 new items, which were reviewed and consolidated to 3 new items. In round 3, a total of 59 participants (43.1% of emailed registrants) responded to survey invitations. The top 20 items from round 3 were selected as the consensus research agenda for future PROs research (Table 3). Ranking of item importance remained stable during the rounds, with only 2 items from the top 20 of round 2 failing to make the final research agenda. Stakeholder participation from rounds 2 and 3 was predominantly from experienced PROs researchers, with 81 participants (43.4%) reporting more than 7 years of experience.

Discussion

Using a modified Delphi approach, we engaged an international stakeholder group of surgeons, researchers, patient advocates, funding agency representatives, health information technology vendors, and regulators to reach consensus on future PROs research priorities in surgery. The final results of the survey contain the top 20 items from more than 450 topics initially submitted by participants. Our results represent the first consensus-driven, surgery-focused PROs research agenda to date. Within the research agenda, 3 themes emerged as priorities for future research in surgery: (1) PROs in the surgical decision-making process, (2) challenges to integration of PROs in the electronic health record (EHR), and (3) PROs and the measurement of quality (Figure 2). Among all items, the decision-making process was rated of high importance, with “impact of PROs on patient and/or provider decision-making” the top-ranked item for both rounds.

Priority 1: Incorporating PROs Data Into the Decision-Making Process

It is not surprising that the use of PROs in the clinical decision-making process was the highest importance. Decision making in surgery has traditionally relied on surgeon experience and established objective measures, such as 30-day mortality and hospital length of stay. Although currently underused, PROs have proven to be effective at measuring subjective outcomes after surgery.13,14 In this respect, PROs data provide an added dimension to the evaluation of new surgical techniques and technology, which may enable surgeons to better understand subjective outcomes. For example, outcomes in randomized clinical trials comparing open and laparoscopic hernia repair techniques have previously focused on visual analog scale scores of pain, hernia recurrence, complications, and operative time.15,16 As the differences in emerging techniques become more nuanced (eg, robotic vs laparoscopic), traditional measures may be insensitive to improvements in patient disability and well-being, both of which are central to arguments for minimally invasive surgery. In addition, the inclusion of patients in shared decision making requires health care that aligns with patient preferences and values.17 Routine and accurate measurement of PROs in surgical trials and clinical care provides a valued outcome for surgeons and expands patient involvement in decision making.

Improving decision making in surgery will require more than just greater implementation of PROs in surgical trials and clinical care. This study identified problematic aspects within the theoretical framework for decision making, specifically, the association between decision making and patient expectations, education, and satisfaction. Understanding the effects of patient expectations on the decision-making process requires accurate measurement of expectations and honest assessment of the ability to recalibrate these expectations through preoperative education.18 A randomized clinical trial of hip and knee arthroplasties found that preoperative education can influence patient expectations of postoperative recovery.19 Furthermore, Ho et al20 reported that patient satisfaction with preoperative information was the strongest predictor of satisfaction with the overall outcome—stronger even than the method of surgery and whether complications occurred. However, thus far, patient expectations have inconsistently correlated with patient satisfaction after surgery, and there is no accepted method for capture of perioperative expectations.21 Studying the association between treatment decision making and patient education, expectations, and satisfaction has become paramount, given that patient satisfaction, which has been measured for the past decade, is now being used to calibrate surgeon and institutional reimbursement.22 Future research will need to further explore the theoretical framework for the decision-making process and identify measurable factors that surgeons and institutions can use to improve care.

Priority 2: Integrating PROs Data Into the EHR

Concerns remain regarding the integration of PROs into the EHR. Paper administration and processing of PROs can be time-consuming, costly, and too burdensome for a busy surgical practice. Integration of EHRs improves the logistics of administration, although it raises additional concerns related to the security of patient and clinician information. Existing research has focused primarily on PROs measurement rather than on the EHR interface with patients and clinicians.23 Effective systems must optimize the presentation of PROs data to enable surgeons to effectively interpret this information for decision making. Likewise, enhanced feedback to patients may help validate the time commitment required to complete patient-reported measures and may potentially improve patient response rates. Without significant collaborative efforts to develop and improve EHR platforms, the effective use of patient-reported data by clinicians is unlikely to increase.

The National Institutes of Health Patient Reported Outcomes Measurement Information System (PROMIS) may provide some insight for institutions collaborating on the electronic integration of PROs data collection and presentation. As a collective effort among institutions, PROMIS uses a centralized, web-based system for PROs data collection and features immediate, standardized scoring using a shared item bank.24,25 Despite the success of PROMIS with electronic administration and scoring across multiple institutions, the PROMIS measures are not uniformly calibrated or validated for measuring the effect of surgical procedures. Future systems that measure PROs in surgery should incorporate the advances made by PROMIS and address the issues identified by our Delphi survey, including presenting PROs data to patients that are responsive to educational level and language abilities, enabling easy interpretation by surgeons for immediate action, and improving the integration of PROs systems into the EHR.

Priority 3: PROs and Quality Assessment

Both PROs and the measurement of quality emerged as the final theme from the Delphi survey and raised concerns about the validity, efficacy, and risk adjustment of PROs measures in surgery. Foremost, the survey identified the validity of measuring quality with PROs vs traditional clinical outcome measures as a significant consideration for stakeholders. Moreover, the efficacy of PROs instruments to improve quality and lower costs has not been extensively studied to date. System-wide introduction of PROs instruments should follow thoughtful research initiatives that demonstrate their effectiveness. To address this need for validation, the CMS Innovation Center could be a potential resource in evaluating the efficacy of selected PROs measures to improve quality because it has already been successful in assessing new reimbursement strategies for accountable care organizations.22 Emerging evidence has begun to reveal a correlation between patient satisfaction and surgical outcomes; however, comparison of selected PROs measures among clinicians, surgical groups, and institutions will require thorough risk and case adjustment.26 Furthermore, to alleviate potential skepticism within the surgical community, the establishment of benchmarks will necessitate adequate transparency with regard to reasoning and methods. Successful use of PROs in performance measurement of surgery will require a thoughtful and open collaboration among stakeholders.

Measuring performance with PROs, however, is not a new concept. During the past decade, the UK National Health Service Patient Reported Outcome Measures initiative has collected health-related quality of life, patient satisfaction, and functional data after inguinal hernia repair, hip and knee arthroplasty, and varicose vein ablation.27 Internationally, the UK National Health Service Patient Reported Outcome Measures initiative is perhaps the most ambitious quality improvement project to date and has started to evaluate the changes in health-related quality of life and patient satisfaction after surgery at the clinician and institution level.28 In a comparable move, CMS plans to encourage clinicians and institutions to routinely collect PROs data through funding models, such as Meaningful Use and Physician Quality and Reporting System. The funding models initially incentivize PROs data collection through physician and institution reimbursements; however, after an introductory period, the models penalize participants who do not meet reporting and collection requirements. It remains to be determined whether routine measurement and comparison of PROs data will improve outcomes and lower costs in the long run.

Study Limitations

This study has several limitations, many of which are inherent to the Delphi process. The qualitative round of the survey was susceptible to influence from the expert panel and reviewers. To address this, the independent reviewers worked separately, without interaction with the expert panel. Final review by the expert panel looked for differences among reviewers, which is an accepted method for item generation and review and has been implemented by other groups in creating a research agenda.11,29 In addition, the survey was susceptible to nonresponder bias. A previous study30 found that the demographic characteristics and survey results of responders are not equivalent to those of nonresponders. No qualitative differences in occupation were found between responders and nonresponders; however, subtle differences may have existed between these groups that were not measured. Despite this limitation, the survey received robust participation for all 3 rounds, and the sample size was comparable to or greater than that of similar conference-based surveys.31,32 Finally, the PROS Conference relied on patient advocacy groups as a surrogate for the patient’s voice. Ideally, one could envision incorporating nonadvocacy patients in a national research agenda; however, patient advocates represent the best practice currently to promote patient concerns in policy discussions, and this study remains one of the few Delphi surveys to incorporate the patient’s voice. Ongoing efforts to increase the patient’s voice in PROs research and implementation will be required for meaningful incorporation of PROs into surgical care, measurement, and research.

Conclusions

In an era of patient-centered care, PROs can serve as a useful complement to ongoing discussions on health care expenditures by including the patient’s voice, and they have considerable potential in the determination of quality in an evolving health care system. The incredible growth of PROs in clinical care and surgical trials has led to many potential research endeavors and collaborations. The PROS Conference developed a research agenda for researchers, surgeons, and funding agencies to help prioritize research on PROs measurement in surgery to direct funding and institutional collaboration. Future research initiatives should address PROs in the decision-making process, challenges to integrating PROs into the EHR, and PROs and the measurement of quality.

Back to top
Article Information

Accepted for Publication: April 2, 2016.

Corresponding Author: Larissa K. Temple, MD, Department of Surgery, Memorial Sloan Kettering Cancer Center, 1275 York Ave, Room C-1079, New York, NY 10065 (templel@mskcc.org).

Published Online: July 20, 2016. doi:10.1001/jamasurg.2016.1640

Author Contributions: Dr Temple had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Pezold, Pusic, Cohen, Hollenberg, Flum, Temple.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Pezold, Pusic, Cohen, Hollenberg, Flum, Temple.

Critical revision of the manuscript for important intellectual content: Pusic, Hollenberg, Butt, Flum, Temple.

Obtained funding: Pusic, Flum, Temple.

Administrative, technical, or material support: Pezold, Hollenberg, Flum, Temple.

Study supervision: Pusic, Hollenberg, Temple.

Conflict of Interest Disclosures: None reported.

Funding/Support: This project was supported by grants R13 HS023357-01 (Drs Pusic and Temple) and T32HS000066 (Dr Pezold) from the Agency for Healthcare Research and Quality and by the Plastic Surgery Foundation. Drs Pezold and Temple are supported in part by grant P30 CA008748 from the National Institutes of Health, National Cancer Institute Cancer Center.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication. The Plastic Surgery Foundation distributed the Delphi survey through their conference listserv.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the Plastic Surgery Foundation or the Agency for Healthcare Research and Quality.

Additional Contributions: David Sewell, MA, MFA, Memorial Sloan Kettering Cancer Center, assisted in the preparation of this article and was compensated by Memorial Sloan Kettering Cancer Center.

References
1.
US Food and Drug Administration.  Guidance for industry: patient reported outcome measures: use in medical product development to support labeling claims.  Fed Regist. 2009;74(235):65132-65133.Google Scholar
2.
Basch  E, Jia  X, Heller  G,  et al.  Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes.  J Natl Cancer Inst. 2009;101(23):1624-1632.PubMedGoogle ScholarCrossref
3.
Möller  E, Weidenhielm  L, Werner  S.  Outcome and knee-related quality of life after anterior cruciate ligament reconstruction: a long-term follow-up.  Knee Surg Sports Traumatol Arthrosc. 2009;17(7):786-794.PubMedGoogle ScholarCrossref
4.
Pusic  AL, Klassen  AF, Scott  AM, Klok  JA, Cordeiro  PG, Cano  SJ.  Development of a new patient-reported outcome measure for breast surgery: the BREAST-Q.  Plast Reconstr Surg. 2009;124(2):345-353.PubMedGoogle ScholarCrossref
5.
Frobell  RB, Roos  HP, Roos  EM, Roemer  FW, Ranstam  J, Lohmander  LS.  Treatment for acute anterior cruciate ligament tear: five year outcome of randomised trial.  BMJ. 2013;346(7895):f232.PubMedGoogle ScholarCrossref
6.
Institute of Medicine.  Crossing the quality chasm: a new health system for the 21st century.  J Inst Med. March 2001:1-8.Google Scholar
7.
Centers for Medicare & Medicaid Services. Hospital value-based purchasing. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Hospital-Value-Based-Purchasing/#main_content. Accessed May 4, 2015.
8.
Selby  JV, Beal  AC, Frank  L.  The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda.  JAMA. 2012;307(15):1583-1584. doi:10.1001/jama.2012.500.PubMedGoogle ScholarCrossref
9.
Bilimoria  KY, Cella  D, Butt  Z.  Current challenges in using patient-reported outcomes for surgical care and performance measurement: everybody wants to hear from the patient, but are we ready to listen?  JAMA Surg. 2014;149(6):505-506.PubMedGoogle ScholarCrossref
10.
Johnson  JA.  Brief History of NIH Funding: Fact Sheet. Bethesda, MD: National Institutes of Health; 2014.
11.
Fink  A, Kosecoff  J, Chassin  M, Brook  RH.  Consensus methods: characteristics and guidelines for use.  Am J Public Health. 1984;74(9):979-983.PubMedGoogle ScholarCrossref
12.
Adler  M, Zigilo  E.  Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. Philadelphia, PA: Jessica Kingsley Publishers; 1996.
13.
Tosteson  ANA, Tosteson  TD, Lurie  JD,  et al.  Comparative effectiveness evidence from the spine patient outcomes research trial: surgical versus nonoperative care for spinal stenosis, degenerative spondylolisthesis, and intervertebral disc herniation.  Spine (Phila Pa 1976). 2011;36(24):2061-2068.PubMedGoogle ScholarCrossref
14.
McCarthy  CM, Klassen  AF, Cano  SJ,  et al.  Patient satisfaction with postmastectomy breast reconstruction: a comparison of saline and silicone implants.  Cancer. 2010;116(24):5584-5591.PubMedGoogle ScholarCrossref
15.
Sajid  MS, Bokhari  SA, Mallick  AS, Cheek  E, Baig  MK.  Laparoscopic vs open repair of incisional/ventral hernia: a meta-analysis.  Am J Surg. 2009;197(1):64-72.PubMedGoogle ScholarCrossref
16.
Karthikesalingam  A, Markar  SR, Holt  PJE, Praseedom  RK.  Meta-analysis of randomized controlled trials comparing laparoscopic with open mesh repair of recurrent inguinal hernia.  Br J Surg. 2010;97(1):4-11.PubMedGoogle ScholarCrossref
17.
Barry  MJ, Edgman-Levitan  S.  Shared decision making--pinnacle of patient-centered care.  N Engl J Med. 2012;366(9):780-781.PubMedGoogle ScholarCrossref
18.
Pusic  AL, Klassen  AF, Snell  L,  et al.  Measuring and managing patient expectations for breast reconstruction: impact on quality of life and patient satisfaction.  Expert Rev Pharmacoecon Outcomes Res. 2012;12(2):149-158.PubMedGoogle ScholarCrossref
19.
Mancuso  CA, Graziano  S, Briskie  LM,  et al.  Randomized trials to modify patients’ preoperative expectations of hip and knee arthroplasties.  Clin Orthop Relat Res. 2008;466(2):424-431.PubMedGoogle ScholarCrossref
20.
Ho  AL, Klassen  AF, Cano  S, Scott  AM, Pusic  AL.  Optimizing patient-centered care in breast reconstruction: the importance of preoperative information and patient-physician communication.  Plast Reconstr Surg. 2013;132(2):212e-220e.PubMedGoogle ScholarCrossref
21.
Waljee  J, McGlinn  EP, Sears  ED, Chung  KC.  Patient expectations and patient-reported outcomes in surgery: a systematic review.  Surgery. 2014;155(5):799-808.PubMedGoogle ScholarCrossref
22.
Centers for Medicare & Medicaid Services. Innovation Center models. https://innovation.cms.gov/initiatives/index.html#views=models. Accessed August 24, 2015.
23.
Hartzler  AL, Fey  BC, Flum  DR.  Integrating patient-reported outcomes into spine surgical care through visual dashboards: lessons learned from human-centered design.  EGEMS (Wash DC). 2015;3(2):1133. PubMedGoogle Scholar
24.
Cella  D, Yount  S, Rothrock  N,  et al; PROMIS Cooperative Group.  The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.  Med Care. 2007;45(5)(suppl 1):S3-S11.PubMedGoogle ScholarCrossref
25.
Assessment Center. What is the assessment center? http://www.assessmentcenter.net. Accessed August 24, 2015.
26.
Sacks  GD, Lawson  EH, Dawes  AJ,  et al.  Relationship between hospital performance on a patient satisfaction survey and surgical quality.  JAMA Surg. 2015;150(9):858-864.PubMedGoogle ScholarCrossref
27.
Health & Social Care Information Centre. Patient reported outcome measures. http://www.hscic.gov.uk/proms. Accessed April 29, 2015.
28.
Hamilton  DF, Lane  JV, Gaston  P,  et al.  What determines patient satisfaction with surgery? a prospective cohort study of 4709 patients following total joint replacement.  BMJ Open. 2013;3(4):e002525.PubMedGoogle ScholarCrossref
29.
Broder  MS, Landow  WJ, Goodwin  SC, Brook  RH, Sherbourne  CD, Harris  K.  An agenda for research into uterine artery embolization: results of an expert panel conference.  J Vasc Interv Radiol. 2000;11(4):509-515.PubMedGoogle ScholarCrossref
30.
Kotaniemi  JT, Hassi  J, Kataja  M,  et al.  Does non-responder bias have a significant effect on the results in a postal questionnaire study?  Eur J Epidemiol. 2001;17(9):809-817.PubMedGoogle ScholarCrossref
31.
Stefanidis  D, Montero  P, Urbach  DR,  et al.  SAGES research agenda in gastrointestinal and endoscopic surgery: updated results of a Delphi study.  Surg Endosc. 2014;28(10):2763-2771.PubMedGoogle ScholarCrossref
32.
Burt  CG, Cima  RR, Koltun  WA,  et al.  Developing a research agenda for the American Society of Colon and Rectal Surgeons: results of a Delphi approach.  Dis Colon Rectum. 2009;52(5):898-905.PubMedGoogle ScholarCrossref
×