Association of Postoperative Readmissions With Surgical Quality Using a Delphi Consensus Process to Identify Relevant Diagnosis Codes

Key Points

Question  Can a modified Delphi process identify postoperative readmissions associated with surgical quality?

Findings  In this modified Delphi consensus process study, 14 experts in medical and surgical readmissions built consensus on the association between postoperative readmissions and quality of surgical procedures using International Classification of Diseases, Ninth Revision codes. The panel’s consensus was that one-third of postoperative readmissions are unlikely to reflect problems with surgical quality.

Meaning  Restricting the definition of postoperative readmission to consensus-defined diagnosis codes might yield a more useful quality measure.

Importance  Postoperative readmission data are used to measure hospital performance, yet the extent to which these readmissions reflect surgical quality is unknown.

Objective  To establish expert consensus on whether reasons for postoperative readmission are associated with the quality of surgery in the index admission.

Design, Setting, and Participants  In a modified Delphi process, a panel of 14 experts in medical and surgical readmissions comprising physicians and nonphysicians from Veterans Affairs (VA) and private-sector institutions reviewed 30-day postoperative readmissions from fiscal years 2008 through 2014 associated with inpatient surgical procedures performed at a VA medical center between October 1, 2007, and September 30, 2014. The consensus process was conducted from January through May 2017. Reasons for readmission were grouped into categories based on International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes. Panelists were given the proportion of readmissions coded by each reason and median (interquartile range) days to readmission. They answered the question, “Does the readmission reason reflect possible surgical quality of care problems in the index admission?” on a scale of 1 (never related) to 5 (directly related) in 3 rounds of consensus building. The consensus process was completed in May 2017 and data were analyzed in June 2017.

Main Outcomes and Measures  Consensus on proportion of ICD-9–coded readmission reasons that reflected quality of surgical procedure.

Results  In 3 Delphi rounds, the 14 panelists achieved consensus on 50 reasons for readmission; 12 panelists also completed group telephone calls between rounds 1 and 2. Readmissions with diagnoses of infection, sepsis, pneumonia, hemorrhage/hematoma, anemia, ostomy complications, acute renal failure, fluid/electrolyte disorders, or venous thromboembolism were considered associated with surgical quality and accounted for 25 521 of 39 664 readmissions (64% of readmissions; 7.5% of 340 858 index surgical procedures). The proportion of readmissions considered to be not associated with surgical quality varied by procedure, ranging from to 21% (613 of 2331) of readmissions after lower-extremity amputations to 47% (745 of 1598) of readmissions after cholecystectomy.

Conclusions and Relevance  One-third of postoperative readmissions are unlikely to reflect problems with surgical quality. Future studies should test whether restricting readmissions to those with specific ICD-9 codes might yield a more useful quality measure.

Hospital readmissions are increasingly used as outcome measures associated with quality of care, and rates are publicly reported for private-sector and Veterans Health Administration (VA) facilities.^1,2 Furthermore, the Centers for Medicare & Medicaid Services and many state governments impose payment penalties on hospitals with readmission rates greater than the national average.³ Recently, attention has focused on identification of the proportion of unplanned hospital readmissions that truly reflect quality of care as opposed to factors that cannot be controlled. Readmissions after surgical procedures may be more associated with quality issues in the index admission than readmissions after medical hospitalizations.⁴ Hospitals trying to reduce overall readmission rates may find value in focusing on surgical care. However, there is little agreement about how to identify which postoperative readmissions truly reflect surgical quality–related problems in the index admission; these cases likely require different readmission reduction efforts than cases not associated with surgical quality.⁵

Early research on VA readmissions suggests that surgical cases contribute to 20% of all hospital readmissions.⁴ Rosen et al⁴ used diagnosis-related groups to determine that 40% of all unplanned surgical readmissions were clinically associated with the index diagnosis; 70% of these had a complication (eg, postoperative infection). Diagnosis-related groups are used for case-mix adjustment and to group patients by International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes, comorbidities, age, and sex.⁶ In contrast, Morris et al⁷ used data from the medical record review–based VA Surgical Quality Improvement Program (VASQIP) and found 56% of postoperative readmissions had a surgical complication. Hechenbleikner et al⁵ found the proportion of postoperative readmission rates associated with quality of surgical procedure (identified through the National Surgical Quality Improvement Program, billing data, and physician review) varying from 51% to 86%. Surgical complications are often used to measure quality of care⁸; however, surgical readmissions are not necessarily the result of complications, nor do they always reflect quality deficits. Identifying which readmissions are associated with surgical quality is difficult.

We approached this challenge by convening a Delphi panel of experts in medical and surgical readmission to evaluate reasons for readmission among a cohort of veterans who underwent selected surgical procedures at VA hospitals and to identify which of these readmissions might signify a surgical quality problem. Delphi consensus processes aggregate expert input to identify areas of agreement and are commonly used to develop quality measures.^9,10 Findings from such expert consensus proceedings have been used to establish valid and reliable criteria for explicit medical record review to identify potentially preventable readmissions.^11-13 We also assessed differences in readmission rates and patient and procedure characteristics between all readmissions in our sample of VA inpatient operations and those with consensus-defined surgical quality–related readmissions. We further examined procedure-level differences in readmission rates between the overall and surgical quality–related cohorts. To our knowledge, this is the first study to use a Delphi consensus process to determine the proportion of surgical readmissions that may be associated with problems in surgical quality. These findings can inform efforts to develop validated quality metrics for surgical care.

We conducted a structured Delphi-based consensus process with clinical experts and a quantitative analysis of postoperative readmissions using VA administrative and clinical databases. We began broadly by assigning reasons for postoperative readmissions using codes from the World Health Organization’s ICD-9. We presented these reasons to our Delphi panel to rate the surgical quality–relatedness of each readmission reason. We used the ICD-9–coded readmission reasons that met our standards for consensus and surgical quality–relatedness to define 2 groups of readmissions; those with potential surgical quality–related problems in the index hospitalization and those without. We then performed statistical analyses to address our study objectives. The VA Central institutional review board approved this research. Informed consent was waived and data were deidentified for the purposes of this analysis.

Veterans Affairs Surgical Quality Improvement Program data were used to identify patient demographics, preoperative comorbidities, operative characteristics, and postoperative complications.¹⁴ Preoperative health care use, postoperative discharge destination, mental health diagnoses, laboratory test results, and vital signs data in the preoperative and postoperative period were obtained from the VA Corporate Data Warehouse, accessed through the VA Informatics and Computing Infrastructure. The Corporate Data Warehouse was also queried for all inpatient diagnoses occurring in the 30-day postsurgical discharge period to categorize reasons for readmission.

The study population included all VASQIP-assessed patients with inpatient surgical procedures performed at a VA Medical Center between October 1, 2007, and September 30, 2014. To accurately assess postdischarge outcomes of readmission, patients who died during hospitalization were excluded from analyses. Patients with a length of stay less than 2 days or greater than 30 days were excluded from the final analytic data set to obtain a study sample representative of a surgical inpatient. The unit of analysis was the surgical procedure, and some patients underwent multiple operations during the study period. In these cases, only the first surgical procedure within a 30-day period was included in the analysis.

Categories of reasons for readmission were developed using a multistep process examining the ICD-9 diagnosis codes from all inpatient readmissions occurring in the 30 days after the index surgery. Diagnosis codes were identified from the VA Corporate Data Warehouse.

For the first step, the Agency for Healthcare Research and Quality Clinical Classifications Software (CCS) multilevel diagnosis categories were applied to the primary, secondary, and length-of-stay diagnosis codes for all readmissions. Three of us (R. H. H., T. W., and C. J.) independently reviewed the combinations of CCS categories, focusing on the most prevalent categories first, and determined an initial list of potential readmission reasons and definitions based on the CCS categories. The clinicians then convened to reconcile their lists of potential categories, incorporating each clinician’s assessment of possible reasons for readmission. Specific ICD-9 diagnoses in addition to the CCS categories were provided to each clinician to help further specify reasons and confirm the appropriateness of reasons defined during the first review. Finally, combinations of primary and secondary diagnoses for the remaining uncoded readmissions were ranked from most to least prevalent, allowing clinicians to review and categorize less frequent reasons.

We followed Delphi process guidelines established by the RAND-UCLA appropriateness model.¹⁵ This approach uses 3 rounds of consensus building and a phone call between the first and second rounds to discuss responses. We recruited a convenience sample of 14 experts: general, vascular, and orthopedic surgeons; internal medicine physicians familiar with treatment of readmitted surgical patients; surgical quality nurses; and surgical social workers from VA and private-sector academic surgical facilities. The identity of panel members was kept confidential to ensure that each member felt free to agree or disagree with other members’ responses. The questionnaire included instructions, readmissions reasons, broad ICD-9 disease categories (eg, bleeding or gastrointestinal tract problems), and the proportion of readmissions by reason with the median (interquartile range [IQR]) number of days to readmission for each reason category.

In the first round, we provided a generic questionnaire; in the second and third rounds, the participants were given individualized questionnaires that presented them with their own previous responses as well as the mean, median, and range of the group’s previous responses. The questions were the same in all 3 rounds.

Participants answered the following question for each readmission reason on a scale of 1 to 5 (1 = never related, 2 = unlikely to be related, 3 = either related or unrelated, 4 = likely related, and 5 = directly related to index quality): “Does the readmission reason reflect possible surgical quality of care problems in the index admission?” We established consensus as an IQR of 1 or less; we retained readmission reasons as related to surgical quality if consensus was reached and the median rating was 4 or greater.

For the final quantitative analyses, we applied our definition of surgical quality–related readmission to all readmissions in our study sample. Bivariate analyses assessed differences between readmission cohorts by patient demographic characteristics; preoperative, intraoperative, and postoperative procedure characteristics; and facility characteristics. An α of .05 was defined a priori, and χ² and unpaired t tests determined significant differences. Finally, we grouped surgical procedures into single-level CCS procedure categories and compared the procedures most frequently associated with any readmission with those considered surgical quality–related by the Delphi panel. All analyses were conducted using SAS version 9.2 (SAS Institute, Inc).

Our sample comprised 340 858 inpatient surgical procedures and 39 664 readmissions (11.6%). Of the 25 521 patients with surgical quality–related readmissions, 1204 (4.7%) were female; 19 769 (77.5%) were of white race/ethnicity; and 4417 (17.3%) were of black race/ethnicity. The mean (SD) age among this subgroup was 65.5 (11.2) years. Of the 14 143 patients whose readmission was not associated with surgical quality, 740 (5.2%) were female; 10 770 (76.2%) were white; 2672 (18.9%) were black; and the mean (SD) age was 63.3 (11.5) years. The most prevalent types of surgical procedures were orthopedic, accounting for 29.8%; general, accounting for 27.3%; and vascular, accounting for 12.3% of all operations. These specialties had readmission rates of 8% (orthopedic), 13% (general), and 17% (vascular).

The group of clinicians developed a list of categories, which identified reasons for 90% of the readmissions in our sample. This list included 50 distinct ICD-9 code–based categories (Table 1). The most prevalent reasons for readmission included other fluid and electrolyte codes (19.7%), other mental health codes (16.5%), anemia (16.9%), pain-related codes (15.8%), and postoperative abscess/septicemia (15.3%). In the first round of the Delphi process, panelists disagreed about the categorization of 2 reasons: amputation neuroma or infection and injury/poisoning; in subsequent rounds, we divided these categories to create amputation neuroma and amputation infection and 6 categories of injury/poisoning. In total, 50 readmission reasons were rated by Delphi panelists.

The Delphi panel included 14 experts: surgeons from general (n = 3), orthopedic (n = 1), vascular (n = 2), and colorectal (n = 2) specialties from within and outside the VA; 2 internal medicine physicians, 1 VA and 1 non-VA, both familiar with readmissions research; 2 VA surgical nurses; and 2 VA social workers familiar with postoperative patients. Panelists rated readmission reasons across 3 rounds and participated in 1 of 2 telephone calls between rounds 1 and 2 of the Delphi process.

Table 1 presents the consensus results for each round of the Delphi process. In round 1, consensus was reached on 28 of the 42 readmission reasons (67%). Seven new readmission reasons were introduced in round 2 and consensus was reached on 18 of the 21 readmission reasons (86%). Thus, in round 3, 3 items were presented and there was 100% consensus. The eTable in the Supplement shows the final list of surgical quality–related readmission reasons and their ICD-9 codes.

Panelists were consistent in how frequently they changed ratings between rounds. Overall, ratings changed after the panelists discussed the readmission reasons on which consensus was not achieved during the phone call between rounds 1 and 2.

Of 39 664 postoperative readmissions, 25 521 (64%) had an ICD-9–coded reason for readmission that our Delphi panel considered to be associated with surgical quality. Among 340 858 inpatient operations performed during the study period, the surgical quality–related readmission rate was 7.5%. In comparing the surgical quality–related and unrelated groups, operations on older, male, and functionally dependent patients were more likely to have problems with quality. Surgical procedures with clean wound sites, shorter operative times, and shorter lengths of hospital stay were more common in the cohort with readmission reasons not associated with surgery (Table 2).

The 25 most common surgical procedures in our sample accounted for 268 082 of all 340858 (78.6%) operations performed. We observed variation in readmission rates by procedure and in the proportion of the readmissions that could be attributed to surgical quality–related problems (Table 3). For example, cholecystectomy and common duct surgical exploration accounted for 1598 of 15 514 surgical procedures, with a 10.3% readmission rate; however, 853 (53.4%) of these readmissions were coded with a surgical quality–related reason by our Delphi panel. In contrast, there were 3538 readmissions after 24 338 colorectal resections, a 14.5% rate, and 2584 (73.0%) were coded with a surgical quality–related readmission reason. The surgical quality–related readmission rate differed from the overall readmission rate for every procedure commonly performed in our sample. Of procedures with the highest readmission rates, amputation of lower extremity and peripheral vascular bypass had readmission rates greater than 20% and surgical quality–related readmission rates of approximately 15%; however, transurethral excision, drainage, or removal of urinary obstruction had 793 readmissions from 3873 surgical procedures, a 20.5% overall readmission rate but 457 (11.8%) of the readmissions were related to surgical quality.

Ranking the procedures by readmission rate revealed consistently higher ranks for amputation of lower extremity, peripheral vascular bypass, and transurethral excisions and consistently lower ranks for arthroplasty procedures in overall as well as surgical quality-related readmissions. Within these bounds, however, there were shifts in rank when readmission rates were restricted to those deemed to be associated with surgical quality problems.

The goal of this study was to use a consensus-based list of readmission reasons to compare the rate of overall readmission with the rate of surgical quality–related readmission. We looked at differences in patient and procedure factors as well as differences in the readmission rate for various surgical specialties and common procedures. In our sample, 64% of postoperative readmissions could be attributed to possible surgical quality–related problems and there were differences in patient and procedure characteristics between quality-related and unrelated readmissions. Although the procedures with the highest and lowest readmission rates were the same in both cohorts, many of the surgical procedures changed rank when readmissions were limited to those deemed surgical quality–related.

Delphi processes for establishing expert consensus have been used to develop clinical quality measures and identify important processes of care associated with medical readmissions.^9,10,13 We extended these methods to the realm of surgical readmissions. Strengths of the Delphi process include the anonymous and iterative nature of the data collection, which allows for reasoned expert feedback with less bias from more forthright participants.¹⁶ Selection of participants is critical to obtaining useful results, and we identified a variety of clinical experts familiar with surgical care, quality problems, and readmissions; however, it is possible that our Delphi panel was not sufficiently representative and that other clinical experts in surgery would have arrived at different conclusions. Therefore, Delphi findings should be considered an early step in establishing quality measures that needs further reliability testing.¹⁷

In addition to conducting a Delphi process, we examined the differences in readmission groups if only the surgical quality–related readmissions were analyzed instead of all readmissions. We found key differences in intraoperative and postoperative characteristics between the 2 groups that reinforce the reliability of the Delphi panel’s definition of surgical quality–related readmissions. We expected that in cases of quality-related postoperative readmission, potential problems would be evident before discharge, and we found higher rates of wound contamination and longer operative times in this group. These factors have been shown to predict surgical complications.^18,19

Our results have implications for quality measurement. The primary concern with using overall readmission rates to measure surgical quality is the likelihood that a fraction of readmissions reflect true quality-related problems. As we demonstrated by comparing the procedures associated with any readmissions with those associated with surgical quality–related readmissions, the types of procedures accounting for potential quality problems varies depending on the definition of readmission. A more precise approach to identifying quality-related readmissions may benefit health care systems and clinicians in their quality improvement efforts.

Research has largely focused on medical readmissions, and this study is an important contribution to the surgical readmissions literature. However, our study has some limitations. Although the Delphi panel’s definition of surgical quality–related readmission detected a different cohort of patients and identified different surgical procedures associated with these readmissions, the definition relied on ICD-9 coding. Previous studies indicate that ICD-9 coding–based detection of postoperative outcomes has limitations with respect to the reliability of the diagnosis code and the reason for admission.^20-22 Our readmission results were limited to care recorded in the VASQIP data and did not include all VA inpatient surgical procedures, per the VASQIP sampling methodology; however, VASQIP captured 83% of all VA inpatient operations during the study period. In addition, we did not include readmissions to non-VA hospitals. Although this omission likely reduced the number of readmissions in our sample, previous VA readmissions research found that the number of patients admitted to non-VA facilities after a VA inpatient stay was small.²³ Finally, explicit medical record review is considered the criterion standard detection method for surgical quality–related readmissions.⁵ Although our study lays the groundwork for developing explicit record review guidelines, further work is necessary to confirm the reliability of the definition of surgical quality–related readmissions developed by the Delphi process and to determine whether this administrative coding–based methodology can be used for quality measurement.

Although the use of readmissions as a quality measure is controverisal,²⁴ they are a readily available outcome that can be used to compare performance across hospitals or to target quality improvement work.^3,7 However, our analysis suggests that one-third of all postoperative readmissions are unlikely to be associated with surgical quality. Future research should examine whether focusing on readmissions for only those admission codes thought to identify surgical quality–related cases would yield a more useful measure of quality. This is the first step toward the development of a surgical readmission quality metric.

Accepted for Publication: January 14, 2018.

Corresponding Author: Hillary J. Mull, PhD, MPP, Center for Healthcare Organization & Implementation Research, VA Boston Healthcare System, 150 S Huntington Ave (152M), Boston, MA 02130 (hillary.mull@va.gov).

Published Online: April 18, 2018. doi:10.1001/jamasurg.2018.0592

Author Contributions: Drs Mull and Graham had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Mull, Graham, Morris, Burns, Itani, Hawn.

Acquisition, analysis, or interpretation of data: Mull, Graham, Morris, Rosen, Richman, Whittle, Burns, Wagner, Copeland, Wahl, Jones, Hollis, Hawn.

Drafting of the manuscript: Mull, Graham, Rosen.

Critical revision of the manuscript for important intellectual content: Graham, Morris, Rosen, Richman, Whittle, Burns, Wagner, Copeland, Wahl, Jones, Hollis, Itani, Hawn.

Statistical analysis: Mull, Graham, Richman.

Obtained funding: Graham, Morris, Rosen, Burns, Hawn.

Administrative, technical, or material support: Mull, Graham, Whittle, Burns, Wagner, Wahl, Jones.

Study supervision: Morris, Rosen, Burns, Itani, Hawn.

Conflict of Interest Disclosures: None reported.

Funding/Support: This research was supported by funding from the Veterans Affairs Health Services Research and Development Service grants CDA 13-270 (Dr Mull) and IIR 12-358 (Dr Hawn).

Role of the Funder/Sponsor: The US Department of Veterans Affairs was not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The opinions expressed are those of the authors and not necessarily those of the US Department of Veterans Affairs.

Additional Contributions: We are grateful to the unnamed Delphi participants who contributed to this work.