Customize your JAMA Network experience by selecting one or more topics from the list below.
Gani F, Canner JK, Pawlik TM. Use of the Modified Frailty Index in the American College of Surgeons National Surgical Improvement Program Database: Highlighting the Problem of Missing Data. JAMA Surg. 2017;152(2):205–207. doi:10.1001/jamasurg.2016.3479
Given the aging surgical population, there is increased interest in preoperatively identifying frail patients who may be at risk for adverse postoperative outcomes.1 In 2013, Velanovich et al2 mapped 70 variables contained within the frailty index (FI) proposed by Rockwood et al3 onto data from the 2005-2009 American College of Surgeons National Surgical Improvement Program (ACS-NSQIP) to develop an 11-point modified frailty index (mFI) to identify patients at risk for adverse postoperative outcomes. While the ACS-NSQIP initially mandated the reporting of all variables contained within the mFI, beginning in 2011, the reporting of some preoperative patient comorbidity variables was made optional. In addition, more recent versions of the ACS-NSQIP since 2012 do not record these variables altogether. Despite the change in ACS-NSQIP variables, the mFI continues to be widely used by surgeons and researchers to assess frailty within the ACS-NSQIP database.4-6 We examined the use of the mFI within the ACS-NSQIP database over time with regard to missing data as well as predictive ability.
Patients undergoing a pancreaticoduodenectomy (Current Procedure Terminology codes 48150 and 48153) between 2005 and 2013 were identified using the ACS-NSQIP database. For each patient, an mFI score was calculated, and patients were categorized based on mFI score into 4 groups: those with an mFI score of 0, 1, 2, or 3 or greater (of a maximum score of 11). Missing data for patient characteristics and mFI scores were compared across the study period. Logistic regression and area under the receiver operating curve analyses were performed to evaluate the discrimination of the mFI against 30-day postoperative mortality as well as changes in discrimination over time. As this study used deidentified data that was compliant with the Health Insurance Portability and Accountability Act of 1996, informed consent was waived, and this study was approved by the Johns Hopkins University Institutional Review Board.
A total of 18 159 patients were identified who underwent a pancreaticoduodenectomy between 2005 and 2013 at a hospital participating in the ACS-NSQIP. The number of patients undergoing a pancreaticoduodenectomy at a participating ACS-NSQIP center increased from 297 patients in 2005 to 3881 patients in 2013 (Table 1). While the proportion of patients presenting with comorbidities was similar over time, the proportion of missing data for 5 of the 11 comorbidity variables included within the mFI increased over time. Specifically, the variables “history of myocardial infarction,” “history of percutaneous intervention, coronary stenting or cardiac surgery,” “history of peripheral vascular disease, rest pain or gangrene,” “history of impaired sensorium,” and “history of transient ischemic attack” were missing for 100% of patients in 2013 compared with 54.9% of patients in 2011. Furthermore, the missing data were not missing at random. Although an increasing mFI score was associated with greater odds for postoperative mortality, the mFI demonstrated poor discrimination for the years between 2005 and 2013 (area under the curve: range, 0.56-0.65) (Table 2).
To assess the extent of missing data within the data set, additional analyses were performed among patients undergoing an esophagectomy (Current Procedure Terminology code 43107, 43112, 43117, 43121, and 43122) or a total hip replacement (Current Procedure Terminology code 27130). Of note, a similar pattern of missing data was also observed among these patients. Of the variables “history of myocardial infarction,” “history of percutaneous intervention, coronary stenting or cardiac surgery,” “history of peripheral vascular disease, rest pain or gangrene,” “history of impaired sensorium,” and “history of transient ischemic attack,” 43.7% and 66.5% of data were missing for esophagectomy and total hip replacement, respectively, in 2011, whereas 100% of data was missing in 2013 for both esophagectomy and total hip replacement (data not shown).
An increasing proportion of missing data for variables used to calculate mFI scores was observed over the period examined. In fact, data for 5 of the 11 composite variables used to calculate mFI scores were noted to be completely missing following 2012. Despite this, multiple studies4,5 continue to apply mFI scores to the ACS-NSQIP to identity frail surgical patients without acknowledging the problem of missing data or evaluating its predictive power and discrimination.
Our data highlight the need for researchers to exercise caution in using mFI scores to avoid misrepresentation and erroneous inferences that cannot be supported by the existing data. Furthermore, given the existing gaps and limitations associated with the mFI, a new tool to assess frailty within ACS-NSQIP data is needed.
Corresponding Author: Timothy M. Pawlik, MD, MPH, PhD, FRACS(Hon), Department of Surgery, The Ohio State University Wexner Medical Center, 395 W 12th Ave, Ste 670, Columbus, OH 43210 (email@example.com).
Published Online: October 26, 2016. doi:10.1001/jamasurg.2016.3479
Author Contributions: Drs Gani and Pawlik had full access to all the data in the study and take full responsibility for the integrity for the data and the accuracy of the data analysis.
Concept and design: Gani, Pawlik.
Acquisition, analysis, or interpretation of data: All Authors.
Drafting of the manuscript: Gani, Pawlik.
Critical revision of the manuscript for important intellectual content: All Authors.
Statistical analysis: Gani, Pawlik.
Administrative, technical, or material support: All Authors.
Study supervision: Pawlik.
Conflict of Interest Disclosures: None reported.
Disclaimer: Dr Pawlik is a deputy editor of JAMA Surgery. He was not involved in the evaluation or decision to accept this article for publication. Additionally, the American College of Surgeons National Surgical Quality Improvement Program and the participating hospitals are the source of the data used herein; they have not verified and are not responsible for the statistical validity of the data analysis or the conclusions derived by the authors.
Create a personal account or sign in to: