Development and Initial Validation of the Risk Analysis Index for Measuring Frailty in Surgical Populations

Key Points

Question  Can frailty be measured rapidly, accurately, and reliably enough to inform surgical decision making?

Findings  In this cohort study, the novel Risk Analysis Index (RAI) measures frailty prospectively (RAI-C) using a questionnaire or retrospectively (RAI-A) using variables from Veterans Affairs or American College of Surgeons National Surgical Quality Improvement Projects. The RAI-C proved feasible for systemwide screening with good predictive power and subsample demonstrated similar predictive power between the RAI-C, RAI-A, and modified Frailty Index.

Meaning  The RAI may measure frailty with predictive ability on par with other frailty tools.

Importance  Growing consensus suggests that frailty-associated risks should inform shared surgical decision making. However, it is not clear how best to screen for frailty in preoperative surgical populations.

Objective  To develop and validate the Risk Analysis Index (RAI), a 14-item instrument used to measure surgical frailty. It can be calculated prospectively (RAI-C), using a clinical questionnaire, or retrospectively (RAI-A), using variables from the surgical quality improvement databases (Veterans Affairs or American College of Surgeons National Surgical Quality Improvement Projects).

Design, Setting, and Participants  Single-site, prospective cohort from July 2011 to September 2015 at the Veterans Affairs Nebraska–Western Iowa Heath Care System, a Level 1b Veterans Affairs Medical Center. The study included all patients presenting to the medical center for elective surgery.

Exposures  We assessed the RAI-C for all patients scheduled for surgery, linking these scores to administrative and quality improvement data to calculate the RAI-A and the modified Frailty Index.

Main Outcomes and Measures  Receiver operator characteristics and C statistics for each measure predicting postoperative mortality and morbidity.

Results  Of the participants, the mean (SD) age was 60.7 (13.9) years and 249 participants (3.6%) were women. We assessed the RAI-C 10 698 times, from which we linked 6856 unique patients to mortality data. The C statistic predicting 180-day mortality for the RAI-C was 0.772. Of these 6856 unique patients, we linked 2785 to local Veterans Affairs Surgeons National Surgical Quality Improvement Projects data and calculated the C statistic for both the RAI-A (0.823) and RAI-C (0.824), along with the correlation between the 2 scores (r = 0.478; P < .001). Of these 2785 patients, there were sufficient data to calculate the modified Frailty Index for 1021, in which the C statistics were 0.865 (RAI-A), 0.797 (RAI-C), and 0.811 (modified Frailty Index). The correlation between the RAI-A and RAI-C was 0.547, and the correlations of the modified Frailty Index to the RAI-A and RAI-C were 0.301 and 0.269, respectively (all P < .001). A cutoff of RAI-C of at least 21 classified 18.3% patients as “frail” with a sensitivity of 0.50 and specificity of 0.82, whereas the RAI-A was less sensitive (0.25) and more specific (0.97), classifying only 3.7% as “frail.”

Conclusions and Relevance  The RAI-C and RAI-A represent effective tools for measuring frailty in surgical populations with predictive ability on par with other frailty tools. Moderate correlation between the measures suggests convergent validity. The RAI-C offers the advantage of prospective, preoperative assessment that is proved feasible for large-scale screening in clinical practice. However, further efforts should be directed at determining the optimal components of preoperative frailty assessment.

Frailty is a geriatric concept identifying those patients (regardless of age) at increased risk of dying in 6 months to 5 years owing to a decline in physiologic reserve.^1-6 This decline in physiologic reserve has important implications for the surgical patient because a diagnosis of frailty is associated with markedly increased risks for postoperative mortality and morbidity. For example, when compared with robust patients, frail surgical patients are less likely to be discharged to home,⁷ more likely to be readmitted within 30 days,^8,9 and have substantially increased rates of perioperative mortality and complications.^9-13 Given these data, growing consensus suggests that frailty-associated risks should inform shared decision making and lead to discussions clarifying realistic goals of care.

Despite the clear clinical significance of frailty in surgical populations, there is no consensus on how best to define or measure frailty, even within the geriatric literature.³ For example, an international panel of experts gathered in 2011 agreed that frailty is a multidimensional construct consisting of 6 domains (physical performance, gait speed, mobility, nutritional status, mental health, and cognition) that together indicate a high risk of mortality in 1 to 2 years, but they could not agree on a definition of frailty.¹⁴ A diversity of measures exists to measure some or all these domains by assessing a combination of grip strength, cognition, walking speed, or the time to get out of a chair and walk 4 m, but only select research-focused tools have been validated in surgical populations.^7,11,15 These tools are sensitive, specific, and well suited to research protocols, but they are too resource-intensive for rapid, cost-effective, preoperative screening of entire populations considering elective surgery.

We therefore sought to develop a frailty index that could function as an effective screening tool for all patients considering elective surgery. We reasoned the tool should effectively distinguish between frail and robust patients in terms of increased risk for mortality and complication. It should also be easy and efficient to incorporate into the existing workflow of preoperative evaluation without causing delays or stressing existing resources. It should thus be assessable by medical personnel at varying levels of training (eg, medical assistant, medical student, licensed practical nurse, resident, and attending surgeon). Ideally, the risk should be calculable both prospectively from patient questionnaires and retrospectively from administrative data.

To address this need, we report the development and initial validation of the Risk Analysis Index (RAI), a novel frailty index for use in surgical populations. The clinical RAI (RAI-C) uses a questionnaire to calculate the risk index based on answers provided by the patient or surrogate, and it is intended for prospective screening of patients considering operative intervention. The administrative RAI (RAI-A) can be calculated retrospectively using frailty-associated variables from the Veterans Affairs or American College of Surgeons National Surgical Quality Improvement Projects (VASQIP/ACS-NSQIP) data sets.

The RAI-C and RAI-A are both adaptations of the Minimum Data Set (MDS) Mortality Risk Index–Revised (MMRI-R).^16,17 From a set of 50 frailty-related variables included in the MDS, the developers of the MMRI-R used stepwise logistic regression to select the 12 variables that most consistently predicted 6-month mortality. They then developed a 15-item survey to measure these variables (4 items to assess activities of daily living and 1 item for each of the other 11 variables from the MDS). The survey relies exclusively on the report of the patient (or surrogate) and can be easily administered by nursing personnel. It requires neither functional assessment of patient performance nor medical record review, but when each item is scored with weights derived from the validated model, it reliably predicts 6-month mortality at the time of admission to a nursing home (C statistic = 0.76).¹⁷

As part of a quality improvement initiative at the Veterans Affairs Nebraska–Western Iowa Heath Care System, we adapted the MMRI-R for use in surgical populations. We eliminated the single survey item assessing current or recent dehydration because we thought that this question would be difficult to assess and interpret in the preoperative population. We also modified the item probing admission to a nursing home in the past 3 months to capture anyone living in a setting other than independent living. We thought this would more expansively capture the range of nonindependent living situations prevalent among surgical populations that might indicate frailty-associated risk. The RAI-C questionnaire includes 14 questions assessing 11 variables and 2 statistical interactions with scores ranging from 0 to 81 (eFigure in the Supplement). The survey is administered by clinical staff based on patient history and report and scored using parameters developed for the MMRI in an MDS sample of nursing home residents.

Pilot testing demonstrated the feasibility of this abbreviated survey. Because most of the questions were already part of standard nursing interviews, it took clinical staff less than 2 minutes to complete and was easily incorporated into the standard intake procedures at surgical clinics. Based on these findings, on July 1, 2011, we began measuring the RAI-C for every patient presenting to outpatient surgery clinics at the Veterans Affairs Nebraska–Western Iowa Health Care System, requiring the score as a precondition for scheduling any elective surgery.

Responses to each item of the RAI-C were recorded along with patient identifiers. As described elsewhere, patients with an RAI-C score of at least 21 were subjected to administrative review aimed at improving perioperative decision making and outcomes.¹⁸ In some cases, this administrative review led to repeated calculation of the RAI-C, often informed by more detailed medical histories. As such, the database includes sequential measurements of the RAI-C on some patients, but for the purposes of this analysis, we used the single RAI-C measurement for each patient that was closest to and antecedent from the date of surgery.

In addition to measuring the RAI-C prospectively, we sought to develop a version of the RAI that could be calculated from variables captured by the VASQIP/ACS-NSQIP. We therefore scrutinized the list of VASQIP variables to identify those that best approximated the 11 variables assessed by the questions in the RAI-C. When definitional clarity was required, we referred back to the definitions of the MDS variables from which the RAI-C and MMRI-R were derived. See the eAppendix in the Supplement for a complete rationale for variable selection. We then developed a scoring convention to calculate the RAI-A score from the identified VASQIP variables (eTable 1 in the Supplement).

To construct our database for analysis, we began with the database recording all responses to the RAI-C survey, checking each item for out-of-range values and missing data, correcting errors when possible by reference to the electronic medical record. We then used each patient’s last name and a portion of his or her social security number to link the RAI-C score to vital status and date of death from the vital statistics file. We then used the patient identifiers and the date of the proposed surgery (recorded on the RAI-C survey) to search the electronic medical record and link records to a specific surgery, recording the surgery’s unique identifier to permit later linkage to VASQIP data. If no surgery was identified for the specific patient and date, we searched the patient’s electronic medical record for any surgery performed within 90 days after the date of RAI-C evaluation, linking the patient’s record to the first surgery (if any) identified in this time frame. The Nebraska–Western Iowa Veterans Affairs Medical Center institutional review board determined these procedures to be “operations activities not constituting research” according to Veterans Affairs Handbook 1058.05, and thus, per VHA Policy, the information presented in this article does not require informed consent or institutional review board approval.

The primary outcome was survival, calculated from the date of surgery to the date of death for those who died and to the last recorded death date (July 24, 2015) for those who survived, excluding all surgeries performed after this date. Dichotomous variables for 30-day, 180-day, and 365-day mortality were calculated, excluding those whose follow-up from date of surgery to July 24, 2015, was less than the specified 30 days, 180 days or 365 days.

We then used each surgery’s unique identifier to link records to local VASQIP data. Not all surgeries were included in VASQIP, but for those that were, we calculated the RAI-A as described in previous paragraphs and the modified Frailty Index (mFI) as described by Adams et al,¹⁰ Karam et al,¹⁹ Tsiouris et al,²⁰ and Velanovich et al.²¹ We also calculated 2 dichotomous composite variables for postoperative complications, indicating the occurrence of (1) any VASQIP-measured complication except urinary tract and superficial wound infections and (2) Clavien-Dindo level IV complications, defined as any VASQIP variable for septic shock, postoperative dialysis, pulmonary embolus, myocardial infarction, cardiac arrest, prolonged ventilation, reintubation, coma, stroke, or return to the operating room.

Statistical analyses began with summary descriptions of the cohort, including rates of mortality and morbidity. We then estimated receiver operating characteristics with nonparametric 95% confidence intervals, C statistics, sensitivity, and specificity. We also plotted Kaplan-Meier curves stratified by frailty scores. Correlation between frailty measures was estimated with nonparametric methods (Spearman ρ). All analyses were made using SPSS, version 23 (IBM Corp).

From July 2011 to September 2015, we assessed the RAI-C 10 698 times in 6856 unique patients with known length of postoperative survival (Figure 1). Of these, 6803, 6419, and 5959 patients were followed up for 30 days, 180 days, and 365 days, respectively. A subsample of 2785 patients were further linked to VASQIP data, from which we calculated the RAI-A. Of these, only 1021 had VASQIP data sufficient to calculate mFI because several mFI variables were phased into VASQIP during our sampling frame.

The demographic characteristics of each sample were similar with regard to age, sex, race/ethnicity, American Society of Anesthesiologists classification, and RAI-C score (eTable 2 in the Supplement). Administrative RAI scores were somewhat lower than RAI-C scores, and this likely reflects the stringent rules for abstracting the VASQIP variables from which the RAI-A is calculated. This sensitivity is apparent when comparing the individual components of the RAI score. For example, the RAI-C questionnaire elucidated a history of heart failure from 3% (n = 90 of 2785) to 4% (n = 274 of 6856) of patients, but only 0.1% (n = 3 of 2785) to 0.2% (n = 2 of 1021) of patients had VASQIP variables indicating a history of congestive heart failure. Similar discrepancies in sensitivity were evident for shortness of breath and cancer (eTable 2 in the Supplement).

Rates of mortality and postoperative complications increased with frailty (Table 1). For example, 180-day mortality rates for RAI-C and RAI-A scores of 15 or less were 0.9% and 0.8%, respectively, but increased to 17.3% and 10.0%, respectively, for scores of at least 36. For each strata of frailty, mortality increased dramatically with lengthening time horizons. For example, the 4.9% 30-day mortality associated with RAI-C scores of at least 36 grew to a staggering 26.8% at 1 year. However, most of even the frailest patients (60%-70%) experienced long-term survival that stabilized 2 to 3 years after the operation (Figure 2).

Complication rates at 30 days were higher than the rates of mortality, ranging from 4.6% to 25% for any complication other than urinary tract or superficial wound infections and 2.5% to 15% for life-threatening Clavien-Dindo level IV complications (Table 1). Similar patterns of outcomes were observed for the mFI.

The C statistics of the RAI-C, RAI-A, and mFI were fairly consistent across the 3 cohorts for both mortality and complications as presented in Table 2. Depending on the cohort or time horizon, the RAI-C predicted mortality with C statistics between 0.704 and 0.824. The RAI-A and mFI were similar, with somewhat higher C statistics ranging from 0.739 to 0.979. In general, sensitivity and specificity of predicting 180-day mortality were maximized at the lower end of the range of frailty scores where a cut point of RAI-C of at least 11 was 72% sensitive and 73% specific (eTable 3 in the Supplement).

Tables 1 and 2 report only those cases linked to a specific date of surgery. To explore the possibility that the excluded cases biased results, we analyzed data from all 9778 patients with known vital status (Figure 1). For cases without a specific date of surgery, we calculated the length of survival from the date of RAI-C assessment. Mortality rates and C statistics for this larger sample (eTable 4 in the Supplement) were similar to those reported in Tables 1 and 2, suggesting that our inability to link patients did not bias our findings among the 6856 patients reported here.

The RAI-A, RAI-C, and mFI were significantly correlated with each other (Table 2). The strength of the correlation between the RAI-A and RAI-C was moderate (Spearman ρ = .547; P <.001, n = 1021), whereas the correlations between the mFI and the RAI-C (Spearman ρ = .269; P <.001) and RAI-A (Spearman ρ = .269; P <.001) were weaker. The Venn diagram in Figure 3 further illustrates these correlations demonstrating the partial overlap between the different frailty measures.

In this study, we developed and validated 2 new frailty indices for use in surgical populations, finding that the RAI-C and RAI-A predict postoperative mortality and morbidity with as good or better predictive ability than other existing measures of frailty. For example, the C statistics for the RAI-C and RAI-A predicting 180-day mortality were 0.797 and 0.865, respectively, which is a moderate improvement over the MMRI-R from which they are derived (C = 0.760) and similar to the mFI (C = 0.811). Although our methods do not permit direct comparison, these data demonstrate similar predictive ability to frailty measures based on physiological performance such as the Hopkins Frail Scale¹¹ and the Timed Up-and-Go.²²

To our knowledge, the RAI-C is the first frailty index used explicitly for systemwide screening of surgical populations, taking only 1 to 2 minutes to complete as part of the standard intake interview and now prospectively validated in cohort of 6856 patients. Makary et al¹¹ used the Hopkins Frail Scale to measure preoperative frailty in 594 elective surgical patients aged 65 years or older, showing strong associations with surgical outcomes. Robinson et al²³ demonstrated the association between frailty and surgical outcomes in cohorts of more than 200 patients by measuring preoperative frailty with a combination of measures including Katz score, Timed Up-and-Go, Charlson Index, Mini-Cog, hemoglobin, and falls reported in the last 6 months. However, the time required to assess frailty with these research-focused tools precludes systemwide screening. In this study, we demonstrated the feasibility of systemwide screening in routine clinical practice with the RAI-C, validating its predictive power in what is, to our knowledge, the single largest cohort of surgical patients published to date.

Similar to the mFI, the RAI-A, is calculated from VASQIP variables and thus suitable for secondary data analyses exploring the associations of frailty and surgical outcomes. Although the overall performance of the 2 tools was similar, the RAI-A generates a wider range of scores, and this may permit more precise selection of cut points for a variety of applications.

The RAI-A and mFI are somewhat more specific than the RAI-C. This is likely owing to the different methods of assessing frailty-associated risk. The RAI-C relies on the clinical judgment of the personnel administering the questionnaire. For this study, clinicians were instructed to use their best judgment in scoring the RAI-C, and as such, it is likely that the presence of specific risk factors was interpreted more liberally than the stringent coding rules required by VASQIP for the variables on which the RAI-A and mFI are based. However, these methods are well-suited to the purposes of the 2 instruments. To the extent that the RAI-C is intended for screening, its sensitivity is an advantage, and to the extent that the RAI-A is used to test associations between frailty and surgical outcomes, its specificity may provide benefit.

As with any imperfect test, the cut point to rule frailty in or out involves a tradeoff between sensitivity and specificity. The advantage of a granular scale, such as the RAI, is that cutoffs could be chosen based on the health care setting, level of precision required, and planned interventions for frail patients. For example, an RAI-C value of 11 is 72% sensitive and 73% specific (eTable 3 in the Supplement), but because 29% of the population would rule in for frailty at this cut point, it is insufficiently specific for use in busy surgical clinics. Cut points between 16 and 21 may strike a more pragmatic balance, identifying 18% to 21% of the population as frail with 52% to 61% sensitivity and 80% to 83% specificity. Cutoffs such as these could rapidly identify most potentially frail patients in the first stage of a 2-stage screening paradigm. If greater specificity is required, the second stage could confirm suspected frailty using 1 or more of the functional measures of frailty that are too labor intensive for screening all patients in predominantly robust populations.

The correlation between RAI-A and RAI-C suggests moderate convergent validity. Correlations with the mFI are weaker and may illustrate the enduring lack of consensus regarding the definition of frailty.¹⁴ However, the weak correlation is in line with other research that demonstrates only limited overlap between different frailty measures,²⁴ suggesting that the overarching syndrome of frailty is larger than any single measure captures.²⁵ As such, we suggest that frailty is best measured with multiple modalities. Screening can be accomplished efficiently with the RAI-C, selecting a cutoff with the desired sensitivity and specificity; those identified as potentially frail can then complete a battery of more time-intensive tests similar to the work by Robinson et al^7,15 that capture multiple domains including functional performance and serological biomarkers.

Although most frail patients live at least 2 to 3 years after their surgery, the risk of mortality increases dramatically with increasing frailty. Because the RAI-C can be calculated in advance of surgery, and because these data provide estimates of the mortality risk, it is now possible to provide risk estimates to patients based solely on their frailty. Whereas the ACS and VA NSQIP risk calculators focus on the immediate perioperative period of 30 days, the RAI-C extends risk estimates out to 6 months and 12 months. As such, the RAI-C may help place traditional 30-day risk estimates into the context of the overarching trajectory of the patient’s life, thus informing the decision-making process shared by patients and surgeons.

Our findings are limited in several ways. First, our data are limited to a single VA medical center and may not generalize to the broader VA or US populations. Second, we were not able to find mortality or surgical data on some of the patients assessed with the RAI-C, and as such, these missing data may represent a source of bias. Third, mortality and morbidity may not be the outcomes of greatest importance for frail patients. Future work should assess the association of frailty with patient-centered outcomes such as independent living, discharge to home, or patient centeredness of care.

The RAI-C and RAI-A represent effective tools for measuring frailty in surgical populations, with predictive abilities on par with other published measures of frailty. The RAI-C offers the additional advantage of assessing frailty-associated risk preoperatively and prospectively, thus presenting an opportunity for feasible, large-scale screening of surgical populations in clinical practice. Its 6- to 12-month time horizon may also help place traditional 30-day risk estimates into the wider context of patients’ lives. However, further efforts are needed to determine the optimal components of preoperative frailty assessment, and future work could more definitively validate the RAI-C and RAI-A in nonveteran, community populations and by comparing their performance with some of the functional measures of frailty.

Corresponding Author: Daniel E. Hall, MD, UPMC Presbyterian, 200 Lothrop St, Ste 1264, Pittsburgh, PA 15213 (hallde@upmc.edu).

Accepted for Publication: July 21, 2016.

Published Online: November 23, 2016. doi:10.1001/jamasurg.2016.4202

Author Contributions: Dr Hall had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Hall, Arya, Schmid, Lynch, Johanning.

Acquisition, analysis, or interpretation of data: Hall, Schmid, Blaser, Carlson, Bailey, Purviance, Bockman, Johanning.

Drafting of the manuscript: Hall, Schmid, Blaser, Carlson, Johanning.

Critical revision of the manuscript for important intellectual content: Hall, Arya, Schmid, Carlson, Bailey, Purviance, Bockman, Lynch, Johanning.

Statistical analysis: Hall, Schmid, Blaser.

Administrative, technical, or material support: Hall, Carlson, Purviance, Bockman, Lynch, Johanning.

Other – Nursing support implementation: Bockman.

Conflict of Interest Disclosures: Dr Hall serves as a consultant to University of Pittsburgh Medical Center on frailty. Dr Johanning holds intellectual property on frailty through FutureAssure, LLC. No other disclosures are reported.

Funding/Support: This research was supported by supported by the US Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development (CDA 08-281; Dr Hall).

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The opinions expressed here are those of the authors and do not necessarily reflect the position of the Department of Veterans Affairs or the US government.

Previous Presentations: This paper was presented at the 40th Annual Surgical Symposium of the Association of VA Surgeons; April 12, 2016; Virginia Beach, Virginia.