A, Kaplan-Meier estimate of overall survival (P < .05; log-rank test). B, Kaplan-Meier estimate of disease-specific survival (P < .05; log-rank test). C, Kaplan-Meier estimate of disease-free survival (P < .05; log-rank test). The clinical care signature is a small group of process-related care practices associated with the overall quality of care. All estimates are stratified by compliance with the clinical care signature quality metrics (100% vs 51%-99% vs ≤50%).
A, Cox multivariable survival analysis for overall survival (100% vs ≤50%: adjusted hazard ratio [aHR], 4.2; 95% CI, 2.1-8.5; 100% vs 51%-99%: aHR, 1.7; 95% CI, 1.0-3.1). B, Cox multivariable survival analysis for disease-specific survival (100% vs ≤50%: aHR, 3.9; 95% CI, 1.7-9.0; 100% vs 51%-99%: aHR, 1.3; 95% CI, 0.6-2.9). C, Cox multivariable survival analysis for disease-free survival (100% vs ≤50%: aHR, 3.0; 95% CI, 1.5-5.8; 100% vs 51%-99%: aHR, 1.6; 95% CI, 0.9-2.7). The clinical care signature is a small group of process-related care practices associated with the overall quality of care. All estimates are stratified by compliance with the clinical care signature quality metrics after adjusting for pT stage, pN stage, extracapsular spread, margin status, and comorbidity.
eTable. Patient and Treatment Characteristics
eFigure. Compliance With Quality Metrics
Customize your JAMA Network experience by selecting one or more topics from the list below.
Graboyes EM, Gross J, Kallogjeri D, et al. Association of Compliance With Process-Related Quality Metrics and Improved Survival in Oral Cavity Squamous Cell Carcinoma. JAMA Otolaryngol Head Neck Surg. 2016;142(5):430–437. doi:10.1001/jamaoto.2015.3595
Quality metrics for patients with head and neck cancer are available, but it is unknown whether compliance with these metrics is associated with improved patient survival.
To identify whether compliance with various process-related quality metrics is associated with improved survival in patients with oral cavity squamous cell carcinoma who receive definitive surgery with or without adjuvant therapy.
Design, Setting, and Participants
A retrospective cohort study was conducted at a tertiary academic medical center among 192 patients with previously untreated oral cavity squamous cell carcinoma who underwent definitive surgery with or without adjuvant therapy between January 1, 2003, and December 31, 2010. Data analysis was performed from January 26 to August 7, 2015.
Surgery with or without adjuvant therapy.
Main Outcomes and Measures
Compliance with a collection of process-related quality metrics possessing face validity that covered pretreatment evaluation, treatment, and posttreatment surveillance was evaluated. Association between compliance with these quality metrics and overall survival, disease-specific survival, and disease-free survival was calculated using univariable and multivariable Cox proportional hazards analysis.
Among 192 patients, compliance with the individual quality metrics ranged from 19.7% to 93.6% (median, 82.8%). No pretreatment or surveillance metrics were associated with improved survival. Compliance with the following treatment-related quality metrics was associated with improved survival: elective neck dissection with lymph node yield of 18 or more, no unplanned surgery within 14 days of the index surgery, no unplanned 30-day readmissions, and referral for adjuvant radiotherapy for pathologic stage III or IV disease. Increased compliance with a “clinical care signature” composed of these 4 metrics was associated with improved overall survival, disease-specific survival, and disease-free survival on univariable analysis (log-rank test; P < .05 for each). On multivariable analysis controlling for pT stage, pN stage, extracapsular spread, margin status, and comorbidity, increased compliance with these 4 metrics was associated with improved overall survival (100% vs ≤50% compliance: adjusted hazard ratio [aHR], 4.2; 95% CI, 2.1-8.5; 100% vs 51%-99% compliance: aHR, 1.7; 95% CI, 1.0-3.1), improved disease-specific survival (100% vs ≤50% compliance: aHR, 3.9; 95% CI, 1.7-9.0; 100% vs 51%-99%: aHR, 1.3; 95% CI, 0.6-2.9), and improved disease-free survival (100% vs ≤50% compliance: aHR, 3.0; 95% CI, 1.5-5.8; 100% vs 51%-99% compliance: aHR, 1.6; 95% CI, 0.9-2.7).
Conclusions and Relevance
Compliance with a core set of process-related quality metrics was associated with improved survival for patients with surgically managed oral cavity squamous cell carcinoma. Multi-institutional validation of these metrics is warranted.
The Agency for Healthcare Research and Quality has defined quality health care as “doing the right thing, at the right time, in the right way, for the right person, and having the best possible results.”1 Measures of quality are being used at the individual physician and hospital-wide levels to determine ratings, accreditation, and reimbursement. However, there remains disagreement about how to define quality in a practical and actionable manner,2,3 as well as how current metrics correlate with true quality care.4,5
According to the Donabedian6 model, the quality of health care can be assessed using 3 types of measures: structure (the environment of health care delivery), process (what a health care professional does for the patient), and outcomes (what happens to the patient). Process-related quality metrics are attractive because they can empower health care professionals to change behavior at the individual level.7
Process-related quality measures have been evaluated in general surgery with variable findings between compliance and the outcome of interest.5,8-10 Nationally endorsed and validated quality metrics by groups such as the National Quality Forum and the American College of Surgeons/National Cancer Database Commission on Cancer exist for patients with colorectal, lung, esophageal, breast, and gastric cancer. To our knowledge, no such measures exist for patients with head and neck cancer.
A call for the development of quality measures for patients with oral cavity squamous cell carcinoma (OCSCC) was approved by the executive council of the American Head and Neck Society in 2007.11 In the interim, several publications have begun to address the topic,12-16 and international efforts to codify quality care performance indicators for head and neck cancer have occurred.17,18 However, 9 years later, nationally endorsed quality measures for patients with head and neck cancer still do not exist in the United States,4 and it is unknown whether compliance with the metrics suggested in 2007 is associated with improved patient outcomes. Given the increasing emphasis placed on quality care,1,19 it is imperative to establish quality metrics with predictive validity for patients with head and neck cancer and then demonstrate that compliance with these measures correlates with improved outcomes.
In this study, using a population of patients with surgically treated OCSCC, we sought to determine compliance with a set of process-related quality metrics that possess face validity, measure the association between compliance and survival, identify the individual quality metrics associated with improved survival, describe a “clinical care signature” group of metrics, and determine if compliance with these metrics correlates with improved survival.
A retrospective review of the medical records of patients 18 years or older who underwent primary surgical intervention for an oral cavity malignant neoplasm at our academic medical center from January 1, 2003, to December 31, 2010, was performed. Inclusion criteria for the study were previously untreated primary oral cavity cancer, no history of head and neck cancer, squamous cell carcinoma histologic findings, definitive surgical management, and a minimum 12-month follow-up (or death). Two hundred sixty-seven patients met inclusion criteria; 75 were excluded owing to insufficient documentation in the medical record to assess compliance with the quality metrics. The final cohort comprised 192 patients with previously untreated OCSCC undergoing surgery with or without adjuvant therapy and a minimum follow-up of 12 months or death. Seven surgeons (including B.N.), along with other members of the multidisciplinary head and neck cancer team, treated these patients. The medical records were reviewed for data related to demographics; clinical variables, including comorbidity as measured by the Adult Comorbidity Evaluation-27 index20; details of surgical and adjuvant treatment; pathologic variables; surveillance; and death.
This study was approved by the Washington University School of Medicine Institutional Review Board. Patient data were deidentified once all relevant data had been abstracted from the medical records.
The quality metrics were developed in a multidisciplinary fashion with 10 members of the head and neck cancer team (surgeons, radiation oncologists, and medical oncologists) at our institution (Table 1). The metrics encompassed the longitudinal care of the patient with head and neck cancer (pretreatment evaluation, treatment, and posttreatment surveillance) and were modeled on metrics suggested by the American Head and Neck Society,11 National Comprehensive Cancer Network guidelines for oral cavity cancer,21 or other publications.12,22,23 The metrics were chosen because they possessed the following characteristics: usability (the information is actionable and understandable), feasibility (the data are easily measurable and collectable), reproducibility, meaningfulness (the metrics are agreed on by stakeholders), promotion of quality improvement (compliance with the metrics can be tracked over time), and possession of face validity (a priori suspicion of association with improved outcomes).
These metrics can apply to any subsite of the head and neck mucosa. The oral cavity was chosen to evaluate these metrics because the American Head and Neck Society previously suggested the oral cavity as the site to test quality metrics11 and because the primary treatment modality of OCSCC is most commonly surgical, so there is little variation in the general approach to therapy.
Table 1 shows the eligibility criteria for the quality metrics. Dental evaluation consisted of documentation of a dental referral before, or dental extractions during, the index hospitalization. Elective neck dissection lymph node yield was calculated on the ipsilateral side. If bilateral elective neck dissections were performed, the average of both sides was used.22 Unplanned surgery within 14 days did not include staged neck dissections or dental extractions but did include re-resection for positive margins. We defined a 30-day unplanned readmission as an admission to any service in the hospital within 30 days of discharge that was not expected.24 For thyroid surveillance, if a patient was dead within 12 months of completing radiation treatment, the metric was not evaluable. For surveillance imaging, if imaging was prompted by clinical suspicion of recurrence or the patient was dead within 6 months of the completion of therapy, the metric was not evaluable.
The outcome measures included compliance rates with quality metrics and measures of survival (overall survival [OS], disease-specific survival [DSS], and disease-free survival [DFS]). For each patient, a compliance score for the group of metrics was calculated based on the opportunity model used by the Centers for Medicare & Medicaid Services.25 If a patient was eligible for all 19 processes and received 10, the compliance score was 53% (10 of 19). If a patient was eligible for only 10 processes and received all 10, the compliance score was 100%.
Overall survival was calculated from the date of surgery to the date of death from any cause. Disease-specific survival was calculated from the date of surgery to the date of death from oral cavity cancer or direct effects of treatment. Disease-free survival was calculated from the date of surgery to the date of either death or first recurrence. A local recurrence was defined as squamous cell carcinoma within 2 cm of the original tumor and within 5 years of the original diagnosis; otherwise it was considered a second primary tumor.26 Patients lost to follow-up were censored at the date last known to be alive.
To determine the association between compliance with the group of quality metrics and survival, compliance scores were calculated for each patient as described above. Compliance scores were divided into 11 groups (1, 0%-9%; 2, 10%-19%; 3, 20%-29%; etc). Estimates of survival were calculated for OS, DSS, and DFS using the Kaplan-Meier method. The association between compliance score for the group of metrics and OS, DSS, and DFS was assessed using the log-rank test. Unadjusted Cox proportional hazards analysis was used to calculate the association between the overall compliance score and OS, DSS, and DFS. Multivariable Cox proportional hazards analyses were then performed for overall compliance, controlling for pT stage, pN stage, extracapsular spread, final margin status, and comorbidity. The Adult Comorbidity Evaluation-27 index comorbidity score was dichotomized to no or mild comorbidities (score of 0 or 1) and moderate or severe comorbidities (score of 2 or 3), pT stage was dichotomized into T1 and T2 vs T3 and T4, and pN stage was dichotomized into N0 and N1 vs N2 and N3. The association between compliance with the overall group of metrics and OS, DSS, and DFS was expressed as an adjusted hazard ratio and 95% CI. The same method was used to determine the association between compliance with individual quality metrics and OS, DSS, and DFS.
The clinical care signature is analogous to a gene expression signature: just as the combined gene expression alteration of a tumor may be associated with the tumor’s overall behavior, a small group of process-related care practices may also be associated with the overall quality of care. The clinical care signature was composed of 4 metrics associated with improved survival on univariable or multivariable analysis. Compliance with the clinical care signature was divided into 3 groups (1, 0%-50%; 2, 51%-99%; and 3, 100%). The method described above was used to determine the association between compliance with the clinical care signature and OS, DSS, and DFS.
P ≤ .05 (2-tailed) was considered statistically significant. Data analysis was performed from January 26 to August 7, 2015, using SPSS, version 20.0 (SPSS Inc).
The demographic, oncologic, and treatment details of the 192 patients are presented in the eTable in the Supplement. Ninety-six of the tumors (50.0%) were located in the oral tongue. Sixty-four patients (33.3%) had advanced pathologic stage disease (stage T3 or T4), and 40 (21.4%) had pathologic N2 or N3 disease. Forty patients (20.8%) received adjuvant radiotherapy and 44 (22.9%) received adjuvant chemoradiation treatment. For the cohort of 192 patients with a median follow-up of 49 months, there were 66 patients who died during the study period, 41 patients who died of disease, and 80 patients with a death or recurrence. The 5-year OS was 68.5%, 5-year DSS was 74.8%, and 5-year DFS was 57.9%.
Patient eligibility and compliance with process-related quality metrics are shown in the eFigure in the Supplement. For the overall group of 19 metrics, the median compliance rate was 82.8%. Compliance for individual care processes ranged from 19.7% for counseling on tobacco cessation to 93.6% for internal review of outside hospital pathologic findings. Compliance was greater than 80% for 10 of the 19 metrics.
When compliance scores for the group of all 19 metrics with face validity were grouped into deciles, there was no association between compliance with the group of metrics and improved OS, DSS, or DFS. On multivariable analysis adjusting for pT stage, pN stage, extracapsular spread, margin status, and comorbidity, there was no statistically significant association between increased compliance and improved survival.
The association between compliance with individual quality metrics and OS, DSS, and DFS on univariable analysis is demonstrated in Table 2. There were no individual pretreatment or surveillance metrics for which increased compliance was associated with improved survival. Three treatment-related quality metrics were associated with improved survival: lymph node yield of 18 or more during elective neck dissection, no 30-day unplanned readmission, and no unplanned surgery within 14 days of the index surgery.
The association between compliance with individual quality metrics and OS, DSS, and DFS was assessed in a multivariable Cox proportional hazards model controlling for pT, pN, extracapsular spread, margin status, and comorbidity. There were no individual pretreatment or surveillance metrics for which increased compliance was associated with improved survival. Three treatment-related quality metrics were associated with improved survival on multivariable analysis (Table 3): lymph node yield of 18 or more during elective neck dissection, no 30-day unplanned readmission, and referral for adjuvant radiotherapy for pathologic stage III or IV disease.
Twenty-five patients (13.0%) had a 30-day unplanned readmission. Of those patients, the most common readmission diagnoses were surgical site infections (n = 6), wound dehiscence (n = 4), hemorrhage (n = 3), re-resection of positive margins (n = 3), and nutritional complications (n = 3). Nineteen patients (9.9%) had an unplanned return to the operating room within 14 days of their index surgery, most commonly for re-resection of positive margins (n = 7), hemorrhage (n = 5), and wound complications (n = 5).
The 4 process-related quality metrics associated with improved survival on univariable or multivariable analysis were used to construct a clinical care signature, a small group of process-related care practices associated with the overall quality of care. Compliance with the clinical care signature was associated with significantly improved OS, DSS, and DFS on univariable analysis (log-rank test, P < .05 for each) (Figure 1). A multivariable model was constructed controlling for pT stage, pN stage, extracapsular spread, margin status, and comorbidity. Compliance with the clinical care signature remained associated with improved OS, DSS, and DFS on multivariable analysis (Figure 2).
In this study, we found that compliance with individual quality metrics for a large group of patients with newly diagnosed oral cavity cancer was variable and ranged from 19.7% to 93.6%. Compliance for many of the metrics was low (eg, counseling patients on tobacco cessation and timely initiation of adjuvant therapy), but compliance for 9 of the metrics was 85% or more. For the 4 metrics that make up the clinical care signature, 135 patients (70.3%) received 100% compliance. The high compliance rate for the metrics in the clinical care signature suggests that they are already achieved in the majority of cases. Other institutions have published compliance rates for similar care processes and found them to be highly achievable.12
To date, little has been published analyzing quality metrics in head and neck cancer.12-15 What has been published for oral cavity cancer assesses compliance with a variety of metrics but does not correlate compliance with an outcome.12 Published studies for larynx cancer have been limited to elderly patients, and most of the metrics were related to surveillance, management of recurrent disease, and end-of-life care.14,15 None of the above-referenced studies assessed how performance of specific quality metrics was associated with survival outcomes.
These compliance data can be incorporated into an audit and feedback system at the surgeon or program level. Data on compliance with this group of 19 quality metrics was easily gathered from the medical record and required approximately 15 minutes per patient to collect. Knowing that compliance with these core quality metrics is associated with improved survival should facilitate motivation for quality improvement.
In this study, 4 treatment-related quality metrics were associated with improved survival: lymph node yield of 18 or more during elective neck dissection, no unplanned readmission within 30 days, no unplanned surgery within 14 days, and referral for adjuvant radiotherapy for pathologic stage III or IV disease.
Lymph node yield has been suggested as a quality metric for colon,27 lung,28 and gastric cancer29 and endorsed as a quality metric for those cancers at a national level.30 Single- and multi-institution retrospective studies have suggested that lymph node yield of 18 or more during elective neck dissection for OCSCC is associated with improved survival.16,22,23,31 Our data lend further support for lymph node yield as a process-related quality metric for these patients.
Compliance with no 30-day unplanned readmissions was associated with improved survival. Unplanned readmissions represent an interaction between patient factors, transitions of care, and postoperative complications.24 In this study, most of the unplanned readmissions were related to surgical site infections, wound dehiscence, postoperative hemorrhage, and re-resection of positive margins. Surgical site infections have been targeted as a quality metric by the Surgical Care Improvement Project, which is endorsed by the Joint Commission and the National Quality Forum.32 Postoperative hemorrhage and positive margins are the criteria used for stopping rules in the ongoing ECOG 3311 trial,33 and thus are tacitly acknowledged as markers of surgical quality.
Compliance with no unplanned surgery within 14 days of the index surgery was associated with improved survival. As with unplanned 30-day readmissions, this metric may not appear to be in the surgeon’s control and thus not be a fair marker of quality. However, the findings of this study may suggest otherwise, as the majority of cases of unplanned surgery within 14 days were related to re-resection of positive margins or postoperative hemorrhage. While both of these circumstances occur because of a complex interplay between surgeon, patient comorbidities, and tumor biology, it seems intuitive that surgical quality is a large factor in determining the frequency of these occurrences. Correlation of this metric and no 30-day unplanned readmission with improved survival are both surprising findings and warrant further investigation.
A major finding in this study was the identification of a group of core quality metrics for which increasing compliance was associated with improved survival. We termed this core group of metrics a clinical care signature to suggest an analogy with gene expression arrays. Just as the combined expression profile of a few key genes can be associated with predicting overall tumor behavior and survival, physician performance on a few key metrics can also be associated with survival outcomes. Prior studies have shown that in addition to the direct effect that compliance with performance metrics has on patient outcomes, it is also a marker of other unmeasured aspects of quality health care.34 We suspect that a similar phenomenon is being observed here.
Despite these promising findings, this study has several limitations. The retrospective nature of the study is limited by the accuracy of the medical record. It is a single-institution study, so the generalizability of the results is not known. It is also unknown whether these data apply to other subsites of the head and neck. Two variables (no unplanned surgery ≤14 days and no unplanned readmission ≤30 days) were dichotomized into yes or no. Future research is needed to determine acceptable risk-adjusted frequencies for these events. It is also possible that compliance with some of the quality metrics assessed in this study are in fact truly associated with improved survival outcomes, but these associations were not captured because the study was underpowered to detect them. Finally, this model requires additional testing in an independent validation cohort of patients.
Although not a limitation of this study itself, application of these data may have unintended consequences (eg, electively dissecting unindicated levels of the neck to increase lymph node yield). Unintended consequences stemming from well-intentioned quality metrics have been previously described, such as an increase in the incidence of Clostridium difficile infection after implementation of a quality metric for antibiotic therapy in community-acquired pneumonia.3
This study identified process-related quality metrics that correlated with improved survival in patients with surgically managed OCSCC. When these quality metrics are grouped together as a clinical care signature, increasing compliance with the bundle of metrics is associated with improved survival even after adjusting for tumor-related prognostic factors and comorbidity. The significant quality metrics in this study were all related to treatment factors rather than pretreatment evaluation or posttreatment surveillance, suggesting that additional research is needed in those areas. Validation of the current findings in an independent population and then in a multi-institutional fashion appears warranted.
Submitted for Publication: October 9, 2015; final revision received November 25, 2015; accepted December 4, 2015.
Corresponding Author: Brian Nussenbaum, MD, Department of Otolaryngology–Head and Neck Surgery, Washington University School of Medicine, 660 S Euclid Ave, Campus Box 8115, St Louis, MO 63110 (email@example.com).
Published Online: February 11, 2016. doi:10.1001/jamaoto.2015.3595.
Author Contributions: Drs Graboyes and Nussenbaum had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Graboyes, Kallogjeri, Al-Gilani, Stadler, Nussenbaum.
Acquisition, analysis, or interpretation of data: Graboyes, Gross, Kallogjeri, Piccirillo, Nussenbaum.
Drafting of the manuscript: Graboyes, Kallogjeri, Nussenbaum.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Graboyes, Kallogjeri, Piccirillo.
Administrative, technical, or material support: Nussenbaum.
Study supervision: Nussenbaum.
Conflict of Interest Disclosures: None reported.
Funding Support: This research was funded in part by the P30 Research Center for Auditory and Vestibular Studies and grant P30DC04665 from the National Institutes of Health NIDCD.
Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: Dr Piccirillo is the editor of JAMA Otolaryngology–Head & Neck Surgery but was not involved in the editorial review or the decision to accept the manuscript for publication.
Additional Contributions: Douglas Adkins, MD, Jason Diaz, MD, Hiram Gay, MD, Wade Thorstad, MD, and Tanya Wildes, MD, Washington University School of Medicine, are the other members of the multidisciplinary head and neck cancer team who helped develop the quality metrics. They were not compensated for their contributions.
Create a personal account or sign in to: