Views 2,400
Citations 0
Invited Commentary
October 17, 2016

The Quest to Improve QualityMeasurement Is Necessary but Not Sufficient

Author Affiliations
  • 1Kaiser Permanente Research, Pasadena, California
  • 2Departments of Veterans Affairs Center for Clinical Management Research and Internal Medicine and Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor

Copyright 2016 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.

JAMA Intern Med. Published online October 17, 2016. doi:10.1001/jamainternmed.2016.6233

Thirteen years ago, we reported that American adults were receiving about half of recommended care for the 30 leading causes of illness and death.1 We used 439 indicators covering inpatient and outpatient care that were validated through a modified Delphi process. Indicators were scored using data abstracted from all physicians seeing participants supplemented by survey data. We also found that quality deficits were similar across very different communities2 and that everyone was at risk for experiencing substandard quality.3 These findings caught the public and health professionals by surprise, although they were consistent with prior small studies. Since then, there has been a proliferation of public and private programs to measure, publicly report, and reward or penalize health plans, hospitals, physicians, and other care settings on quality.

So, is quality any better today than it was in 2003? Unfortunately, the substantial costs associated with replicating the methodology have deterred funders from supporting a study to fully answer this question. However, the article in this issue by Levine et al4 using a different methodology suggests that, on average, quality has not improved dramatically. Their study has several limitations. First, they used measures that were valid across all years of their study, which means they did not necessarily address best clinical practice in 2013. Second, the relatively small number of measures used were selected because they could be scored using administrative data and not because of their importance to overall quality outcomes. So, we probably do not know how good quality is overall in 2013.

Using this limited set of measures, the authors4 report anemic improvements in quality. For example, over the 11 years of their study, improvements in composite scores for use of cardiovascular medications and counseling for smoking cessation, weight loss, and exercise ranged from 2 to 7 percentage points. Most of the composites they constructed remained below the overall performance of 55% we reported in 2003 with the exception of cancer screening and selected interventions for diabetes, preventive, or diagnostic care. In those areas, performance was similar to the levels at the beginning of the study. This will likely disappoint many readers. For those actively engaged in efforts to improve quality, the results may not be as surprising.

Measurement Isn’t Enough

Measurement combined with public reporting can draw attention to particular areas of concern and stimulate improvement efforts, but measurement in the absence of other changes will not produce different results. For example, you can weigh yourself every day but if you do not change your diet or exercise routine, the results (measurements) probably won’t change. Further measurement approaches today are deficient. Many measures are simplistic approximations of what clinicians and patients believe represents high quality of care. Much of today’s quality measurement enterprise operates separately from the workflows associated with delivering health care services. But the workflows have to change to get different results. Physicians generally know what constitutes best practices and show up every day to do the best for their patients, but reliably and consistently offering those services at the point of care delivery requires a systems approach. This means integrating clinically meaningful measurement into care delivery at appropriates points of interaction with patients combined with specific actions to ensure delivery of optimal care.5

Payment Reform May Be Helpful but Not Sufficient

Levine and colleagues4 point to both insurance coverage and payment reform as potential opportunities to encourage future improvements in quality but the evidence on this is not clear. For example, Asch et al3 showed no significant differences in performance related to either the presence or type of health insurance coverage. An older body of work on appropriateness of care has demonstrated rates of inappropriate care that were similar in the United States and in countries such as Canada, the United Kingdom, and Israel with universal coverage. Our study of 12 US communities with very different health economic profiles (eg, managed care penetration, insurance rates, unemployment rates, physician-to-patient population ratios, beds per 1000 population) found no relationship between quality performance and these factors.2 Furthermore, we and others have found that quality of care in the VA, the largest integrated health care system in the country, has exceeded that in the private sector, despite the lack of financial incentives for doctors to deliver evidence based care.6 The theory is that economic incentives and public reporting will magically enable individual clinicians and systems to self-organize to transform care. These approaches may remove some barriers but they are a long way from the work required to ensure more reliable care delivery.

It Takes a System

Health care systems that have achieved substantial and sustained improvements in health care quality have devoted time, people, and resources to creating more reliable systems. High-performing systems require leadership that sets priorities and creates a culture committed to continuous improvement, engagement of all members of the health services delivery team, capacity building that enables staff to execute plans effectively and consistently, and a mechanism for ongoing learning from efforts to improve care at all levels.7 This approach requires that everyone in the organization is engaged in improvement from the leadership to the front lines, which is difficult to accomplish in systems that are not integrated and provide an uncertain and unpredictable portion of care for an individual. Because many people in the United States still receive care in nonintegrated systems, the failure to observe progress in quality in light of what is required for high-performing systems is perhaps not surprising.

To provide a concrete example, about 15 years ago, Kaiser Permanente leaders in Southern and Northern California set out to improve members’ blood pressure control. The exact interventions and the sequence in which they were implemented varied, but they included developing treatment algorithms integrated into workflows, creating a registry that enabled real time tracking, providing performance feedback to physicians, integrating pharmacists and nurses with primary care teams, promoting fixed-dose combination drugs, creating drop-in blood pressure visits that did not require copayments, and several other changes. Both regions today have achieved blood pressure control rates exceeding 85%, up from less than 50% in 2001. These systems approaches are difficult to achieve in smaller practices and in nonintegrated settings and offer reusable infrastructure that enables improvement activities across a wide range of clinical areas.


Much of the work in quality improvement has focused on approaches that are driven by payers and policy makers. These have included measurement and public reporting, payment incentives, investments in electronic medical records, and developing virtual systems of care in select areas. None of these approaches by itself is likely to fundamentally alter the level of quality delivered throughout the nation. To do so requires significant work by health professionals on the front lines in collaboration with their patients. And those approaches require time, resources, and energy that are beyond what is available to many practices that are struggling to keep up with a rapidly changing world. We need to find a more effective way to transform the delivery of health care so that physicians and patients can achieve the outcomes that both desire. This will be hard work and will require engagement on the ground and not simply exhortations from those paying the bills.

Back to top
Article Information

Corresponding Author: Elizabeth A. McGlynn, PhD, Kaiser Permanente Research, 100 S Los Robles, Third Floor, Pasadena, CA 91101 (

Published Online: October 17, 2016. doi:10.1001/jamainternmed.2016.6233

Conflict of Interest Disclosures: None reported.

Funding/Support: Dr Kerr is partly supported by the Department of Veterans Affairs (VA) and has VA- and National Institutes of Health–funded grants on quality assessment and improvement.

Role of the Funder/Sponsor: The VA and NIH had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The opinions expressed in this article reflect those of the authors and do not represent those of Kaiser Permanente, the Veterans Administration, or the University of Michigan.

McGlynn  EA, Asch  SM, Adams  J,  et al.  The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635-2645.
Kerr  EA, McGlynn  EA, Adams  J, Keesey  J, Asch  SM.  Profiling the quality of care in 12 communities: results from the CQI study. Health Aff. 2004;23(3):247-256. doi:10.1377/hlthaff.23.3.247Article
Asch  SM, Kerr  EA, Keesey  J,  et al.  Who is at greatest risk for receiving poor-quality health care? N Engl J Med. 2006;354(11):1147-1156.
Levine  DM, Linder  JA, Landon  BE.  The quality of outpatient care delivered to adults in the United States, 2002 to 2013  [published online October 17, 2016]. JAMA Intern Med. 10.1001/jamainternmed.2016.6217
McGlynn  EA, Kerr  EA.  Creating safe harbors for quality measurement innovation and improvement. JAMA. 2016;315(2):129-130. doi:10.1001/jama.2015.16858
Asch  SM, McGlynn  EA, Hogan  MM,  et al.  Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med. 2004;141(12):938-945.
Schilling  L, Chase  A, Kehrli  S, Liu  AY, Stiefel  M, Brentari  R.  Kaiser Permanente’s performance improvement system, part 1: From benchmarking to executing on strategic priorities. Jt Comm J Qual Patient Saf. 2010;36(11):484-498.