[Skip to Content]
[Skip to Content Landing]
Invited Commentary
Obstetrics and Gynecology
February 12, 2020

Unintended Consequences of Obstetric Quality Metrics—Do Not Throw the Baby Out With the Bathwater

Author Affiliations
  • 1Department of Obstetrics and Gynecology, University of Texas Southwestern Medical Center, Dallas
JAMA Netw Open. 2020;3(2):e1919749. doi:10.1001/jamanetworkopen.2019.19749

Using health care quality metrics to promote improvements, influence payment for services, and increase transparency has been at the forefront of medicine for decades.1 However, the interest in obstetric care is relatively new and has largely been focused on care for mothers (eg, cesarean delivery rates, third- and fourth-degree perineal laceration rates) or preterm infants (eg, corticosteroid administration).2 Because term births represent 90% of all deliveries in the United States (ie, 3.4 million births in 2018), the Joint Commission implemented a new quality metric, Unexpected Complications in Term Newborns (PC-06), on January 1, 2019.2,3 The purpose of this metric was to assess adverse outcomes in otherwise healthy term infants without preexisting conditions and also to serve as a balance against the maternal-focused quality metrics already in place.2

Clapp et al4 report the findings from a population-based study examining severe unexpected newborn complications among counties within the United States with at least 1 obstetric hospital. Similar to the PC-06 metric, severe unexpected newborn complication rates were defined as neonatal death, a 5-minute Apgar score of 3 or less, seizure, assisted ventilation for at least 6 hours, or transfer to another facility. The authors found a wide range of hospital complication rates (ie, 0.6-89.9 per 1000 births) with measurable between-hospital variation.4 There was little association with case mix to explain the observed variation following adjustment. Importantly, neonatal transfer represented nearly 60% of complications overall and two-thirds of complications in the cohort of hospitals with high complication rates.4 Not surprisingly, hospitals with higher levels of neonatal care—thus not using neonatal transfer—had lower measured complication rates. Specifically, in the patient-level analysis, a patient’s risk of an unexpected complication was increased by 50% when born in a hospital without a neonatal intensive care unit. When transfers were excluded from the primary outcome, there was no difference between the comparison groups (5.1 vs 4.8 complications per 1000 births in counties without and with neonatal intensive care units, respectively; P = .61).4 The authors concluded that this metric was potentially useful in reflecting quality of care because of the wide variation among hospitals. However, the authors cautioned that adjustment for a hospital’s level of neonatal care should be considered to avoid disincentivizing appropriate transfers to higher levels of neonatal care.

The findings by Clapp et al4 highlight the challenges in developing obstetric quality metrics. To be a valid indicator, the chosen complication should reflect not just an adverse event but also an event that could occur less often with improvements in clinical care.5 Furthermore, implementing a measure should not lead to unintended adverse consequences.1 Unexpected newborn complication rates—measured in part by neonatal transfers—fail both of these principles. It is intuitive that lower-level neonatal care facilities would have higher rates of neonatal transfer. Thus, facilities without neonatal intensive care services are inadvertently penalized because neonatal transfers dominate the composite. Indeed, when transfer was excluded from the primary outcome, Clapp et al4 found no differences between cohorts. This serves as a caution to not throw the baby out with the bathwater, given that the unintended consequences of disincentivizing necessary neonatal transfers to higher levels of care are obvious. Furthermore, comparing isolated birthing facilities in a county skews the hospital comparison to counties in less-populated (ie, rural) regions. Suggesting that lower-level neonatal care facilities perform poorly simply because of appropriate transfers to higher levels of care may accelerate the already declining access to care in rural areas. Between 2004 and 2014, the percentage of rural counties in the United States with hospital-based obstetric services decreased from 55% to 46%, leaving women in many areas of the United States without local access to basic obstetric care.6 More frightening, what if hospitals defer necessary transfers to avoid such labels? The irony of the PC-06 metric is that it was created to be a quality metric for infants balanced against other obstetric care measures. Because of the skewed composite components, this metric may inadvertently become a detriment if birthing facilities are closed because of perceived poor performance. For some, like those in the cohort studied by Clapp et al,4 these facilities are the only existing obstetric resources for delivery in the county.

Defining relevant quality metrics is paramount given the tremendous costs of American health care. Put simply, how do we justify a nearly $3 500 000 000 000 annual price tag for services? More importantly, how do we collectively improve the quality of care to our patients? In 2013, the Eunice Kennedy Shriver National Institute of Child Health and Human Development Maternal-Fetal Medicine Units Network attempted to answer this question across 25 hospitals in the United States using 5 obstetric performance metrics—including a neonatal composite of adverse outcomes.7 Rather than administrative billing codes, research nurses entered clinical outcome data using a manual of operations to ensure data accuracy. None of the associations between hospital-adjusted outcome frequencies were significant. For example, a hospital with the best composite neonatal outcome ranking was also ranked 21st for severe perineal lacerations at vaginal delivery. Furthermore, up to 4 risk-adjusted outcomes did not allow for assessment of hospital obstetric quality. The authors concluded that their findings “underscore the complexity of quality measurement.”7

Although the reports by the Maternal-Fetal Medicine Units Network7 and Clapp et al4 serve as cautionary examples, the bathwater does need changing. Obstetric quality measures need to be reappraised. The analysis by Clapp et al4 demonstrates that severe unexpected newborn complications are affected by neonatal transfers and, if not considered, may lead to misinterpretation. Better measures of obstetric quality are needed—especially when examining outcome (ie, performance) measures. Indeed, some outcomes may be too infrequent to draw meaningful conclusions or comparisons of quality. For example, severe unexpected newborn complications excluding transfers were found in less than 1% of term deliveries (approximately 5 per 1000 births) and, thus, may only be identified a few times a year at a given hospital.4 Fortunately, the rate of maternal mortality is so low (measured per 100 000 livebirths) that it is not a metric that can be tracked at the hospital level to affect meaningful change. Therefore, a starting point could be to review process—rather than performance—measures. Other specialties have faced similar issues with quality measurement and taken this path, using process measures that align with outcomes. An example is thromboembolic therapy in the setting of stroke.8 Obstetrics could use similar metrics, such as the presence of standardized protocols and time intervals to deployment (eg, time from blood pressure measurement to antihypertensive administration). Regardless of the chosen obstetric metrics, more data are needed because the evidence to suggest improved outcomes, even using metrics that seem useful in other specialties, has been disappointing.8

The report by Clapp et al4 provides data and highlights the challenges and consequences of selecting obstetric quality measures. With public reporting of these metrics, the dilemma is whether to provide quality care for the patient or chase contrived metrics of quality. The consequences cannot be overstated.

Back to top
Article Information

Published: February 12, 2020. doi:10.1001/jamanetworkopen.2019.19749

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Nelson DB et al. JAMA Network Open.

Corresponding Author: David B. Nelson, MD, Department of Obstetrics and Gynecology, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390-9032 (davidb.nelson@utsouthwestern.edu).

Conflict of Interest Disclosures: None reported.

References
1.
Chassin  MR, Loeb  JM, Schmaltz  SP, Wachter  RM.  Accountability measures: using measurement to promote quality improvement.  N Engl J Med. 2010;363(7):683-688. doi:10.1056/NEJMsb1002320PubMedGoogle ScholarCrossref
2.
Joint Commission. Specifications Manual for Joint Commission National Quality Measures. https://manual.jointcommission.org/releases/TJC2018B/MIF0393.html. Accessed November 28, 2019.
3.
Martin  JA, Hamilton  BE, Osterman  MJK, Driscoll  AK. Births: Final data for 2018. https://www.cdc.gov/nchs/data/nvsr/nvsr68/nvsr68_13-508.pdf. Accessed January 8, 2020.
4.
Clapp  MA, James  KE, Bates  SV, Kaimal  AJ.  Patient and hospital factors associated with unexpected newborn complications among term neonates in US hospitals.  JAMA Netw Open. 2020;3(2):e1919498. doi:10.1001/jamanetworkopen.2019.19498Google Scholar
5.
Grobman  WA, Feinglass  J, Murthy  S.  Are the Agency for Healthcare Research and Quality obstetric trauma indicators valid measures of hospital safety?  Am J Obstet Gynecol. 2006;195(3):868-874. doi:10.1016/j.ajog.2006.06.020PubMedGoogle ScholarCrossref
6.
Society for Maternal-Fetal Medicine (SMFM).  Executive summary: Reproductive Services for Women at High Risk for Maternal Mortality Workshop, February 11-12, 2019, Las Vegas, Nevada.  Am J Obstet Gynecol. 2019;221(4):B2-B5. doi:10.1016/j.ajog.2019.06.032PubMedGoogle ScholarCrossref
7.
Bailit  JL, Grobman  WA, Rice  MM,  et al; Eunice Kennedy Shriver National Institute of Child Health and Human Development Maternal–Fetal Medicine Units Network.  Risk-adjusted models for adverse obstetric outcomes and variation in risk-adjusted outcomes across hospitals.  Am J Obstet Gynecol. 2013;209(5):446.e1-446.e30. doi:10.1016/j.ajog.2013.07.019PubMedGoogle ScholarCrossref
8.
Parker  C, Schwamm  LH, Fonarow  GC, Smith  EE, Reeves  MJ.  Stroke quality metrics: systematic reviews of the relationships to patient-centered outcomes and impact of public reporting.  Stroke. 2012;43(1):155-162. doi:10.1161/STROKEAHA.111.635011PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    ×