aN = 41 hospitals. Top 10 on the quality measures indicates lowest rate for elective and cesarean deliveries; bottom 10, highest rates.
eTable. ICD-9 Codes Used to Define Comorbid Conditions
eAppendix. Computation of Hospital Risk–Standardized Rates of Severe Maternal Morbidity and Severe Neonatal Morbidity
Elizabeth A. Howell, Jennifer Zeitlin, Paul L. Hebert, Amy Balbierz, Natalia Egorova. Association Between Hospital-Level Obstetric Quality Indicators and Maternal and Neonatal Morbidity. JAMA. 2014;312(15):1531–1541. doi:10.1001/jama.2014.13381
In an effort to improve the quality of care, several obstetric-specific quality measures are now monitored and publicly reported. The extent to which these measures are associated with maternal and neonatal morbidity is not known.
To examine whether 2 Joint Commission obstetric quality indicators are associated with maternal and neonatal morbidity.
Design, Setting, and Participants
Population-based observational study using linked New York City discharge and birth certificate data sets from 2010. All delivery hospitalizations were identified and 2 perinatal quality measures were calculated (elective, nonmedically indicated deliveries at 37 or more weeks of gestation and before 39 weeks of gestation; cesarean delivery performed in low-risk mothers). Published algorithms were used to identify severe maternal morbidity (delivery associated with a life-threatening complication or performance of a lifesaving procedure) and morbidity in term newborns without anomalies (births associated with complications such as birth trauma, hypoxia, and prolonged length of stay). Mixed-effects logistic regression models were used to examine the association between maternal morbidity, neonatal morbidity, and hospital-level quality measures while risk-adjusting for patient sociodemographic and clinical characteristics.
Main Outcomes and Measures
Individual- and hospital-level maternal and neonatal morbidity.
Severe maternal morbidity occurred among 2372 of 115 742 deliveries (2.4%), and neonatal morbidity occurred among 8057 of 103 416 term newborns without anomalies (7.8%). Rates for elective deliveries performed before 39 weeks of gestation ranged from 15.5 to 41.9 per 100 deliveries among 41 hospitals. There were 11.7 to 39.3 cesarean deliveries per 100 deliveries performed in low-risk mothers. Maternal morbidity ranged from 0.9 to 5.7 mothers with complications per 100 deliveries and neonatal morbidity from 3.1 to 21.3 neonates with complications per 100 births. The maternal quality indicators elective delivery before 39 weeks of gestation and cesarean delivery performed in low-risk mothers were not associated with severe maternal complications (risk ratio [RR], 1.00 [95% CI, 0.98-1.02] and RR, 0.99 [95% CI, 0.96-1.01], respectively) or neonatal morbidity (RR, 0.99 [95% CI, 0.97-1.01] and RR, 1.01 [95% CI, 0.99-1.03], respectively).
Conclusions and Relevance
Rates for the quality indicators elective delivery before 39 weeks of gestation and cesarean delivery performed in low-risk mothers varied widely in New York City hospitals, as did rates of maternal and neonatal complications. However, there were no correlations between the quality indicator rates and maternal and neonatal morbidity. Current quality indicators may not be sufficiently comprehensive for guiding quality improvement in obstetric care.
Although great progress has been made in reducing obstetric complications, they persist. Severe maternal complications include renal failure and eclampsia or the need for lifesaving interventions such as prolonged mechanical ventilation or transfusions.1,2 Neonatal complications may occur in low-risk term infants and include hypoxia and shock.3 Severe maternal morbidity occurs in about 60 000 women (1.6 per 100 deliveries) annually in the United States, and 1 in 10 term infants experience neonatal complications.3,4 Variation in complication rates between hospitals exists and suggests that the quality of obstetric care can be improved.5 More than one-third of maternal deaths and severe morbidities, and a significant proportion of neonatal mortality and morbidity, may be preventable by changes in patient, clinician, and system factors.4,6- 9
As part of its core measure set, The Joint Commission now recommends 2 perinatal quality measures that address important aspects of obstetric care during childbirth: elective deliveries performed prior to 39 weeks of gestation and cesarean deliveries performed in low-risk nulliparous women.10 The elective delivery measure, which includes nonmedically indicated deliveries associated with medical induction or cesarean delivery at more than 37 weeks and prior to 39 weeks of gestation, is also mandated by the Centers for Medicare & Medicaid Services.11 The elective delivery before 39 weeks of gestation indicator is intended to reduce neonatal complications among term infants. Assessing rates of cesarean delivery performed in low-risk patients is intended to reduce unnecessary variation in rates of cesarean delivery. Both of these measures may be associated with maternal outcomes.12,13 However, how well hospital performance on these quality indicators correlates with maternal or neonatal morbidity is not known.
We investigated whether elective deliveries performed prior to 39 weeks of gestation and cesarean deliveries performed in low-risk nulliparous women were associated with severe maternal or neonatal morbidity in New York City hospitals.
We used birth certificate data linked with New York State discharge abstract data reported in the Statewide Planning and Research Cooperative System (SPARCS) for all delivery and newborn hospitalizations in New York City in 2010. Data linkage was conducted by the New York State Department of Health. Ninety-seven percent of maternal and 98% of infant discharge abstracts were linked with infant birth certificates. Institutional review board approvals were obtained from the New York City Department of Health and Mental Hygiene, the New York State Department of Health, and the Icahn School of Medicine at Mount Sinai. A waiver of consent was approved by the Icahn School of Medicine. Delivery hospitalizations were identified based on International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes and diagnosis related group delivery codes.14 Singleton nonanomalous term newborns were identified using the newborn flag. Gestational age was ascertained from birth certificate data and congenital anomalies from SPARCS data.
We identified severe maternal morbidity using a Centers for Disease Control and Prevention (CDC)–endorsed surveillance algorithm (available at http://www.cdc.gov/reproductivehealth/MaternalInfantHealth/SevereMaternalMorbidity.html), using diagnoses for life-threatening conditions and procedure codes for life-saving procedures.2 Codes were selected by experts and reviewed for their association with in-hospital maternal mortality. The algorithm excludes hospitalizations with a length of stay less than the 90th percentile as calculated separately for vaginal and primary and repeat cesarean deliveries.2 As recommended, length of stay reclassification was not applied to in-hospital deaths, transfers, or delivery hospitalizations with severe complications identified by procedure codes.2
We identified neonatal morbidity at term based on diagnoses and procedure codes as defined by Korst et al3 to monitor childbirth morbidity using hospital discharge data.3,15,16 The list includes ICD-9-CM code–based categories that capture indicators of death or neonatal complications (eg, birth trauma, disseminated intravascular coagulation, neonatal intensive care unit [NICU] procedures, renal failure, respiratory conditions, necrotizing enterocolitis, shock, neonatal length of stay >5 days, neonatal death) and was ascertained from SPARCS data.3
We used birth certificate data and SPARCS data to calculate 2 perinatal quality measures using algorithms designated by the Joint Commission.10 The first measure, elective deliveries at 37 or more weeks and prior to 39 weeks of gestation, was defined as all deliveries associated with medical inductions of labor or cesarean delivery at 37 or more weeks and prior to 39 weeks of gestation as a proportion of all deliveries at 37 or more weeks and prior to 39 weeks. All conditions possibly justifying delivery prior to 39 weeks of gestation were excluded, as specified.10 We also excluded cesarean deliveries that were associated with a trial of labor but were not inductions. The second measure, cesarean deliveries in low-risk women, was defined as the proportion of cesarean deliveries among nulliparous patients with singleton vertex deliveries of newborns at 37 or more weeks of gestation. We excluded patients with ICD-9-CM codes that signify contraindications to vaginal deliveries.10 Gestational age, parity, multiple birth, vertex presentation, and trial of labor were ascertained from birth certificate data. ICD-9-CM codes were obtained from SPARCS.
For each hospital we calculated the rate of elective deliveries and low-risk cesarean deliveries. No risk adjustment is required for the elective delivery measure, but direct standardization by maternal age is suggested for the cesarean delivery measure. We standardized this indicator for each hospital by 5-year age groups using the observed maternal age distribution in the entire sample.10
To risk-adjust hospital-level rates of maternal morbidity we used variables from birth certificate records, including mothers’ sociodemographics (maternal age, self-identified race/ethnicity, parity, education), prenatal care visits, and clinical and obstetric factors (multiple births, history of previous cesarean delivery). We ascertained patient insurance status from SPARCS. We also included diagnoses codes for patient risk factors that could lead to maternal morbidity but were likely present on admission to the hospital (eg, diabetes, hypertension, obesity, premature rupture of membranes, disorders of placentation). These conditions have been used to risk-adjust for severe maternal morbidity,17 cesarean deliveries, and other maternal outcomes (eTable in the Supplement).18,19
We risk-adjusted neonatal morbidity by maternal sociodemographics (maternal age, self-identified race and ethnicity, parity, education), behaviors (prenatal care visit, tobacco use, illicit drug use), maternal clinical factors (hypertension or diabetes during or before pregnancy), obstetric factors (previous cesarean delivery, premature rupture of membranes), and infant factors (gestational age, size for gestational age).20 Small for gestational age was defined as a birth weight in less than the 10th percentile; large for gestational age was defined as a birth weight in more than the 90th percentile for gestational age and sex.21
We obtained teaching status from the American Hospital Association, ownership and nursery level from the New York State Department of Health, and volume of deliveries in each hospital from SPARCS to assess how other hospital characteristics are correlated with measures of quality and morbidity.
We compared the sociodemographic characteristics and clinical conditions of women with and without severe maternal morbidity and of infants with and without neonatal morbidity using χ2 tests.
We used mixed-effects logistic regression with a random hospital-specific intercept to generate risk-standardized severe maternal morbidity rates and risk-standardized neonatal morbidity rates at term for each hospital. The models included covariates described above. Hospital risk-standardized rates were computed from these models using methods recommended by Centers for Medicare & Medicaid Services Hospital Compare22 (eAppendix in the Supplement).
We used Spearman rank correlations to assess correlations among hospital-level rates of severe maternal morbidity, neonatal morbidity, and the 2 measures of hospital quality. We divided hospitals into quartiles based on each quality measure and examined hospital rankings on both risk-standardized morbidity measures using the nonparametric Kruskal-Wallis Test. Quality measures were coded as continuous for our primary multivariable analyses. Our final models explored the association of the 2 quality measures with severe maternal morbidity and then with neonatal morbidity using mixed-effects logistic regression in 3 steps, with (1) no adjustment, (2) adjustment for patient characteristics, and (3) adjustment for patient and hospital-level characteristics, including ownership, volume, level of nursery, and teaching status. Risk ratios were calculated using recycled predictions, which yields the marginal effect of a variable while holding all other patient characteristics constant23; confidence intervals were estimated using the delta method.24 Intraclass correlations were calculated to measure the between-hospital variance as a proportion of the total variance.
Sensitivity analyses included coding quality measures in quartiles; examining the association of the quality measures with severe maternal morbidity without blood transfusion; and using 2 simpler measures of neonatal outcomes (prolonged length of stay [>5 days] and NICU admission).
Statistical analyses were performed using SAS version 9.3 (SAS Institute Inc) and Stata version 12 (StataCorp). P < .05 (2-sided) was considered statistically significant.
Of the 43 hospitals with deliveries, we excluded 2 hospitals with annual delivery volumes fewer than 5 births and 1360 deliveries with missing hospital identifiers. The final sample included 115 742 deliveries, of which 2732 (2.4%) were associated with severe maternal morbidity. Of the 119 793 newborns identified, we excluded 4672 multiple births, 4447 with congenital anomalies, and 7258 with gestational age less than 37 weeks. The final sample included 103 416 newborns; of these, 8057 (7.8%) were associated with neonatal morbidity. The associations of sociodemographic and clinical characteristics with severe maternal morbidity and neonatal morbidity are described in Table 1 and Table 2.
Table 3 shows that the majority of hospitals were private, had level 3/4 nurseries, and were teaching hospitals. Hospital performance per 100 deliveries ranged from 15.5 to 41.9 for elective deliveries before 39 weeks; 11.7 to 39.3 for age-standardized rates of cesarean deliveries in low-risk nulliparous women; 0.9 to 5.7 for risk-standardized severe maternal morbidity rates; and 3.1 to 21.3 for risk-standardized neonatal morbidity rates among these hospitals. Both quality measures were correlated with each other (Spearman ρ = 0.45; P = .003). Severe maternal morbidity and neonatal morbidity at term were also correlated (Spearman ρ = 0.39; P = .01).
Hospital rankings on both quality measures were not associated with hospital rankings for maternal or neonatal morbidity as demonstrated in the Figure. In fact, among the 10 hospitals with the best performance (lowest rates) on elective deliveries, only 3 were in the lowest quartile of risk-standardized severe maternal morbidity. Among the 10 hospitals with the best performance (lowest rates) on the low-risk cesarean delivery measure, only 3 were in the lowest quartile of risk-standardized severe maternal morbidity. The rankings were similarly discordant for neonatal morbidity.
Nonparametric correlations between obstetric quality indicators and both maternal and neonatal morbidity were not significant (Spearman ρ = −0.01, P = .94 for cesarean delivery and maternal morbidity; Spearman ρ = −0.10, P = .52 for cesarean delivery and neonatal morbidity; Spearman ρ = 0.14, P = .39 for elective delivery and maternal morbidity; Spearman ρ = −0.14, P = .39 for elective delivery and neonatal morbidity).
In mixed-effects models including both quality measures as independent variables, neither quality indicator was associated with severe maternal morbidity (model 1, Table 4); the risk ratio (RR) for elective delivery was 1.00 (95% CI, 0.97-1.03) and for low-risk cesarean deliveries was 0.98 (95% CI, 0.95-1.01). In models that also included patient-level variables (model 2) and patient-level and other hospital-level variables (model 3), the RRs for both quality measures remained essentially unchanged. Sensitivity analyses with the quality measures coded in quartiles and analyses excluding blood transfusions from the severe maternal morbidity outcome corroborated our findings. There was a substantial between-hospital variation in severe maternal morbidity: the intraclass correlation coefficient (ICC) was 28% in the unadjusted model; after adjusting for patient characteristics, ICC was reduced to 22%.
Our findings were similar in models examining the 2 obstetric quality measures and neonatal morbidity (Table 4). Volume of deliveries and nursery level were associated with neonatal morbidity. The association between the 2 obstetric quality measures with neonatal length of stay longer than 5 days and NICU admission showed similar results. In a multilevel model without adjustment, the ICC was 20% for neonatal morbidity. The variance between hospitals remained (ICC = 16%) after accounting for patient characteristics.
Severe maternal morbidity and neonatal morbidity at term remain important health issues. We found that in New York City hospitals 2.4% of all mothers and 7.8% of term nonanomalous newborns have major complications and that these rates vary substantially between hospitals. Severe maternal morbidity rates varied 4- to 5-fold between hospitals, and there was a 7-fold variation in neonatal morbidity at term between hospitals. Although there was substantial variation in morbidity rates, they were not correlated with the performance measures designed to assess hospital-level obstetric quality of care.
Measuring quality of care in obstetrics is complex: it involves assessing care for 2 separate individuals (mother and infant). Improving the quality of care requires reducing the use of obstetric interventions that can harm infants (eg, early delivery) and mothers (eg, unnecessary cesarean delivery) and avoiding suboptimal care such as underutilization of antenatal steroids or antibiotics, which can lead to neonatal or maternal complications or death. Measures of overutilization may be important obstetric quality indicators and enable the tracking of utilization of obstetric interventions; however, the measures we assessed do not reflect the quality of care in terms of severe maternal or neonatal morbidity.
The Joint Commission set of perinatal quality measures includes 3 other indicators in addition to the 2 we studied. Those measures assess other dimensions of perinatal care—breastfeeding, health care–associated infections among very low-birth-weight infants, and antenatal steroid use among premature deliveries. The measures may be indicators of obstetric quality, but the outcomes primarily concern newborns and not their mothers and 2 of the measures only assess care provided in the neonatal period. Our findings suggest that other quality measures should be developed that focus on suboptimal care. Examples include whether hemorrhage and preeclampsia protocols are used in the delivery suite.25- 29 Given that we did not find a relationship between the quality indicators we examined and maternal or neonatal morbidity, these measures do not adequately capture obstetric hospital quality.
Our results are consistent with findings from the Maternal Fetal Medicine Network and studies in NICUs demonstrating that performance assessment based on isolated measures do not accurately characterize the overall quality of care in a hospital.20,30 Numerous other studies have found a lack of correlation between quality indicators and the clinical outcomes they are supposed to reflect. Hospital Compare measures only predict small differences in hospital risk-adjusted mortality rates for heart failure, pneumonia, and acute myocardial infarction.31 Adherence with infection control practices was not associated with lower perioperative infection rates (http://www.ncbi.nlm.nih.gov/pubmed/20571014). There was no association between adherence to home management care quality measures and emergency department visits and readmissions for pediatric asthma.32 Adherence to perioperative prophylaxis for venous thromboembolism was not associated with venous thromboembolism rates, and surveillance bias resulted in unfair penalization of hospitals that more rigorously looked for the disorder.33 The growing list of quality measures often does not reflect the care these measures are supposed to measure and capture only a narrow slice of hospital quality.34,35 There is a need to reassess how these measures are designed and implemented and to think more broadly about constructing meaningful quality measures tightly linked with patient outcomes.
In our analyses, the number of deliveries at a hospital and the availability of high-level nursery facilities were associated with neonatal morbidity at term. These hospital characteristics have been consistently related to outcomes for very low-birth-weight infants.36 The association is less clear for outcomes of lower-risk infants.37,38 Few studies have investigated how hospital characteristics affect severe maternal morbidity.
Our analysis has limitations. We used administrative (ICD-9-CM procedure and diagnosis codes) and birth certificate data rather than medical chart review for our analysis. Both birth certificate and SPARCS data may be limited by the reliability of certain types of information contained in these databases. Previous assessments demonstrated good reliability for maternal age, race, and gestational age.39 Previous studies of birth certificate data used for hospital quality assessment studies showed that they perform as well as medical records information.40
Algorithms for morbidity based on diagnosis and procedure codes are imperfect. However, discharge data in obstetrics are reliable for delivery and procedure codes and moderately reliable for comorbidities.41,42 It is possible that the codes used in our analyses inadequately captured the outcomes of interest. However, many of the diagnosis codes included in the severe maternal morbidity algorithm (eg, eclampsia, shock, acute renal failure) are designated as major complications and comorbidities codes in diagnosis related group coding and when present may affect diagnosis related group assignment for obstetric deliveries, making these diagnoses less likely to be underreported. In a sensitivity analyses we found that coding for prolonged neonatal length of stay and NICU admission (outcomes known to be less subject to coding problems) were not associated with the 2 quality measures. If administrative codes inadequately captured the clinical information we assessed, we would have expected null findings in the patient-level analysis. This was not the case. We were able to demonstrate significant associations between age, race/ethnicity, parity, education, prenatal care, pregnancy type, and multiple comorbidities with severe maternal morbidity.
Although we have no reason to believe that systematic coding biases (eg, the probability that a hospital undercodes maternal morbidity is a positive function of its cesarean or elective delivery rates) affected our results, the existence of random coding errors could bias our findings toward the null. Our estimate for severe maternal morbidity is higher than the recent national estimate of 1.6% that was derived using similar methods4 but is consistent with data from New York City showing significantly higher than average maternal mortality rates,43 and the rates of neonatal morbidity at term are consistent with previous findings.3 Further, without medical chart review we were unable to address the recent Joint Commission recommendation to exclude women with specific types of previous uterine surgery (eg, classic cesarean delivery and myomectomy) from the measure on elective delivery before 39 weeks. We constructed risk-adjustment models that included multiple comorbidities and clinical conditions consistent with those in previous studies. Our sensitivity analyses examining the association of the quality measures with subcomponents of both outcomes measures (severe maternal morbidity without blood transfusion, prolonged neonatal length of stay, and NICU admission) reinforced our findings. By linking hospital discharge and birth certificate data we were able to control for maternal confounders, such as self-identified race/ethnicity, education, and prenatal visits associated with maternal and neonatal morbidity.
Performance on elective delivery before 39 weeks of gestation, cesarean deliveries performed on low-risk mothers, and maternal and neonatal morbidity varied widely between New York City hospitals. The obstetric quality indicators we examined were not associated with lower morbidity. Our findings highlight the need for an expanded array of obstetric quality measures.
Corresponding Author: Elizabeth A. Howell, MD, MPP, Departments of Population Health Science and Policy and Obstetrics, Gynecology, and Reproductive Science, PO Box 1077, Icahn School of Medicine at Mount Sinai, One Gustave L. Levy Pl, New York, NY 10029 (email@example.com).
Author Contributions: Dr Egorova had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Howell, Zeitlin, Hebert, Egorova.
Acquisition, analysis, or interpretation of data: Howell, Zeitlin, Balbierz, Egorova.
Drafting of the manuscript: Howell, Zeitlin.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Zeitlin, Hebert, Egorova.
Obtained funding: Howell, Balbierz.
Administrative, technical, or material support: Balbierz.
Study supervision: Howell.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.
Funding/Support: This study was funded by grant R21HD068765 from the Eunice Kennedy Shriver National Institute of Child Health and Human Development. Dr Zeitlin received funding from the European Commission, Research Directorate, Marie Curie, IOF Fellowship grant 254171.
Role of the Funders/Sponsors: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.