Effects of the World Health Organization Safe Childbirth Checklist on Quality of Care and Birth Outcomes in Aceh, Indonesia

This cluster randomized clinical trial examines the effect of implementing the Safe Childbirth Checklist with medium-intensity coaching on the performance of essential birth practices in health care facilities in Aceh, Indonesia.


Statistical analysis
For treatment assignment, we used the publicly available minMSE code in Stata. 26 We estimated Intention To Treat (ITT) effects and Complier Average Causal Effects (CACE) using Stata, whereas supplementary Generalized Linear Models (GLM) estimates were computed via the statistical computing package R. Finally, we used a publicly available program in Stata to adjust p-values for false discovery rates. 33,34 Intention To Treat (ITT) Effect Analysis The basic estimation equation for the Intention to Treat (ITT) effect reads as follows: Outcome is a binary indicator with the value 1, if the respective practice was applied at facility i for birth j. indicates if the facility was in the treatment group, is a vector of covariates at the facility-level (facility type, urban-rural, CEmONC-status, district dummy), and is the error term.

Complier Average Causal Effect (CACE) Analysis
Our CACE approach builds on a two-step analysis, where the treatment allocation serves as an instrument to predict compliance in the first stage: Where indicates compliance (checklist use) at facility i for birth j. Predicted compliance ̂ is inserted into the 2nd stage to estimate Equation (3) analogous to Equation (1): where refers to outcomes at facility i for birth j, which are regressed on predicted compliance. Compliance is measured at the birth level by midwives actively using or looking at the SCC during clinical observations. When analyzing facility-level outcomes with a CACE, we measured compliance by calculating the number of completed SCCs over the total numbers of births. While individuals from the control group may also theoretically qualify as compliers if they would use the SCC (also labeled "always-takers" in the literature), this case did not materialize during observed births. 31 Our clustered trial design (provision of SCC at facility-level) made spill-overs very unlikely to happen. Cragg-Donald Wald F statistics were on both levels larger than 10, which suggests that the treatment is a sufficiently strong predictor of compliance to warrant reliable inference (e.g., we do not face weak instrumental variable issues which may inflate our estimates).

Penalized maximum likelihood logistic regression
Although neonatal and maternal mortality, as well as stillbirths in particular, impose a burden for the considered sample, their frequencies still qualify them as rare events in a statistical sense.. To account for this, we apply a penalized maximum likelihood logistic regression estimator, which applies a penalization to generalized linear models to correct for potential bias. 32 This approach is implemented via the firthlogit package in Stata.
Minimal detectable effects eTables 1 and 2 describe minimal detectable effect sizes for the essential practices covered under the SCC as well as for health outcomes. We estimate minimal detectable effects based on the following formula: where MDE refers to the minimal detectable effect, t 1− ∝ 2 to the level of intended statistical significance and ρ to the intended power (in our case 0.8). 2 is the variance, N indicates the number of observations per cluster and f the fraction of observations in treatment and control group. Effect sizes suggest that the study is sufficiently powered to analyze effects for essential practices, but that the sample was not sufficiently large to draw broader conclusions with regard to mortality and morbidity outcomes.
Mortality and morbidity (Facility-level estimates) eTable 4 presents the outcomes for maternal and neonatal mortality and stillbirths (Rows 1, 2 and 3) as well as complications (Rows 4 and 5). Columns 2 and 4 in eTable 4 display the mean values in the treatment and control groups, whereas columns 5 and 7 depict our regression estimates from the ITT analysis and from the CACE estimation in columns 9 and 11. In the CACE estimations, the coefficients' confidence interval [-68.0 | 4.0] includes a larger range of negative values and turns out to be significant at the 10%-level when adding covariates. As only one maternal death was recorded in the 32 facilities over the six months intervention period results for maternal mortality are insignificant. With regard to stillbirths and neonatal mortality, eTable 3 indicates consistently negative coefficients for the ITT, which are in line with the expectations, but are generally insignificant. In the CACE estimations, the coefficient for neonatal mortality turns significant when adding covariates. However, those results should be treated cautiously as a reduction in the mean neonatal mortality rate in the treatment facilities from 2/1000 at baseline to 1.1/1000 at endline is contrasted by an increase in the mean of the control facilities from 2.7/1000 at baseline to 15/1000 at endline. Similarly, the stillbirth rate both increased in treatment and control group between base-and endline from 11/1000 to 17/1000 and from 13/1000 to 21/1000 respectively. Nonetheless, in line with the consistently negative signs across specifications, this provides some weak evidence that the SCC could contribute to a reduction in neonatal mortality. Effects for complication rates (Rows 4 and 5) are statistically insignificant, for a list of complications see eTable 9.

Background informationcoaching approach:
The coaching comprised of three visits in the first month, two visits in the second, third and fourth month and finally one visit in the fifth and sixth month. The coaching consisted of a two-hour visit of the coach at the facility and (i) a meeting with the checklist quality coordinator (CQC), (ii) filling out a short survey on usage and barriers and (iii) an opportunity for consultation on correct SCC use. Additionally, when possible, the coaches (iv) provided feedback on observed births and (v) gave input regarding the previously collected checklists. CQCs were selected among the midwives to ensure regular use of the SCC and support other midwives with the application. The CQCs did not receive any remuneration.

Checklist adjustment
Together with local health staff, our team adjusted the checklist to the local context and needs as listed below: -Change item -Pause Point 2: o Confirm essential supplies are at bedside and prepare for delivery: Adapted item "Sterile blade to cut cord" to "Sterile scissors/knife to cut cord" to account for local practice -Pause Point 3: o Is mother bleeding abnormally?: Added items "Yes, treat, but if cause cannot be treated, refer," and "If cause cannot be treated, refer based on your criteria" to accommodate the fact that community health centers (puskesmas) may not be able to address abnormal bleeding o Start breastfeeding and skin-to-skin contact (IMD) (if mother and baby are well): Added item "No, start later because mother or baby are not well" to provide midwives with an option to check if patient's status does not allow the starting of breastfeeding and skin to skin contact -Pause Point 4: o Discuss and offer family planning options to mother: Add item "No, already done in antenatal care" and "No, will be done later" to allow midwives to check item if the point cannot be addressed immediately Besides contextual adjustments, we added two major modifications according to practitioners' feedback. First, we included a slot for SCC users to note the time and date of given medication. This documentation should make it easier for the next user to assess which medication was needed and at what time. Second, we included a field to add information on the mother to attach it to the patient file (e.g., mother's name, age, weight, height). Third, we prepared a separate sheet for the danger signs to hand over to mothers and relatives before discharge, as a means for families to be better able to remember and assess situations in which they should return to the health facilities. An English version of the adapted SCC can be found in eFigure 2. For two facilities, we had to impute rates based on one observation month (March 2017), as data at the facility-level was not available for multiple months. Yearly birth volumes from the baseline data collection, however, strongly correspond to the monthly births in March 2017. Notes: Confidence intervals based on ordinary least squares (Intention To Treat) and two-stage-least-squares (Complier Average Causal Effect) estimations in brackets. p-val *<10%, **<5% and ***<1%. The set of covariates include District, Urban-Rural, CEmONC and Facility Type. Notes: Confidence intervals based on ordinary least squares (Intention To Treat) and two-stage-least-squares (Complier Average Causal Effect) estimations in brackets. p-val *<10%, **<5% and ***<1%. The set of covariates include District, Urban-Rural, CEmONC and Facility Type. ^ One facility did not report any outcomes with regard to complications in the endline data collection. Rates refer to # x/100,000, ## x/1,000.

eTable 5: Mortality and morbidity (subsample of facilities where observations were conducted)
For two facilities, we had to impute rates based on one observation month (March 2017), as data at the facility-level was not available for multiple months. Yearly birth volumes from the baseline data collection, however, strongly correspond to the monthly births in March 2017. -1 x Checklist Introduction event (2 hours): Presentation| checklist explanation| role play (no training)| selection of (non-remunerated) checklist quality coordinators -11 x Monitoring visits over six months (2 hours each): SCC provision & collection| interviews with providers| feedback on previous performance| opportunity to ask questions -2 x Meetings of facility-based checklist quality coordinators (2 hours each) threeand six-months post introduction: Focus group discussions to exchange best practice -Checklist provision and provision of danger sign sheets Exposure to research team: -Information event for facility leadership to present study design -Observations (24 hours over six days in larger facilities and one month on call in smaller facilities) -Survey on provider characteristics and perceptions (30-40 minutes) SCC exposure: -None Exposure to research team: -Information event for facility leadership to present study design -Observations (24 hours over six days in larger facilities and one month on call in smaller facilities) -Survey on provider characteristics and perceptions (30-40 minutes)