Development and Use of a Calculator to Measure Pediatric Low-Value Care Delivered in US Children’s Hospitals

Key Points

Question  What are the prevalence and cost associated with a set of low-value services across US children's hospitals?

Findings  This cross-sectional study used an evidence-based low-value care calculator to estimate prevalence and cost of low-value services among 1 011 950 encounters across 49 children’s hospitals contributing to the Pediatric Health Information System database. The prevalence of low-value care ranged from less than 1% to 60% across measures, with nearly $17 million in standardized cost attributable to 30 measured low-value services.

Meaning  This study found that low-value care was costly, but prevalence varied widely across measured services; use of this calculator may aid in prioritization of deimplementation initiatives.

Importance  The scope of low-value care in children’s hospitals is poorly understood.

Objective  To develop and apply a calculator of hospital-based pediatric low-value care to estimate prevalence and cost of low-value services.

Design, Setting, and Participants  This cross-sectional study developed and applied a calculator of hospital-based pediatric low-value care to estimate the prevalence and cost of low-value services among 1 011 950 encounters reported in 49 US children’s hospitals contributing to the Pediatric Health Information System (PHIS) database. To develop the calculator, a multidisciplinary stakeholder group searched existing pediatric low-value care measures and used an iterative process to identify and operationalize relevant hospital-based measures in the PHIS database. Children with an eligible encounter in 2019 were included in the calculator-applied analysis. Two cohorts were analyzed: an emergency department cohort (with encounters resulting in emergency department discharge) and a hospitalized cohort.

Exposures  Eligible condition-specific hospital encounters.

Main Outcomes and Measures  The proportion and volume of encounters in which low-value services were delivered and their associated standardized costs. Measures were ranked by those outcomes.

Results  There were 1 011 950 encounters eligible for 1 or more of 30 calculator-included measures in 2019; encounters were incurred by 816 098 unique patients with a median age of 3 years (IQR, 1-8 years). In the emergency department cohort, low-value services delivered in the greatest percentage of encounters were Group A streptococcal testing among children younger than 3 years with pharyngitis (3679 of 9785 [37.6%]), computed tomography scan for minor head injury (7541 of 42 602 [17.7%]), and bronchodilators for treatment of bronchiolitis (8899 of 55 616 [16.0%]). In the hospitalized cohort, low-value care was most prevalent for broad-spectrum antibiotics in the treatment of community-acquired pneumonia (3406 of 5658 [60.2%]), acid suppression therapy for infants with esophageal reflux (3814 of 7507 of [50.8%]), and blood cultures for uncomplicated community-acquired pneumonia (2277 of 5823 [39.1%]). Measured low-value services generated nearly $17 million in total standardized cost. The costliest services in the emergency department cohort were computed tomography scan for abdominal pain (approximately $1.8 million) and minor head injury (approximately $1.5 million) and chest radiography for asthma (approximately $1.1 million). The costliest services in the hospitalized cohort were receipt of 2 or more concurrent antipsychotics (approximately $2.4 million), and chest radiography for bronchiolitis ($801 680) and asthma ($625 866).

Conclusions and Relevance  This cross-sectional analysis found that low-value care for some pediatric services was prevalent and costly. Measuring receipt of low-value services across conditions informs prioritization of deimplementation efforts. Continued use of this calculator may establish trends in low-value care delivery.

Low-value care, or delivery of health services offering limited benefit as compared with harm, is an important domain of health care waste.^1-3 Consequences associated with such care range from physical effects, including adverse medication effects and procedural complications, to psychosocial and financial effects of incidental findings, false diagnoses, and downstream health care utilization. The prevalence and impact of low-value care remain poorly understood in pediatrics; measurement has proven challenging owing to the number and diversity of low-value practices, fragmented data sources, and a dearth of quality measures focused on low-value service delivery.^4-6

Administrative databases containing billing information for a large number of encounters and offering accessible data for longitudinal measurement have emerged as sources for quantifying low-value care.^7-9 One proprietary tool measuring nearly 50 low-value services primarily delivered to adults has been applied to state- and payer-level data sets, identifying common and costly services and describing temporal low-value care trends.^7,9,10 Studies using administrative data have established low-value care as an important pediatric problem,^11-15 but most studies describe care at a single time point or for a limited set of measures. With child health spending estimated to be equivalent to half the US defense budget at more than $300 billion¹⁶ and increasing recognition of harms associated with low-value care, understanding the extent of this problem in pediatrics is imperative. Tools leveraging large data sources for longitudinal measurement of low-value care and benchmark setting may prove valuable in scoping the issue.

As strategies to measure pediatric low-value care evolve, hospital-based care warrants particular attention. This care is increasingly costly,¹⁷ and literature on overuse of nonrecommended hospital-based pediatric services is robust, suggesting improvement opportunities.^14,15,18-21 Given this context, our specific aim was to develop a calculator to measure low-value care within US children’s hospitals. In this report, we describe the development of this calculator and apply the calculator to estimate prevalence and cost associated with low-value services in US children’s hospitals during 2019.

Following the principles of the National Quality Forum for quality measure set development,²² we convened a multidisciplinary stakeholder group of 9 subject matter experts (SMEs) consisting of pediatricians practicing hospital medicine (S.A.H., S.L.R., E.R.C., A.R.S., M.C.G., M.M., and M.A.R.), emergency medicine (J.R.M.), critical care (A.R.S.), and neonatology (T.H.) to develop a low-value care calculator. All SMEs had research experience with health care value and quality measurement. We identified and operationalized evidence-based low-value care measures and recommendations within the Pediatric Health Information System (PHIS; Children’s Hospital Association [CHA], Lenexa, Kansas) database. We selected inpatient, emergency department (ED), and neonatal intensive care unit (NICU) settings as our areas of focus owing to availability of measures relevant to these settings. The calculator development process is outlined in the Figure.^11,23 This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline²⁴ and was deemed not human subjects research by the Dartmouth College Institutional Review Board.

We first identified published pediatric quality measures or recommendations targeting reduction of nonevidence-based services (ie, low-value care measures). Relevant measure sources were identified using the Measure Use Tool from CHA,²⁵ which is a repository of pediatric quality measures, and through peer-reviewed literature describing or categorizing pediatric quality measures. We identified 5 candidate sources including low-value care measures (Figure).^11,23,26-28 Two sources contained only low-value care measures^11,26; 2 other sources categorized measures by type, explicitly identifying low-value care measures.^23,28,29 The Pediatric Quality Measures Program measures²⁷ were not categorized; 3 authors (S.A.H., S.L.R., and M.A.R.) categorized these measures to identify those targeting low-value service delivery.

After duplicate measures were excluded, a set of unique low-value care measures was distributed to all SMEs. The SMEs were first asked to determine whether each measure was relevant in at least 1 target clinical setting. Measures for which there was universal agreement on setting applicability continued to the next round of review (if deemed applicable) or were removed (not applicable). All other measures were iteratively discussed until consensus was reached. This method was then repeated, with SMEs determining whether individual measures could be operationalized within PHIS. Measures were excluded if SMEs felt that the clinical information needed to determine whether a service was low value was more nuanced than that provided in PHIS. Final candidate measures were then reviewed with members of the PHIS analytic team (M.H., H.G.D.S., A.D., and P.D.) to ensure feasibility of operationalization within the database.

Our SME group determined an approach to measure construction a priori. For measures with clear specifications, we matched original inclusion and exclusion criteria as closely as possible. For measures without clear specifications or measures appearing in multiple measure sets with conflicting specifications, we constructed definitions that were as narrow, or specific, as possible. Prior literature shows that estimates of low-value care vary with approach to measure definition.¹¹ Narrow measure definitions with multiple restrictions prioritize specificity, capturing care that is likely to be low-value but potentially underestimating low-value service delivery; broader measures with minimal restrictions prioritize sensitivity while potentially misidentifying some appropriate use as low value.¹¹ The narrow measures we used were intended to capture consensus-defined low-value care and to minimize misclassification of appropriate care, acknowledging possible underestimation of low-value care for some measures.

For all measures, we excluded patients older than 18 years and patients with International Statistical Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes documenting a complex chronic condition³⁰ or neurologic impairment³¹ within the year prior to the included encounter. For hospitalized patients, we additionally excluded encounters with an All Patient Refined Diagnosis Related Group (3M) extreme severity of illness and patients admitted to an intensive care unit at any point during hospitalization (with the exception of NICU-specific measures). These exclusions were determined a priori given that the primary literature sources supporting included measures often exclude these populations.

Measure definitions are shown in the eTable in the Supplement. Inclusion and exclusion criteria were derived from ICD-10-CM and Current Procedural Terminology codes. To achieve narrow definitions, we excluded encounters with diagnostic codes that SMEs felt may justify service delivery. Clinical services were defined by Clinical Transaction Classification codes specific to PHIS.

After calculator development, we conducted a cross-sectional, observational cohort study using the PHIS database. This database contains deidentified administrative data detailing demographic characteristics, diagnostics, procedures, and daily billing information from 49 tertiary referral care children’s hospitals, accounting for approximately 20% of all annual pediatric hospitalizations and approximately 12% of all ED visits in the US. Data quality is ensured through a joint effort between CHA and participating hospitals.

Results were analyzed for 2 cohorts: (1) the ED cohort, including encounters resulting in discharge from the ED, and (2) the hospitalized cohort, including encounters for patients admitted to a medical department (inpatient or observation status) or to the NICU. In the hospitalized cohort, care delivered during the inpatient encounter was not separable from that delivered in the associated ED encounter within the same center; as such, results for the hospitalized cohort reflect care delivered in both settings, if applicable.

Data were analyzed from January 1 to December 31, 2019, and hospitals were included only if they consistently contributed data during this period. Encounters were eligible if they met inclusion criteria for at least 1 included measure and no exclusions were identified.

We used the low-value care calculator to assess 3 outcomes for each measure: (1) percentage of eligible encounters in which a low-value service was delivered, (2) number of encounters in which a low-value service was delivered, and (3) standardized unit cost associated with low-value care. Standardized unit costs were previously developed by the CHA as a measure for comparison of resource utilization across hospitals in the setting of interhospital variation in cost definitions and are determined by calculating the median cost for services across PHIS hospitals; a full description is published elsewhere.³²

To inform deimplementation efforts, we ranked measures in each setting by these 3 outcomes. As a subanalysis, we grouped measures into 4 categories (medications, imaging, labs, and procedures) and calculated standardized cost associated with low-value care for each category. We also calculated category-specific standardized cost for all eligible encounters (ie, a sum of cost for medications provided in all eligible encounters), allowing determination of the percentage of total standardized cost within a category that was attributable to low-value care. Statistical analyses were performed with SAS, version 9.4 (SAS Institute Inc).

The final low-value care calculator included 30 measures. Of these measures, 22 were applicable to the ED cohort; 774 584 encounters by 621 633 unique patients were eligible for these measures from 47 hospitals. There were 26 measures applicable to the hospitalized cohort (including 2 NICU measures), for which there were 237 366 eligible encounters by 194 465 patients from 49 hospitals. The median age of patients with included encounters was 3 years (IQR, 1-8 years).

Table 1 describes low-value care delivery for the ED cohort. Measures with the greatest percentage of low-value care delivery among eligible encounters were testing for group A streptococcus among children younger than 3 years with pharyngitis (3679 of 9785 [37.6%]), computed tomography (CT) scan for minor head injury (7541 of 42 602 [17.7%]), and bronchodilator treatment of bronchiolitis (8899 of 55 616 [16.0%]).

Measures for which low-value care was associated with the greatest number of encounters were chest radiography for asthma (n = 10 971), followed by bronchodilators for treatment of bronchiolitis (n = 8899), and chest radiography for bronchiolitis (n = 8676). The measures associated with the greatest cost were CT scan for abdominal pain (approximately $1.8 million), CT scan for minor head injury (approximately $1.5 million), and chest radiography for asthma (approximately $1.1 million). Magnetic resonance imaging for febrile seizure (n = 11; $2928) and antibiotics for treatment of asthma (n = 138; $8518) were measures for which low-value care delivery was infrequent and associated costs were low.

Table 2 describes low-value care delivery for the hospitalized cohort. Measures with the greatest percentage of low-value care delivery among eligible encounters were antibiotics broader than ampicillin for treatment of community-acquired pneumonia (CAP; 3406 of 5658 [60.2%]), acid suppression therapy for infants younger than 1 year with esophageal reflux (3814 of 7507 [50.8%]), and blood cultures for uncomplicated CAP (2277 of 5823 [39.1%]).

Measures for which low-value care was associated with the greatest number of encounters was bronchodilator treatment of bronchiolitis (n = 6964) and chest radiography for bronchiolitis (n = 6254) and asthma (n = 6203). The costliest measures were receipt of 2 or more concurrent antipsychotics (approximately $2.4 million), chest radiography for bronchiolitis ($801 680) and chest radiography for asthma ($625 866).

Measures showing the lowest proportion of low-value care delivery included antipsychotics for children younger than 5 years (110 of 109 538 eligible encounters [0.1%]), ipratropium bromide after 24 hours of hospitalization for treatment of asthma (402 of 19 145 [2.1%]), and antibiotics for treatment of asthma (405 of 18 417 [2.2%]). The NICU measures were ranked separately (Table 2).

Across all conditions, measured low-value care delivery generated nearly $17 million in standardized cost; 55% of this cost was generated by low-value services in the ED cohort. Bronchiolitis measures generated the greatest standardized cost at more than $3.6 million, followed by behavioral health measures at nearly $2.4 million. Low-value care for pneumonia, abdominal pain, and asthma generated substantial cost in both cohorts (Table 3). The median standardized cost of low-value care per hospital was $306 018 (IQR, $157 397-$481 965).

Table 4 gives the standardized cost associated with each category of low-value care. The 9 imaging measures accounted for the largest standardized cost overall (>$9.5 million) and the greatest proportional cost by category, accounting for 27.3% of all standardized imaging costs among eligible encounters.

The development and application of a calculator incorporating 30 pediatric, hospital-based, low-value care measures revealed nearly $17 million in standardized costs attributable to these practices in 2019. A wide range of performance was observed across measures, with group A streptococcus testing for young children and broad-spectrum antibiotic use for treatment of CAP being delivered in the highest proportion of encounters in the ED and hospitalized cohorts, respectively. Our results support prior assertions that low-value pediatric care warrants focused measurement and improvement efforts.¹⁶

Our work was informed by prior efforts to describe pediatric low-value care. Chua et al¹¹ developed 20 claims-based measures of pediatric low-value care that have been applied to multiple data sources.^12,13 An analysis of care delivered in 2014 found that at least 10% of commercially insured children received 1 or more of these services, accounting for $27 million in spending; 34% of this total was paid out of pocket.¹¹ Modestly higher rates of low-value service delivery were identified among publicly insured and military-insured children.^12,33 Reyes et al,^14,15 using the PHIS database, created a report card to measure performance on the original measures included in the pediatric Choosing Wisely Campaign by the Society of Hospital Medicine among hospitalized patients and found that low-value care delivery ranged from 12% to 49% across these measures. Our work incorporates a broad set of hospital-based, pediatric, low-value care measures into a tool capable of sustaining these measurement efforts. The low-value care identified in our study highlights the persistent potential for value improvement in pediatrics through deimplementation of nonevidence-based practices.

Bronchiolitis, CAP, and asthma measures had a relatively high prevalence of low-value care among both cohorts. These are among the most common and costly conditions treated in the pediatric hospital setting³⁴ and are popular targets for quality improvement initiatives, yet low-value care persists. Comparisons between our data and those previously published reveal important trends. For example, in the hospitalized cohort, use of broad-spectrum antibiotic therapy for CAP was only slightly lower than that observed in PHIS in 2012,¹⁹ supporting a need for innovation on this measure. This need is reinforced by the inclusion of this measure in the 2021 Pediatric Hospital Medicine Choosing Wisely recommendations.³⁵ Blood culture rates for CAP were even higher than some prior PHIS estimates for the hospitalized cohort.³⁶ On the contrary, broad-spectrum antibiotic and blood culture use in the ED were considerably lower than rates previously described in the ED setting using varying data sources.^37-40 These results highlight the importance of assessing the trajectory of low-value care over time; our calculator can facilitate the longitudinal measurement needed to establish such trends.

Our investigation also identified low-value care for conditions that have historically not been prioritized for deimplementation. In the inpatient cohort, acid suppression was used in more than one-half of encounters by children with gastroesophageal reflux. This rate is similar to that observed by Reyes et al^14,15 in 2017. Despite a clinical practice guideline recommending against this treatment,⁴¹ published quality improvement initiatives targeting this service are limited; inpatient clinicians may be in a unique position to effect change in this practice. Concurrent antipsychotic administration occurred in 21% of eligible encounters. Although multiple psychotropic medications may be deemed necessary to maintain patient and staff safety in some clinical circumstances, evidence for the effectiveness of this practice has not been established, and the potential for harm related to adverse effects and drug-drug interactions is high.^42,43 Efforts should be made to explore whether additional hospital-based behavioral health resources may decrease this practice.

In the ED, group A streptococcus testing was performed in more than one-third of patients younger than 3 years with pharyngitis. With low rates of pathogenic streptococcal pharyngitis and very low risk of complications, such as acute rheumatic fever, in this population,^44,45 this practice places children at risk for unnecessary antibiotics and associated adverse effects.

As efforts increase to alleviate measurement burden in health care,⁴⁶ data identifying measures that might be deprioritized are also useful. In the ED cohort, head imaging for febrile seizures and blood cultures for bronchiolitis were observed relatively infrequently in eligible encounters. In the hospital cohort, antipsychotic administration to children younger than 5 years and ipratropium delivery after 24 hours of hospitalization were also infrequent.

This single-year analysis represents an initial step; several future steps may enhance understanding of low-value care patterns in US children’s hospitals. Continued application of this tool will aid in establishing and monitoring temporal low-value care trends and identifying services in need of ongoing deimplementation efforts. Hospital-specific reports have been distributed to PHIS-participating centers to facilitate benchmarking and local quality improvement work. Finally, further analyses will characterize variation in low-value care by hospital and aim to identify facilitators and barriers to value improvement.

Our work has important limitations. Our measure definitions rely on diagnostic codes representing the discharge diagnosis for a particular encounter. These codes are influenced by services provided during the encounter and their findings. As a result, it is possible that some inappropriately prescribed services influenced discharge diagnosis codes such that low-value care was underestimated. For example, our narrow measure definitions would not identify scenarios in which inappropriate chest radiography in the setting of bronchiolitis led to overdiagnosis and overtreatment of pneumonia, as pneumonia is an exclusionary diagnosis for this measure. Conversely, our approach may have overestimated low-value care when appropriately prescribed services return normal results. For example, a significant mechanism of injury or subtle behavioral change in a child with head trauma may warrant a CT scan, but a normal CT scan may result in a diagnosis of minor head injury and thus be deemed of low-value. It is our hope that hospitals will use individualized data in the context of peer hospitals to better understand their practice patterns and track improvements over time. Further efforts to validate included measures with robust clinical data will also strengthen conclusions that can be drawn from calculator use.

In addition, our data reflect only practices associated with published low-value care measures in specific clinical settings; as such, they should not be viewed as a comprehensive picture of all hospital-based low-value care. Our analysis includes only data from US children’s hospitals participating in PHIS, and our findings may not be generalizable to other settings. With a majority of pediatric hospital-based care being delivered outside of these centers, our results reflect a minority of pediatric low-value care delivery. Efforts to expand low-value care measurement beyond the children’s hospital setting are critical to gaining a more robust understanding of how such care may impact children. Finally, we have not assessed harms associated with low-value care beyond direct financial cost; further exploration of outcomes, including related downstream health care utilization, is needed.

We identified nearly $17 million in cost associated with low-value services delivered in US children’s hospitals during a single year. Our analysis identified some low-value services that are frequent and costly and other low-value services with lesser associated impact, offering data for prioritization of deimplementation efforts.

Accepted for Publication: September 23, 2021.

Published: December 30, 2021. doi:10.1001/jamanetworkopen.2021.35184

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 House SA et al. JAMA Network Open.

Corresponding Author: Samantha A. House, DO, MPH, Children’s Hospital at Dartmouth-Hitchcock Medical Center, One Medical Center Dr, Lebanon, NH 03750 (Samantha.A.House@hitchcock.org).

Author Contributions: Drs House and Hall had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: House, Hall, Ralston, Coon, Schroeder, De Souza, Ho, Genies, Reyes.

Acquisition, analysis, or interpretation of data: House, Hall, Ralston, Marin, Schroeder, De Souza, Davidson, Duda, Genies, Mestre, Reyes.

Drafting of the manuscript: House, Hall, Ralston, De Souza, Ho, Genies.

Critical revision of the manuscript for important intellectual content: House, Ralston, Marin, Coon, Schroeder, De Souza, Davidson, Duda, Ho, Genies, Mestre, Reyes.

Statistical analysis: House, Hall, De Souza, Genies, Reyes.

Administrative, technical, or material support: House, Davidson, Duda, Ho, Reyes.

Supervision: House, Ralston, Coon.

Conflict of Interest Disclosures: Dr Marin reported serving on a medical review committee for Highmark Inc outside the submitted work. No other disclosures were reported.