Clinical Prediction Models for Suspected Pediatric Foreign Body Aspiration: A Systematic Review and Meta-analysis

Key Points

Question  What is the evidence for the use of clinical predictions models in diagnosing pediatric foreign body aspiration?

Findings  This systematic review and meta-analysis identified models using predictors based on clinical history, physical examination, and radiographic findings. A meta-analysis of model performance suggests that there is a potential benefit of such decision-making tools, but is overall limited because of concerns of model overfitting, inconsistent reporting of performance, and lack of model validation.

Meaning  Current prediction models for the diagnosing pediatric foreign body aspiration are at high risk of bias and are not recommended to guide clinical decision-making; thus, adherence to recognized guidelines for conducting and reporting modeling studies is recommended.

Importance  Although various clinical prediction models (CPMs) have been described for diagnosing pediatric foreign body aspiration (FBA), to our knowledge, there is still no consensus regarding indications for bronchoscopy, the criterion standard for identifying airway foreign bodies.

Objective  To evaluate currently available CPMs for diagnosing FBA in children.

Data Sources  Performed in Ovid MEDLINE, Ovid Embase, PubMed, Web of Science, and CINAHL database with citation searching of retrieved studies.

Study Selection  Prediction model derivation and validation studies for diagnosing FBA in children were included. Exclusion criteria included adult studies; studies that included variables that were not available in routine clinical practice and outcomes for FBA were not separate or extractable.

Data Extraction and Synthesis  We followed the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies and the Prediction Model Risk of Bias Assessment Tool framework. Data were pooled using a random-effects model.

Main Outcomes and Measures  The primary outcome was the diagnosis of FBA as confirmed by bronchoscopy. Characteristics of CPMs and individual predictors were evaluated. The final model presentation with available measures of performance was provided by narrative synthesis. A meta-analysis of individual predictor variables and prediction models was performed.

Results  After screening 4233 articles, 7 studies (0.2%; 1577 patients) were included in the final analysis. There were 6 model derivation studies and 1 validation study. Air trapping (odds ratio [OR], 8.3; 95% CI, 4.4-15.5), unilateral reduced air entry (OR, 4.8; 95% CI, 3.5-6.5), witnessed choking (OR, 3.1; 95% CI, 1.0-9.6), wheezing (OR, 2.5; 95% CI, 1.2-5.2), and suspicious findings suggestive of FBA on radiography (OR, 18.5; 95% CI, 5.0-67.7) were the most commonly used predictor variables. Model performance varied, with discrimination scores (C statistic) ranging from 0.74 to 0.88. The pooled weighted C statistic score of all models was 0.86 (95% CI, 0.80-0.92). All studies were deemed to be at high risk of bias, with overfitting of models and lack of validation as the most pertinent concerns.

Conclusions and Relevance  This systematic review and meta-analysis suggests that existing CPMs for FBA in children are at a high risk of bias and have not been adequately validated. No current models can be recommended to guide clinical decision-making. Future CPM studies that adhere to recognized standards for development and validation are required.

Quiz Ref IDPediatric foreign body aspiration (FBA) is a potentially life-threatening emergency that is most frequently seen in patients younger than 5 years.^1-3 Foreign body aspiration can lead to partial or complete obstruction of the airway, resulting in pneumonia, atelectasis, bronchiectasis, anoxic brain injury, or death.^4,5 Rapid diagnosis and management of FBA is essential to prevent these sequelae.^6,7 Unfortunately, the diagnosis can be challenging because of a vague presentation with subtle physical examination and radiological findings, leading to delayed diagnosis and risking complications.^8-15Quiz Ref ID A definitive diagnosis of FBA can only be made with flexible or rigid bronchoscopy.¹⁶ However, because bronchoscopy is invasive, involves a general anesthetic, is often performed on children only in pediatric centers, and can have a 16% to 57% rate of negative findings, an algorithm for predicting FBA would be very beneficial.¹⁷

Clinical prediction models (CPMs) help health care professionals to evaluate the probability of a diagnosis to assist with patient stratification.^18-20 Clinical predication models incorporate at least 2 predictors that are derived from clinical history, physical examination, test results, or responses to treatment to estimate the probability of different outcomes based on the clinical presentation.^21,22 Several CPMs for FBA in children have been proposed, but to our knowledge, they are heterogenous and have not been combined into 1 accepted decision algorithm. The aim of this study was to analyze and consolidate the literature on prediction models for the diagnosis of suspected FBA.

This systematic review was completed in accordance with the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) and Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guidelines.^23,24 A systematic search was completed of the Ovid MEDLINE, Ovid Embase, PubMed, Web of Science, and CINAHL databases to identify all studies that described CPMs for diagnosing FBA in children. The search keywords and medical subject headings terms were used under the topics of (1) clinical prediction models, (2) foreign body aspiration, and (3) pediatrics (eTable 1 in the Supplement).

The study inclusion criteria were the following: (1) study type: randomized clinical trials, prospective or retrospective observational studies, cross-sectional/case-control trials, prediction model derivation studies with or without external validation, and external model validation studies; (2) population: pediatric (age <18 years); (3) intervention/exposure: CPMs developed for use in clinical practice to support the diagnostic decision-making of a health care professional during their assessment of pediatric patients with symptoms that were suggestive of FBAs; (4) comparator: not applicable; and (5) setting: inpatient setting/emergency department. Only original research studies that were published in English in peer-reviewed journals were included. The citations of studies included in the full-text analysis were screened to ensure all relevant studies were included, and the primary outcome was the diagnosis of FBA as confirmed by bronchoscopy. Studies were excluded if they included the following: study participants younger than 18 years or adult prediction models, a foreign language, variables used in the model that would not be available in routine clinical practice (eg, use of multidetector computed tomography scans)^25,26 or not clearly reported, outcomes for FBA that were not separate, or data that were associated with the FBA outcome were not extractable. Additionally, nonoriginal studies, such as editorials, expert views, nonresearch letters, unpublished abstracts, conference posters, reviews, and letters to editors were excluded.

Three independent reviewers (J.P., M.L., and A.N.) selected articles for inclusion in the order of title screening, abstract screening, and full text screening. Any article inclusion disagreements were resolved by consensus. Data were collected based on the CHARMS checklist,²⁴ and extracted items included (1) study demographic characteristics, (2) definition and number of outcome events: foreign body diagnosed on bronchoscopy, and (3) clinical prediction model: description of prediction model/algorithm/protocol, statistical methods to derive the CPM, number of patients included in the CPM, number and description of candidate predictors, measure of model performance (area under the receiver operating characteristic curve [AUC], sensitivity, specificity, positive predictive value, negative predictive value, odds ratios [ORs]/likelihood ratios), and methods for internal or external validation.

The characteristics of included CPM studies and clinical predictors were evaluated with descriptive analysis. The final model presentation with available measures of performance was provided and summarized by narrative synthesis. Events per variable (EPV) were derived by dividing the number of outcome events by the number of candidate predictor variables. Performance measures, such as sensitivity, specificity, and C statistics, were calculated if not provided and data were available. The C statistic quantifies the ability of a model to distinguish individuals who are developing and not developing an outcome and corresponds to the AUC.²⁷ A meta-analysis of model performance was performed by summarizing retrieved estimates of C statistics as a weighted average.²⁸ The derivation of the C statistic and standard error from retrieved measures is described in eAppendix in the Supplement. Individual predictor variables that were included in more than 50% of CPMs were evaluated by calculating the pooled sensitivity and specificity values and performing meta-analysis. A random-effects model was adopted for all analyses given its suggested advantages of accounting for between-study heterogeneity for meta-analysis of prediction models.²⁹ The I² statistic was calculated to assess the degree of heterogeneity, with a value exceeding 50% implying substantial heterogeneity. Meta-analysis was performed and forest plots were constructed using Review Manager, version 5.4 (Cochrane).³⁰

The risk of bias (ROB) and applicability assessment within individual studies was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST).²⁴ Organized into 20 signaling questions across 4 domains, including participants, predictors, outcomes, and analysis, each domain is scored and given a bias assessment of low, high, or unclear, with a study being deemed high ROB if any domain is determined to have a high score. An overall rating for applicability was determined by evaluating the domains of participant selection, predictors, and outcomes.

The literature search identified 4233 studies, of which 71 (1.7%) were included for full-text review (Figure 1). Of these, 64 articles did not meet eligibility criteria and 7 were included in the systematic review.^31-37 Details of excluded studies are provided eTable 2 in the Supplement.

Of the included articles, 6 described newly developed CPMs^31-35,37 and 1 externally validated 1 of these models³⁶ (Table 1). No internal validation was performed in the model derivation studies. Multivariable logistic regression was used in 6 studies,^31-36 with 1 developing a decision tree algorithm³⁷ (Table 2).

All studies were performed exclusively in children (age <18 years) with a mean age ranging from 1.5 to 2.8 years. The total number of patients was 1577, with a median (interquartile range) of 150 (77-720) per study. Diagnostic bronchoscopy (flexible or rigid) was performed in all patients in 5 studies,^31-35 whereas 2 studies included patients with suspected FBA who did not ultimately undergo bronchoscopy.^36,37 Airway foreign body was identified in 653 children (41.4%) across all studies (range, 12%-76%) (Table 1).

Using the PROBAST checklist, all studies were deemed to have a high risk of bias (eTable 3 in the Supplement). Across nearly all models, ROB was associated with predictors, and outcomes were evaluated to be low. By contrast, all CPMs were deemed to be high ROB in the statistical analysis domain. Specifically, this was associated with a low EPV in 6 of 7 studies (86%), incomplete reporting of relevant performance measures (6 [86%]), lack of internal or external model validation (5 [71%]), and no correction or adjustment of models to prevent overfitting when indicated (5 [71%]).

Of the CPMs, 6 models were deemed to have high³⁷ or unclear concerns^31-35 of applicability associated with participant selection. The exclusion of children who did not undergo diagnostic bronchoscopy because of their presumed low risk of FBA raises concerns of the applicability of the models. Specifically, it is unclear if these prediction models would be applicable for children who are suspected to have a FBA with noncontributory or equivocal clinical finding. The CPM with a low concern of applicability included children who were deemed low risk of FBA on initial evaluation and followed accordingly with clinical surveillance.³⁶ A detailed ROB and applicability assessment is provided in the eMethods in the Supplement.

Of the 6 CPM derivation studies, 4 presented a scoring system based on the number of risk variables (Table 2).^31-33,35 The scoring systems differed based on the variables included and scores assigned to each predictor variable; however, diagnostic bronchoscopy was recommended for scores of 2 or higher in all 4 scoring systems based on CPMs. One study presented a computerized scoring system in which the probability of a positive diagnosis was generated from the sum of numeric coefficients derived from the logistic regression analysis.³⁴ The last model derivation study provided a binary decision tree algorithm based on the presence of positive predictor variables.³⁷

All 7 studies determined a positive outcome of FBA as diagnosed based on bronchoscopy. A negative outcome was defined as no airway foreign body identified in children who received bronchoscopy. In 2 studies, children at low risk of FBA who did not receive some form of bronchoscopy (rigid or flexible) were also determined to be negative for FBA based on asymptomatic or minimally symptomatic presentation with clinical observation.^36,37 Rigid and flexible bronchoscopy were used, with flexible being used as a diagnostic tool in suspected cases and conversion to rigid for therapeutic removal of FBAs.^31-34,36,37 One study exclusively used rigid bronchoscopy.³⁵

Predictor variables were mostly derived from the clinical history, physical examination, and radiographic findings from chest radiography (eTables 4 and 5 in the Supplement). Quiz Ref IDThe most common predictors (>50% of models) were focal hyperinflation/air trapping, uniliteral reduced air entry, witnessed choking event, wheezing, and radiopaque/suspicious finding on radiography results for FBA (Figure 2). Variables of cough, dyspnea, cyanosis, pneumonia, and atelectasis were evaluated in most studies but were not selected forward as predictors in the CPMs. A derived model and its validation study used patient demographic characteristics of age and sex, although only male sex was significantly associated with the outcome in just 1 of the 2 cohorts.^34,36 No other studies utilized demographic factors as a predictor variable. In 1 study, an elevated white blood cell count (>10 × 10⁹/L) was found to be significantly associated with FBA (OR, 3.3; 95% CI, 1.4-7.7) and included into their final model.³²

Quiz Ref IDFocal hyperinflation or air trapping on chest radiography results demonstrated a strong association with an OR of 8.3 (95% CI, 4.4-15.5) across all 7 studies (Table 3).^31-37 There was considerable variation among studies (I², 70%), with the ORs of individual studies ranging from 2.7 to 61.1 (eFigure in the Supplement).^32,33 Janahi et al³³ did not include focal hyperinflation in their final model and reported that 34% of patients with diagnosed FBA had air trapping vs 16.3% of children who were determined to be negative for FBA. Focal hyperinflation as a variable across all studies had a sensitivity of 57.5% and specificity of 84.2% to predict FBA.

Physical examination findings of unilateral reduced air entry on auscultation were included in CPMs as a predictor variable.^31,33-37 The prediction of FBA with unilateral decreased air entry demonstrated a consistent association across all studies, with an OR of 4.8 (95% CI, 3.5-6.5) and low heterogeneity (I², 24%) (eFigure in the Supplement). The definition of unilateral auscultation findings in 2 studies included not only decreased breath sounds but also any other abnormal findings, such as crackles, wheezes, or bronchial sounds, raising some ROB to these estimates.^34,36 Unilateral decreased air entry as a predictor variable alone had a sensitivity of 58.1% and specificity of 69.4%.

Witnessed choking as a predictor based on clinical history was collected in all studies and included in 5 CPMs.^31-34,36 The strength of the association for choking as a predictor variable in studies in which individual data were available demonstrates a significant association, with an OR of 3.1 (95% CI, 1.0-9.6; I², 90%) (eFigure in the Supplement). Across all available studies, choking as a predictor variable alone had a sensitivity of 45.9% and specificity of 52.4% (Table 3).

Wheezing on physical examination was collected in 6 studies and selected forward in 4 of the final CPMs.^34-37 Wheezing as a variable had a significant association, with an OR of 2.5 (95% CI, 1.2-5.2; I², 82%) (eFigure in the Supplement); the sensitivity was low at 33.1%, with a specificity at 77.5%.

Radiopaque findings that were suspicious for FBA were evaluated in 6 studies and included in 4 of the final CPMs.^34-37 The presence of this variable was strongly associated with the diagnosis of FBA, with an OR of 18.5 (95% CI, 5.0-67.7) (eFigure in the Supplement) and specificity of 100% across all studies, although it had a low sensitivity of 7.5%.

Performance measures of individual models are presented in Table 2. Overall, the reporting of performance measures for the CPMs varied across studies. Discrimination ability by C statistic scores was provided in 4 models^33-36 and derived in 2 studies,^31,37 whereas calibration scores were only reported in a single study.³⁴ Classification measures, such as sensitivity and specificity of CPMs based on the probability threshold to perform bronchoscopy, were reported in 5 studies^33-37 and calculated for 1 CPM.³¹ One model provided only cumulative proportions of FBA by the number of risk factors using a graphical representation; thus, there were inadequate data available to derive performance measures.³²

Individually reported or derived C statistics of 5 derivation and 1 validation models are provided in Figure 3. Discrimination performance for these CPMs ranged from 0.74 to 0.94 (C statistic).^31,33-36 A pooled C statistic of all CPMs was estimated to be 0.86 (95% CI, 0.8-0.92; I², 83%). When evaluating the only CPM with external validation by Kadmon et al³⁴ and Stafler et al,³⁶ respectively, the combined estimated C statistic was 0.83 (95% CI, 0.70-0.97). The remaining 5 CPMs did not have any form of external validation, raising concerns of overestimation of model performance.

This systematic review aimed to analyze and consolidate the literature on CPMs for FBA in children. In total, 6 diagnostic prediction models from 7 studies were evaluated, with model validation conducted in only 1 CPM. All CPMs were presented to guide decision-making for diagnostic bronchoscopy based on scores derived from history, physical examination, and radiographic findings. Individual and pooled model discrimination performance were found to be excellent overall (C statistic, >0.75). However, all CPMs were found to be at high ROB, specifically in association with concerns of analytic limitations leading to model overfitting, inadequate reporting of performance, and an overall lack of internal and external validation.

Regarding model development, several methodological issues were identified. The number of EPV is commonly used to evaluate adequate sample size for CPM studies, with a score of 10 or higher recommended to avoid overfitting.^24,38,39 In validation studies, a minimum of 100 events and 100 nonevents have been suggested. Only 1 study by Ozyuksel et al³⁵ met this criterion. Furthermore, the use of shrinkage techniques, such as adjustments to the estimated weights of predictors in the multivariable model, can address possible overfitting. In our review, 5 CPMs selected variables from univariate analysis, with no adjustments in their estimated weights.^2,4,6-8 The derivation model proposed by Kadmon et al³ and its subsequent validation study by Stafler et al⁵ included all collected variables in their final model, with adjusted weights for each variable. However, there appear to be differences in the coefficients of the same variables between the 2 studies, which reflects more of a model redevelopment rather than a true validation. In terms of assessing model performance, this review found the reporting of discrimination and calibration measures to be lacking. A model’s effectiveness at differentiating between those with or without outcomes, also known as discrimination, was reported in 4 studies.^33-36 Calibration, or how well the predicted risk scores compare with the observed outcomes, was provided in only 1 study.³⁴ The absence of either calibration or discrimination measures makes a full appraisal and meta-analysis of prediction models difficult.²⁴ Finally, a substantial limitation of most models included in our review is the lack of internal and external validation, which may be considered the most important aspect of developing prediction models.^24,40 No studies performed internal validation, which can be problematic, as using the same population cohort and data set to develop the model and derive model performance leads to model overinflation and overfitting.^41,42 Internal validation techniques, such as cross-validation and bootstrapping, can be particularly useful for small data sets with multiple candidate predictors to capture overfitting of model performance.²⁴ A model developed by Kadmon et al³⁴ was validated in an external prospective cohort from the same hospital institution; however, discrepancies of regression coefficient between the 2 studies raise concerns of model redevelopment vs validation, which may be a source of bias for performance.³⁶ Robust external validation studies are crucial, as they are thought to provide the best insight into the prediction of a model and its use in other populations, hospital settings, and geographical locations.²⁴

Finally, the applicability of individual studies with the review question was assessed with PROBAST. Only 1 study by Stafler et al³⁶ had low overall concerns, and the rest of the studies were mostly rated to be unclear because of participant selection.^31-35 These studies excluded patients with low suspicion of FBA at the time of presentation and subsequently did not receive bronchoscopy. With this, these models may not be applicable for children who present with vague, noncontributory, or equivocal clinical findings of FBA.

A meta-analysis of all identified CPMs was performed, with the findings highlighting the overall potential of prediction models for the diagnosis of FBA. However, several factors limit the interpretation of these summary results to guide further model development and use in clinical practice. Quantitative synthesis of prediction models is challenging because of incomplete reporting of relevant summary statistics, often leading to many systematic reviews of CPMs refraining from performing such analyses.⁴³ These issues highlight the importance of adhering to guidelines, such as the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement, which provide recommendations for performing and reporting CPM studies.⁴⁴ For example, methods to restore missing information because of unreported performance measures are recommended to facilitate meta-analyses and were used in this review.^28,29,45-47 In addition, most of the included CPMs were distinct derivation studies, with varying predictors included in each model. Intermodel heterogeneity associated with study design and population differences, varying predictor effects across studies, and discrete statistical methods substantially limits the interpretation of model performance summary statistics.²⁹ Standardized comparisons of C statistics or AUCs with suggested methods (ie, DeLong test) are considered inappropriate when comparing models with different clinical predictors.⁴⁸ As a result, meta-analysis of prediction models may be more appropriate when summarizing validation studies.

This review also quantitively evaluated the ability of individual predictors of FBA in children. Witnessed choking, unilateral decreased breath sounds, focal hyperinflation or air trapping, and findings suspicious for a foreign object on chest radiography were significantly associated predictors selected for more than half of the prediction models. Meta-analysis of these individual variables revealed robust significant associations, corroborating their frequent use in current clinical practice.⁴⁹ However, their use as independent predictors is limited, as demonstrated by their individual discrimination values. Clinical variables from other sources of investigations have been suggested for use in diagnosing FBA. Computed tomography (CT) imaging and virtual bronchoscopy have been frequently reported to be effective and consistent for detecting airway foreign bodies.^50-52 The largest series to date, to our knowledge, of 69 patients who underwent CT evaluation demonstrated a sensitivity and specificity of 100% and 98%, respectively.⁵¹ Despite these findings, the use of CT imaging has not been widely adopted in clinical practice, highlighting the concerns of radiation exposure in children, need for sedation, and potential accessibility issues for volumetric CT scanners.^35,51 One of the models incorporated laboratory findings, with an elevated white blood cell count identified as a predictor for FBA in children.³² The role of blood and biochemical testing in FBA is unclear, with only a few studies reporting an inconsistent association between levels of white blood cell counts and inflammatory markers, such as C-reactive protein levels and erythrocyte sedimentation rate, with the presence of an airway foreign body.^53,54

Although use of the CHARMS checklist, the PROBAST framework (Cochrane Prognosis Group), and adherence to PRISMA guidelines contributed to the overall strength of the methods of this systematic review and meta-analysis, they revealed numerous study limitations. Quiz Ref IDQuantitative synthesis of the prediction models was limited because of a lack of validation studies of a specific CPM. With this, further studies that are involved in robust validation of models will be essential for translating such diagnostic tools in clinical practice. Furthermore, summary estimates of calibration performance were not incorporated, as these were not widely reported and information was not available for derivation. This highlights the importance of following guidelines, such as CHARMS and TRIPOD, to design, perform, and report prediction model studies to facilitate the development and improvements of such efforts.

Existing CPMs for FBA in children are at high ROB and have not been adequately validated. As a result, no current models can be recommended to guide clinical decision-making. Witnessed choking event, unilateral decreased breath sounds, air trapping, and radiographic findings that are suspicious for a foreign object can raise suspicion for a foreign body, but still require astute clinical care by an experienced clinician. Future studies aimed at developing a CPM for FBA that adheres to recognized standards for development and validation are required.

Accepted for Publication: May 26, 2021.

Published Online: July 15, 2021. doi:10.1001/jamaoto.2021.1548

Corresponding Author: Nikolaus E. Wolter, MD, MSc, Department of Otolaryngology–Head and Neck Surgery, Faculty of Medicine, University of Toronto, The Hospital for Sick Children, 555 University Ave, Room 6133, Burton Wing, Toronto, ON, Canada M5G 1X8 (nikolaus.wolter@sickkids.ca).

Author Contributions: Dr Lee had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Lee, Philteos, Levin, Propst, Wolter.

Acquisition, analysis, or interpretation of data: Lee, Philteos, Levin, Namavarian, Wolter.

Drafting of the manuscript: Lee, Philteos, Levin, Namavarian, Wolter.

Critical revision of the manuscript for important intellectual content: Lee, Levin, Propst, Wolter.

Statistical analysis: Lee, Philteos, Wolter.

Administrative, technical, or material support: Namavarian, Wolter.

Supervision: Propst, Wolter.

Conflict of Interest Disclosures: None reported.

Meeting Presentation: Abstract submitted for the virtual 2021 American Society of Pediatric Otolaryngology Annual Meeting; July 16, 2021.

Data Sharing Statement: The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Additional Contributions: We thank Erica Nekolaichuk, MA, MLIS, librarian at The Hospital for Sick Children, for her help with the development of the systematic review search protocol. She was not compensated for her contributions.