aReview articles, unpublished theses or dissertations, and articles that were unrelated to pediatric mHealth were removed.
bTwo unique studies were included in 1 article.
The size of the data marker represents weighting of the study in the analyses. SMD indicates standardized mean difference.
eMethods. PubMed Electronic Search Strategy
eTable 1. Studies Included in the Meta-analysis
eTable 2. Assigned Coding of Moderator Variables by mHealth Intervention Study
Customize your JAMA Network experience by selecting one or more topics from the list below.
Fedele DA, Cushing CC, Fritz A, Amaro CM, Ortega A. Mobile Health Interventions for Improving Health Outcomes in Youth: A Meta-analysis. JAMA Pediatr. 2017;171(5):461–469. doi:10.1001/jamapediatrics.2017.0042
How effective are mobile health interventions for improving health outcomes in youth 18 years or younger?
Thirty-seven studies evaluating the use of mobile health interventions in approximately 30 000 participants were included in the meta-analysis. Mobile health interventions had a small but significant positive effect on health outcomes in youth.
Mobile health interventions appear to be a viable health behavior change intervention modality for youth.
Mobile health interventions are increasingly popular in pediatrics; however, it is unclear how effective these interventions are in changing health outcomes.
To determine the effectiveness of mobile health interventions for improving health outcomes in youth 18 years or younger.
Studies published through November 30, 2016, were collected through PubMed, Cumulative Index to Nursing and Allied Health Literature, Educational Resources Information Center, and PsychINFO. Backward and forward literature searches were conducted on articles meeting study inclusion criteria. Search terms included telemedicine, eHealth, mobile health, mHealth, app, and mobile application.
Search results were limited to infants, children, adolescents, or young adults when possible. Studies were included if quantitative methods were used to evaluate an application of mobile intervention technology in a primary or secondary capacity to promote or modify health behavior in youth 18 years or younger. Studies were excluded if the article was an unpublished dissertation or thesis, the mean age of participants was older than 18 years, the study did not assess a health behavior and disease outcome, or the article did not include sufficient statistics. Inclusion and exclusion criteria were applied by 2 independent coders with 20% overlap. Of 9773 unique articles, 36 articles (containing 37 unique studies with a total of 29 822 participants) met the inclusion criteria.
Data Extraction and Synthesis
Of 9773 unique articles, 36 articles (containing 37 unique studies) with a total of 29 822 participants met the inclusion criteria. Effect sizes were calculated from statistical tests that could be converted to standardized mean differences. All aggregate effect sizes and moderator variables were tested using random-effects models.
Main Outcomes and Measures
Change in health behavior or disease control.
A total of 29 822 participants were included in the studies. In studies that reported sex, the total number of females was 11 226 (53.2%). Of those reporting age, the average was 11.35 years. The random effects aggregate effect size of mobile health interventions was significant (n = 37; Cohen d = 0.22; 95% CI, 0.14-0.29). The random effects model indicated that providing mobile health intervention to a caregiver increased the strength of the intervention effect. Studies that involved caregivers in the intervention produced effect sizes (n = 16; Cohen d = 0.28; 95% CI, 0.18-0.39) larger than those that did not include caregivers (n = 21; Cohen d = 0.13; 95% CI, 0.02-0.25). Other coded variables did not moderate study effect size.
Conclusions and Relevance
Mobile health interventions appear to be a viable health behavior change intervention modality for youth. Given the ubiquity of mobile phones, mobile health interventions offer promise in improving public health.
Mobile phones are commonly used by health professionals as a platform to deploy interventions to elicit health behavior change and improve health outcomes.1 In the United States, more than 75% of adults 49 years or younger own a smartphone and an estimated 73% of adolescents own or have access to a smartphone.2,3 Moreover, on an average day, adolescents spend more than 2.5 hours using a smartphone,4 with their device always in close proximity. The ubiquitous nature of mobile phone use coupled with continual technological advances have given rise to a proliferation in mobile health (mHealth) interventions in which mobile devices target a range of health promotion and disease management foci.5,6
There are several intuitive factors that have led to the expeditious growth of mHealth interventions. First, mHealth programs can collect dynamic health-related data and deliver intervention content to individuals in their natural environment, outside of a clinical encounter, at key times that have a higher likelihood of modifying behavior. Second, the kinds of high-throughput data that are available from sensors and smartphones, including activity trackers and digital pillboxes, increase the ability for researchers to tailor interventions to participants in real time to encourage engagement in positive health behaviors (eg, prompt exercise when an activity tracker has been stationary). Third, the app marketplace has quickly become saturated with a wide range of mHealth-related platforms because businesses have recognized the value of the wealth of data (eg, data on populations can be leveraged for health management and marketing) that these apps can provide.
To date, however, the mHealth ecosystem has not met its potential to disseminate evidence-based interventions focused on health promotion.7 A critical initial step toward facilitating the maturation of the mHealth ecosystem is for the scientific community to increase our understanding of what existing mHealth interventions are most effective and elucidate ways that the interventions can be improved. Understanding the aggregate effectiveness of mHealth interventions in the empirical literature may help to accelerate the rate of scientific discovery and then improve the incorporation of evidence-based practices in the marketplace.
Reviews of the effectiveness of mHealth interventions have predominantly focused on adult populations.5,8,9 These reviews suggest that mHealth interventions appear to be promising for a number of health-related areas, including chronic illness management, adherence, and appointment attendance. A small but growing number of studies have used mHealth technologies in pediatric populations. These mHealth interventions have targeted a range of outcomes (eg, improving diabetes control and physical activity promotion) and have produced mixed findings.10,11 To our knowledge, the only systematic review of mHealth intervention effectiveness focused exclusively on pediatric obesity.12 Given the burgeoning interest in mHealth and the increasing number of pediatric-focused mHealth interventions in recent years, it is an ideal time to more broadly assess how effective mHealth interventions are at eliciting meaningful change in health outcomes. The objectives of the present meta-analysis were to (1) determine the effectiveness of mHealth interventions at improving health outcomes in youth 18 years or younger, (2) assess study- and sample-level moderating factors that may be critical drivers of mHealth intervention effectiveness, and (3) characterize the risk of bias of the extant pediatric mHealth intervention literature.
PubMed/MEDLINE, Cumulative Index to Nursing and Allied Health Literature, Educational Resources Information Center, and PsycINFO databases were searched on November 30, 2016, with the assistance of a research librarian. Search terms included telemedicine, eHealth, mobile health, mHealth, app, and mobile application. Search results were limited to infants, children, adolescents, or young adults when possible (eMethods in the Supplement). Only articles published in English were included.
The initial search produced a total of 11 280 articles. Two additional articles13,14 from a recent review of mHealth interventions for pediatric obesity12 were identified as relevant to the present study. After the removal of duplicate studies, the titles and abstracts of 9773 unique articles were screened by 2 authors (D.A.F. and A.O.). Review articles, unpublished theses or dissertations, and articles that were unrelated to pediatric mHealth were removed. Following this level of screening, 318 full-text articles were evaluated. Articles were included if quantitative methods were used to evaluate the use of mobile intervention technology in a primary or secondary capacity to promote or modify health behavior (eg, physical activity promotion) in youth 18 years or younger. Quiz Ref IDExclusion criteria were applied in the following order: (1) the article did not address the use of mobile intervention technology in a primary or secondary capacity to promote or modify health behavior, (2) the mean age of the participants was older than18 years, (3) the article did not assess a health behavior and disease outcome, or (4) the article did not include sufficient statistics to report effect sizes.
A random 20% of the full-text articles (n = 64) were assessed for inclusion and exclusion criteria by 2 independent coders (D.A.F. and C.C.C.) to evaluate interrater reliability. There was 100% agreement between the raters when assessing whether an article should be included or excluded in a dichotomous fashion. Agreement declined to 98% with regard to the specific reason for exclusion because each criterion was applied in order, making it possible that raters could exclude studies for different reasons. Twenty-two articles met the inclusion criteria. Backward and forward reference searches were conducted, resulting in the identification of 14 additional articles meeting the inclusion criteria. The final sample of 36 articles included 37 unique studies (Figure 1).
Any statistical information that could be converted into a standardized mean difference was used in the analysis. These included P values, sample sizes, t tests, F tests, odds ratios, and nonparametric tests. Study effect sizes were calculated using Comprehensive Meta-Analysis, version, 2.2.064 (Biostat Inc).
Two authors (C.C.C. and C.M.A.) and a trained research assistant separately categorized all articles and extracted data using a standardized codebook. Disagreements were resolved through discussion until there was 100% agreement.
Only variables measuring a health behavior or disease functioning associated with a health behavior were coded. For instance, outcomes could include laboratory values (eg, hemoglobin A1c level) or behaviors (eg, dietary intake). A total of 93 effect sizes were calculated from 37 studies. A given study could contribute more than 1 effect size as described below in the statistical approach to aggregation.
Because study procedures allowed for the inclusion of multiple outcomes, it was necessary to aggregate effect sizes at the study level before computing an overall aggregate effect size. Including multiple outcomes for each study provides a more conservative estimate of the effect of the individual study and serves as a weak protection against “P hacking” (ie, conducting analyses on multiple dependent variables in the hopes that 1 will be significant). After aggregating effect sizes at the study level, a standardized mean difference was computed across each study.
The Cochrane Collaboration risk of bias tool15 was used to categorize risk of bias in (1) random sequence generation, (2) blinding of participants and personnel, (3) incomplete outcome data, and (4) selective reporting. Categorical ratings of high, unclear, or low were assigned for each domain within the 37 individual studies. Specifically, a high risk of bias rating was given if the report made it clear that the method used introduced the potential that the findings could be biased. An unclear rating was assigned if the report made it unclear whether the study findings were likely to be biased. The risk of bias was rated as low if it was clear from the method and reporting that the issues assessed could not have biased the study.
Several moderator variables were coded. Intervention length was coded as a statistically continuous moderator. A categorical moderator indicating whether caregivers were active recipients of the mHealth intervention was coded. Studies were also coded for the use of only text message interventions in comparison with interventions that included text messaging in addition to other components (eg, smartphone apps or online surveys). Another moderator examined whether studies used a theoretical framework for the intervention. Specifically, authors needed to indicate in the article that the mHealth intervention components were guided by a theoretical model (eg, health belief model). The type of outcome (ie, occurring once or behaviors that were repeated) was also coded. A separate categorical moderator variable was created for studies in which the mean age of the participants fell in the preadolescence (<13 years) or adolescence (13-18 years) range (eg, received a 0 if mean age was younger and/or if only the caregiver received the intervention). Finally, whether or not the study was developed with patient stakeholder input or used intent-to-treat analysis were coded as moderators.
Studies reporting multiple outcomes were aggregated as a single effect size in the analysis. This approach helps to ensure accurate SEs as well as to reduce bias that can occur by treating studies as independent when data were gathered from the same sample.16 Effect sizes were expressed as a standardized mean difference (ie, Cohen d). Cohen17 provided general guidelines for interpreting the magnitude of study effect sizes noting that 0.20 to 0.49 qualifies as a small effect, 0.50 to 0.79 constitutes a medium effect, and above 0.80 is a large effect. Potential moderator variables were tested using the random effects analog to the analysis of variance technique described by Lipsey and Wilson.18
Of 36 unique articles, the final sample of 37 studies included a total of 29 822 participants. In studies that reported sex, the total number of females was 11 226 (53.2%) and the average age was 11.35 years. Other relevant demographic information, study characteristics, and mHealth components for all included studies are provided in eTable 1 in the Supplement. Reporting of demographic information was suboptimal across the study sample. Fifteen studies (40.5%) did not report mean participant age and 14 studies (37.8%) did not report race/ethnicity. Of participants with race/ethnicity reported, 4887 (43.7%) were Hispanic/Latino, 1616 (14.4%) were African American, and 341 (3.15%) were white; 4354 (38.8%) of the participants were of another, unknown, or preferred-not-to-state category of race/ethnicity.
Quiz Ref IDA large proportion of studies was rated as having a high risk of bias regarding the blinding of participants and personnel (21 [56.8%]) and attrition (16 [43.2%]). Risk of selection bias was also relatively high (14 [37.8%]); however, 11 (29.7%) studies did not provide information to determine risk. All but 1 study19 was coded as having a low risk of bias for selective reporting. The Table provides the risk of bias at the individual study level for all articles.
The aggregate effect size of mHealth interventions compared with the controls was small, but significant (n = 37; Cohen d = 0.22; 95% CI, 0.14-0.29) (Figure 2). However, significant heterogeneity was observed in the effect sizes (Q = 106.40; P < .001). The amount of systematic variability in study effect size was substantial (I2 = 66.17), suggesting that moderators of effect size could be used to better understand drivers of treatment effectiveness.
Assigned coding of assessed moderator variables for each study is presented in eTable 2 in the Supplement. Caregiver involvement in the intervention emerged as a significant moderator in the random effects analysis (P = .05). Specifically, studies that involved caregivers in the intervention produced larger effect sizes (n = 16; Cohen d = 0.28; 95% CI, 0.18-0.39) compared with those that did not (n = 21; Cohen d = 0.13; 95% CI, 0.02-0.25). Moreover, the variability remaining in the study effect sizes was trivial beyond what could be explained by the moderator. Use of theory to guide intervention development (Q = 77; P = 0.38), use of text messaging (Q = 0.46; P = .50), preadolescent vs adolescent as the intervention target (Q = 3.03; P = .08), recurrence of the outcome (Q = 0.68; P = .41), length of study (Q = 0.49; P = .48), use of intent-to-treat analysis (Q = 0.62; P = .43), and involvement of stakeholders in intervention development (Q = 0.03; P = .84) did not moderate study effect sizes.
A number of factors may lead to studies not appearing in the literature. Some unpublished studies are likely to contain null findings. Therefore, it is important to understand the number of null results that it would take to overturn the findings in the present study. Based on the effect size observed for the omnibus test, 634 null findings would be necessary to overturn the findings presented herein.51
The primary objective of the present study was to determine the effectiveness of mHealth interventions in improving health-related outcomes in youth 18 years or younger. Results indicate that mHealth interventions can be effective in eliciting meaningful improvements in pediatric health behavior and associated pediatric health outcomes. The small but significant aggregate effect size of mHealth interventions (Cohen d = 0.22) aligns well with the findings from previous meta-analyses of adherence promotion52 and eHealth53 health behavior change interventions. These findings suggest that mHealth interventions may be a viable modality for health care professionals to effect health-related changes in pediatric populations.
Quiz Ref IDModerator analyses revealed that mHealth interventions with caregiver involvement produced larger changes in health outcomes, on average, than did those that exclusively targeted youth. Youth health behaviors and outcomes are multidetermined and influenced by a combination of factors within the individual, family, community, and health care system domains.54 Among these factors, caregiver involvement has been shown to be an integral component in both health promotion55,56 and chronic illness management in youth.57 Results from the present study indicate that mHealth interventions that include caregiver involvement may have an advantage in producing positive health outcome changes in youth. However, there are 2 important points regarding this finding that should be considered. First, 9 of the 16 mHealth interventions that included caregivers targeted children 5 years or younger—a developmental period during which caregivers are primarily responsible for their child’s health. Second, a number of these interventions focused on immunizations (68.8%) and were designed so that caregivers were the predominant recipient of mHealth intervention content (eg, caregiver received text message reminders for their child’s immunization). Given the early stage of the mHealth literature, it will be necessary to replicate and extend this finding to determine whether the extent to which caregivers are involved in a mHealth intervention is a driver of effectiveness.
Specifically, it remains an open question whether parental involvement increases the effect of an mHealth intervention beyond the technology alone when the child or adolescent is the predominant recipient of the intervention content. Of particular importance will be studies that distinguish the effect of caregiver involvement in adolescence against early childhood since developmental stage is likely to be an important moderator. These findings fit well with a recent meta-analysis of health promotion interventions for youth reporting that interventions that cut across multiple domains (eg, family and community) were superior in eliciting health behavior change.58 Future research is needed to determine whether mHealth interventions that incorporate additional domains beyond caregiver involvement may produce greater changes in health outcomes.
Quiz Ref IDUse of theory to guide intervention development was not a moderator of mHealth intervention effectiveness. This finding runs counter to previous research indicating that interventions that incorporate health behavior change theory to guide development are often superior in web-based studies.1 In anticipation of a modest number of published mHealth intervention studies, we coded studies as using theory to guide intervention development if the authors broadly mentioned a theoretical framework during study development. This inclusive coding scheme may have attenuated study findings given that the degree of incorporation of theory into interventions can modulate effectiveness.1 Despite our coding scheme, most studies (n = 24) were coded as not using a guiding theoretical model. There have been calls for the incorporation of health behavior change theory into pediatric behavioral interventions to increase optimization and advance our understanding of which theoretical domains may be integral to effectiveness.59 Additional assessment of the effectiveness of mHealth interventions is needed as future studies incorporate established theoretical models of health behavior change.
Quiz Ref IDSeveral other coded demographics and study characteristics did not moderate the effectiveness of mHealth interventions, including youth age, use of text messaging, type of health outcome (recurring vs nonrecurring), length of study, and patient stakeholder involvement in intervention development. These findings may suggest that mHealth interventions can be effective for a wide range of youth and for a variety of pediatric health outcomes. However, these conditional effects should be interpreted with caution in light of a rapidly developing mHealth intervention literature base. Subgroup sample sizes across moderator categories were sometimes small, thereby limiting the power to assess conditional differences in effectiveness. Reassessment of these moderators is needed as mHealth interventions continue to move beyond text messaging and use differing study designs.
The findings of the present meta-analysis should be interpreted in light of the quality of the pediatric mHealth intervention literature. The use of randomized controlled trial designs (n = 24) and intent-to-treat analyses (n = 19) was relatively modest. Furthermore, consistent with previous findings in the adult literature,5,9 we identified a risk of bias in some areas that is notable given the small but meaningful aggregate effect size of pediatric mHealth interventions. Our ratings revealed that the most frequent risks of bias among the reviewed studies were potential failure to blind participants and personnel, problems with attrition, and potential biased allocation of participants to treatment groups. We found infrequent instances of selective reporting. The use of less-stringent research designs (eg, pre-post) and the frequency of bias in the extant pediatric mHealth literature are limitations and introduce a degree of uncertainty regarding the efficacy of mHealth interventions and the possibility that the effect size will change as more studies are conducted.
These findings reflect a clear need for reducing bias in subsequent mHealth studies. First, participants should be randomized to reduce selection bias. Second, blinding of participants and study staff to intervention conditions should occur whenever possible. Although double-blinding is typically impractical in face-to-face behavioral interventions, a unique benefit of mHealth is the possibility to develop and deploy attention control conditions (eg, sham text messages about general health information vs messages focused on health behavior change techniques). Third, to reduce bias related to attrition, participant flow throughout a trial should be clearly reported and the use of intent-to-treat analyses is encouraged. Fourth, given that only 37.8% of the studies reported on the racial and ethnic composition of the sample, assessment and transparent reporting of demographic characteristics are imperative in future mHealth interventions. Implementation of these recommendations and thorough reporting of methodologic processes would improve confidence in the reported effects of mHealth interventions on pediatric health outcomes.
Despite the strengths of the present study, there are several limitations. Due to the modestly sized pediatric mHealth literature, we were not able to examine the effectiveness of mHealth interventions for specific disease groups or health topics, health outcomes, or other intervention components (eg, personalized feedback). These are important issues to examine as the mHealth literature grows. mHealth interventions that did not report on changes in at least 1 youth health outcome were excluded. Although this exclusion resulted in a more stringent test of the effectiveness of mHealth interventions, studies that focused solely on changes in knowledge, self-efficacy, motivation, and other related outcomes were not included.
Inclusive definitions were used in the coding of moderator variables for this initial meta-analysis given the nascent state of pediatric mHealth intervention literature. As previously noted, broader coding may have reduced the ability to find meaningful associations between moderators and mHealth intervention effectiveness. Furthermore, some of the moderator categories included a small number of studies (eg, nonrecurring outcomes). Moderation effects may shift with additional studies and should therefore be interpreted with caution. mHealth intervention studies that had a sample of participants whose mean age was older than 18 years were excluded from the meta-analysis. In doing so, we acknowledge that studies that spanned a broader age range or focused on young adults (eg, 18-25 years) were excluded from analyses. Finally, the meta-analysis did not focus on the effectiveness of mHealth interventions to improve mental health outcomes; this is an important area for future investigation.
To our knowledge, the present study is the first to examine the aggregate effects of mHealth interventions on improving health outcomes in youth. The findings indicate that mHealth interventions are a promising and potentially effective route for pediatric health care professionals to use with patients and their caregivers. Given the ubiquity of mobile phone use and the willingness of youth to use their mobile devices for health-related activities, mHealth interventions appear poised to be a viable health behavior change intervention modality. To extend the promise of pediatric mHealth interventions, future work is needed to explicitly incorporate and test the effect of competing and complementary theoretical mechanisms of behavior change.59 Moreover, additional studies are needed to determine the optimal positioning of mHealth interventions within a larger ecologic context (ie, whether health systems, families, or schools should be included and the appropriate developmental levels). To increase confidence in the robustness of the findings, the full extent of methodologic rigor should be incorporated into mHealth studies. As the research community executes the agenda outlined herein, future work will be better positioned to provide policy recommendations for hospitals and governments seeking to use mHealth approaches to population-level pediatric health management.
Corresponding Author: David A. Fedele, PhD, Department of Clinical & Health Psychology, University of Florida, 101 S Newell Dr, PO Box 100165, Gainesville, FL 32610 (firstname.lastname@example.org).
Accepted for Publication: January 4, 2017.
Correction: This article was corrected on May 1, 2017, to remove a duplicate sentence from the Results section; this article was corrected on January 16, 2018, to fix reference errors in Figure 2.
Published Online: March 20, 2017. doi:10.1001/jamapediatrics.2017.0042
Author Contributions: Dr Fedele had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Fedele, Cushing.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Fedele, Cushing, Fritz, Amaro.
Critical revision of the manuscript for important intellectual content: Fedele, Cushing, Amaro, Ortega.
Statistical analysis: Fedele, Cushing, Amaro.
Administrative, technical, or material support: Cushing, Ortega.
Supervision: Fedele, Cushing.
Conflict of Interest Disclosures: None disclosed.