Effectiveness of the Extension for Community Health Outcomes Model as Applied to Primary Care for Autism: A Partial Stepped-Wedge Randomized Clinical Trial

Key Points

Question  Is the Extension for Community Health Outcomes (ECHO) telementoring model effective in improving clinical practice among primary care clinicians caring for children with autism?

Findings  In this large-scale, multisite partial stepped-wedge study of the ECHO model as applied to autism, significant changes in autism screening or treatment of comorbidities were not demonstrated. However, primary care clinicians demonstrated significant improvements in knowledge and self-efficacy immediately following and 3 months after completing the ECHO program.

Meaning  The results provide support for the ECHO model in improving clinician knowledge and confidence in caring for patients with autism in primary care practices, but measurable changes in practice were not observed.

Importance  The Extension for Community Health Outcomes (ECHO) model is a widely adopted technology-based model for training primary care physicians and practitioners (PCPs) to care for patients with complex conditions. Despite its popularity, to our knowledge, direct effects of ECHO on clinical practice have not been tested in a large-scale study.

Objective  To test the effectiveness of the ECHO model as applied to primary care for autism and whether it resulted in improved clinical practice, knowledge, and self-efficacy regarding autism screening and comorbidity management.

Design, Setting, and Participants  Primary care physicians and practitioners were recruited to participate in a 6-month ECHO Autism program delivered by 1 of 10 academic medical center sites. A sequential, staggered rollout of ECHO Autism was delivered to 5 cohorts of participants (15 per site; 2 sites per cohort). Sites were randomized after recruitment to cohort/start time. Cohorts launched every 3 months. The ECHO Autism program used videoconferencing technology to connect community-based PCPs with interdisciplinary expert teams at academic medical centers. There were 148 participants (PCPs [family practice physicians, pediatricians, nurse practitioners, and physician assistants] providing outpatient services to underserved children) studied between December 2016 and November 2018.

Interventions  The 6-month ECHO Autism program included twelve 2-hour sessions connecting PCP participants with an interdisciplinary expert team. Sessions included didactics, case-based learning, guided practice, and discussion.

Main Outcomes and Measures  Coprimary outcomes were autism screening practices and comorbidity management (assessed by medical record review). Secondary outcomes were knowledge (assessed by direct testing) and self-efficacy (assessed by self-report survey). Assessments were conducted at baseline, mid-ECHO, post-ECHO, and follow-up (3 months after ECHO).

Results  Ten sites were randomized to 1 of 5 cohorts. Participants were 82% female (n = 108), 76% white (n = 100), and 6% Hispanic or Latino (n = 8); the median age was 46 years (interquartile range, 37-55 years). Significant changes in autism screening and treatment of comorbidities in children with autism were not observed. Participants demonstrated significant improvements in knowledge (9%; 95% CI, 4-13; P < .001) and self-efficacy (29%; 95% CI, 25-32; P < .001).

Conclusions and Relevance  The ECHO model was developed to increase access to high-quality health care for underserved patients with complex conditions. Study results provide support for the model in improving clinician knowledge and confidence but little support for achieving practice change.

Trial Registration  ClinicalTrials.gov Identifier: NCT03677089

Despite rapid advancements in medical knowledge, patients with complex conditions from rural and underserved areas face significant barriers in accessing specialty care.¹ The Extension for Community Health Outcomes (ECHO) model was developed to address this gap by infusing specialty knowledge into local primary care practices.² Through ECHO, groups of community-based primary care physicians and practitioners (PCPs) connect with a team of experts using secure videoconferencing and receive best-practice knowledge through didactics, case-based learning, comanagement, and a virtual community of practice.^3,4 Since 2003, the framework has been adopted by more than 500 programs globally. Originally developed to improve care of hepatitis C, the model has now been applied to more than 100 complex conditions (ranging from diabetes to addiction).^5,6 Despite widespread adoption of the model, evidence of effectiveness remains preliminary. Evidence to date derives from tests of single-project ECHO clinics rather than from more robust multisite designs and primarily from self-reported outcomes rather than direct measures of practice change, as was extensively summarized for Congress.⁵ More rigorous tests of the model are needed to assess its effectiveness in improving the quality of health care for patients with complex conditions. Autism represents one such increasingly common and complex condition.

Autism is a neurodevelopmental disorder associated with high rates of medical and psychiatric comorbidities.^7-9 Increased autism prevalence over the past 2 decades¹⁰ has resulted in health care demands that greatly exceed the capacity of specialty centers.¹¹ Children with autism experience greater unmet health care needs, higher health care costs, and worse access to medical homes and specialty care than children with other special health care needs.^12-16 Children with autism from underserved populations face disproportionate health care barriers owing to clinician shortages, high costs of services, and language or cultural barriers.^17-19 Poor access to care results in diagnostic and intervention delays, unmanaged comorbid conditions, poor health, and reduced quality of life for children with autism.^20-22

Community-based PCPs are ideally positioned to address these disparities. Primary care, by definition, consists of “integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and community.”²³ Unfortunately, many PCPs lack training in autism and feel unprepared to provide best-practice care.^24,25 Although the American Academy of Pediatrics issued clear recommendations for developmental surveillance and screening for autism,^26,27 physician compliance has been strikingly poor.^28-31 Many PCPs lack knowledge about autism^20,28,32 and lack confidence in managing common co-occurring conditions in children with autism.³³ Equipping PCPs with the skills to provide effective community-based screening and management of autism has the potential to enhance outcomes for children with autism.

A 2017 pilot study^34,35 was conducted to examine the application of the ECHO model to enhance primary care for autism. The study consisted of a small sample of 14 PCPs who attended 12 biweekly ECHO Autism clinics focused on best-practice medical care for children with autism. The results were promising, showing significant improvements in participant self-efficacy and self-reported practice change. However, objective measures of improvement were not collected.³⁴ Thus, a more rigorous examination of the model is needed.

This study was conducted to test the effectiveness of ECHO Autism in a large and geographically diverse randomized sample using objective measures of improvement in clinical practice, while delivering a potentially important educational intervention to all participants. The specific hypotheses were that participating in ECHO Autism would result in significant improvements among PCPs in (1) clinical practice, (2) knowledge, and (3) self-efficacy regarding autism screening and management of comorbidities.

Ten academic medical centers (eAppendix in Supplement 1) participating in the Autism Treatment Network and the Autism Intervention Research Network on Physical Health served as individual ECHO Autism clinic sites (each training a target sample of 15 PCP participants). A sequential, staggered rollout of the ECHO Autism program was studied in 5 cohorts of participants (2 sites every 3 months) between December 2016 and November 2018. Sites were randomized to startup order after recruitment into study. This design allowed us to distribute limited resources to assess the effect of the ECHO Autism program across a broad range of sites. Assessments were completed at 4 points separated by 3-month intervals: Baseline/pre-ECHO, mid-ECHO, post-ECHO, and a post-ECHO follow-up to assess persistence of effect. Figure 1 shows a CONSORT diagram of participation in the study. Figure 2 describes the timeline for data collection. The study was registered at ClinicalTrials.gov (NCT03677089), and the formal trial protocols are in Supplement 2.

Each of the 10 ECHO Autism sites (9 US and 1 Canadian site) planned to recruit a sample of 15 PCPs from their respective geographic region. Inclusion criteria were current practice as a PCP (ie, family practice physician, pediatrician, nurse practitioner, or physician assistant), providing outpatient services to children, and providing care to underserved populations (at least 50% of patients were underserved). Only 1 PCP from a single-practice location was allowed to participate in the study. Recruitment methods included professional distribution lists, professional meetings, and face-to-face and word-of-mouth recruitment. Participants received no-cost continuing medical education credit for attendance at each ECHO Autism clinic, and modest financial incentives were provided for completion of study measures at each assessment point. Informed consent followed procedures dictated by the institutional review board at the specific site; generally verbal consent was considered adequate because the study involved no more than minimal risk. The study was approved by the institutional review board at each site and at the Network Coordinating Center.

ECHO Autism expert teams were formed at each of the 10 participating sites, consisting of at least 5 autism specialists (physician/autism medical specialist, psychologist, family resource specialist, dietician, and parent expert). One site included an additional Family Resource Specialist because participants were practicing in 2 different states. Teams were trained to fidelity in the ECHO model and in the ECHO Autism curriculum.³⁴ Training included in-person training and orientation at the ECHO Institute, regular learning network videoconference meetings, attendance and observation of at least 3 ECHO Autism clinics facilitated by the leadership team, and facilitation of at least 2 mock clinics with live coaching and feedback prior to clinic launch.

Each ECHO autism clinic site provided twice-monthly 2-hour ECHO Autism clinics during a 6-month period. Clinics were conducted using high-quality secure Health Insurance Portability and Accountability Act–compliant videoconferencing technology (Zoom), enabling screen-sharing (for document viewing) and real-time interactions among participants and ECHO Autism expert team members. All clinics followed the established ECHO Autism curriculum, which included a standard set of didactic presentation slides and content, case presentation forms, and materials.³⁴ Each clinic consisted of a brief 15-minute didactic presentation, 2 PCP-generated case presentations, and collaborative discussion among experts and participants. Didactic content focused on guidelines and algorithms for evidence-based best-practice medical care for children with autism, with particular emphasis on autism screening and identification and on medical management of comorbid conditions^27,36-41 (didactic materials are available on request). The intent of the program was for participants to develop new clinical skills through guided practice and collaborative case-based learning while maintaining responsibility for care of their patients. Implementation fidelity was assessed by study team members at 2 randomly selected sessions per site using a form developed by the University of New Mexico Project ECHO Team to assess fidelity; adequate fidelity was defined as “agree” or “strongly agree” for at least 80% of the relevant items.

Unconstrained random allocation was generated in SAS (SAS Institute Inc) by the senior study statistician after sites were enrolled by principal investigator. Sites were allocated to a cluster based on the random allocation.

Some operational problems occurred during the study. Owing to scheduling issues, 2 sites started 1 month after the scheduled start of their cohort. This was ignored for the analysis. Because Canadian guidelines for screening for autism were not consistent with US guidelines, autism screening for the Canadian site was excluded from the primary analysis. Full details of all operational problems are provided in section 12 of the statistical analysis plan in Supplement 3.

Clinical practice/behavior was assessed through medical record review conducted by study staff in each PCP practice. Appropriate autism screening was assessed at 18-month and 24-month well-child visits for the 30 days prior to the scheduled date of medical record review. Medical record reviews were done at baseline (before the start of ECHO, but on rare occasions after the first but before the second ECHO session), at mid-ECHO (between the sixth and seventh ECHO session), post-ECHO (within 30 days after the end of the ECHO program), and for long-term effect (between 3 and 4 months after the end of the ECHO program). Appropriate autism screening was defined as administering an autism screening tool at 18 and 24 months, as recommended by the American Academy of Pediatrics²⁷ for the 9 US sites (all other measures included both US and Canadian sites). Treatment of 4 common autism comorbidities (constipation, sleep disturbance, behavior problems, and anxiety) was assessed for all visits for children with autism seen within the 60 days prior to the scheduled date of medical record review. We considered any treatment addressing a recognized problem (from a predefined list of appropriate treatment options for each condition, which are shown in eTable 1 in Supplement 1) as appropriately addressing the problem. Secondary outcome measures (knowledge, self-efficacy, and barriers to care) are described further in the trial protocol in Supplement 2. Self-reported demographic information (including age, sex, and race/ethnicity) was collected to characterize the sample (Table 1).

Given the complexity of the proposed analysis, power calculations were based on simulations. The data generation process allowed for random effects for site, PCP within site, and nominal period. There was no time trend in the data, although a potential time trend as a fixed effect was included in the model. Simulations were done for 10 randomly selected seeds (from several different websites and different random number tables) at 1000 simulations per seed. The data-generating process allowed for approximately a 50% intraclass correlation for the PCP within-group effect, reflecting the possibility that the effect of ECHO would be correlated within each site, even with good fidelity to the intervention program. Simulations allowed for varying numbers of patients per PCP practice.

If PCPs on average had 15 patients with well-child visits seen in the past month, we would have more than 90% power to detect an increase from 50% to 60% in autism screening (α = .025; all tests were 2-sided). If PCPs on average had seen 5 patients with autism in the last 60 days, we would have more than 90% power to detect an increase from 50% to 65% in appropriate comorbidity management (α = .025; all tests were 2-sided).

Data are summarized as median (interquartile range [IQR]) or counts. The primary analysis used a generalized linear mixed model with data from baseline, mid-ECHO, post-ECHO, and 9-month follow-up to assess program effect, with fixed effects for period (time since start of study) and time in study (categorical time since PCP participant entered study) including site-specific random intercept for each PCP participant. We accounted for repeated measures within PCP participants with unstructured covariance. This is a modified version of the standard model for a stepped-wedge design put forward by Hussey and Hughes.⁴² Model estimates for baseline, month 3, month 6, and month 9 are presented are presented with the primary analysis test of change over 6 months of study. Additional results test changes from month 6 to month 9 to assess any regression after ECHO is completed. Results are reported for a modified intent-to-treat population that includes all participants who consented and provided basic demographics at baseline. Although we initially planned an analysis restricted to participants who more closely followed protocol, we agreed with a reviewer to present mITT analyses because this was better suited to the study design; results were almost identical between the 2 analyses. The SAP called for baseline data to be imputed if more than 5% of data were missing for a primary outcome. Because less than 5% of baseline data were missing, no baseline data were imputed. Missing follow-up data had the baseline carried forward as a conservative estimate (5 PCPs had autism screening follow-up information imputed, and 15 PCPs had follow-up information imputed for the proportion of comorbidities addressed outcome). Because we had 2 coprimary end points, we used P < .025 to determine statistical significance for proportion screened appropriately and proportion of comorbidities addressed appropriately at 6 months; otherwise P < .05 was considered statistically significant. All tests were 2-sided.

The study team received training and consultation from ECHO Institute members on ECHO implementation. Members of the ECHO Institute were not involved in study design, analyses, interpretation of results, or decision to publish.

A total of 148 participants were enrolled across the 10 ECHO Autism clinic sites, ranging from 11 to 22 per site. See Table 1 for demographic data by recruitment cohort. Of the 129 participants (87%) who attended at least 1 session, the median attendance was 10 of 12 sessions (interquartile range [IQR], 8-11). Regarding implementation, all sessions assessed during random fidelity checks demonstrated fidelity to the ECHO model. Median fidelity at individual clinics was 96% (range, 82%-100%).

Some demographic information was provided by 132 participants (89%), and baseline medical record review was completed for 118 participants (80%). The number of medical records available for review at each point varied across participants and visit type. For the baseline medical record review window, participants had a median of 5 well-child visits (IQR, 2-9) reviewed for autism screening (18-month or 24-month visits) and 3 visits (IQR, 1-6) with children with autism. Of the 118 participants having a medical record review attempted at baseline, 25 had no relevant well-child visits within the 30-day review period, 18 had no visits with a child with autism within the 60-day review period, and 24 provided a medical record for a child with autism but did not identify any of the listed problems. Baseline outcome measures by cohort are presented in eTable 2 in Supplement 1.

Model-based estimates of the outcomes are presented in Table 2, while effect estimates and statistical significance are shown in Table 3. Our primary analysis models estimated a small 6-month increase in autism screening rates from 65% (95% CI, 40%-84%) to 71% (95% CI, 52%-84%) (P = .63) and a small 6-month decrease in mean percentage of comorbidities addressed from 81% (95% CI, 67%-90%) to 77% (95% CI, 66%-86%) (P = .59). There was substantial improvement in knowledge (9% increase; 95% CI, 4%-13%; P < .001) and overall self-efficacy (29% increase; 95% CI, 25%-32%; P < .001) and a substantial decrease in the number of barriers identified by participants (2.2 decrease; 95% CI, −2.7 to −1.7; P < .001) (Table 3).

The significant effects seen in knowledge, overall self-efficacy, and barriers identified did not subside by month 9, and there were no significant differences observed from the 6-month post-ECHO visit to the 9-month follow-up visit (Table 3).

This study was designed to evaluate the effectiveness of a widely adopted technology-based continuing education model (project ECHO) in improving direct clinical care. In what is to our knowledge the first large-scale prospective multisite study of this model, we directly tested the effects of ECHO as applied to primary care for autism. Contrary to predictions, participants did not demonstrate statistically significant improvements in either autism screening or in treatment of comorbidities in children with autism after participation in the 6-month ECHO autism program. However, consistent with previous studies,⁵ participants did demonstrate significant increases in both knowledge and self-efficacy as well as significant reductions in perceived barriers to caring for children with autism in their practices.

The ECHO model was developed to increase access to high-quality health care for underserved patients with complex conditions. The results of this study do not provide strong support for the model in achieving direct practice change among PCPs, at least as it relates to the autism screening and comorbidity management. However, marked improvements in clinician knowledge and confidence in treating children with autism were observed. Although less direct than the specific practice behaviors we assessed through medical record review, these outcomes may ultimately enhance overall care and patient-clinician interactions for a population at high risk for poor health care experiences and outcomes.^11-15

This study benefitted from a large, multisite, and geographically diverse sample of PCPs. Additional strengths included randomization at the level of cohort, staggered implementation of the program across cohorts, and tests of effectiveness under baseline, intervention, and follow-up conditions. There are also several limitations that should be noted. Participants in this study had some motivation to improve their care of children with autism and may not be representative of the larger population of PCPs. However, their screening rates at baseline were generally consistent with 2016/2017 clinician-reported estimates,^43,44 suggesting that they did not differ substantially from their peers with regard to screening practices prior to participation in the program.

The ECHO Autism curriculum was delivered for a predefined period (6 months) for the purposes of the research study. However, typical ECHO implementation is ongoing, with no clearly defined stopping point. We also prohibited multiple participants from the same practice to minimize confounds; however, traditional ECHO models encourage participation from multiple clinicians from within a single practice. We also observed a relatively high rate of dropout, which may indicate issues with feasibility or acceptability of the model for some practicing PCPs.

Finally, our measure of comorbidity management may have been too simplistic. We were unable to assess whether PCPs were actively screening for each condition or whether they were noted only when concerns were raised by caregivers. As such, we were unable to determine the extent to which comorbid conditions were accurately screened and identified. Because the medical record review process did not allow for determination of best treatment approach for an individual patient (because this would have required thorough clinical review of each case), our measure simply noted whether the PCP administered 1 or more potentially appropriate treatments when a comorbid condition was observed. In future studies, a more fine-grained analysis of treatment strategies would be more helpful in determining whether a clinician followed a best-practice treatment approach or algorithm.

In conclusion, the results of this large-scale study provide some support for the effectiveness of the ECHO model in improving primary care knowledge and self-efficacy regarding treatment of children with autism but little evidence of practice change in specific screening and management domains. Future research is needed to further evaluate the cost, benefit, and effectiveness of ECHO. A more comprehensive assessment of screening and management practices, patient-clinician interactions, and patient outcomes would be particularly informative in future studies.

Accepted for Publication: December 12, 2018.

Corresponding Author: Micah O. Mazurek, PhD, Curry School of Education and Human Development, University of Virginia, 417 Emmet St S, PO Box 400267, Charlottesville, VA 22904 (mazurekm@virginia.edu).

Published Online: March 9, 2020. doi:10.1001/jamapediatrics.2019.6306

Author Contributions: Dr Parker and Mr Chan had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Mazurek, Parker, Kuhlthau, Sohl.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Mazurek, Parker, Chan.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Parker, Chan.

Obtained funding: Kuhlthau, Sohl.

Administrative, technical, or material support: Mazurek, Parker, Kuhlthau, Sohl.

Supervision: Mazurek, Parker, Kuhlthau.

Conflict of Interest Disclosures: Dr Mazurek reported grants from Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services during the conduct of the study. Dr Parker reported other support from HRSA and grants from Autism Speaks during the conduct of the study. Since completing the study, Dr Parker has become a consultant to the ECHO Institute, University of New Mexico. Dr Kuhlthau reported grants from Autism Intervention Research Network on Physical Health grant from MCHB HRSA and funding from the Autism Speaks Foundation during the conduct of the study. Dr Sohl reported other support from Quadrant Biosciences, Autism Navigator, the National Institutdes of Health/National Institute of Neurological Disorders and Stroke; and Cognoa and personal fees from university-based grand rounds; in addition, Dr Sohl had a patent to Autism Ready Communities pending. No other disclosures were reported.

Funding/Support: This project was supported by the Health Resources and Services Administration of the US Department of Health and Human Services under cooperative agreement UA3 MC11054–Autism Intervention Research Network on Physical Health.

Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: The members of the ECHO Autism Collaborative are Evdokia Anagnostou, MD, University of Toronto, Toronto, Ontario, Canada; Amanda Bennett, MD, MPH, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania; Kristin Dalope, MD, University of Pittsburgh, Pittsburgh, Pennsylvania; Susan Hyman, MD, University of Rochester, Rochester, New York; Heather Johnson, PsyD, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio; Maya Lopez, MD, University of Arkansas for Medical Sciences, Little Rock; Beth Malow, MD, MS, Vanderbilt University Medical Center, Nashville, Tennessee; Ann Neumeyer, MD, Lurie Center for Autism, Lexington, Massachusetts; Karen Ratliff-Schaub, MD, Nationwide Children’s Hospital, Columbus, Ohio; Robin Steinberg-Epstein, MD, University of California, Irvine.

Disclaimer: This information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by Health Resources and Services Administration of the US Department of Health and Human Services or the US government.

Data Sharing Statement: See Supplement 4.

Additional Contributions: We thank the members of the Clinical Coordinating Center and Data Coordinating Center at Massachusetts General Hospital, and the project staff and hub team members at each ECHO Autism Collaborative site. We also gratefully acknowledge all the primary care clinicians who participated in this study.

Additional Information: This work was conducted through the Autism Speaks Autism Treatment Network serving as the Autism Intervention Research Network on Physical Health.