Academic Outcomes 2 Years After Working Memory Training for Children With Low Working Memory: A Randomized Clinical Trial

Importance  Working memory training may help children with attention and learning difficulties, but robust evidence from population-level randomized controlled clinical trials is lacking.

Objective  To test whether a computerized adaptive working memory intervention program improves long-term academic outcomes of children 6 to 7 years of age with low working memory compared with usual classroom teaching.

Design, Setting, and Participants  Population-based randomized controlled clinical trial of first graders from 44 schools in Melbourne, Australia, who underwent a verbal and visuospatial working memory screening. Children were classified as having low working memory if their scores were below the 15th percentile on either the Backward Digit Recall or Mister X subtest from the Automated Working Memory Assessment, or if their scores were below the 25th percentile on both. These children were randomly assigned by an independent statistician to either an intervention or a control arm using a concealed computerized random number sequence. Researchers were blinded to group assignment at time of screening. We conducted our trial from March 1, 2012, to February 1, 2015; our final analysis was on October 30, 2015. We used intention-to-treat analyses.

Intervention  Cogmed working memory training, comprising 20 to 25 training sessions of 45 minutes’ duration at school.

Main Outcomes and Measures  Directly assessed (at 12 and 24 months) academic outcomes (reading, math, and spelling scores as primary outcomes) and working memory (also assessed at 6 months); parent-, teacher-, and child-reported behavioral and social-emotional functioning and quality of life; and intervention costs.

Results  Of 1723 children screened (mean [SD] age, 6.9 [0.4] years), 226 were randomized to each arm (452 total), with 90% retention at 1 year and 88% retention at 2 years; 90.3% of children in the intervention arm completed at least 20 sessions. Of the 4 short-term and working memory outcomes, 1 outcome (visuospatial short-term memory) benefited the children at 6 months (effect size, 0.43 [95% CI, 0.25-0.62]) and 12 months (effect size, 0.49 [95% CI, 0.28-0.70]), but not at 24 months. There were no benefits to any other outcomes; in fact, the math scores of the children in the intervention arm were worse at 2 years (mean difference, −3.0 [95% CI, −5.4 to −0.7]; P = .01). Intervention costs were A$1035 per child.

Conclusions and Relevance  Working memory screening of children 6 to 7 years of age is feasible, and an adaptive working memory training program may temporarily improve visuospatial short-term memory. Given the loss of classroom time, cost, and lack of lasting benefit, we cannot recommend population-based delivery of Cogmed within a screening paradigm.

Trial Registration  anzctr.org.au Identifier: ACTRN12610000486022

Low academic achievement is a major public health issue because it is prevalent,^1,2 alters the life opportunities of individuals via poorer mental and physical health and financial hardship,^3-6 and threatens the economic and societal functioning of nations. By the time it becomes evident, low academic achievement is often entrenched, and it may be too late to intervene.^1,7,8 Preventing low academic achievement is a public health priority,^9,10 but solutions remain elusive.

Novel interventions that can be widely applied can be attractive, even in the absence of evidence of long-term benefits and cost-effectiveness. One such approach is “brain training” to improve working memory, a cognitive function responsible for temporarily storing and manipulating information needed to support learning.^11-13 Children with low working memory often fail classroom activities¹² and are at high risk of low academic achievement.^14,15 For example, more than 90% of children 6 to 7 years of age with reading difficulties have low working memory,¹⁶ and children with mathematical difficulties are more likely than their peers to have low working memory,¹⁷ and this association persists after adjusting for IQ.¹¹ Children at educational risk can be identified at school entry on the basis of low working memory scores.¹⁵

Commercially available computerized programs to improve working memory or cognition are widely implemented, despite a lack of supporting evidence. The cognitive training program “Elevate” was ranked by Apple as the best iPhone app in 2014, and the cognitive training sector has been forecast to profit by more than $500 million internationally in 2015. Administered to tens of thousands of clients each year, the Cogmed Working Memory Training program (Pearson) is the most widely used and evaluated working memory training program. Randomized clinical trials have shown the benefits to working memory for children with attention-deficit/hyperactivity disorder^18,19 and for other clinical and nonclinical populations.^20-22 The benefits of training may transfer to other cognitive domains,²³ academic functioning,^22,24 and behavior.^25-27 Cognitive training may induce neuroplasticity, such as increased activation in the frontal and parietal areas of the brain,²⁸ and increased connectivity of key components of the attentional control network at rest.²⁹

While there is evidence for short-term working memory enhancement following Cogmed,^25,30 there are no rigorous randomized clinical trials with large samples and long-term follow-up, nor is there any consistent evidence of long-term transfer effects on educational attainment.^26,30-32 Furthermore, the economic and opportunity costs of implementation of a working memory training program as a population-level prevention strategy are unknown, calling the effectiveness of working memory training programs into question.³³

To our knowledge, we report the first large-scale population-based randomized clinical trial of a working memory intervention for children 6 to 7 years of age screened as having low working memory. We aimed to (1) determine the efficacy of Cogmed, at 12 and 24 months, on reading, spelling, and mathematics (our primary outcome measured at 12 months) and a broad range of secondary outcomes for these children (the intervention arm) compared with children receiving the usual classroom education (the control arm), and (2) evaluate the costs and benefits.

Box Section Ref ID

We have previously reported our trial protocol (Supplement 1).³⁴ The Memory Maestros study³⁵ is a population-based randomized controlled clinical trial comparing a computerized working memory intervention (Cogmed) with usual classroom teaching in Grade 1 students with low working memory(Grade 1 refers to the second year of formal primary school education in Victoria, Australia). Figure 1 shows the CONSORT diagram, and Figure 2 shows a graphical representation of the trial. Approval was obtained from the Human Research Ethics Committee at the Royal Children’s Hospital in Melbourne, Australia, the Victorian Department of Education and Training, and the Catholic Education Office. All parents provided written informed consent.

Grade 1 students from 44 primary schools in metropolitan Melbourne (with a population of 4.1 million in 2012), Australia, participated. Schools were approached according to a random sequence that was generated to recruit a sample representative of each of the 3 Victorian school sectors (government, Catholic, and independent) and from a range of sociodemographic backgrounds.

Recruitment was carried out over all 4 terms of the 2012 school year (from February to December). Screening and intervention occurred in succession within each school. Teachers sent home recruitment packets containing a baseline questionnaire, information statement, and consent form. We simultaneously obtained consent to screen the working memory of the child and, in the event of it being low, consent for the child to enroll in the trial and follow-up.

Research assistants administered a 10-minute computerized working memory screening during school hours to each child whose parents had consented, within 2 weeks of completing the baseline questionnaire. Children were classified as having low working memory if their scores on either the Backward Digit Recall or Mister X subtest from the Automated Working Memory Assessment were below the 15th percentile, or if both scores were below the 25th percentile.³⁶ The cut points were based on internally generated percentile ranks because our sample was much larger than the Automated Working Memory Assessment normative sample. Eligibility cut points were revised midyear, to reflect the developmental progression in working memory over time.³⁵

Children with low working memory were eligible to be randomized. We excluded children with disabilities (eg, cerebral palsy, vision/hearing impairments, or pervasive developmental disorders) that were likely to prevent participation in the intervention program and children and families with insufficient English language abilities that were likely to prevent participation in the consent, assessment, or intervention procedures of the trial.

An independent statistician randomized children into usual classroom teaching (control arm) or the working memory training program (intervention arm) using a computerized random number sequence. Assignment was at a ratio of 1:1 intervention to control, stratified by school; thus, each school included a balanced number of children participating in the intervention and control arms. Researchers assessing outcomes were blinded to group allocation.

Children were taken out of class in groups (up to 4 students) by a research assistant to participate in the Cogmed RM program, comprising 20 to 25 sessions delivered over 5 to 7 weeks. Each child had their own computer and noise-cancelling headphones to limit distractions. Sessions took 45 minutes, on average (range, 35-60 minutes). Children completed 8 of 12 tasks. A new task was introduced every 5 to 6 days. This adaptive program matches difficulty level to the child’s performance on a trial-by-trial basis. Tasks involve the temporary storage and manipulation of verbal and/or visuospatial information in a computer game format, such as recalling a sequence of numbers that light up in a certain order. Motivational features include verbal feedback, the displaying of “high scores,” and the accumulation of “points” when tasks are successfully completed.

Research assistants (“training aides”) were trained through role-playing and field observations before trial commencement. Weekly meetings with the research assistants and project manager (a trained Cogmed “coach”) were held to discuss any issues that may arise (eg, child motivation), and a standard approach was agreed on, as recommended in the Cogmed training manual.³⁷

Table 1 summarizes the trial measures. The primary outcomes were the word reading, spelling, and math computation subscales of the Wide Range Achievement Test 4 at 12 months (also measured at 24 months), each with a normative mean (SD) of 100 (15).³⁸ Secondary outcomes measured at 12 and 24 months included working memory performance (also measured at 6 months), behavior, and health-related quality of life.

Costs in 2014 Australian dollars (A$) included training of Cogmed coach and training aides, screening, and intervention delivery. Research staff members prospectively recorded time and the materials used. Staff time was valued at staff wage rates (including 20% on-costs), child time was valued at A$0, car travel was valued at standard medium car unit costs, computer hardware and software were valued at the unit costs experienced in the study, and intervention software (provided at zero cost) was valued at A$0.

We used intention-to-treat analyses to compare outcomes at 6, 12, and 24 months between the intervention and control arms. Mean outcomes were compared using linear regression in unadjusted analyses and analyses adjusted for factors specified a priori that may have affected outcomes and may not have been fully balanced by randomization: child’s sex, IQ, and primary caregiver’s education. Clustering of children within schools was accounted for using robust regression techniques in which the variance estimates are adjusted to account for the similarity between children within schools.³⁹

Ancillary sensitivity analyses were also conducted. First, for the working memory outcomes that were repeatedly measured at 3 or more time points, we ran random-effects regression analyses to reexamine treatment effects by time within a longitudinal regression model. This was precluded for our principal analysis because, in keeping with the “screen plus intervene” model being tested, most of our key outcomes were not measured at baseline. Second, we reran all analyses using a multiply imputed data set to estimate possible effects of missing data within the intention-to-treat analyses.⁴⁰ The multiple imputation model included all variables used in the complete data analysis, with a series of 50 data sets imputed using chained equations. All analyses were undertaken using Stata version 14 (StataCorp).

To detect a clinically important difference of 0.3 SD in the primary outcome measures at a significance level of .05 with 80% power, we required 175 children in each of the 2 trial arms (350 in total) at our primary end point. Allowing for 20% attrition, we found that our target sample size at recruitment was 438 children.

Of the 67 schools approached, 44 (65.7%) agreed to participate. Figure 2 shows that, of the 2747 parents approached, 1761 (64.1%) consented, and 1723 of 1761 children (97.8%) completed the screening assessment (ie, the parent to child ratio was 1:1). Table 2 shows that boys and girls were similarly represented. About two-thirds of parents had completed a tertiary education, somewhat higher than the 43% expected from Australian census data.⁴¹

Of the 1723 children screened, 452 met the inclusion criteria and were randomized, 226 in each study arm. At 6, 12, and 24 months postrandomization, 90.5%, 89.5%, and 87.8% of children remained in the study. Demographic characteristics between children who did and children who did not participate in outcome assessments were similar at all time points.

Of the 226 children randomly assigned to the intervention arm, 204 (90.3%) completed at least 20 training sessions. The children who competed the training sessions and the children who did not were comparable on baseline screening assessment and characteristics.

Table 3 shows the complete-case outcomes comparisons. At 6 months, children in the intervention arm had higher visuospatial short-term memory (mean difference, 5.47 [95% CI, 2.87-8.07]; P < .001) and verbal working memory scores (mean difference, 2.91 [95% CI, 0.02-5.79]; P = .04) than children in the control arm. Relationships were similar after adjustment. Only the visuospatial short-term memory benefits remained at 12 months, and none were apparent by 24 months.

Despite this transient short-term memory benefit, there was little evidence for improved academic outcomes (Table 3). Children in the intervention arm had poorer word reading (mean difference, −1.81 [95% CI, −3.78 to 0.15]; P = .07) and math computation scores (mean difference, −2.64 [95% CI, −5.48 to 0.20]; P = .07) at 12 months than children in the control arm. The evidence for lower scores in math computation for the intervention group was stronger at 24 months (mean difference, −3.03 [95% CI, −5.39 to −0.67]; P = .01), although the effect size was small (0.2). We note the inflated potential for chance findings due to the number of outcomes and time points. All other academic outcomes were similar at both 12 and 24 months, although a consistent theme was for slightly lower scores in the intervention group. Despite low working memory at baseline, mean word reading and spelling scores of children in both groups were slightly above the mean US normative population value of 100, while the mean math computation score was around a third of a standard deviation below. The rate of completion of the required 20 sessions was so high (>90%) that we did not run dose-response analyses. The parent, child, and teacher ratings spanning attention problems, social-emotional difficulties, and quality of life were similar in the intervention and control groups at 12 and 24 months, with mean scores typical for that age.

The eTable in Supplement 2 shows the treatment effects reestimated using random-effects regression models and also the principal analysis repeated using the multiply imputed data set, which is also presented graphically in the eFigure in Supplement 2. These reanalyses did not substantively change the outcome values or any conclusions.

Costs of training (A$12 919), materials (A$15 939), screening (A$54 205), and intervention delivery (A$150 956) summed to A$234 020: A$1035 per child randomly assigned to the intervention arm. If offered to all Victorian Grade 1 children in the lowest quartile for working memory, this would equate to over A$18 million per annum ($1035 × 25% × 75 000 + annual school intake). This does not include the costs of the Cogmed program, which was made available to the trial at no charge but currently retails for around A$1500 per child.

This randomized controlled clinical trial examined the effectiveness of an adaptive working memory intervention in improving population-based academic outcomes for children with low working memory. Despite an advantage to some working memory measures at 6 and 12 months in line with other studies of Cogmed,^22,31 there were no evident benefits to academic outcomes at 12 or 24 months. This lack of effect is also seen in the parent and teacher ratings of attention, social-emotional difficulties, and quality of life. This lack of benefit must be considered in light of the costs in terms of price (over A$1000 per child, plus cost of the program itself), the loss of around 15 to 20 hours of usual classroom teaching for each child, and opportunity (other remediation that could have been offered instead).

Our trial was randomized and controlled, used an intention-to-treat analysis, and is the largest trial of working memory training to date, to our knowledge. Allocation and outcomes assessment were blinded. The study groups were recruited from a large population-based cohort, allowing generalizability. The retention rate was high, maximizing power and minimizing bias. The completion rate for the intervention arm was high, allowing us to evaluate long-term efficacy with confidence; this is supported by the similar conclusions from ancillary analyses using multiply imputed outcomes data sets. We exceeded our required sample size for the directly assessed primary outcome measure but not for the parent- or teacher-reported outcomes. However, complete-case analyses closely resembled those using the multiply imputed outcomes data set, and neither showed any trends toward effectiveness for these secondary outcomes.

A potential limitation of our study is that the screening protocol used only 2 subtests of the Automated Working Memory Assessment. This mimicked what we felt would be possible if this model was to be rolled out on a large scale: screening all children quickly, with rapid progression to intervention. However, this comes at a likely cost in terms of measurement precision. Although our working memory screening tests have been widely used in previous research in this field,^22,24 like many standardized cognitive measures, they suffer from task impurity and tap skills additional to working memory (eg, Mister X subtest requires visuoperceptual ability).

The generalizability of these results may not extend to children who do not speak English or to children whose parents have a lower educational level than those in our study. The children in our study were at the lowest end of the recommended age range for the Cogmed RM program. Lack of a nonadaptive control group could have been seen as a limitation had there been an intervention effect, but it is not relevant for a noneffective intervention.

Our 6-month working memory effect sizes are smaller than those in previous trials in this field. This is not an uncommon finding when scaling up to a population level with a rigorous design⁴²; previous Cogmed studies varied in participant characteristics (including participants with attention-deficit/hyperactivity disorder, learning difficulties, special education needs, and low working memory) and/or used weaker designs (including nonrandom allocation and lack of a control group). Most of these studies test the intervention on older children. Our findings sit between those of the initial, small school-based studies reporting that Cogmed training can enhance visuospatial^{22,24,31,43-45} and verbal short-term memory⁴⁴ and visuospatial^22,24,31,43 and verbal working memory,^{22,24,31,43-45} and an increasing number of studies indicating little benefit for verbal short-term memory.^{22,31,43,44,46}

Our 12-month working memory and academic outcomes are comparable to those of the largest randomized trial, to date, but go further by virtue of our larger sample size, higher retention, and subsequent longer follow-up. In the trial by Dunning et al,³¹ more than 800 children 7 to 9 years of age from 9 UK schools were screened for low working memory; 34 participated in Cogmed, 30 in a nonadaptive version, and 30 received teaching as usual. At 12 months, moderate-to-large gains in verbal working memory were sustained in the Cogmed group (n = 15) compared with the nonadaptive group (n = 19), but there was little evidence of improved academic performance. Smaller, methodologically less rigorous studies have reported some academic benefits. Egeland et al⁴⁷ reported a slight increase in reading scores at 8 months in a small randomized controlled trial (N = 57) of children with attention-deficit/hyperactivity disorder, and Holmes et al^22,43 and Dahlin^22,43 reported an increase in mathematics scores at 6 and 7 months, respectively, in small (N = 42 and N = 57) nonrandomized trials.

A controversial issue^33,39 is whether targeting one specific cognitive skill (eg, working memory) can improve another (eg, academic function). Klingberg⁴⁸ argues that working memory training can result in changes in activity (“plasticity”) in neural networks that may then result in a transfer of effects to nontrained tasks. However, a primary school classroom offers a complex interplay between the temperaments of students and teachers, cognitive abilities, and engagement, which may not correspond with the highly structured and predictable computer-based training environment. Future interventions may benefit from extending programs such as Cogmed to include real-world activities and incorporate explicit strategy training paired with education intervention.

With no evidence of benefit in our primary outcomes, the intervention is not cost-effective. Our intervention delivery costs were inflated by high travel costs, as city-center–based training aides traveled to schools. In practice, individual schools may deliver the intervention using local staff. However, any reduction in travel costs would be balanced against the cost of the intervention program, which was waived for this trial. Given the consistently, slightly lower academic scores in the intervention group, particularly for word reading and math computation, our results raise questions regarding the potential for harm by taking children out of class on a regular basis for several weeks to provide an intervention such as this. On balance, we cannot recommend this intervention as a population-level selective prevention strategy.

It is feasible to implement population-based working memory screening for children 6 to 7 years of age and deliver a working memory training program (Cogmed). Although this benefitted some elements of memory at 6 months, given the high cost and the lack of benefit to academic outcomes or any other outcomes 12 or 24 months after randomization, we cannot recommend its population-based delivery as a selective prevention program. Longer-term follow-up of the trial’s cohort will clarify any lasting effects, whether harmful or beneficial. In the meantime, we recommend that equally rigorous trials test the other indications currently targeted by adaptive computerized training programs.

Accepted for Publication: December 4, 2015.

Corresponding Author: Gehan Roberts, MPH, PhD, Centre for Community Child Health, Royal Children’s Hospital, Flemington Road, Parkville VIC 3052, Australia (gehan.roberts@rch.edu.au).

Published Online: March 7, 2016. doi:10.1001/jamapediatrics.2015.4568.

Author Contributions: Drs Roberts and Quach had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Roberts, Anderson, Gathercole, Gold, Mensah, Rickards, Ainley, Wake.

Acquisition, analysis, or interpretation of data: Roberts, Quach, Spencer-Smith, Anderson, Gathercole, Gold, Sia, Mensah, Wake.

Drafting of the manuscript: Roberts, Quach, Spencer-Smith, Gathercole, Wake.

Critical revision of the manuscript for important intellectual content: Roberts, Spencer-Smith, Anderson, Gathercole, Gold, Sia, Mensah, Rickards, Ainley, Wake.

Statistical analysis: Quach, Anderson, Mensah.

Obtained funding: Roberts, Gathercole, Gold, Mensah, Wake.

Administrative, technical, or material support: Roberts, Quach, Gold, Sia, Wake.

Study supervision: Roberts, Spencer-Smith, Anderson, Gold, Rickards, Wake.

Conflict of Interest Disclosures: None reported.

Funding/Support: The trial is funded by the National Health Medical Research Council in Australia, as follows: Project Grant 1005317; Early Career Fellowship 607384 (Dr Roberts); Senior Research Fellowship 1046518 (Dr Wake); Capacity Building Grant 425855 and Early Career Fellowship 1035100 (Dr Gold); Senior Research Fellowship 1081288 (Dr Anderson); and Capacity Building Grant 436914 and Early Career Fellowship 1037449 (Dr Mensah). Dr Quach is funded by an Australian Research Council Discovery Early Career Award DE140100751. The project also received support through the Centre for Research Excellence in Child Language, which is funded by the National Health Medical Research Council in Australia and based at the Murdoch Childrens Research Institute, which is supported by the Victorian Government’s Operational Infrastructure Program.

Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: We thank Bibi Gerner, DEdPsych, at Population Health, Murdoch Childrens Research Institute, for assisting Dr Quach, who was a postdoctoral research fellow and the project manager. She received no compensation outside of her regular manager salary.