Does Problem-Based Learning Improve Residents' Self-directed Learning?

THIS STUDY¹ investigated the effect of a problem-based learning (PBL) curriculum on self-directed learning behaviors among a group of pediatric residents. Following an initial period of daily lectures (lecture-based learning [LBL]), 80 residents were then divided into 2 groups: 1 group of residents (n = 39) underwent a period of twice-weekly PBL sessions, while the other group (n = 41) continued receiving LBL. After this intervention, all residents once again received a period of LBL. Throughout each 3-month period (termed the pre-exposure, exposure, and follow-up periods, respectively), residents were given self-administered questionnaires to evaluate self-directed learning behaviors, including the amount of time they spent on independent study, medical discussions, and computer literature searches. The investigators found that while the PBL group reported significantly more time spent in these self-directed endeavors during the exposure period, both groups reported equal baseline levels durign the follow-up period. The investigators concluded that residents exposed to PBL engaged in significantly higher levels of self-directed learning than their counterparts.

We evaluated this study with respect to design, results, and analysis according to the Journal of the American Medical Association users' guide format for a therapeutic trial.^2,3 This evaluation was done to assess the quality of the evidence, the strength of the subsequent conclusions, the generalizability of the outcomes, and the consideration of benefits vs risk and cost.

This was a quasi-experimental design using residents' rotations to determine group assignment. The PBL group consisted of residents on inpatient rotations at the central residency site, while the LBL group consisted of residents on other rotations. This raises questions regarding potential confounding factors that were not accounted for in the analysis. For example, we do not know any characteristics of the subjects in each group (ie, postgraduate training level, differences in call schedules, other obligations or time commitments associated with each rotation). It is generally acknowledged that interns have less time for independent learning than do upper-level residents, and residents with more frequent call schedules will have potentially less time than others. The investigators did not control for or address any of these issues in their article. In their defense, they did find that both groups reported similar self-directed learning behaviors in the pre-exposure period, despite being on inpatient vs outpatient rotations. However, simply because both groups used similar durations for independent learning does not automatically reflect whether one group may have had more opportunity or interest in pursuing independent learning than the other.

We asked the following questions: (1) Were all subjects who entered the trial properly accounted for and attributed at its conclusion? (2) Was follow-up complete? and (3) Were subjects analyzed in the groups to which they were randomized? All subjects were accounted for and attributed at the conclusion of the study as indicated by the statement that "complete data were available on all 80 subjects." We are not told explicitly that subjects were analyzed in the groups to which they were assigned initially, although we assume this to be the case. Unfortunately, we were not able to deduce this with certainty because results are reported only in the form of percentages, without including the actual number of subjects for the intervention and control groups.

The article states that the subjects were blinded to the hypothesis of the study. It obviously would not be possible to blind them to the intervention itself. Yet it is unclear whether the faculty members leading the sessions were aware of the hypothesis, and whether those analyzing the data (subject questionnaires) were blinded as well. The specific format of the questionnaire used to evaluate self-directed learning behaviors was not included in this article. Nevertheless, it is possible that residents receiving questions about learning habits in the setting of a "new" format of lectures would be more conscientious of and potentially biased in their responses. One possible solution would have been to add sham questions to the questionnaire in an attempt to avert any transparency bias.

In addition, if the lecturers in the 2 groups were aware of the goals of the study, it may have influenced their effort and performance in the teaching sessions, via the Hawthorne effect. Lecturers may have been motivated to increase self-directed learning in the intervention group, and to de-emphasize it in the control group.

Finally, it is important that the evaluators of the questionnaires be blinded to the group assignment. The more objective the outcome data, the less critical this issue becomes. As stated earlier, the specific questions in the questionnaire were not included in the article; thus, it is impossible to assess how the evaluators interpreted the data. For example, if residents were told to write in the number of hours they spent on independent study, any ambiguity in the penmanship could potentially lead to bias in interpretation. However, if the questionnaires included numbers that subjects circled, there would be a lower potential for bias of interpretation.

There were no statistically significant differences between the 2 groups at the start of the trial with respect to the outcomes of interest. However, the groups are not compared with respect to stage of training or mean hours of work per week. As we mentioned before, these could be potential confounders or effect modifiers.

There was no additional intervention offered to either group, although, because this was not a randomized controlled trial, there were likely to be significant differences in other variables, such as patient contact hours, sleep deprivation, and the complexity of patients encountered. Some of these potential confounders might serve to bias toward the null hypothesis (eg, working longer hours, as might be expected on the inpatient wards, might decrease the incentive to conduct literature searches). However, other factors, such as the added complexity of inpatients, might serve to increase the incentive to do so.

The study found a statistically significant increase in several measures of self-directed learning among the PBL group vs the LBL group during the intervention period. These included more residents reporting 5 or more hours of study, 5 or more hours of medical discussions, and 2 or more computer literature searches, with P<.01. The number needed to treat (NNT) for 5 or more hours of study is approximately 5, indicating that 5 residents would need to undergo PBL for each additional resident to demonstrate this behavior. The NNT for 5 or more hours of medical discussion and 2 or more computer literature searches are 4 and 5, respectively.

It is unclear whether the increased amount of self-directed learning was part of the work assigned in the PBL sessions or in addition to these assignments. As described in the "Materials and Methods" section, residents in the PBL group attended weekly meetings to work through various medical cases. The first meeting of each case introduced the clinical scenario, ending in "self-assignment of the learning issues" to investigate further information necessary to develop an appropriate differential diagnosis and plan of care, which was discussed at the next meeting. If the results indicating increased self-directed learning among these residents with respect to reading, research, and discussion were because of these assignments, then it is no surprise that this behavior disappeared after the intervention was completed. That is to say, once the assignments were discontinued, the impetus for pursuing more self-directed study returned to baseline.

The precision of the treatment effect is determined by the confidence interval (CI) reported in the analysis. While CIs were not provided in this study, P values were included, and results were judged to be statistically significant with P<.01. This implies a 99% confidence that the results were not due to chance alone.

It seems that these results can be generalized to other residency programs, although the overall benefit would be somewhat questionable in some very small residency programs given the calculated NNT (roughly 5).

This study did not demonstrate any long-term benefit to PBL, as evidenced by the return to baseline of self-directed learning in the follow-up period. Results from the study showed that self-directed learning behaviors in the follow-up period were comparable to both the pre-exposure period and between the PBL and LBL groups. This indicates that while exposure to ongoing PBL demonstrated increased independent learning, once this format was discontinued (as would be the case when a resident completes the residency program) the subjects did not maintain this increased level of independent learning. The authors acknowledged the apparent short-lived gains in self-directed learning behavior, as well as the question as to whether residents would benefit from more prolonged or ongoing exposure to PBL to sustain these improved behaviors.

The likely benefits of PBL seem to be worth any potential harms or costs. One would definitely need to take into account the cost of training instructors in PBL, and the potential changes in resident occupational and recreational schedules (and quality of life). In addition, further study into maintaining long-term increases in self-directed learning (implicit in a change in attitude toward independent learning that would continue throughout one's career) would need to be carried out to fully assess the benefit of this intervention vs the associated risk or cost.

Despite various limitations in the design of the investigation, the results of this study have several important implications for the future training of physicians nationwide. Despite the questionable benefit of PBL in terms of persistent change in residents' self-directed learning behaviors, the potential benefits of PBL in any residency program would seem to outweigh the minimal risks and costs. Further investigation is warranted to study the long-term effects of PBL— in particular, focusing on whether additional or more intensive exposure could sustain the increase in self-directed learning beyond the intervention period alone.