Medicine Residents' Understanding of the Biostatistics and Results in the Medical Literature

Context Physicians depend on the medical literature to keep current with clinical information. Little is known about residents' ability to understand statistical methods or how to appropriately interpret research outcomes.

Objective To evaluate residents' understanding of biostatistics and interpretation of research results.

Design, Setting, and Participants Multiprogram cross-sectional survey of internal medicine residents.

Main Outcome Measure Percentage of questions correct on a biostatistics/study design multiple-choice knowledge test.

Results The survey was completed by 277 of 367 residents (75.5%) in 11 residency programs. The overall mean percentage correct on statistical knowledge and interpretation of results was 41.4% (95% confidence interval [CI], 39.7%-43.3%) vs 71.5% (95% CI, 57.5%-85.5%) for fellows and general medicine faculty with research training (P < .001). Higher scores in residents were associated with additional advanced degrees (50.0% [95% CI, 44.5%-55.5%] vs 40.1% [95% CI, 38.3%-42.0%]; P < .001); prior biostatistics training (45.2% [95% CI, 42.7%-47.8%] vs 37.9% [95% CI, 35.4%-40.3%]; P = .001); enrollment in a university-based training program (43.0% [95% CI, 41.0%-45.1%] vs 36.3% [95% CI, 32.6%-40.0%]; P = .002); and male sex (44.0% [95% CI, 41.4%-46.7%] vs 38.8% [95% CI, 36.4%-41.1%]; P = .004). On individual knowledge questions, 81.6% correctly interpreted a relative risk. Residents were less likely to know how to interpret an adjusted odds ratio from a multivariate regression analysis (37.4%) or the results of a Kaplan-Meier analysis (10.5%). Seventy-five percent indicated they did not understand all of the statistics they encountered in journal articles, but 95% felt it was important to understand these concepts to be an intelligent reader of the literature.

Conclusions Most residents in this study lacked the knowledge in biostatistics needed to interpret many of the results in published clinical research. Residency programs should include more effective biostatistics training in their curricula to successfully prepare residents for this important lifelong learning skill.

Physicians must keep current with clinical information to practice evidence-based medicine (EBM). In doing so, most prefer to seek evidence-based summaries, which give the clinical bottom line,¹ or evidence-based practice guidelines.^1-3 Resources that maintain these information summaries, however, currently include a limited number of common conditions.⁴ Thus, to answer many of their clinical questions, physicians need to access reports of original research. This requires the reader to critically appraise the design, conduct, and analysis of each study and subsequently interpret the results.

Several surveys in the 1980s demonstrated that practicing physicians, particularly those with no formal education in epidemiology and biostatistics, had a poor understanding of common statistical tests and limited ability to interpret study results.^5-7 Many physicians likely have increased difficulty today because more complicated statistical methods are being reported in the medical literature.⁸ They may be able to understand the analysis and interpretation of results in only 21% of research articles.⁸

Educators have responded by increasing training in critical appraisal and biostatistics throughout the continuum of medical education. Many medical schools currently provide some formal teaching of basic statistical concepts.⁹ As part of the Accreditation Council for Graduate Medical Education's practice-based learning and improvement competency, residents must demonstrate ability in “locating, appraising, and assimilating evidence from scientific studies related to their patients' problems and apply knowledge of study designs and statistical methods to the appraisal of clinical studies.”¹⁰ Most residency programs address this competency through EBM curricula or journal clubs.^11-13 In 2000, the majority of these programs included training in appraisal of studies and study conduct, but fewer specifically addressed the selection and interpretation of statistical tests.^11,14 In addition, the majority of published assessments of residents' knowledge and skills in EBM were performed at single programs, were conducted in the context of determining the impact of a specific curriculum, evaluated critical appraisal skills more commonly than biostatistics, and found that residents scored well below EBM “experts” on evaluation instruments.¹⁵ We performed a multiprogram assessment of residents' biostatistics knowledge and interpretation of study results using a new instrument developed for this study.

We developed an instrument to reflect the statistical methods and results most commonly represented in contemporary research studies (Appendix). Thus, we reviewed all 239 original articles published from January to March of 2005 in each issue of 6 general medical journals (American Journal of Medicine, Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine) and summarized the frequency of statistical methods described (Table 1). From this review, we developed questions that focused on identifying and interpreting the results of the most frequently occurring simple statistical methods (eg, χ², t test, analysis of variance) and multivariate analyses (eg, Cox proportional hazards regression, multiple logistic regression).

The survey (Appendix) contained 4 sets of questions: (1) 11 demographic questions that included age, sex, current training level, past training in biostatistics and EBM, and current journal-reading practices; (2) 5 attitude questions regarding statistics; (3) 4 confidence questions about interpreting and assessing statistical concepts; and (4) a 20-question biostatistics knowledge test that assessed understanding of statistical methods, study design, and interpretation of study results. Statistical attitudes and confidence questions were adapted from surveys on the Assessment Resource Tools for Improving Statistical Thinking (ARTIST) Web site, which is a resource for teaching statistical literacy, reasoning, and thinking.¹⁶ Attitudes regarding statistics were rated on a 5-point Likert scale. Confidence questions were assessed using a 5-point scale in which 1 indicated no confidence and 5 indicated complete confidence. The remaining 20 knowledge test questions addressed understanding of statistical techniques, study design, and interpretation of study results most commonly represented in our journal review. These questions were multiple-choice, clinically oriented with a case vignette, and required no calculations. Two questions were adapted from a study of Danish physicians' statistical knowledge.⁷ Seven questions were adapted from course materials used in statistics courses at the Johns Hopkins Bloomberg School of Public Health.¹⁷ The remaining questions were developed by one of the study authors (D.M.W.). The knowledge questions addressed research variable types, statistical methods, confidence intervals, P values, sensitivity and specificity, power and sample size, study design, and interpretation of study results.

The original test contained 22 knowledge questions and was pilot tested with 5 internal medicine faculty with advanced training in epidemiology and biostatistics and 12 primary care internal medicine residents at 1 residency program. Faculty reviewed the instrument for content validity, completed the test, and provided feedback. Residents completed the test and provided written and oral feedback. Four of the 5 faculty answered 21 of 22 questions correctly and 1 faculty member correctly answered 19 questions. This resulted in an overall mean score of 94%. Incorrect responses did not favor any particular question. Residents answered 53% of questions correctly. Based on feedback, 1 question was modified to improve clarity, 3 questions were eliminated to avoid duplicating similar concepts, and 1 question was added to further assess interpretation of results. Therefore, the final version of the test consisted of 20 questions.

We conducted an anonymous cross-sectional survey from February through July 2006 of 11 internal medicine residency programs in Connecticut, including 7 traditional internal medicine programs, 2 primary care medicine programs, 1 medicine/pediatrics program, and 1 medicine/preventive medicine program. We initially contacted all 15 internal medicine residency programs in Connecticut to ask for their participation in the study. All programs were successfully contacted and expressed interest. However, 3 programs could not accommodate the study because of scheduling conflicts, and 1 program was not included because its residents (medicine/pediatrics) were distributed to different training sites and therefore were not present at the conferences used for the survey.

Included residencies were both university affiliated (7 programs) and community based (4 programs). Residents at all postgraduate levels of training were invited to participate. Oral consent was obtained from each participant after providing a description of the survey's purpose. The survey was administered during the first 25 minutes of an inpatient noon conference lecture for current residents. After all questionnaires were collected, the remainder of the time was devoted to a seminar in statistical methods and interpretation of the literature. Four residency programs also allowed us to survey their entering intern classes during their orientations. To provide data for validity testing, an additional 10 faculty and fellows trained in clinical investigation also completed the final survey. The Yale University human investigation committee approved the study protocol.

In addition to assessing the content validity, the psychometric properties of the 20-question knowledge test were determined by assessing internal consistency using Cronbach α. Discriminative validity was assessed by comparing the difference in mean scores obtained between residents and research-trained fellows and faculty using the t test.

The biostatistics knowledge test was scored by determining the percentage of questions correct, weighting each question equally. Missing values were counted as incorrect responses. The t test or a 1-way analysis of variance was used to compare survey scores by respondent characteristics. We calculated the percentage of residents who agreed or strongly agreed with each attitudinal question. We determined the percentage of respondents with fair to high confidence for each confidence question and the mean confidence score based on the sum of all 4 questions.

Correlation analyses were performed to test for multicollinearity between 3 sets of factors we hypothesized might be highly correlated (training outside of the United States and years since medical school graduation; training level and age; and past biostatistics training and past epidemiology training). Bivariate analyses were performed to identify factors that might be associated with knowledge scores. Candidate variables included sex, age, academic affiliation of residency program, advanced degrees, years since medical school graduation, training outside of the United States, current level of training, past biostatistics training, past epidemiology training, past EBM training, and currently reading medical journals. We also tested for effect modification for pairs of factors including past biostatistics training and past EBM training; past biostatistics training and past epidemiology training; and past biostatistics training and sex. The results of the correlation, bivariate, and effect modification analyses were used to determine which demographic variables to include in the multivariable model. Decisions to include factors in the multivariable regression analysis were based on the strength of correlated factors (r <0.75) or a P value <.05 on bivariate analyses. Forward stepwise regression was subsequently used to identify which demographic factors were independently associated with biostatistics knowledge scores.

To adjust for multiple pairwise comparisons, a 2-sided level of statistical significance was set at P < .01 using a Bonferroni correction. With a sample size of 277 and a P value of .01, the study had 80% power to detect a 4.4% difference in mean knowledge scores. All analyses were performed using Stata release 8.2 (StataCorp, College Station, Texas).

The 11 targeted training programs had 532 residents, with a mean of 53.6 trainees (range, 12-118).^18,19 In comparison, the 388 internal medicine training programs in the United States have a total of 21 885 residents, with a mean of 56.4 trainees (range, 4-170) (P = .76 compared to targeted programs).²⁰ The study programs had 41.9% women residents, compared with 42.1% nationally (P = .96), and 49.9% of residents with training outside of the United States vs 52.3% nationally (P = .51).¹⁹ Comparing targeted programs with all internal medicine programs, no statistically significant differences were seen for postgraduate year 1 trainees in mean duty hours per week (61.9 vs 65.2, P = .13), mean consecutive work hours (30 vs 27.5, P = .09), and mean number of days off per week (1.3 vs 1.2, P = .31).¹⁸ Targeted programs also did not differ in these characteristics from the remaining 4 Connecticut training programs.

Three hundred sixty-seven residents in the 11 targeted programs were on rotations that would make them available to attend their respective noon conferences on the day of the survey. Of these, 309 (84.2%) were in attendance. Of the total available residents, 277 (75.5%) completed the assessment. The response rate for individual programs ranged from 28.1% to 80%. No differences in response rates or attendance were seen based on sex, level of training, or past training outside of the United States. Table 2 lists the respondents' demographic characteristics. Approximately equal numbers of men and women were represented. Fifty-eight percent were enrolled in traditional internal medicine programs, 76.5% participated in university-based programs, 50.6% had some training outside of the United States, and 14.8% had advanced degrees. More than 68% of respondents had some training in biostatistics, with approximately 70% of this training occurring during medical school.

The survey instrument had high internal consistency (Cronbach α = 0.81). Fellows and general medicine faculty with advanced training in biostatistics had a significantly higher score than residents (mean percentage correct, 71.5% [95% confidence interval {CI}, 57.5%-85.5%] vs 41.1% [95% CI, 39.7%-43.3%]; P < .001), indicating good discriminative validity.

The overall mean resident knowledge score was 41.1% (SD, 15.2%; range, 10%-90%). Residents scored highest in recognition of double-blind studies (87.4% [95% CI, 83.5%-91.3%] answering correctly) and interpretation of relative risk (81.6% [95% CI, 77.0%-86.2%] answering correctly) (Table 3). They were least able to interpret the results of a Kaplan-Meier analysis, with 10.5% (95% CI, 6.9%-14.1%) answering correctly. Only 37.4% (95% CI, 31.9%-43.3%) understood how to interpret an adjusted odds ratio from a multivariate regression analysis, while 58.8% (95% CI, 53.0%-64.6%) could interpret the meaning of a P value.

Training outside of the United States had moderate correlation with years since medical school graduation (r = 0.59), as did past epidemiology training with past biostatistics training (r = 0.53). Training level had a fair correlation with age (r = 0.46). No effect modification was seen for the 3 sets of factors assessed. In bivariate analyses, differences in scores were seen based on residency program type, with medicine/pediatric residents scoring the highest (Table 4). Residents with advanced degrees performed better than those without advanced training (50.0% [95% CI, 44.5%-55.5%] vs 40.1% [95% CI, 38.3%-42.0%]; P < .001). Statistically significant higher scores were also seen in residents who were just entering residency, had prior biostatistics training, were enrolled in a university-based training program, and were men (Table 4).

Using forward stepwise regression, 5 factors were found to be independently associated with knowledge scores (Table 4). An advanced degree was associated with an absolute increase of 9.2% questions correct after adjustment for other factors (P < .001). Successive years since medical school graduation were associated with decreasing knowledge scores, with 11 years or more postgraduation associated with a 12.3% absolute decrease in score compared with less than 1 year postgraduation. Male sex, belonging to a university-based program, and past biostatistics training were all associated with higher scores.

The majority of residents agreed or strongly agreed that to be an intelligent reader of the literature it is necessary to know something about statistics (95%) and indicated they would like to learn more about statistics (77%). Seventy-five percent reported they did not understand all of the statistics they encountered in the literature, whereas only 15% felt that they do not trust statistics “because it is easy to lie.” More than 58% of respondents indicated that they use statistical information in forming opinions or when making decisions in medical care.

The mean confidence score in understanding certain statistical concepts was 11.4 (SD, 2.7) (maximum possible confidence score, 20). The majority of residents reported fair to complete confidence in understanding P values (88%). Fewer were confident in interpreting results of statistical methods used in research (68%), identifying factors influencing a study's power (55%), or assessing if a correct statistical procedure was used (38%).

Respondents with higher confidence in their statistical knowledge (a score higher than the mean confidence score) performed better on the knowledge questions than those with lower confidence (43.6% [95% CI, 40.8%-46.3%] vs 39.3% [95% CI, 37.0%-41.6%]; P = .02). Those who reported fair to high confidence in interpreting a P value were more likely to correctly interpret its meaning (62.8% [95% CI, 56.8%-67.2%] vs 38.2% [95% CI, 24.3%-51.7%]; P = .006). No differences were seen in a resident's ability to appropriately identify the correct statistical procedure used based on their confidence to do so.

In this multiprogram survey of internal medicine residents' confidence in, attitudes toward, and knowledge of statistical methods and interpretation of research results, 95% believed that it was important to understand these concepts to be an intelligent reader of the literature, yet three-fourths of residents acknowledged low confidence in understanding the statistics they encounter in the medical literature. This lack of confidence was validated by their low knowledge scores, in which on average only 8 of 20 questions were answered correctly. Although past instruction in biostatistics and advanced degrees were associated with better performance, knowledge scores appeared to decline with progression through training.

The poor knowledge in biostatistics and interpretation of study results among residents in our study likely reflects insufficient training. Nearly one-third of trainees indicated that they never received biostatistics teaching at any point in their career. When training did occur, the majority of instruction took place during undergraduate medical education and was not reinforced in residency. The most recent comprehensive survey of medical school biostatistics teaching was conducted in the 1990s and found that more than 90% of medical schools focused their biostatistics teaching in the preclinical years without later instruction and that the depth and breadth of this education varied greatly among schools.²¹ That review reported that familiar concepts such as P values, t tests, and χ² analyses were frequently addressed (95%, 92%, and 88%, respectively), but advanced methods (such as Cox proportional hazards regression, multiple logistic regression, and Kaplan-Meier analyses) were not included in instruction.²¹ If biostatistics teaching has continued at the same level in recent years, it would not be surprising that only a small percentage of residents in our survey (10.5%-37.6%) understood the results and use of these analyses.

The correlates of differences in knowledge scores might have been expected. Residents with prior biostatistical training and those with advanced instruction through a master's or PhD degree scored better than their counterparts. More senior residents performed worse than junior residents, potentially reflecting loss of knowledge over time, lack of reinforcement, or both. Although fourth-year residents were an exception to this pattern, these residents were part of a single medicine/pediatrics program that outperformed all other training programs. The higher scores in university-based residency programs may reflect exposure to faculty with more biostatistical training or teaching experience. In a survey study, community faculty considered EBM less important, were less confident in their EBM knowledge, and demonstrated poorer EBM skills than full-time faculty.²²

Although sex was associated with a difference in scores, this finding is not supported by other literature. Studies of evidence-based practice knowledge and skills rarely report analyses by sex. In 2 studies, investigators found no sex differences in critical appraisal skills among family physicians²³ or in use of online evidence databases among public health practitioners.²⁴ Six studies assessing the biostatistics and epidemiology knowledge of physicians and trainees did not conduct comparisons by sex.^5-7,25-27 Furthermore, our result was not a confirmation of an a priori hypothesis and so should be interpreted with caution.

Our final regression model found 5 predictors of knowledge scores: advanced degrees, academic affiliation, prior biostatistics training, sex, and years since medical school graduation. The proportion of explained variation for the model was small, with R² = 0.18. This likely reflects in part the low variance in resident scores.

Our results suggest the need for more effective training in biostatistics in residency education. Such training has proven difficult, with systematic reviews showing only limited effectiveness of many journal clubs and EBM curricula.^14,28-32 Thus, it is not surprising that prior EBM experience, which in the past has not included biostatistics training,^11,14 was not associated with higher scores in our multivariable analysis. Interactive, self-directed, and clinically instructional strategies seem to stand the best chance of success.³³ Involvement in hypothesis-driven research during training that requires comprehensive reading of the literature may also enhance residents' knowledge and understanding.³⁴

Faculty who are implementing biostatistics curricula can access several teaching resources. In internal medicine, the American College of Physicians' ACP Journal Club has presented a series of reports emphasizing basic study designs and statistics.³⁵CMAJ has published a series of EBM “teaching tips” for learners and teachers.³⁶ A guide designed to help medical educators choose and interpret statistical tests when developing educational studies or when reading the medical literature is also available.³⁷

Limitations of this study should be considered. First, while our instrument showed good content validity, internal consistency, and discriminative validity, these psychometric properties were not known in advance but were established in the current study. Second, our survey was purposely kept brief, thus limiting our ability to assess understanding of all biostatistical concepts and research results. Nonetheless, our questions focused on the most commonly used methods and results found in the contemporary literature. Third, we attempted to survey only those residents who were present at the time of their inpatient conference. Residents who did not attend, either by choice or by chance, might have scored differently. However, since we found no differences in demographic characteristics between responders and nonresponders, this is less likely. Fourth, our study was confined to internal medicine residents, limiting generalizability to other resident physicians. Nevertheless, we were able to assess multiple types of internal medicine training programs and found similar results.

Despite these limitations, this study also has several strengths. First, it was a multiprogram study that captured information on a wide range of internal medicine residents at different types of residency programs. Second, the residents in our survey, although limited to 1 state, possessed characteristics similar to all other trainees in internal medicine programs across the United States. Third, the 11 residency programs were similar in size and composition to the average US internal medicine program, and thus our study appears to be generalizable to internal medicine trainees and training programs in the United States.

Higher levels of statistical methods are being used in contemporary medical literature, but basic concepts, frequently occurring tests, and interpretation of results are not well understood by resident physicians. This inadequate preparation demonstrates lack of competence in meeting part of the Accreditation Council for Graduate Medical Education's practice-based learning and improvement requirement.¹⁰ If physicians cannot detect appropriate statistical analyses and accurately understand their results, the risk of incorrect interpretation may lead to erroneous applications of clinical research. Educators should reevaluate how this information is taught and reinforced in order to adequately prepare trainees for lifelong learning, and further research should examine the effectiveness of specific educational interventions.

Corresponding Author: Donna M. Windish, MD, MPH, Yale Primary Care Residency Program, 64 Robbins St, Waterbury, CT 06708 (donna.windish@yale.edu).

Author Contributions: Dr Windish had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Windish, Huot, Green.

Acquisition of data: Windish.

Analysis and interpretation of data: Windish, Green.

Drafting of the manuscript: Windish, Green.

Critical revision of the manuscript for important intellectual content: Windish, Huot, Green.

Statistical analysis: Windish, Green.

Administrative, technical, or material support: Huot.

Study supervision: Green.