Estimating Time to Conduct a Meta-analysis From Number of Citations Retrieved

To the Editor: The number of meta-analyses in the health sciences has been dramatically increasing. By 1996, approximately 300 meta-analyses in medicine had been published,¹ and in 1997, this number had increased to more than 400 (Bruce Kupelnick, written communication, August 1998). The movement toward evidence-based medicine suggests that these numbers will continue to increase. The reliability and value of meta-analysis methods have been questioned since its introduction.² User friendly meta-analytic tools may exacerbate the problem, as they have for regression analyses.³

When a researcher prepares to perform a meta-analysis, knowledge of the time a meta-analysis will take would be useful both for grant proposals and for realistic planning. One published meta-analysis on the association among ovarian cancer, reproductive variables, and contraceptive use⁴ measured time spent on task for completing a meta-analysis of summary data compared with known time to do a meta-analysis of individual patient data, and the documented time for meta-analysis was more than 1000 hours. How does one estimate the length of time required to carry out a first-rate research synthesis? The size of the body of evidence, quality and complexity, the reviewer pool, and support services all may have an effect. We suggest that a useful and practical correlate to the time necessary to carry out a meta-analysis is the size of the total body of literature (before any deletions) in the field.

MetaWorks, a private company engaging in meta-analysis, documented time for various tasks in 37 meta-analyses (unpublished project summary, MetaWorks Inc, Boston, Mass, August 1998). The mean total number of hours was 1139 (median, 1110), with a wide range from 216 to 2518 hours.

The component mean (SD) times were (1) preanalysis search, retrieval, and database development: 588 (337) hours; (2) statistical analysis: 144 (106) hours; (3) report and manuscript writing: 206 (125) hours; and (4) other (administrative): 201 (193) hours. Item 1 includes protocol development, searches, library retrieval, abstract management, study matrix construction, paper screening and blinding, data extraction and quality scoring, data entry, and data matrix construction. Item 4 includes proposal development, project-specific correspondence, project meetings and administration, project management, and training.

In these 37 meta-analyses, we have observed a reasonable association between the number of initial citations (before exclusion criteria are invoked) and total time that it takes to complete a meta-analysis (Figure 1). A quadratic regression plus a 95% confidence band was fit to these data. Although we might have expected the total time to be zero when there were no citations, we noted that there was a certain amount of start-up time even when there were few citations. The quadratic equation is as follows: Total time=721 + 0.243x − 0.0000123x², where x denotes the number of citations before exclusion criteria are applied. The predicted start-up time is 721 hours (95% confidence interval, 478-964 hours) as can be seen in Figure 1. Variability increases when the number of citations is less than 1000, in part because some parts of the initial stages of the analysis require relatively constant periods of time. However, the scatterplot does convey the major time investment required to conduct a meta-analysis.

A researcher might use the following scenario. Suppose that an initial search uncovers 2500 citations. The predicted time to complete the meta-analysis is 1251 hours. From the proportion (1251/1139)=1.10, we obtain, as predicted time for tasks 1 through 4, 647, 158, 226, and 221 hours, respectively, yielding a total of 1251 hours. These estimates may be useful to researchers embarking on a meta-analysis or writing a grant proposal to fund a meta-analysis.

Financial Disclosure: Dr Allen is a stockholder in MetaWorks Inc.