Quality of Systematic Reviews of Economic Evaluations in Health Care

Context Reviews performed almost a decade ago showed considerable gaps in the quality of reporting and methods applied to economic evaluations of health care interventions. Measures taken by the research community to address the issue included the promulgation of guidelines and the publicizing of good practice in economic evaluation.

Methods To assess the quality of methods of systematic reviews, economic evaluations in health care, and reporting methods, we conducted full-text searches of private and public databases for the period 1990 through March 2001 and corresponded with researchers active in the field. A total of 102 reports were identified, but only 39 were included. Quality of systematic reviews was assessed by a 6-item checklist.

Results Quality of review methods was reasonable, but more attention needs to be paid to search methods and standardization of evaluation instruments. The reviews found consistent evidence of serious methodological flaws in a significant number of economic evaluations. Lack of clear descriptions of methods, lack of explanation and justification for the framework and approach used, and low-quality estimates of effectiveness for the interventions evaluated were the most frequent flaws. Modest improvements in quality of conducting and reporting economic evaluations appear to have taken place in the last decade.

Conclusions Proper allocation of resources on the basis of economic evaluations remains uncertain. Editorial teams and regulatory bodies should perform quality assurance based on a single widely accepted and validated standard instrument.

Economic evaluations (analytical studies comparing costs and outcomes of investing resources in ≥1 alternatives) have increased in availability and acceptance as a tool for decision making in health care in the last 2 decades.^1,2 However, the costs of decisions based on methodologically weak evidence are widely accepted.³ A number of reviews published in the period 1990-1994 illustrated the variability of the methods used in conducting and reporting economic evaluations.³ Although the findings could be partly explained by possible variations in review methods and by the known absence of editorial policies to assess economic evaluations prior to publication,^4,5 initiatives aimed at increasing the uniformity, quality, and reporting of economic evaluations were undertaken. The initiatives (production of guidelines for regulatory bodies for submission and editorial management in medical journals and further research into the quality of economic evaluation methods) should have led to an increase in the quality of economic evaluations during the last years of the previous decade. We examined systematic reviews of economic evaluations in health care to assess the quality of methods used in the reviews and the quality of conducting and reporting economic evaluations in the last decade.⁶

We searched for studies from the period 1990 to March 2001 in all languages on a variety of databases, corresponded with members of the International Health Economist Association, and handsearched issues of Health Economics from 1992 to March 2001. A detailed description of search strategy, sources, and terms used is available in the online Appendix (available in PDF format).

Two reviewers examined each citation for relevance. Those deemed relevant were retrieved in full. Two reviewers compared each study against the selection criteria independently, resolving disagreements by discussion and, when necessary, the third reviewer adjudicated. We included systematic reviews of economic evaluations of health care interventions defined as studies assessing methodological quality using explicit criteria. We identified and retrieved 102 reports of reviews possibly satisfying our inclusion criteria. Fifty-four were excluded from further analysis, 9 are awaiting assessment, and the remaining 39 were included. References to the 54 excluded reviews and 9 awaiting assessment are available in the online Appendix (available in PDF format).

For each included review, we extracted author(s) and year of study, topic and study question, type (ie, cost-benefit analysis) and number of included economic evaluations, year of publication or preparation of included economic evaluations, instrument used to assess quality of included economic evaluations, and main study conclusions. Quality of systematic review methods was assessed using the following criteria that were adapted from different sources^7-9: (1) Is it unlikely that important relevant studies were missed? (2) Were the inclusion criteria used to select articles appropriate? (3) Was the assessment of studies reproducible? (4) Were the design and/or methods and/or topic of included studies broadly comparable? (5) How reproducible are the overall results? (6) Will the results help resource allocation in health care? Each question was answered with "impossible to judge," "no," "partly," or "yes."

We performed a calculation of Spearman rank-order coefficient correlating inter-reviewer agreement on an initial sample of 20 studies using 2 independent reviewers. As correlation was high (0.98), the remaining studies were assessed by a single reviewer.

Two reviewers extracted data on methods of assessing the quality of economic evaluations included in each of the reviews in our study. As a wide variety of assessment instruments were used, the criteria used in each instrument were grouped and analyzed by variables listed in the BMJ checklist for editors and authors of economic evaluations.^3,10 We hoped that this would enable us to find some common methodological quality items used for assessment in the reviews to allow us to draw some conclusions. We further subdivided grouped items into methodological quality and reporting quality items.

We grouped reviews according to whether they assessed general methodological quality or by intervention, by study design, or by specific methods used in economic evaluations. A summary of the 39 included reviews is in the online Table A (available in PDF format).

Four of the 6 quality criteria (inclusion criteria, reproducibility of assessment, comparability of included economic evaluations, and impact on resource allocation in health care) were fulfilled in at least 75% of reviews. The remaining 2 criteria (thoroughness of searches and reproducibility of overall results) were completely fulfilled in 12% and 73.5% and partly in 52.9% and 23.5% of reviews, respectively. A detailed methodological assessment of each review is available from the corresponding author.

Common search weaknesses were restricted use of databases and lack of efforts to identify unpublished material. Reproducibility of overall review results was hampered by the disparate nature of quality assessment instruments used in the reviews. Twenty-six reviews used ad hoc instruments with a variable number of items (3-25), 5 used the Drummond et al¹¹ 10-item checklist, 5 used the BMJ 35-item checklist,³ and 2 used US panel recommendations.^12-14 One review used a checklist of unclear structure and origin.¹⁵

Funding sources were available for 28 (71%) reviews. Twenty-one (53%) were publicly funded, 2 (5%) were privately funded, and 5 (13%) had mixed funding.

Quality assessment criteria used in each review were compared with those in the BMJ checklist. Many reviews did not use quality assessment instruments that covered all the criteria as the BMJ checklist. In some cases this was because a review focused on a narrow methodological issue.^16-18 Overall, the instruments used appeared to be appropriate to the scope of the reviews.

We included 6 reviews assessing the quality of 644 economic evaluations in health care across a wide range of general and specialty medical journals, different countries and settings, including industry submissions to a reimbursement authority (unpublished data, 2000).^19-24 All identified major flaws in a substantial number of evaluations. The prevalence of major methodological flaws appeared higher in the population assessed by Hill et al,²³ probably because of a higher degree of scrutiny by the Australian reimbursement authority.

We included 19 reviews assessing the quality of 776 economic evaluations (not allowing for the overlap between Demicheli²⁵ and Jefferson²⁶) focusing on vaccines, preventive interventions for human immunodeficiency virus, adjuvant therapy for breast cancer, vascular and orthopedic surgery, and antenatal screening (unpublished data, 2000).^21,25-42 The evaluated interventions were mainly preventive. All included studies reached the same conclusions, albeit with different emphasis, such as the presence of uncertainty due to variable epidemiological assumptions, estimates of effect of evaluated interventions, and poor reporting, writing, or use of methods.

Six reviews assessing the quality of specific economic study design included 5 studies that assessed 362 cost-utility analyses over a time span of 20 years.^43-47 Overall results show a small and slow improvement over the years, but the authors raise concerns about the standard of peer review in some of the smaller specialty journals. We were unable to identify similar depth of scrutiny for other economic study types.

Nine reviews assessing the quality of a broad range of specific methods (statistical analysis of costs, health status measurement, contingent valuation, and cost estimation) in 1407 economic evaluations reported poor methods.^15-18,48-52 All reviews cast serious doubts on the validity of the conclusions reached by the economic evaluations assessed and all propose stricter criteria for quality control.

Eleven reviews assessed and commented on changes in quality of economic evaluations over time. Six reported improvements mostly up to the late 1990s,^{15,23,34,44,47} one reported quality improvement over the 1980s,¹⁹ one the opposite,²⁴ and 4 reported no improvement.^27,28,50,51

The major methodological findings of the reviews are: lack of clarity on study questions, viewpoint, and epidemiological assumptions; unclear conceptual and decision-making context; lack of clear descriptions of methods used to define effectiveness, utilities, benefits, and resource and cost estimates; basic calculation errors in a significant minority of studies; variability in the assumptions underlying the choice of estimates of effect; choice of discount rate and perspective often not explained; and sensitivity analysis more likely to be performed in more recent evaluations.

Although the provision of some descriptive information (study viewpoint, cost basis) may be improving over time, a sizeable proportion of economic evaluations could not justify their conclusions on the basis of methods used. There appeared to be no difference in the methodological quality of conducting and reporting economic evaluations, although evaluation of the former was difficult as few reviews had raw data from the evaluations at their disposal.

Although overall quality of reviews is satisfactory, more attention needs to be paid to search strategies and the use of comparable instruments to assess quality of included studies.

The findings of the reviews indicate the presence of serious methodological flaws in a significant number of economic evaluations of health care interventions, regardless of publication status, period of preparation or publication, topic, or type of evaluation. Overall, there appear to have been some modest, but slow, improvements in quality in the last decade, but the evidence for this observation is thin. There is evidence of lower quality in evaluations published in specialty journals. There is no evidence of language bias, but there is evidence of low quality of unpublished evaluations submitted by the pharmaceutical industry within a reimbursement scheme.

There is evidence of considerable confusion in the design, reporting, and description of economic evaluations. Reviews found a proportion of evaluations of unclassifiable study design, studies that ignored basic research and economic methodological principles, and ones that reported results lacking clarity.

There could be many explanations for our findings, ranging from lack of appreciation by researchers and editorial teams of the complexities of economic evaluation method^23,47 to resistance in accepting that "any method" will no longer suffice,³⁵ or lack of direction in the quality control of economic submissions to journals,⁴ with the exception of the BMJ.^3,46

There are 2 possible major limitations to our descriptive synthesis of results. First, it is possible that a number of primary studies were included more than once in the research synthesis studies included in our review. For example, a cost-utility analysis included in Gerard et al⁴⁶ also could have been included in the analysis by Demicheli and Jefferson.²⁵ If this kind of double counting were extensively present, a proportion of the same poor-quality evaluations could bias the results of our review.

Second, few methodological studies used the same instrument to assess quality, possibly leading to lack of overall comparability of their results. We believe these problems not to have had a major impact on our findings. All included systematic reviews unequivocally point to the variable nature of methods for conducting and reporting economic evaluations and to the slow and modest progress in overall quality over the last decade. This finding appears to be independent of review focus or assessment methods. There appears to be little difference in the conclusions of those reviews using disparate instruments and those which used the same instrument.

We believe that urgent action should be taken to address the problem of poor methods in economic evaluations. First, absolute transparency of reporting is needed, with maximum use of journal Web sites to obviate space constraints.⁵³ Economic models used in evaluations should be readily accessible to reviewers and readers. Second, basic formal training in economic evaluation should be given to all those involved in economic evaluation or their assessment. Third, the use of a validated and accepted instrument for quality assessment is a priority for any future monitoring of economic evaluations. In our view, the BMJ checklist could be adopted by general and specialty journals and regulatory and grant-giving institutions as a quality assessment instrument. Modifications of the BMJ checklist for in-depth scrutiny of particular methodological aspects, such as the ones described by Gerard et al,⁴⁶ should be performed on the basis of the research results. Lastly, we propose continuous monitoring of the quality of economic evaluation methods and more research into specific study designs, often-used interventions, and comparisons of economic evaluations in decision-making and editorial settings.

Caution should be taken when deciding or justifying allocation of resources on the basis of economic evaluations, especially if based on unpublished studies or studies published in specialty journals. Editorial teams, regulatory institutions, and researchers should implement and assess quality assurance based on a single widely accepted and validated standard instrument.