Author Affiliations: Cochrane Health Economics Methods Group, Rome, Italy (Drs Jefferson and Demicheli, and Mr Vale); Health Reviews Ltd, Rome, Italy (Dr Jefferson); Azienda Sanitaria Locale 20, Alessandria, Italy (Dr Demicheli); and Health Services Research Unit and Health Economics Research Unit, Aberdeen, Scotland (Mr Vale).
Context Reviews performed almost a decade ago showed considerable gaps in the
quality of reporting and methods applied to economic evaluations of health
care interventions. Measures taken by the research community to address the
issue included the promulgation of guidelines and the publicizing of good
practice in economic evaluation.
Methods To assess the quality of methods of systematic reviews, economic evaluations
in health care, and reporting methods, we conducted full-text searches of
private and public databases for the period 1990 through March 2001 and corresponded
with researchers active in the field. A total of 102 reports were identified,
but only 39 were included. Quality of systematic reviews was assessed by a
Results Quality of review methods was reasonable, but more attention needs to
be paid to search methods and standardization of evaluation instruments. The
reviews found consistent evidence of serious methodological flaws in a significant
number of economic evaluations. Lack of clear descriptions of methods, lack
of explanation and justification for the framework and approach used, and
low-quality estimates of effectiveness for the interventions evaluated were
the most frequent flaws. Modest improvements in quality of conducting and
reporting economic evaluations appear to have taken place in the last decade.
Conclusions Proper allocation of resources on the basis of economic evaluations
remains uncertain. Editorial teams and regulatory bodies should perform quality
assurance based on a single widely accepted and validated standard instrument.
Economic evaluations (analytical studies comparing costs and outcomes
of investing resources in ≥1 alternatives) have increased in availability
and acceptance as a tool for decision making in health care in the last 2
decades.1,2 However, the costs
of decisions based on methodologically weak evidence are widely accepted.3 A number of reviews published in the period 1990-1994
illustrated the variability of the methods used in conducting and reporting
economic evaluations.3 Although the findings
could be partly explained by possible variations in review methods and by
the known absence of editorial policies to assess economic evaluations prior
to publication,4,5 initiatives
aimed at increasing the uniformity, quality, and reporting of economic evaluations
were undertaken. The initiatives (production of guidelines for regulatory
bodies for submission and editorial management in medical journals and further
research into the quality of economic evaluation methods) should have led
to an increase in the quality of economic evaluations during the last years
of the previous decade. We examined systematic reviews of economic evaluations
in health care to assess the quality of methods used in the reviews and the
quality of conducting and reporting economic evaluations in the last decade.6
We searched for studies from the period 1990 to March 2001 in all languages
on a variety of databases, corresponded with members of the International
Health Economist Association, and handsearched issues of Health Economics from 1992 to March 2001. A detailed description of
search strategy, sources, and terms used is available in the online Appendix
(available in PDF format).
Two reviewers examined each citation for relevance. Those deemed relevant
were retrieved in full. Two reviewers compared each study against the selection
criteria independently, resolving disagreements by discussion and, when necessary,
the third reviewer adjudicated. We included systematic reviews of economic
evaluations of health care interventions defined as studies assessing methodological
quality using explicit criteria. We identified and retrieved 102 reports of
reviews possibly satisfying our inclusion criteria. Fifty-four were excluded
from further analysis, 9 are awaiting assessment, and the remaining 39 were
included. References to the 54 excluded reviews and 9 awaiting assessment
are available in the online Appendix (available in PDF format).
For each included review, we extracted author(s) and year of study,
topic and study question, type (ie, cost-benefit analysis) and number of included
economic evaluations, year of publication or preparation of included economic
evaluations, instrument used to assess quality of included economic evaluations,
and main study conclusions. Quality of systematic review methods was assessed
using the following criteria that were adapted from different sources7- 9: (1) Is it unlikely
that important relevant studies were missed? (2) Were the inclusion criteria
used to select articles appropriate? (3) Was the assessment of studies reproducible?
(4) Were the design and/or methods and/or topic of included studies broadly
comparable? (5) How reproducible are the overall results? (6) Will the results
help resource allocation in health care? Each question was answered with "impossible
to judge," "no," "partly," or "yes."
We performed a calculation of Spearman rank-order coefficient correlating
inter-reviewer agreement on an initial sample of 20 studies using 2 independent
reviewers. As correlation was high (0.98), the remaining studies were assessed
by a single reviewer.
Two reviewers extracted data on methods of assessing the quality of
economic evaluations included in each of the reviews in our study. As a wide
variety of assessment instruments were used, the criteria used in each instrument
were grouped and analyzed by variables listed in the BMJ checklist for editors and authors of economic evaluations.3,10 We hoped that this would enable us
to find some common methodological quality items used for assessment in the
reviews to allow us to draw some conclusions. We further subdivided grouped
items into methodological quality and reporting quality items.
We grouped reviews according to whether they assessed general methodological
quality or by intervention, by study design, or by specific methods used in
economic evaluations. A summary of the 39 included reviews is in the online Table A
(available in PDF format).
Four of the 6 quality criteria (inclusion criteria, reproducibility
of assessment, comparability of included economic evaluations, and impact
on resource allocation in health care) were fulfilled in at least 75% of reviews.
The remaining 2 criteria (thoroughness of searches and reproducibility of
overall results) were completely fulfilled in 12% and 73.5% and partly in
52.9% and 23.5% of reviews, respectively. A detailed methodological assessment
of each review is available from the corresponding author.
Common search weaknesses were restricted use of databases and lack of
efforts to identify unpublished material. Reproducibility of overall review
results was hampered by the disparate nature of quality assessment instruments
used in the reviews. Twenty-six reviews used ad hoc instruments with a variable
number of items (3-25), 5 used the Drummond et al11
10-item checklist, 5 used the BMJ 35-item checklist,3 and 2 used US panel recommendations.12- 14
One review used a checklist of unclear structure and origin.15
Funding sources were available for 28 (71%) reviews. Twenty-one (53%)
were publicly funded, 2 (5%) were privately funded, and 5 (13%) had mixed
Quality assessment criteria used in each review were compared with those
in the BMJ checklist. Many reviews did not use quality
assessment instruments that covered all the criteria as the BMJ checklist. In some cases this was because a review focused on a
narrow methodological issue.16- 18
Overall, the instruments used appeared to be appropriate to the scope of the
We included 6 reviews assessing the quality of 644 economic evaluations
in health care across a wide range of general and specialty medical journals,
different countries and settings, including industry submissions to a reimbursement
authority (unpublished data, 2000).19- 24
All identified major flaws in a substantial number of evaluations. The prevalence
of major methodological flaws appeared higher in the population assessed by
Hill et al,23 probably because of a higher
degree of scrutiny by the Australian reimbursement authority.
We included 19 reviews assessing the quality of 776 economic evaluations
(not allowing for the overlap between Demicheli25
and Jefferson26) focusing on vaccines, preventive
interventions for human immunodeficiency virus, adjuvant therapy for breast
cancer, vascular and orthopedic surgery, and antenatal screening (unpublished
data, 2000).21,25- 42
The evaluated interventions were mainly preventive. All included studies reached
the same conclusions, albeit with different emphasis, such as the presence
of uncertainty due to variable epidemiological assumptions, estimates of effect
of evaluated interventions, and poor reporting, writing, or use of methods.
Six reviews assessing the quality of specific economic study design
included 5 studies that assessed 362 cost-utility analyses over a time span
of 20 years.43- 47
Overall results show a small and slow improvement over the years, but the
authors raise concerns about the standard of peer review in some of the smaller
specialty journals. We were unable to identify similar depth of scrutiny for
other economic study types.
Nine reviews assessing the quality of a broad range of specific methods
(statistical analysis of costs, health status measurement, contingent valuation,
and cost estimation) in 1407 economic evaluations reported poor methods.15- 18,48- 52
All reviews cast serious doubts on the validity of the conclusions reached
by the economic evaluations assessed and all propose stricter criteria for
Eleven reviews assessed and commented on changes in quality of economic
evaluations over time. Six reported improvements mostly up to the late 1990s,15,23,34,44,47
one reported quality improvement over the 1980s,19
one the opposite,24 and 4 reported no improvement.27,28,50,51
The major methodological findings of the reviews are: lack of clarity
on study questions, viewpoint, and epidemiological assumptions; unclear conceptual
and decision-making context; lack of clear descriptions of methods used to
define effectiveness, utilities, benefits, and resource and cost estimates;
basic calculation errors in a significant minority of studies; variability
in the assumptions underlying the choice of estimates of effect; choice of
discount rate and perspective often not explained; and sensitivity analysis
more likely to be performed in more recent evaluations.
Although the provision of some descriptive information (study viewpoint,
cost basis) may be improving over time, a sizeable proportion of economic
evaluations could not justify their conclusions on the basis of methods used.
There appeared to be no difference in the methodological quality of conducting
and reporting economic evaluations, although evaluation of the former was
difficult as few reviews had raw data from the evaluations at their disposal.
Although overall quality of reviews is satisfactory, more attention
needs to be paid to search strategies and the use of comparable instruments
to assess quality of included studies.
The findings of the reviews indicate the presence of serious methodological
flaws in a significant number of economic evaluations of health care interventions,
regardless of publication status, period of preparation or publication, topic,
or type of evaluation. Overall, there appear to have been some modest, but
slow, improvements in quality in the last decade, but the evidence for this
observation is thin. There is evidence of lower quality in evaluations published
in specialty journals. There is no evidence of language bias, but there is
evidence of low quality of unpublished evaluations submitted by the pharmaceutical
industry within a reimbursement scheme.
There is evidence of considerable confusion in the design, reporting,
and description of economic evaluations. Reviews found a proportion of evaluations
of unclassifiable study design, studies that ignored basic research and economic
methodological principles, and ones that reported results lacking clarity.
There could be many explanations for our findings, ranging from lack
of appreciation by researchers and editorial teams of the complexities of
economic evaluation method23,47
to resistance in accepting that "any method" will no longer suffice,35 or lack of direction in the quality control of economic
submissions to journals,4 with the exception
of the BMJ.3,46
There are 2 possible major limitations to our descriptive synthesis
of results. First, it is possible that a number of primary studies were included
more than once in the research synthesis studies included in our review. For
example, a cost-utility analysis included in Gerard et al46
also could have been included in the analysis by Demicheli and Jefferson.25 If this kind of double counting were extensively
present, a proportion of the same poor-quality evaluations could bias the
results of our review.
Second, few methodological studies used the same instrument to assess
quality, possibly leading to lack of overall comparability of their results.
We believe these problems not to have had a major impact on our findings.
All included systematic reviews unequivocally point to the variable nature
of methods for conducting and reporting economic evaluations and to the slow
and modest progress in overall quality over the last decade. This finding
appears to be independent of review focus or assessment methods. There appears
to be little difference in the conclusions of those reviews using disparate
instruments and those which used the same instrument.
We believe that urgent action should be taken to address the problem
of poor methods in economic evaluations. First, absolute transparency of reporting
is needed, with maximum use of journal Web sites to obviate space constraints.53 Economic models used in evaluations should be readily
accessible to reviewers and readers. Second, basic formal training in economic
evaluation should be given to all those involved in economic evaluation or
their assessment. Third, the use of a validated and accepted instrument for
quality assessment is a priority for any future monitoring of economic evaluations.
In our view, the BMJ checklist could be adopted by
general and specialty journals and regulatory and grant-giving institutions
as a quality assessment instrument. Modifications of the BMJ checklist for in-depth scrutiny of particular methodological aspects,
such as the ones described by Gerard et al,46
should be performed on the basis of the research results. Lastly, we propose
continuous monitoring of the quality of economic evaluation methods and more
research into specific study designs, often-used interventions, and comparisons
of economic evaluations in decision-making and editorial settings.
Caution should be taken when deciding or justifying allocation of resources
on the basis of economic evaluations, especially if based on unpublished studies
or studies published in specialty journals. Editorial teams, regulatory institutions,
and researchers should implement and assess quality assurance based on a single
widely accepted and validated standard instrument.
Jefferson T, Demicheli V, Vale L. Quality of Systematic Reviews of Economic Evaluations in Health Care. JAMA. 2002;287(21):2809-2812. doi:10.1001/jama.287.21.2809