Background
Medication toxic effects and drug-related problems can have profound medical and safety consequences for older adults and economically affect the health care system. The purpose of this initiative was to revise and update the Beers criteria for potentially inappropriate medication use in adults 65 years and older in the United States.
Methods
This study used a modified Delphi method, a set of procedures and methods for formulating a group judgment for a subject matter in which precise information is lacking. The criteria reviewed covered 2 types of statements: (1) medications or medication classes that should generally be avoided in persons 65 years or older because they are either ineffective or they pose unnecessarily high risk for older persons and a safer alternative is available and (2) medications that should not be used in older persons known to have specific medical conditions.
Results
This study identified 48 individual medications or classes of medications to avoid in older adults and their potential concerns and 20 diseases/conditions and medications to be avoided in older adults with these conditions. Of these potentially inappropriate drugs, 66 were considered by the panel to have adverse outcomes of high severity.
Conclusions
This study is an important update of previously established criteria that have been widely used and cited. The application of the Beers criteria and other tools for identifying potentially inappropriate medication use will continue to enable providers to plan interventions for decreasing both drug-related costs and overall costs and thus minimize drug-related problems.
TOXIC EFFECTS of medications and drug-related problems can have profound medical and safety consequences for older adults and economically effect the health care system. Thirty percent of hospital admissions in elderly patients may be linked to drug-related problems or drug toxic effects.1 Adverse drug events (ADEs) have been linked to preventable problems in elderly patients such as depression, constipation, falls, immobility, confusion, and hip fractures.1,2 A 1997 study of ADEs found that 35% of ambulatory older adults experienced an ADE and 29% required health care services (physician, emergency department, or hospitalization) for the ADE.1 Some two thirds of nursing facility residents have ADEs over a 4-year period.3Of these ADEs, 1 in 7 results in hospitalization.4
Recent estimates of the overall human and economic consequences of medication-related problems vastly exceed the findings of the Institute of Medicine (IOM) on deaths from medical errors, estimated to cost the nation $8 billion annually.5 In 2000, it is estimated that medication-related problems caused 106 000 deaths annually at a cost of $85 billion.6 Others have calculated the cost of medication-related problems to be $76.6 billion to ambulatory care, $20 billion to hospitals, and $4 billion to nursing home facilities.2,7,8If medication-related problems were ranked as a disease by cause of death, it would be the fifth leading cause of death in the United States.9 The prevention and recognition of drug-related problems in elderly patients and other vulnerable populations is one of the principal health care quality and safety issues for this decade.
The aforementioned IOM report has focused increased attention on finding solutions for unsafe medication practices, polypharmacy, and drug-related problems in the care of older adults. There are many ways to define medication-related problems in elderly patients, including the use of lists containing specific drugs to avoid in the elderly and appropriateness indexes applied by pharmacists or clinicians.1,10,11 Systematic review of the evidence-based literature on medication use in elderly patients is another approach to defining the problem, but the number of controlled studies on medication use in elderly patients is limited.
The use of consensus criteria for safe medication use in elderly patients is one approach to developing reliable and explicit criteria when precise clinical information is lacking. The two most widely used consensus criteria for medication use in older adults are the Beers criteria and the Canadian criteria.12-14 The Beers criteria are based on expert consensus developed through an extensive literature review with a bibliography and questionnaire evaluated by nationally recognized experts in geriatric care, clinical pharmacology, and psychopharmacology using a modified Delphi technique to reach consensus. The Beers criteria have been used to survey clinical medication use, analyze computerized administrative data sets, and evaluate intervention studies to decrease medication problems in older adults. The Beers criteria were also adopted by the Centers for Medicare & Medicaid Services (CMS) in July 1999 for nursing home regulation. Previous studies have shown these criteria to be useful in decreasing problems in older adults.15-19 These criteria, though controversial at times, have been widely used over the past 10 years for studying prescribing patterns within populations, educating clinicians, and evaluating health outcomes, cost, and utilization data.20-23
A recently published study of potentially inappropriate medication (PIM) use with the Beers criteria in a Medicare-managed care population found a PIM prevalence of 23% (541/2336). Those receiving a PIM had significantly higher total, provider, and facility costs and a higher mean number of inpatient, outpatient, and emergency department visits than comparisons after controlling for sex, Charlson Comorbidity Index, and total number of prescriptions.20 Other studies have found that specific PIMs such as nonsteroidal anti-inflammatory drugs (NSAIDs) and benzodiazepines have been associated with adverse outcomes and increased costs.18 In contrast, a recent study on the relationship between inappropriate drug use, functional status decline, and mortality in 3234 patients from the Duke cohort did not find an association with mortality and inappropriate drug use as determined by the Beers criteria after controlling for covariates.24
In summary, these criteria have been used extensively for evaluating and intervening in medication use in older adults over the past decade. However, with the continuous arrival of new drugs on the market, increased knowledge about older drugs, and removal of older drugs from the market, these criteria must be updated on a regular basis to remain useful. Since the criteria were published in 1997, there has been an increase in the number of scientific studies addressing drug use and appropriateness in older adults, but there is still a lack of controlled studies in the older population and particularly in patients older than 75 years and patients with multiple comorbidities.23
The purpose of this initiative was to revise and update the Beers criteria for ambulatory and nursing facility populations older than 65 years in the United States. There were 3 main aims: (1) to reevaluate the 1997 criteria to include new products and incorporate new information available from the scientific literature, (2) to assign or reevaluate a relative rating of severity for each of the medications, and (3) to identify any new conditions or considerations not addressed in the 1997 criteria.
There were 5 phases in the data collection for this study: (1) the review of the literature, (2) creation and mailing of the round 1 questionnaire, (3) creation of the second-round questionnaire based on round 1 and expert panel feedback, (4) convening of the expert panel and panel responses to the second-round questionnaire, and (5) completion and analysis of a third and final mailed questionnaire that measured the severity ratings of the PIMs to create the final revised list.
The criteria reviewed covered 2 types of statements: (1) medications or medication classes that should generally be avoided in persons 65 years or older because they are either ineffective or they pose unnecessarily high risk for older persons and a safer alternative is available and (2) medications that should not be used in older persons known to have specific medical conditions. The 2 statements each used a 5-point Likert scale and ask respondents to rate their agreement or disagreement with the statement from strongly agree (1) to strongly disagree (5), with the midpoint (3) expressing equivocation. The second type of question asked the respondents to evaluate the medication appropriateness given certain conditions or diagnoses (Figure 1). All questions included an option to not answer if the respondent did not feel qualified to answer. This methodology was similar to that used by Beers et al13 in the creation of the first 2 iterations of the criteria. The methodology used in the third iteration of the Beers criteria only differed in the number of panelists (13 in 1991; 6 in 1997; and 12 in 2002) and the use of a third-round survey for the severity ratings, which was done (in person) in the 1997 update of the criteria.
The modified Delphi method is a technique to arrive at a group consensus regarding an issue under investigation that was originally developed at the RAND Corporation (Santa Monica, Calif) by Olaf Helmer and Norman Dalkey.25 The Delphi method is a set of procedures and methods for formulating a group judgment for a subject matter in which precise information is lacking (such as medication use in older adults). The Delphi method provides a means to reach consensus within a group of experts. The method relies on soliciting individual (often anonymous) answers to written questions by survey or other type of communication. A series of iterations provides each individual with feedback on the responses of the others in the group. The final responses are evaluated for variance and means to determine which questions the group has reached consensus about, either affirmatively or negatively.
The selection of articles for formulating the survey involved 3 steps and was phase 1 of the study. First, we identified literature published since January 1994 in English, describing or analyzing medication use in community-living (ambulatory) older adults and older adults living in nursing homes. From that, we created a table and bibliography. We used MEDLINE, searching with the following key terms adverse drug reactions, adverse drug events, medication problems, and medications and elderly for all relevant articles published between January 1994 and December 2000. Second, we hand searched and identified additional references from the bibliographies of relevant articles. Third, all the panelists were invited to add references and articles after the first survey to add to the literature review. Each study was systematically reviewed by 2 investigators using a table to outline the following information: type of study design; sample size; medications reviewed; summary of results and key points; quality, type and category of medication addressed; and severity of the drug-related problem.
The panel of members were invited to participate via letter by the 4 investigators and a consultant and represented a variety of experience and judgment including extensive clinical practice, extensive publications in this area, and/or senior academic rank. They were also chosen to represent acute, long-term, and community practice settings with pharmacological, geriatric medicine, and psychiatric expertise. Lastly, they were selected from geographically diverse parts of the United States. We initially invited (via regular mail) 16 potential participants with nationally and/or internationally recognized expertise in psychopharmacology, pharmacoepidemiology, clinical geriatric pharmacology, and clinical geriatric medicine to complete our survey. Our response rate for the initial invitation to participate as a panelist was 75% (12/16). Our final panel thus consisted of 12 experts who completed all rounds of the survey.
Data collection and analysis
We used the systematic review of the literature to construct the first round questionnaire. The first-round survey contained 4 sections. Parts 1 and 2 reviewed the latest 1997 criteria. Parts 3 and 4 were medications added for the 2002 update for medications alone (part 3) and medications considering diagnoses and conditions. Parts 3 and 4 included 29 new questions about medications or medication classes and conditions. The last question in part 4 asked panel members to add medications to the list. The panel was then surveyed via Delphi technique to determine concordance/consensus with the round 1 survey and invited to add additional medications prior to and during the second-round meeting.
We created the second and third questionnaires (severity ratings) from panel input and the results of the previous round survey. We completed all mailed and face-to-face rounds between October 2001 and February 2002. We constructed the questionnaire statements according to the original Beers criteria published in 1991 and the updated criteria published in 1997. The instructions accompanying the survey asked the respondents to consider the use of medications only in adults 65 years and older. The second-round survey included the statements included from round 1 and any statements added by the experts from the first round. In the second round and the face-to-face meeting, the respondents were given information about their answers and the anonymous answers of the other members of the group and were given the opportunity to reconsider their previous response.
After analyzing the responses from the first round of the survey, we examined each question for inclusion or exclusion in the revised criteria or for further consideration in the second round of the survey. We calculated the mean rating and corresponding 95% confidence interval (CI) of each statement or dosing question collected from the first round of the survey. Those statements whose upper limit of the 95% CI was less than 3.0 were included in the updated criteria. Those statements or dosing questions whose lower limit of the 95% CI was greater than 3.0 were excluded from the updated criteria. Statements whose 95% CI included the value of 3.0 were included for further determination in the second-round face-to-face meeting.
The face-to-face meeting was convened on December 10, 2001, in Atlanta, Ga. Each panel member was given the results of the first-round survey and the added medications (from the other panel members) to review approximately 10 days before the meeting. For statements that needed further examination (neither included or excluded during round 1), each rater was given his or her previous rating and the mean rating of the group of experts in the second survey.
Any additional statements or dosing questions that had been made on the open-ended portion of the first round of the survey by any expert was included in the survey for the second round. Forty-four questions were added by expert panelists during round 1 of the survey, and 9 questions were added during the round 2 in-person survey and voted on during the in-person meeting. These questions/medications made up part 5 of the survey. Twenty-four questions from parts 3 and 4 had 95% CIs greater than 3.0 after the round 1 survey. During the second-round face-to-face meeting, the group debated these remaining statements and then rerated them using the same Likert scale. The mean rating and 95% CI were calculated. The technique used for the first round for inclusion or exclusion of the statement or dosing question in the updated criteria was used. Those statements whose 95% CI included 3.0 were excluded from the updated criteria. Lastly, in January 2002, we surveyed panelists on a 5-point scale for the severity of the potential medication problem.
The final criteria are listed in Table 1 and Table 2. Table 1 contains 48 individual medications or classes of medications to avoid in older adults and their potential concerns. Table 2 lists 20 diseases or conditions and medications to be avoided in older adults with these conditions. Sixty-six of these potentially inappropriate drugs were considered by the panel to have adverse outcomes of high severity. New conditions and diagnoses that were addressed this time included depression, cognitive impairment, Parkinson disease, anorexia, and malnutrition, syndrome of inappropriate antidiuretic hormone secretion, and obesity.
A total of 15 medications/medication classes were dropped or modified from the 1997 to the 2002 update from the round 1 survey. Most of the medications dropped since 1997 were those that were associated with diagnoses or conditions. The following medications were voted to be dropped or modified from the criteria by the panelists since the 1997 publication: phenylbutazone, oxybutynin chloride, β-blockers, corticosteroids with persons with diabetes; sedative-hypnotics in persons with chronic obstructive pulmonary disease; β-blockers in persons with asthma; β-blockers in persons with peripheral vascular disorder; β-blockers in persons with syncope and falls; narcotics in persons with bladder outflow obstruction; and theophylline sodium glycinate in persons with insomnia (Table 3). Oxybutinin was modified by not including the extended-release formula, which the panel believed had fewer adverse effects. Reserpine was changed to be avoided only at doses greater than 0.25 mg, and disopyramide phosphate avoidance now only refers to the non–extended release formulation. New information about β-blockers in elderly patients led the panel to drop this class of drugs from the list. The other criteria dropped involved use of drugs in the setting of a comorbid condition or drugs that are off the market. The expert panelists could not reach consensus about adding questions regarding setting maximum dosages for sedative-hypnotics, antipsychotics, selective serotonin reuptake inhibitors, and tricyclic antidepressants that do not have specific recommendations from the manufacturer, though there was agreement that consideration of changes in pharmacokinetics were important in older patients in preventing problems caused by excessive dosages and usage.
This update also includes several medications that have new information or have come to market since the last study of the Beers criteria was published (1997), including selective serotonin reuptake inhibitors, amiodarone, and fluoxetine hydrochloride. The panel also voted to add methyltestosterones, amphetamines, and bupropion hydrochloride to the list of medications to be avoided in older adults.Table 1 and Table 2 state why medications were added since 1997, and Table 3 summarizes all the changes to the Beers criteria since 1997, including medications that were added, dropped, or modified.
This study is an important update of previously established criteria that have been widely used and cited.16,20,22,23,26-29 The application of the Beers criteria and other tools for identifying PIM use will continue to enable providers to plan interventions for decreasing both drug-related costs and overall costs and thus minimize drug-related problems.9,30 Such tools are also vitally important to managed care organizations, pharmacy benefit plans, and both acute and long-term health care institutions. However, to remain useful, criteria must be regularly updated and must take into account the ever-increasing, evidence-based literature in the area of medication use in older adults.
The argument in favor of using explicit criteria in prescribing practice is overwhelming: improvements in therapeutic practices and reduction in medication-related ADEs will increase the quality of care and enhance patient outcome at the same time as optimizing resource utilization and promoting fiscal prudence. These criteria, though widely used, have been controversial because of their adoption by nursing home regulators and have been criticized at times as too simplistic and limiting the freedom of physicians to prescribe.31-35 However, we believe that thoughtful application of the updated 2002 Beers criteria and other tools for identifying PIM use can enable providers and insurers to plan interventions aimed at decreasing drug-related costs and overall health care costs, while reducing ADE-related admissions in elderly patients9,30 and improving care. The updated Beers criteria will enable everyone from individual physicians to health care systems to integrate the new criteria-based prescribing recommendations into their organic, mechanical, and electronic information systems.
The proponents of explicit criteria and evidence-based prescribing are among the biggest players in the health care industry: the IOM, the CMS, the Agency for Healthcare Research and Quality (AHRQ), and the American Association of Health Plans (AAHP), to name but four.36,37 Indeed, finding a voice of dissent is challenging. In "Crossing the Quality Chasm" the IOM38 presents a template for the future, when the traditional values of physician integrity, altruism, knowledge, skill, and dedication to lifelong patient care are seamlessly integrated into an information era of point-of-care, computerized decision support that facilitates appropriate care using the available resources. The updated Beers criteria are one component of that movement, enabling all parties, from providers to insurers, to integrate our recommendations into their clinical information systems.
Given the aforementioned, there appears to be a potential niche for the Beers criteria in fulfilling the missions of the IOM, CMS, AHRQ, and AAHP. However, translating research into measurable quality improvement may be more challenging. In the first instance, despite the much-lauded public statements about quality by many (including the above organizations), there is widespread recognition that perhaps cost containment is the principal driver of change in the health care world.39 Individual health care providers and organizations will demand objective evidence that implementation of the updated Beers criteria (or, indeed, other inappropriate medication guides) will result in objective, quantifiable improvements in the clinical effectiveness and cost-effectiveness of health care services. To date, despite extensive literature demonstrating association—based on retrospective studies on administrative data—there is an absence of rigorous, prospective research in this field. We (D.M.F., J.L.W., and J.R.M.) are completing a randomized controlled study among a Medicare managed care population at this time, using the 1997 medication criteria for older adults. Well-controlled studies are needed that show prospectively that using these criteria make a difference in patient outcomes.31
These criteria have some limitations, however, and must be regularly updated to remain useful to both clinicians, health care administrators, and researchers. These criteria are meant to apply to the general population of patients 65 years and older, thus some that are not appropriate for significantly older or more frail persons do not appear in this list. These criteria are not meant to regulate practice in a manner to which they supersede the clinical judgment and assessment of the physician or practitioner. In addition, defining inappropriate medications by specific lists of medications rather than other mechanisms may miss some problems such as the underuse and interactions of drugs in older people.26,40 A true meta-analysis was not conducted for this study. Lastly, this study has the same limitations previously documented regarding the use of the Delphi technique.25,41
A further challenge to adoption of the Beers criteria will come from the information systems and information technology sector. Despite phenomenal advances in hardware and software, decision support systems continue to have significant limitations, and presenting the right information to the right person at the point of clinical need remains a challenge for the information systems and information technology engineer, the behavior change specialist, and the medical profession.42
Corresponding author and reprints: Donna M. Fick, PhD, RN, Center for Health Care Improvement, Department of Medicine, Medical College of Georgia, HB 2010, 1467 Harper St, Augusta, GA 30912 (e-mail: dfick@mail.mcg.edu).
Accepted for publication March 28, 2003.
This research was supported by a grant from the Medical College of Georgia (Augusta) and University of Georgia (Athens) Combined Intramural Grant Program.
We thank Judy Johnson, MAT, R. C. Robinson, BS, and Alison Maclean, BA, for assistance with data management and manuscript preparation. We acknowledge the following individuals for contributing their expertise to this study as panel members: Maude Babington, PharmD (Babington Consulting, LLC, Boulder, Colo); Manju T. Beier, PharmD (The University of Michigan, Ann Arbor); Richard W. Besdine, MD (Brown University, Providence, RI); Jack Fincham, PhD (University of Kansas, Lawrence); F. Michael Gloth III, MD (Johns Hopkins University School of Medicine, Baltimore, Md); Thomas Jackson, MD (Medical College of Georgia, Augusta); John E. Morley, MD (Saint Louis University Health Sciences Center, St Louis, Mo); Becky Nagle, PharmD, BCPC (Medco Health Solutions, Franklin Lakes, NJ); Todd Semla, PharmD, MS (Evanston Northwestern Healthcare, Evanston, Ill); Mark A. Stratton, PharmD (University of Oklahoma, Oklahoma City); Andrew D. Weinberg, MD (Emory University School of Medicine, Atlanta, Ga).
This article was corrected on 12/18/03.
1.Hanlon
JTSchmader
KEKornkowski
MJ
et al. Adverse drug events in high risk older outpatients.
J Am Geriatr Soc. 1997;45945- 948
PubMedGoogle Scholar 2.Bootman
JLHarrison
DLCox
E The health care cost of drug-related morbidity and mortality in nursing facilities.
Arch Intern Med. 1997;1572089- 2096
PubMedGoogle ScholarCrossref 3.Cooper
JW Probable adverse drug reactions in a rural geriatric nursing home population: a four-year study.
J Am Geriatr Soc. 1996;44194- 197
PubMedGoogle Scholar 4.Cooper
JW Adverse drug reaction-related hospitalizations of nursing facility patients: a 4-year study.
South Med J. 1999;92485- 490
PubMedGoogle ScholarCrossref 6.Perry
DP When medicine hurts instead of helps.
Consultant Pharmacist. 1999;141326- 1330
Google Scholar 7.Bates
DWSpell
NCullen
DJ
et al. the Adverse Drug Events Prevention Study Group, The costs of adverse drug events in hospitalized patients.
JAMA. 1997;277307- 311
PubMedGoogle ScholarCrossref 9.Lazarou
JPomeranz
BHCorey
PN Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies.
JAMA. 1998;2791200- 1205
Google ScholarCrossref 10.Hanlon
JTLandsman
PBCowan
K
et al. Physician agreement with pharmacist-suggested drug therapy changes for elderly outpatients.
Am J Health Syst Pharm. 1996;532735- 2737
PubMedGoogle Scholar 11.Hanlon
JTSchmader
KESamsa
GP
et al. A method for assessing drug therapy appropriateness.
J Clin Epidemiol. 1992;451045- 1051
PubMedGoogle ScholarCrossref 12.Beers
MH Explicit criteria for determining potentially inappropriate medication use by the elderly.
Arch Intern Med. 1997;1571531- 1536
PubMedGoogle ScholarCrossref 13.Beers
MHOuslander
JGRollingher
JReuben
DBBeck
JC Explicit criteria for determining inappropriate medication use in nursing home residents.
Arch Intern Med. 1991;1511825- 1832
PubMedGoogle ScholarCrossref 14.McLeod
JPHuang
ARTamblyn
RM Defining inappropriate practices in prescribing for elderly people: a national consensus panel.
CMAJ. 1997;156385- 391
PubMedGoogle Scholar 15.Doucet
JChassagne
PTrivalle
C
et al. Drug-drug interactions related to hospital admissions in older adults: a prospective study of 1000 patients.
J Am Geriatr Soc. 1996;44944- 948
PubMedGoogle Scholar 16.Golden
AGPreston
RABarnett
SDLiorente
MHamdan
KSilverman
MA Inappropriate medication prescribing in homebound older adults.
J Am Geriatr Soc. 1999;47948- 953
PubMedGoogle Scholar 17.Mort
JRAparasu
RR Prescribing potentially inappropriate psychotropic medications to the ambulatory elderly.
Arch Intern Med. 2000;1602825- 2831
PubMedGoogle ScholarCrossref 18.Smalley
WEGriffin
MR The risks and costs of upper gastrointestinal disease attributable to NSAIDs.
Gastroenterol Clin North Am. 1996;25373- 396
PubMedGoogle ScholarCrossref 19.Thapa
PBGideon
PCost
TWMilam
ABRay
WA Antidepressants and the risk of falls among nursing home residents.
N Engl J Med. 1998;339875- 882
PubMedGoogle ScholarCrossref 20.Fick
DMWaller
JLMaclean
JR
et al. Potentially inappropriate medication use in a Medicare managed care population: association with higher costs and utilization.
J Managed Care Pharm. 2001;7407- 413
Google Scholar 21.Kaufman
DWKelly
JPRosenberg
LAnderson
TEMitchell
AA Recent patterns of medication use in the ambulatory adult population of the United States: the Slone survey.
JAMA. 2002;287337- 344
PubMedGoogle ScholarCrossref 24.Hanlon
JFillenbaum
GKuchibhatla
M
et al. Impact of inappropriate drug use on mortality and functional status in representative community dwelling elders.
Med Care. 2002;40166- 176
PubMedGoogle ScholarCrossref 25.Dalkey
NBrown
BCochran
S The Delphi Method, III: Use of Self Ratings to Improve Group Estimates. Santa Monica, Calif Rand Corp November1969;Publication RM-6115-PR.
27.Aparasu
RRFliginger
SE Inappropriate medication prescribing for the elderly by office-based physicians.
Ann Pharmacother. 1997;31823- 829
PubMedGoogle Scholar 28.Dhalla
IAnderson
GMandani
MBronskill
SSykora
KRochon
P Inappropriate prescribing before and after nursing home admission.
J Am Geriatr Soc. 2002;50995- 1000
PubMedGoogle ScholarCrossref 29.Sloane
PZimmerman
SBrown
LIves
TWalsh
J Inappropriate medication prescribing in residential care/assisted living facilities.
J Am Geriatr Soc. 2002;501001- 1011
PubMedGoogle ScholarCrossref 30.Cooper
JWWade
WE Repeated unnecessary NSAID-associated hospitalizations in an elderly female: a case report.
Geriatr Drug Ther. 1997;1295- 97
Google ScholarCrossref 38.Institute of Medicine, Crossing the Quality Chasm. Washington, DC Institute of Medicine Press2001;
39.DesHarnais
SIFortham
MTHoma-Lowry
JMWooster
LD Risk-adjusted clinical quality indicators: indices for measuring and monitoring rates of mortality, complications, and readmissions.
Qual Manag Health Care. 2000;914- 22
PubMedGoogle ScholarCrossref 40.Hanlon
JTSchmader
KRuby
CWeinberger
M Suboptimal prescribing in older inpatients and outpatients.
J Am Geriatr Soc. 2001;49200- 209
PubMedGoogle ScholarCrossref 41.Hasson
FKeeney
SMckenna
H Research guidelines for the Delphi survey technique.
J Adv Nurs. 2000;321008- 1015
PubMedGoogle Scholar