[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
February 12, 1997

Clinical Prediction RulesA Review and Suggested Modifications of Methodological Standards

Author Affiliations

From the Departments of Medicine (Drs Laupacis, Sekar, and Stiell) and Epidemiology and Community Medicine (Drs Laupacis and Stiell), University of Ottawa, and the Clinical Epidemiology Unit, Loeb Medical Research Institute, Ottawa Civic Hospital (Drs Laupacis and Stiell), Ottawa, Ontario.

JAMA. 1997;277(6):488-494. doi:10.1001/jama.1997.03540300056034

Background.  —Clinical prediction rules are decision-making tools for clinicians, containing variables from the history, physical examination, or simple diagnostic tests.

Objective.  —To review the quality of recently published clinical prediction rules and to suggest methodological standards for their development and evaluation.

Data Sources.  —Four general medical journals were manually searched for clinical prediction rules published from 1991 through 1994.

Study Selection.  —Four hundred sixty potentially eligible reports were identified, of which 30 were clinical prediction rules eligible for study. Most methodological standards could only be evaluated in 29 studies.

Data Abstraction.  —Two investigators independently evaluated the quality of each report using a standard data sheet. Disagreements were resolved by consensus.

Data Synthesis.  —The mathematical technique was used to develop the rule, and the results of the rule were described in 100% (29/29) of the reports. All the rules but 1 (97% [28/29]) were felt to be clinically sensible. The outcomes and predictive variables were clearly defined in 83% (24/29) and 59% (17/29) of the reports, respectively. Blind assessment of outcomes and predictive variables occurred in 41% (12/29) and 79% (23/29) of the reports, respectively, and the rules were prospectively validated in 79% (11/14). Reproducibility of predictive variables was assessed in only 3% (1/29) of the reports, and the effect of the rule on clinical use was prospectively measured in only 3% (1/30). Forty-one percent (12/29) of the rules were felt to be easy to use.

Conclusions.  —Although clinical prediction rules comply with some methodological criteria, for other criteria, better compliance is needed.

Mark DB, Shaw L, Harrell FE, et al.  Prognostic value of a treadmill exercise score in outpatients with suspected coronary artery disease . N Engl J Med . 1991;325:849-853.Article
Stiell IG, McKnight RD, Greenberg GH, et al.  Implementation of the Ottawa Ankle Rules . JAMA . 1994;271:827-832.Article
Wasson JH, Sox HC, Neff RK, Goldman L.  Clinical prediction rules: application and methodological standards . N Engl J Med . 1985;313:793-799.Article
Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Worthington JR.  A study to develop clinical decision rules for the use of radiography in acute ankle injuries . Ann Emerg Med . 1992;21:384-390.Article
Stiell IG, Greenberg GH, McKnight RD, et al.  Decision rules for the use of radiography in acute ankle injuries: refinement and prospective validation . JAMA . 1993;269:1127-1132.Article
Stiell I, Wells G, Laupacis A, et al.  Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries . BMJ . 1995;311:594-597.Article
Stiell IG, Greenberg GH, Wells GA, et al.  Prospective validation of a decision rule for the use of radiography in acute knee injuries . JAMA . 1996;275:611-615.Article
Sauve JS, Thorpe KE, Sackett DL, et al.  Can bruits distinguish high-grade from moderate symptomatic carotid stenosis? Ann Intern Med . 1994;120:633-637.Article
Farr BM, Sloman AJ, Fisch MJ.  Predicting death in patients hospitalized for community-acquired pneumonia . Ann Intern Med . 1991;115:428-436.Article
Marottoli RA, Cooney LM, Wagner R, Doucette J, Tinetti ME.  Predictors of automobile crashes and moving violations among elderly drivers . Ann Intern Med . 1994;121:842-846.Article
Silver MT, Rose GA, Paul SD, O'Donnell CJ, O'Gara PT, Eagle KA.  A clinical rule to predict preserved left ventricular ejection fraction in patients after myocardial infarction . Ann Intern Med . 1994;121:750-756.Article
Pryor DB, Shaw L, McCants CB, et al.  Value of the history and physical in identifying patients at increased risk for coronary artery disease . Ann Intern Med . 1993;118:81-90.Article
Williams JW, Simel DL, Roberts L, Samsa GP.  Clinical evaluation of sinusitis: making the diagnosis by history and physical examination . Ann Intern Med . 1992;117:705-710.Article
Ashton CM, Petersen NJ, Wray NP, et al.  The incidence of perioperative myocardial infarction in men undergoing noncardiac surgery . Ann Intern Med . 1993;118:504-510.Article
The Stroke Prevention in Atrial Fibrillation Investigators.  Predictors of thromboembolism in atrial fibrillation, I: clinical features of patients at risk . Ann Intern Med . 1992;116:1-5.Article
The Stroke Prevention in Atrial Fibrillation Investigators.  Predictors of thromboembolism in atrial fibrillation, II: echocardiographic features of patients at risk . Ann Intern Med . 1992;116:6-12.Article
Morrow K, Morris CK, Frolicher VF, et al.  Prediction of cardiovascular death in men undergoing noninvasive evaluation for coronary artery disease . Ann Intern Med . 1993;118:689-695.Article
Zarchy TM, Ershoff D.  Which clinical variables predict an abnormal double-contrast barium enema result? Ann Intern Med . 1991;114:137-141.Article
Inouye SK, Viscoli CM, Horwitz RI, Hurst LD, Tinetti ME.  A predictive model for delirium in hospitalized elderly medical patients based on admission characteristics . Ann Intern Med . 1993;119:474-481.Article
Nguyen T, Sambrook P, Kelly P, et al.  Prediction of osteoporotic fractures by postural instability and bone density . BMJ . 1993;307:1111-1115.Article
Medical Research Council Antiepileptic Drug Withdrawal Study Group.  Prognostic index for recurrence of seizures after remission of epilepsy . BMJ . 1993;306:1374-1378.Article
van Duijn NP, Brouwer HJ, Lamberts H.  Use of symptoms and signs to diagnose maxillary sinusitis in general practice: comparison with ultrasonography . BMJ . 1992;305:684-687.Article
Skene AI, Smith JM, Dore CJ, Charlett A, Lewis JD.  Venous leg ulcers: a prognostic index to predict time to healing . BMJ . 1992;305:1119-1121.Article
Poungvarin N, Viriyavejakul A, Komontri C.  Siriraj stroke score and validation study to distinguish supratentorial intracerebral hemorrhage from infarction . BMJ . 1991;302:1565-1567.Article
Celani MG, Righetti E, Migliacci R, et al.  Comparability and validity of two clinical scores in the early differential diagnosis of acute stroke . BMJ . 1994;308:1674-1676.Article
Dramaix M, Hennart P, Brasseur D, et al.  Serum albumin concentration, arm circumference, and oedema and subsequent risk of dying in children in Central Africa . BMJ . 1993;307:710-713.Article
Parsons RW, Jamrozik KD, Hobbs MST, Thompson DL.  Early identification of patients at low risk of death after myocardial infarction and potentially suitable for early hospital discharge . BMJ . 1994;308:1006-1010.Article
Nouri FM, Lincoln NB.  Predicting driving performance after stroke . BMJ . 1993;307:482-483.Article
Bates DW, Lee TH.  Rapid classification of positive blood cultures: prospective validation of a multivariate algorithm . JAMA . 1992;267:1962-1966.Article
Marcantonio ER, Goldman L, Mangione CM, et al.  A clinical prediction rule for delirium after elective noncardiac surgery . JAMA . 1994;271:134-139.Article
Browner WS, Li J, Mangano DT, for the Study of Perioperative Ischemia Research Group.  In-hospital and long-term mortality in male veterans following noncardiac surgery . JAMA . 1992;268:228-232.Article
Chodak GW, Vogelzang NJ, Caplan RJ, Soloway M, Smith JA.  Independent prognostic factors in patients with metastatic (stage D2) prostate cancer . JAMA . 1991;265:618-621.Article
Hollenberg M, Mangano DT, Browner WS, et al.  Predictors of postoperative myocardial ischemia in patients undergoing noncardiac surgery . JAMA . 1992;268:205-209.Article
Cooper-Patrick L, Crum RM, Ford DE.  Identifying suicidal ideation in general medical patients . JAMA . 1994;272:1757-1762.Article
The International Non-Hodgkins Lymphoma Prognostic Factors Project.  A predictive model for aggressive non-Hodgkin's lymphoma . N Engl J Med . 1993;329:987-994.Article
Lee TH, Juarez G, Cook EF, et al.  Ruling out acute myocardial infarction: a prospective multicenter validation of a 12-hour strategy for patients at low risk . N Engl J Med . 1991;324:1239-1246.Article
Williams M, Warwick D, Bannister GC.  Death and thromboembolic disease after total hip replacement without routine chemical prophylaxis . Thromb Haemost . 1995;73:953.
Wells PS, Hirsh J, Anderson DR, et al.  Accuracy of clinical assessment of deep-vein thrombosis . Lancet . 1995;345:1326-1330.Article
Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine . 2nd ed. Boston, Mass: Little Brown & Co Inc; 1991:179-180.
Jutras D.  Clinical practice guidelines as legal norms . Can Med Assoc J . 1993;148:905-908.
Ciampi A, Hogg SA, McKinney S, Thiffault J.  RECPAM: A computer program for recursive partitioning and amalgamation for censored survival data and other situations frequently occurring in biostatistics, I: methods and program features . Comput Methods Programs Biomed . 1988;26:239-256.Article
Ciampi A, Chang CH, Hogg S, McKinney S.  Recursive partitioning: a versatile method for exploratory data analysis in biostatistics . In: MacNeill IB, Umphrey GJ, eds. Time Series and Econometric Modelling/Biostatistics . Boston, Mass: D Reidel Publishing Co; 1987:23-50.
Concato J, Feinstein AR, Holford TR.  The risk of determining risk with multivariable models . Ann Intern Med . 1993;118:201-210.Article
Baxt WG.  Use of an artificial neural network for the diagnosis of myocardial infarction . Ann Intern Med . 1991;115:843-848.Article
Shufflebarger CM, Young W.  What is a neural network? Ann Emerg Med . 1992;21:1461-1462.Article
The PIOPED Investigators.  Value of the ventilation/perfusion scan in acute pulmonary embolism: results of the prospective investigation of pulmonary embolism diagnosis (PIOPED) . JAMA . 1990;263:2753-2759.Article
Rifkin MD, Zerhouni EA, Gatsonis CA, et al.  Comparison of magnetic resonance imaging and ultrasonography in staging early prostate cancer: results of a multi-institutional cooperative trial . N Engl J Med . 1990;323:621-626.Article
Hanley JA, McNeil BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve . Radiology . 1982;143:29-36.
Fletcher RH, Fletcher SW, Wagner EH. Clinical Epidemiology: The Essentials . 2nd ed. Baltimore, Md: Williams & Wilkins; 1988.
Kerr L, Kelly A, Grant J, et al.  Failed validation of a clinical decision rule for the use of radiography in acute ankle injury . N Z Med J . 1994;107:294-295.
Stiell IG, Greenberg GH, McKnight RD, Wells GA.  Ottawa ankle rules for radiography of acute ankle injuries . N Z Med J . 1995;108:111.
Fleiss JL. Statistical Methods for Rates and Proportions . 2nd ed. New York, NY: John Wiley & Sons Inc; 1981.
Feinstein AR. Clinimetrics . New Haven, Conn: Yale University Press; 1987.
Kirshner B, Guyatt G.  A methodological framework for assessing health indices . J Chronic Dis . 1985;38:27-36.Article
The Standards of Reporting Trials Group.  A proposal for structured reporting of randomized controlled trials . JAMA . 1994;272:1926-1931.Article
The Asilomar Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature.  Checklist of information for inclusion in reports of clinical trials . Ann Intern Med . 1996;124:741-743.Article