Context Comparisons of risk-adjusted hospital performance often are important components of public reports, pay-for-performance programs, and quality improvement initiatives. Risk-adjustment equations used in these analyses must contain sufficient clinical detail to ensure accurate measurements of hospital quality.
Objective To assess the effect on risk-adjusted hospital mortality rates of adding present on admission codes and numerical laboratory data to administrative claims data.
Design, Setting, and Patients Comparison of risk-adjustment equations for inpatient mortality from July 2000 through June 2003 derived by sequentially adding increasingly difficult-to-obtain clinical data to an administrative database of 188 Pennsylvania hospitals. Patients were hospitalized for acute myocardial infarction, congestive heart failure, cerebrovascular accident, gastrointestinal tract hemorrhage, or pneumonia or underwent an abdominal aortic aneurysm repair, coronary artery bypass graft surgery, or craniotomy.
Main Outcome Measures C statistics as a measure of the discriminatory power of alternative risk-adjustment models (administrative, present on admission, laboratory, and clinical for each of the 5 conditions and 3 procedures).
Results The mean (SD) c statistic for the administrative model was 0.79 (0.02). Adding present on admission codes and numerical laboratory data collected at the time of admission resulted in substantially improved risk-adjustment equations (mean [SD] c statistic of 0.84 [0.01] and 0.86 [0.01], respectively). Modest additional improvements were obtained by adding more complex and expensive to collect clinical data such as vital signs, blood culture results, key clinical findings, and composite scores abstracted from patients' medical records (mean [SD] c statistic of 0.88 [0.01]).
Conclusions This study supports the value of adding present on admission codes and numerical laboratory values to administrative databases. Secondary abstraction of difficult-to-obtain key clinical findings adds little to the predictive power of risk-adjustment equations.
Risk-adjusted hospital mortality rates for specified conditions and procedures frequently are used in public reports and pay-for-performance programs as indicators of the quality of hospital care.1-3 Risk adjustment often is based solely on administrative claims data from uniform bills that hospitals submit to payers. These data lack clinically important pathophysiological information and do not distinguish between conditions that were present on admission (POA; ie, potential risk factors) and complications that occurred during hospitalization. The validity of risk-adjustment systems that use only administrative data has been challenged repeatedly,4-9 and there is general agreement10-23 that additional data are required to predict accurately an individual patient's risk of dying.
Physicians are particularly concerned that inadequate risk adjustment penalizes the practitioners and facilities that care for the sickest patients and may result in the denial of needed care to high-risk patients.24-26 Consumer advocates and payers, through initiatives such as the Consumer-Purchaser Disclosure Project,27 are attempting to expand administrative data sets to include clinical information to ensure that incentives reward high-quality clinical care. On the other hand, many hospital administrators have complained that the cost of retrieving supplementary clinical data from medical records is prohibitive, and some researchers have argued that risk adjustment using only administrative data can be made sufficiently accurate to support valid comparisons among hospitals.28,29
The addition of a POA modifier for secondary diagnosis codes first was proposed in 199130 and was successfully adopted in New York State in 1994 and in California in 1996.31 The planned implementation in March 2007 of new standards for hospital claims data includes nationwide adoption of this modifier, which distinguishes conditions that develop during hospital stays (potential complications of care) from conditions that were present at admission (potential treatment-independent risk factors). Inclusion of POA codes in administrative data sets should permit analysts to incorporate important predictors of inpatient mortality into administrative risk-adjustment equations without improperly designating patients as having high intrinsic risks at admission when their increased vulnerability resulted from hospital-acquired complications.31
The adoption of Logical Observation Identifiers Names and Codes32 for laboratory data and advances in electronic health data technology have lowered the cost of retrieving numerical laboratory data at many hospitals.33 Because of substantial differences in the cost of obtaining various types of clinical data, limited enhancement of administrative data sets appears to be both practical and desirable. Ideally, clinical data elements selected for this purpose will be relatively inexpensive to obtain and will be useful predictors of mortality for multiple conditions and procedures.
This study was designed to test the hypothesis that the combination of POA modifiers for secondary diagnoses and a limited set of numerical laboratory data would improve risk adjustment of inpatient mortality for a diverse set of clinical conditions and procedures. We also hypothesized that further additions of highly specific, difficult-to-obtain clinical data sometimes considered important predictors of inpatient mortality by clinicians would add little to the accuracy of predictive models.
Risk-adjustment models were created and analyzed using data from July 2000 through June 2003 from 188 Pennsylvania hospitals supplied by the Pennsylvania Health Care Cost Containment Council.34 Case-level claims data were supplemented with clinical data abstracted from medical records by specially trained personnel using MediQual's proprietary Atlas clinical information system.35 This system defines a broad array of clinical data elements, including historical information, laboratory results, vital signs, clinical symptoms and signs, pathophysiological abnormalities, and composite pathophysiological scores, which are collected and stored along with the hospital day on which each clinical finding was observed.
Risk-adjusted mortality rates were analyzed for 5 health conditions (acute myocardial infarction, congestive heart failure, acute cerebrovascular accident, gastrointestinal tract hemorrhage, or pneumonia) and 3 surgical procedures (abdominal aortic aneurysm repair, coronary artery bypass graft surgery, or craniotomy). The Agency for Healthcare Research and Quality's Inpatient Quality Indicator software version 2.1 was used to identify cases that met criteria for inclusion in each group.36
Four models were constructed for each condition and procedure (1 set of data for each of the 5 conditions and 1 set of data for each of the 3 procedures). The first model termed administrative used standard claims data. The second model termed POA used data abstracted from medical records to determine whether coded secondary diagnoses had been present at admission. The third model termed laboratory used POA codes and numerical laboratory data (often available in electronic form; eg, creatinine, hematocrit level) documented on the first day of hospitalization prior to a procedure requiring general or regional anesthesia. The fourth model termed clinical used the criteria in the third model plus vital signs, other laboratory data not included in the third model (eg, bacterial culture results), Atlas key clinical findings abstracted from medical records (eg, immunocompromised, lethargy), and composite clinical scores (ie, American Society of Anesthesiologists classification, Glasgow Coma Score) documented on the first day of hospitalization prior to a procedure requiring general or regional anesthesia.
The administrative model was based solely on data from hospital bills (ie, age, sex, and principal diagnoses, secondary diagnoses, and procedures coded according to the International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM]). To avoid using hospital-acquired complications as risk factors, hospital bills from New York and California (secondary diagnoses were modified by POA codes in these states) were used to help identify which secondary diagnoses were generally present at admission. Secondary diagnoses were eligible for inclusion as risk factors in the administrative model only when they were coded as hospital-acquired complications in fewer than 20% of cases in which they occurred.
The POA model included additional secondary diagnoses excluded from the administrative model because of their association with unacceptably high rates of complications. Because Pennsylvania claims data do not include POA codes, clinical data in the Atlas database were used to determine whether coded secondary diagnoses were present at admission. In the creation of surrogate POA codes, the Atlas database served as a substitute for the complete medical record available to coders in New York and California. For example, posthemorrhagic anemia was excluded from the administrative model for congestive heart failure because hospitals in New York and California coded it as acquired during hospitalization in more than 30% of the cases in which it occurred. However, posthemorrhagic anemia was eligible for inclusion as a risk factor in the POA model for congestive heart failure when the Atlas database documented that anemia was present on the day of admission.
For each condition or procedure, candidate risk factors were constructed from principal diagnosis codes, up to 8 secondary diagnosis codes, up to 6 procedure codes, and clinical data elements associated with higher than average mortality rates. Infrequently occurring codes were combined with codes for clinically similar conditions or procedures that had similar mortality rates. Continuous measures (eg, age, creatinine level) were transformed into 1 or more categorical variables based on clinical judgment and empirical evaluation of associated mortality rates.
For each condition or procedure, stratified random samples of live discharges and fatalities were combined to create 3 mutually exclusive data sets: a training set (50%), a validation set (25%), and a test set (25%). Partitioning the data in this way facilitated the construction of more robust models.
A preliminary predictive equation was developed on the training set for each condition or procedure using only age categories and individual hospital identifiers. (Including hospitals as risk factors during model development is a standard technique for reducing possible bias caused by the associations between the prevalence of potential risk factors at individual hospitals and the quality of care provided by those hospitals.) For the administrative, POA, and laboratory models, additional potential risk factors were added in a sequence determined by forward stepwise logistic regression.37 To avoid overfitting, variables added after the minimum value of the Schwarz criterion38 was attained were removed from models. (This criterion weighs the trade-off between the fit of a model and its complexity.) The remaining predictive variables and their coefficients were evaluated for clinical plausibility. On rare occasions, clinically problematic variables were eliminated or modified. To avoid substituting more expensive clinical variables for less costly ones with almost equivalent predictive power, predictive variables selected for the laboratory model were retained and additional clinical risk factors were added in sequence as described above.
For each of the 4 models (administrative, POA, laboratory, and clinical) for each condition or procedure, a nested sequence of models was created first with 1 variable selected using the training data set, then with 2 variables, and lastly with all the variables. Variables were added to successive models in the order in which they were entered in the minimum Schwarz criterion model. From each nested sequence of models, the validation set was used to select the model with the smallest average prediction error39 as the final validated model. Finally, the coefficients of the variables in the validated models were retained, the hospital variables were removed, and the intercepts were recalculated to equate observed and predicted mortality rates.
Case-level discriminatory power (ie, the ability of a model to distinguish cases that died from those that survived) was computed on the test set using c statistics.40 All data management and statistical analyses were performed using SAS software versions 8 and 9.1 (SAS Institute Inc, Cary, NC). The study design was approved by the Abt Associates' institutional review board.
The numbers of cases ranged from 5309 (abdominal aortic aneurysm repair) to 200 506 (congestive heart failure) (Table). Mortality rates ranged from 3.2% for coronary artery bypass graft surgery to 10.8% for acute cerebrovascular accident.
Designating secondary diagnoses as present at admission increased the average number of secondary diagnosis variables included from 8.6 in the administrative model to 15.4 in the POA model. This increase occurred because secondary diagnoses such as acute renal failure in patients admitted to the hospital with pneumonia, who also had elevated creatinine levels on the day of admission, were eligible for inclusion as risk factors in the POA model. Comparison of the POA model and the laboratory model revealed that the addition of an average of 11.1 numerical laboratory values present on the first hospital day was accompanied by an average reduction of 4.5 secondary diagnosis variables. This reduction reflected the substitution of more specific laboratory values for less specific secondary diagnosis variables (eg, pH ≤7.25 or pH >7.25 but ≤7.35 replaced the ICD-9-CM secondary diagnosis code for acidosis in acute myocardial infarction). Compared with the laboratory model, an average of 9 additional clinical findings present on the first hospital day were incorporated into the clinical model.
The final models included a total of 20 numerical laboratory determinations, 3 other laboratory determinations (eg, blood cultures), 5 vital signs, 22 key clinical findings, and 2 composite scores. Many individual numerical laboratory results and vital signs appeared in the clinical models for 4 or more conditions or procedures (eg, pH and prothrombin time were risk factors in the clinical models for all 5 conditions and 3 procedures). On the other hand, few key clinical findings appeared in the models for more than 2 of the conditions and procedures. The Glasgow Coma Score was a risk factor for 3 of the 5 conditions and 1 of the 3 procedures and the American Society of Anesthesiologists classification was a risk factor for the 2 other procedures.
The receiver operating characteristic curves reflecting the average c statistics of alternative models are shown in the Figure. The average c statistic increased from 0.50 for no risk adjustment to a mean (SD) of 0.79 (0.02) for the administrative model, to 0.84 (0.01) for the POA model, to 0.86 (0.01) for the laboratory model, to 0.88 (0.01) for the clinical model.
This study was designed to guide the selection of a cost-effective set of clinical data elements to improve the validity of comparisons of risk-adjusted hospital mortality rates. Because these comparisons often are important components of public reports, pay-for-performance programs, and quality improvement initiatives, it is essential that they accurately reflect the quality of care provided by each facility.41 Unlike most previous studies that attempted to derive the most parsimonious or most sophisticated risk-adjustment model for a single condition or procedure or to compare models based on administrative data to corresponding models based on clinical data, the principal goal of this study was to evaluate the relative performances of alternative equations based on progressively more detailed data sets and identify one that could meet the sometimes conflicting needs of physicians, hospital administrators, and payers. Therefore, a diverse sample of conditions and procedures was evaluated, and methodological uniformity was emphasized to minimize the confounding effects of differences in analytic technique and to obtain precise estimates of improvements in the risk adjustment directly attributable to changing the type of data available for use in the predictive equations.
In deriving the administrative and POA models, care was taken to avoid using risk factors based on conditions or procedures that reflected potentially avoidable hospital-acquired complications rather than intrinsic patient risks at the time of admission. For both the administrative and POA models, the use of procedure codes as risk factors was limited to situations in which they were found to be irreplaceable surrogates for intrinsic patient risk. In the administrative model, secondary diagnoses were eligible for use as risk factors only when a separate analysis documented that they only rarely reflected hospital-acquired complications. In the POA model, secondary diagnoses ineligible for the administrative model were considered as potential risk factors only if the Atlas clinical data substantiated their presence on the first day of hospitalization.
A national standard for adding a POA code to administrative claims data in the UB-04 (the uniform bill used to submit all hospital claims to payers) is planned as part of the revised ICD-9-CM coding modifications for 2007.31,42 In this study, the use of a surrogate for this code resulted in noteworthy improvements in the performance of the risk-adjustment models, confirming the value of this new coding convention. Substantial additional improvements in the performance of risk-adjustment models occurred when numerical laboratory values were added to the POA codes. The sum of all further improvements from adding other clinical data elements was substantially less than improvements achieved by adding surrogate POA coding and numerical laboratory values to the standard administrative data.
Data collected by the Pennsylvania Health Care Cost Containment Council43 demonstrated that when hospitals routinely collect a specified set of clinical data elements on a large number of discharges, they can reduce the cost of retrieving these data by investing in standardized record formats and electronic aids to data collection and by training less expensive personnel to obtain required data quickly and accurately. In addition, a recent study44 by HIMSS Analytics found that 80.8% of hospitals had the computerized laboratory systems required to support the laboratory models. Therefore, many hospitals currently should be capable of electronically merging administrative and numerical laboratory data, thereby reducing their costs of acquiring laboratory data by eliminating the need for manual abstraction. In contrast, only 10.6% of hospitals currently have computerized nursing documentation required to support models that include vital signs and other clinical data, although rapid improvements in health information technology are anticipated over the next few years.33
The present study was limited by its use of only 1 indicator of hospital quality (ie, mortality) and by its failure to evaluate directly the effects of variations in coding practices and in the number of secondary diagnosis codes included in the centralized databases. Recommendations about the inclusion of specific data elements within each level of clinical data may not apply in all circumstances because some data elements not identified in this study might prove to be important for outcomes and conditions outside the scope of this investigation. In addition, measures of function were not available but have been shown to have value in predicting outcomes.45
In summary, this analysis strongly supports the value of enhancing administrative claims data with POA codes and a limited set of numerical laboratory values obtained at admission. These data provide information required to avoid errors in the designation of hospitals and their medical staffs as delivering better than average or worse than average care. On the other hand, secondary abstraction of difficult-to-obtain key clinical findings appears to add little to the risk adjustment of inpatient mortality rates.
Corresponding Author: Michael Pine, MD, MBA, 1210 Chicago Ave, Suite 503, Evanston, IL 60202 (mpine@aol.com).
Author Contributions: Dr Pine had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Pine, Jordan, Elixhauser, Hoaglin.
Acquisition of data: Jordan, Elixhauser, Jones.
Analysis and interpretation of data: Pine, Jordan, Fry, Hoaglin, Jones, Meimban, Warner, Gonzales.
Drafting of the manuscript: Pine, Jordan, Fry, Jones, Meimban, Warner.
Critical revision of the manuscript for important intellectual content: Jordan, Elixhauser, Hoaglin, Gonzales.
Statistical analysis: Pine, Jordan, Hoaglin, Jones, Meimban.
Obtained funding: Pine, Jordan, Elixhauser.
Administrative, technical, or material support: Pine, Jordan, Elixhauser, Warner.
Study supervision: Pine, Jordan, Elixhauser.
Financial Disclosures: Dr Hoaglin reported owning shares of stock in Cardinal Health Inc. None of the other authors reported any disclosures.
Funding/Support: This study was supported by the Agency for Healthcare Research and Quality contract 233-02-0088 (task order 13).
Role of the Sponsor: The Agency for Healthcare Research and Quality (AHRQ) specified the overarching study design in a request for proposal. Data were collected by Pennsylvania hospitals, which were required by law to submit these data to the Pennsylvania Health Care Cost Containment Council. Data were transmitted to the AHRQ by the Council and the AHRQ transmitted them to the authors. Further data management, analyses, interpretation, and preparation of the manuscript were the independent work of the authors. The manuscript was reviewed by the Council and by the director of the Center for Delivery, Organizations, and Markets at AHRQ prior to the initial submission. Complete specifications of data elements, potential risk factors, and risk-adjustment equations are available in the final report submitted to the AHRQ. Administrative and Atlas clinical data were provided by the Council, an independent state agency responsible for addressing the problem of escalating health costs, ensuring the quality of health care, and increasing access to health care for all citizens regardless of ability to pay. The Council provided data in an effort to further its mission of educating the public and containing health care costs in Pennsylvania. The Council, its agents and staff, have made no representation, guarantee, or warranty, express or implied, that the data are error-free, or that the use of the data will avoid differences of opinion or interpretation. Analyses reported in this article were not prepared by the Council.
Disclaimer: The Pennsylvania Health Care Cost Containment Council, its agents and staff, bear no responsibility or liability for the results of these analyses, which are solely the opinion of the authors.
3.McKay NL, Deily ME. Comparing high- and low-performing hospitals using risk-adjusted excess mortality and cost inefficiency.
Health Care Manage Rev. 2005;30:347-36016292012
Google ScholarCrossref 4.Hannan EL, Kilburn H Jr, Lindsey ML.
et al. Clinical versus administrative data bases for CABG surgery: does it matter?
Med Care. 1992;30:892-9071405795
Google ScholarCrossref 5.Jollis JG, Ancukiewicz M, DeLong ER.
et al. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research.
Ann Intern Med. 1993;119:844-8508018127
Google ScholarCrossref 6.Iezzoni LI, Ash AS, Coffman GA.
et al. Predicting in-hospital mortality: a comparison of severity measurement approaches.
Med Care. 1992;30:347-3591556882
Google ScholarCrossref 7.Iezzoni LI, Ash AS, Shwartz M.
et al. Predicting who dies depends on how severity is measured: implications for evaluating patient outcomes.
Ann Intern Med. 1995;123:763-7707574194
Google ScholarCrossref 8.Hannan EL, Racz MJ, Jollis JG.
et al. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough?
Health Serv Res. 1997;31:659-6789018210
Google Scholar 9.Pine M, Norusis M, Jones B.
et al. Predictions of hospital mortality rates: a comparison of data sources.
Ann Intern Med. 1997;126:347-3549054278
Google ScholarCrossref 10.Knaus WA, Zimmerman JE, Wagner DP.
et al. APACHE-acute physiology and chronic health evaluation: a physiologically based classification system.
Crit Care Med. 1981;9:591-5977261642
Google ScholarCrossref 11.Brewster AC, Karlin BG, Hyde LA.
et al. MEDISGRPS: a clinically based approach to classifying hospital patients at admission.
Inquiry. 1985;22:377-3872934332
Google Scholar 12.Daley J, Jencks S, Draper D.
et al. Predicting hospital-associated mortality for Medicare patients: a method for patients with stroke, pneumonia, acute myocardial infarction, and congestive heart failure.
JAMA. 1988;260:3617-36243057251
Google ScholarCrossref 13.Hannan EL, Kilburn H Jr, O'Donnell JF.
et al. Adult open heart surgery in New York State: an analysis of risk factors and hospital mortality rates.
JAMA. 1990;264:2768-27742232064
Google ScholarCrossref 14.Smith DW, Pine M, Bailey RC.
et al. Using clinical variables to estimate the risk of patient mortality.
Med Care. 1991;29:1108-11291943271
Google ScholarCrossref 15.Higgins TL, Estafanous FG, Loop FD.
et al. Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients: a clinical severity score.
JAMA. 1992;267:2344-23481564774
Google ScholarCrossref 16.Lemeshow S, Teres D, Klar J.
et al. Mortality probability models (MPM II) based on an international cohort of intensive care unit patients.
JAMA. 1993;270:2478-24868230626
Google ScholarCrossref 17.Van Ruiswyk J, Hartz A, Kuhn E.
et al. A measure of mortality risk for elderly patients with acute myocardial infarction.
Med Decis Making. 1993;13:152-1608483400
Google ScholarCrossref 18.Khuri SF, Daley J, Henderson W.
et al. The Department of Veterans Affairs' NSQIP [National VA Surgical Quality Improvement Program]: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care.
Ann Surg. 1998;228:491-5079790339
Google ScholarCrossref 19.Pine M, Jones B, Lou YB. Laboratory values improve predictions of hospital mortality.
Int J Qual Health Care. 1998;10:491-5019928588
Google ScholarCrossref 20.Gibbs J, Cull W, Henderson W.
et al. Preoperative serum albumin level as a predictor of operative mortality and morbidity.
Arch Surg. 1999;134:36-429927128
Google ScholarCrossref 21.Ghali WA, Faris PD, Galbraith PD.
et al. Sex differences in access to coronary revascularization after cardiac catheterization: importance of detailed clinical data.
Ann Intern Med. 2002;136:723-73212020140
Google ScholarCrossref 22.Grover FL, Shroyer AL, Hammermeister K.
et al. A decade's experience with quality improvement in cardiac surgery using the Veterans Affairs and Society of Thoracic Surgeons national databases.
Ann Surg. 2001;234:464-47411573040
Google ScholarCrossref 23.Lee DS, Austin PC, Rouleau JL.
et al. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model.
JAMA. 2003;290:2581-258714625335
Google ScholarCrossref 24.Schneider EC, Epstein AM. Influence of cardiac surgery performance reports on referral practices and access to care: a survey of cardiovascular specialists.
N Engl J Med. 1996;335:251-2568657242
Google ScholarCrossref 25.Moscucci M, Eagle KA, Share D.
et al. Public reporting and case selection for percutaneous coronary interventions: an analysis from two large multicenter percutaneous coronary intervention databases.
J Am Coll Cardiol. 2005;45:1759-176515936602
Google ScholarCrossref 26.Narins CR, Dozier AM, Ling FS.
et al. The influence of public reporting of outcome data on medical decision making by physicians.
Arch Intern Med. 2005;165:83-8715642879
Google ScholarCrossref 28.Krumholz HM, Wang Y, Mattera JA.
et al. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction.
Circulation. 2006;113:1683-169216549637
Google ScholarCrossref 29.Krumholz HM, Wang Y, Mattera JA.
et al. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure.
Circulation. 2006;113:1693-170116549636
Google ScholarCrossref 30.Naessens JM, Brennan MD, Boberg CJ.
et al. Acquired conditions: an improvement to hospital discharge abstracts.
Qual Assur Health Care. 1991;3:257-2621790324
Google ScholarCrossref 31.Coffey R, Milenkovic M, Andrews RM.US Agency for Healthcare Research and Quality. The case for the present on admission (POA) indicator: HCUP methods series report 2006-01 [June 26, 2006].
http://www.hcup-us.ahrq.gov/reports/2006_1.pdf. Accessed November 6, 2006 33.Berner ES, Detmer DE, Simborg D. Will the wave finally break? a brief view of the adoption of electronic medical records in the United States.
J Am Med Inform Assoc. 2005;12:3-715492029
Google ScholarCrossref 34. Pennsylvania Health Care Cost Containment Council Web site.
http://www.phc4.org. Accessed November 6, 2006 37.Hosmer DW, Lemeshow S. Logistic regression. In: Applied Logistic Regression. 2nd ed. New York, NY: John Wiley and Sons; 2000:91-142
39.Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. New York, NY: Springer-Verlag; 2001
40.Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Radiology. 1982;143:29-367063747
Google Scholar 41.Lee TH, Meyer GS, Brennan TA. A middle ground on public accountability.
N Engl J Med. 2004;350:2409-241215175444
Google ScholarCrossref 43.Pennsylvania Healthcare Cost Containment Council. Independent accountant's [Parente Randolph] report on applying agreed-upon procedures [August 5, 2004].
http://www.phc4.org. Accessed December 1, 2006 45.Lee SJ, Lindquist K, Segal MR, Covinsky KE. Development and validation of a prognostic index for 4-year mortality in older adults.
JAMA. 2006;295:801-80816478903
Google ScholarCrossref