[Skip to Navigation]
Sign In
Figure.  Variation in Low-Value Health Care Services Across Physicians
Variation in Low-Value Health Care Services Across Physicians

A and B, Distributions of annual rates of low-value services (predicted modes) adjusted for beneficiary sociodemographic and clinical characteristics, local area economic and educational characteristics, year, and hospital referral region, as well as health care provider organization (B only). These graphs represent adjusted rates of low-value services for physicians practicing within the same region (A) and within the same health care provider organization (B). C and D, Corresponding predicted distributions of annual rates of low-value services, adjusted for the same factors, based on physician characteristics.

Table 1.  Measures of Low-Value Services Related to Primary Care
Measures of Low-Value Services Related to Primary Care
Table 2.  Beneficiary and Physician Characteristicsa
Beneficiary and Physician Characteristicsa
Table 3.  Estimates of Across-Physician Variation, With and Without Chronic Condition Adjustment
Estimates of Across-Physician Variation, With and Without Chronic Condition Adjustment
Table 4.  Association Between Physician Characteristics and Low-Value Services
Association Between Physician Characteristics and Low-Value Services
1.
Centers for Medicare & Medicaid Services. The Medicare Access & CHIP Reauthorization Act of 2015 Quality Payment Program. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/Quality-Payment-Program-MACRA-NPRM-Slides.pdf. 2017. Accessed May 21, 2017.
2.
Schwartz  AL, Landon  BE, Elshaug  AG, Chernew  ME, McWilliams  JM.  Measuring low-value care in Medicare.  JAMA Intern Med. 2014;174(7):1067-1076. doi:10.1001/jamainternmed.2014.1541PubMedGoogle ScholarCrossref
3.
Colla  CH, Morden  NE, Sequist  TD, Schpero  WL, Rosenthal  MB.  Choosing Wisely: prevalence and correlates of low-value health care services in the United States.  J Gen Intern Med. 2015;30(2):221-228. doi:10.1007/s11606-014-3070-zPubMedGoogle ScholarCrossref
4.
Schwartz  AL, Chernew  ME, Landon  BE, McWilliams  JM.  Changes in low-value services in year 1 of the Medicare Pioneer Accountable Care Organization Program.  JAMA Intern Med. 2015;175(11):1815-1825. doi:10.1001/jamainternmed.2015.4525PubMedGoogle ScholarCrossref
5.
Schwartz  AL, Zaslavsky  AM, Landon  BE, Chernew  ME, McWilliams  JM.  Low-value service use in provider organizations.  Health Serv Res. 2018;53(1):87-119. doi:10.1111/1475-6773.12597PubMedGoogle ScholarCrossref
6.
Mehrotra  A, Reid  RO, Adams  JL, Friedberg  MW, McGlynn  EA, Hussey  PS.  Physicians with the least experience have higher cost profiles than do physicians with the most experience.  Health Aff (Millwood). 2012;31(11):2453-2463. doi:10.1377/hlthaff.2011.0252PubMedGoogle ScholarCrossref
7.
O’Neill  L, Kuder  J.  Explaining variation in physician practice patterns and their propensities to recommend services.  Med Care Res Rev. 2005;62(3):339-357. doi:10.1177/1077558705275424PubMedGoogle ScholarCrossref
8.
Meyer  CM, Ladenson  PW, Scharfstein  JA, Danese  MD, Powe  NR.  Evaluation of common problems in primary care: effects of physician, practice, and financial characteristics.  Am J Manag Care. 2000;6(4):457-469.PubMedGoogle Scholar
9.
Pham  HH, Landon  BE, Reschovsky  JD, Wu  B, Schrag  D.  Rapidity and modality of imaging for acute low back pain in elderly patients.  Arch Intern Med. 2009;169(10):972-981. doi:10.1001/archinternmed.2009.78PubMedGoogle ScholarCrossref
10.
Stafford  RS, Misra  B.  Variation in routine electrocardiogram use in academic primary care practice.  Arch Intern Med. 2001;161(19):2351-2355. doi:10.1001/archinte.161.19.2351PubMedGoogle ScholarCrossref
11.
Rosen  MP, Davis  RB, Lesky  LG.  Utilization of outpatient diagnostic imaging: does the physician’s gender play a role?  J Gen Intern Med. 1997;12(7):407-411. doi:10.1046/j.1525-1497.1997.00071.xPubMedGoogle ScholarCrossref
12.
Landon  BE, Reschovsky  J, Reed  M, Blumenthal  D.  Personal, organizational, and market level influences on physicians’ practice patterns: results of a national survey of primary care physicians.  Med Care. 2001;39(8):889-905. doi:10.1097/00005650-200108000-00014PubMedGoogle ScholarCrossref
13.
Chen  YA, Gray  BG, Bandiera  G, MacKinnon  D, Deva  DP.  Variation in the utilization and positivity rates of CT pulmonary angiography among emergency physicians at a tertiary academic emergency department.  Emerg Radiol. 2015;22(3):221-229. doi:10.1007/s10140-014-1265-6PubMedGoogle ScholarCrossref
14.
Obermeyer  Z, Powers  BW, Makar  M, Keating  NL, Cutler  DM.  Physician characteristics strongly predict patient enrollment in hospice.  Health Aff (Millwood). 2015;34(6):993-1000. doi:10.1377/hlthaff.2014.1055PubMedGoogle ScholarCrossref
15.
Doyle  JJ  Jr, Ewer  SM, Wagner  TH.  Returns to physician human capital: evidence from patients randomized to physician teams.  J Health Econ. 2010;29(6):866-882. doi:10.1016/j.jhealeco.2010.08.004PubMedGoogle ScholarCrossref
16.
Choudhry  NK, Fletcher  RH, Soumerai  SB.  Systematic review: the relationship between clinical experience and quality of health care.  Ann Intern Med. 2005;142(4):260-273. doi:10.7326/0003-4819-142-4-200502150-00008PubMedGoogle ScholarCrossref
17.
Pham  HH, Schrag  D, Hargraves  JL, Bach  PB.  Delivery of preventive services to older adults by primary care physicians.  JAMA. 2005;294(4):473-481. doi:10.1001/jama.294.4.473PubMedGoogle ScholarCrossref
18.
Streja  DA, Rabkin  SW.  Factors associated with implementation of preventive care measures in patients with diabetes mellitus.  Arch Intern Med. 1999;159(3):294-302. doi:10.1001/archinte.159.3.294PubMedGoogle ScholarCrossref
19.
Tsugawa  Y, Jena  AB, Figueroa  JF, Orav  EJ, Blumenthal  DM, Jha  AK.  Comparison of hospital mortality and readmission rates for Medicare patients treated by male vs female physicians.  JAMA Intern Med. 2017;177(2):206-213.PubMedGoogle Scholar
20.
Kerfoot  BP, Holmberg  EF, Lawler  EV, Krupat  E, Conlin  PR.  Practitioner-level determinants of inappropriate prostate-specific antigen screening.  Arch Intern Med. 2007;167(13):1367-1372. doi:10.1001/archinte.167.13.1367PubMedGoogle ScholarCrossref
21.
Kim  C, McEwen  LN, Gerzoff  RB,  et al.  Is physician gender associated with the quality of diabetes care?  Diabetes Care. 2005;28(7):1594-1598. doi:10.2337/diacare.28.7.1594PubMedGoogle ScholarCrossref
22.
Berthold  HK, Gouni-Berthold  I, Bestehorn  KP, Böhm  M, Krone  W.  Physician gender is associated with the quality of type 2 diabetes care.  J Intern Med. 2008;264(4):340-350. doi:10.1111/j.1365-2796.2008.01967.xPubMedGoogle ScholarCrossref
23.
Henderson  JT, Weisman  CS.  Physician gender effects on preventive screening and counseling: an analysis of male and female patients’ health care experiences.  Med Care. 2001;39(12):1281-1292. doi:10.1097/00005650-200112000-00004PubMedGoogle ScholarCrossref
24.
Flocke  SA, Gilchrist  V.  Physician and patient gender concordance and the delivery of comprehensive clinical preventive services.  Med Care. 2005;43(5):486-492. doi:10.1097/01.mlr.0000160418.72625.1cPubMedGoogle ScholarCrossref
25.
Asch  DA, Nicholson  S, Srinivas  S, Herrin  J, Epstein  AJ.  Evaluating obstetrical residency programs using patient outcomes.  JAMA. 2009;302(12):1277-1283. doi:10.1001/jama.2009.1356PubMedGoogle ScholarCrossref
26.
Ko  DT, Austin  PC, Chan  BTB, Tu  JV.  Quality of care of international and Canadian medical graduates in acute myocardial infarction.  Arch Intern Med. 2005;165(4):458-463. doi:10.1001/archinte.165.4.458PubMedGoogle ScholarCrossref
27.
Rhee  SO, Lyons  TF, Payne  BC, Moskowitz  SE.  USMGs versus FMGs: are there performance differences in the ambulatory care setting?  Med Care. 1986;24(3):248-258. doi:10.1097/00005650-198603000-00007PubMedGoogle ScholarCrossref
28.
Lurie  N, Slater  J, McGovern  P, Ekstrum  J, Quam  L, Margolis  K.  Preventive care for women: does the sex of the physician matter?  N Engl J Med. 1993;329(7):478-482. doi:10.1056/NEJM199308123290707PubMedGoogle ScholarCrossref
29.
Roter  DL, Hall  JA, Aoki  Y.  Physician gender effects in medical communication: a meta-analytic review.  JAMA. 2002;288(6):756-764. doi:10.1001/jama.288.6.756PubMedGoogle ScholarCrossref
30.
Hodgson  DC, Fuchs  CS, Ayanian  JZ.  Impact of patient and provider characteristics on the treatment and outcomes of colorectal cancer.  J Natl Cancer Inst. 2001;93(7):501-515. doi:10.1093/jnci/93.7.501PubMedGoogle ScholarCrossref
31.
Barnett  ML, Linder  JA, Clark  CR, Sommers  BD.  Low-value medical services in the safety-net population.  JAMA Intern Med. 2017;177(6):829-837. doi:10.1001/jamainternmed.2017.0401PubMedGoogle ScholarCrossref
32.
Mafi  JN, Wee  CC, Davis  RB, Landon  BE.  Association of primary care practice location and ownership with the provision of low-value care in the United States.  JAMA Intern Med. 2017;177(6):838-845. doi:10.1001/jamainternmed.2017.0410PubMedGoogle ScholarCrossref
33.
Jena  AB, Khullar  D, Ho  O, Olenski  AR, Blumenthal  DM.  Sex differences in academic rank in US medical schools in 2014.  JAMA. 2015;314(11):1149-1158. doi:10.1001/jama.2015.10680PubMedGoogle ScholarCrossref
34.
Jena  AB, Olenski  AR, Blumenthal  DM.  Sex differences in physician salary in US public medical schools.  JAMA Intern Med. 2016;176(9):1294-1304. doi:10.1001/jamainternmed.2016.3284PubMedGoogle ScholarCrossref
35.
Tsugawa  Y, Jena  AB, Orav  EJ, Jha  AK.  Quality of care delivered by general internists in US hospitals who graduated from foreign versus US medical schools: observational study.  BMJ. 2017;356:j273. doi:10.1136/bmj.j273PubMedGoogle ScholarCrossref
36.
Tsugawa  Y, Newhouse  JP, Zaslavsky  AM, Blumenthal  DM, Jena  AB.  Physician age and outcomes in elderly patients in hospital in the US: observational study.  BMJ. 2017;357:j1797. doi:10.1136/bmj.j1797PubMedGoogle ScholarCrossref
37.
McWilliams  JM, Hatfield  LA, Chernew  ME, Landon  BE, Schwartz  AL.  Early performance of accountable care organizations in Medicare.  N Engl J Med. 2016;374(24):2357-2366. doi:10.1056/NEJMsa1600142PubMedGoogle ScholarCrossref
38.
Choosing Wisely. Lists of five things physicians and patients should question. http://www.choosingwisely.org/patient-resources/. 2014. Accessed February 16, 2015.
39.
US Preventive Services Task Force. Published recommendations. https://www.uspreventiveservicestaskforce.org/BrowseRec/Index/brouse-recommendations. 2014. Accessed February 16, 2016.
40.
Canadian Agency for Drugs and Technologies in Health. Health technology assessments. https://www.cadth.ca/resources/hta-database-canadian-search-interface. December 23, 2014. Accessed February 16, 2016.
41.
Elshaug  AG, Moss  JR, Littlejohns  P, Karnon  J, Merlin  TL, Hiller  JE.  Identifying existing health care services that do not provide value for money.  Med J Aust. 2009;190(5):269-273.PubMedGoogle Scholar
42.
Chronic Conditions Data Warehouse. https://www.ccwdata.org/. Accessed February 16, 2015.
43.
US Census Bureau. 2009-2011 American Community Survey Summary File. https://www2.census.gov/acs2011_5yr/summaryfile/?sec_ak_reference=18.940a1160.1538683639.9f97ba9. 2012.
44.
US News and World Report.  US News and World Report Best Graduate Schools.  April 2013:50-55.
45.
Adams  JL, Mehrotra  A, Thomas  JW, McGlynn  EA.  Physician cost profiling—reliability and risk of misclassification.  N Engl J Med. 2010;362(11):1014-1021. doi:10.1056/NEJMsa0906323PubMedGoogle ScholarCrossref
46.
Fay  R, Herriot  R.  Estimates of income for small places: an application of James-Stein procedures to census data.  J Am Stat Assoc. 1979;74(366):269-277. doi:10.1080/01621459.1979.10482505Google ScholarCrossref
47.
Agency for Healthcare Research and Quality. Instructions for Analyzing Data from CAHPS Surveys. https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/helpful-resources/analysis/2015-instructions-for-analyzing-data.pdf. Updated June 1, 2017. Accessed October 11, 2018.
48.
Cutler  DM, Skinner  JS, Stern  AD, Wennberg  DE.; National Bureau of Economic Research.  Physician Beliefs and Patient Preferences: A New Look at Regional Variation in Health Care Spending. 2013:19320. http://www.nber.org/papers/w19320. Revised February 2018. Accessed October 11, 2018. doi:10.3386/w19320
49.
Gogineni  K, Shuman  KL, Chinn  D, Gabler  NB, Emanuel  EJ.  Patient demands and requests for cancer tests and treatments.  JAMA Oncol. 2015;1(1):33-39. doi:10.1001/jamaoncol.2014.197PubMedGoogle ScholarCrossref
Original Investigation
Less Is More
January 2019

Analysis of Physician Variation in Provision of Low-Value Services

Author Affiliations
  • 1Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
  • 2Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, Massachusetts
  • 3Department of Medicine, Massachusetts General Hospital, Boston
JAMA Intern Med. 2019;179(1):16-25. doi:10.1001/jamainternmed.2018.5086
Key Points

Questions  Does extensive variation exist among physicians in the provision of low-value health care services, and can levels of provision be predicted by observed physician characteristics?

Findings  In this observational study using Medicare claims data of 3 159 834 beneficiaries served by 41 773 generalist physicians, the rates of low-value services were 60% higher for primary care physicians at the 90th percentile of their provider organization than for physicians at the 10th percentile. Only 1.4% of physician variation within organizations could be explained by observable physician characteristics.

Meaning  Although differences in physician practice patterns may substantially contribute to low-value service use, it is difficult to predict which physicians are more wasteful without measuring their behavior.

Abstract

Importance  Facing new financial incentives to reduce unnecessary spending, health care organizations may attempt to reduce wasteful care by influencing physician practices or selecting more cost-effective physicians. However, physicians’ role in determining the use of low-value services has not been well described.

Objectives  To quantify variation in provision of low-value health care services among primary care physicians and to estimate the proportion of variation attributable to physician characteristics that may be used to predict performance.

Design, Setting, and Participants  This retrospective analysis included national Medicare fee-for-service claims of 3 159 834 beneficiaries served by 41 773 generalist physicians from January 1, 2008, through December 31, 2013 (data were analyzed in 2016 through 2018). Multilevel modeling was used to estimate the extent of variation in service use across physicians within their region and provider organization, adjusted for patient clinical and sociodemographic characteristics and sampling variation. The proportion of variation attributable to physician characteristics that may be used to predict performance (age, sex, academic degree, professorship, publication record, trial investigation, grant receipt, pharmaceutical or device manufacturer payment, and panel size) was estimated via additional regression analysis.

Main Outcomes and Measures  Annual count per beneficiary of 17 primary care–associated services that provide minimal clinical benefit.

Results  Among the 3 159 834 beneficiaries (58.3% women; mean [SD] age, 73.2 [11.0] years) served by 41 773 physicians (74.9% men; mean [SD] age, 48.0 [10.1] years), the mean annual rate of low-value services was 33.1 services per 100 beneficiaries. Considerable variation across physicians within the same region was found (SD, 8.8 [95% CI, 8.7-8.9]; 90th:10th percentile ratio, 2.03 [95% CI, 2.01-2.06]) and across physicians within the same organization (SD, 6.1 [95% CI, 6.0-6.2]; 90th:10th percentile ratio, 1.61 [95% CI, 1.60-1.63]). The corresponding rates at the 10th percentile of physicians within region and within organization respectively were 21.8 and 25.3 services per 100 beneficiaries. Observable physician characteristics accounted for only 4.4% of physician variation within region and 1.4% of physician variation within organization.

Conclusions and Relevance  Physician practices may substantially contribute to low-value service use, which is prevalent even among the least wasteful physicians. Because little variation is predicted by measured physician characteristics, direct measures of low-value care provision may aid organizational efforts to encourage high-value practices.

Introduction

New payment models reward health care provider organizations for reducing wasteful care. These models include accountable care organization programs that place spending for a provider organization’s patient population under a global budget and provide incentives to bring spending below the budget while achieving high performance on quality measures.1 Understanding physicians’ propensity to provide unnecessary medical services independent of their patients’ needs could promote success under these models. For example, knowing physicians’ practice patterns, accountable care organizations could target more wasteful physicians for retraining. They could also include less wasteful physicians in their risk contracts with payers or in their referral networks. The premise of such physician-focused strategies is that physician-specific factors (eg, education, preferences) may influence their providing unnecessary care. However, efforts to describe the role physicians play in generating unnecessary care have been limited.

Recent studies have demonstrated the feasibility of directly measuring low-value services that provide minimal benefits for patients.2-5 Such direct measurement offers distinct advantages over alternative approaches, such as less specific measures of the use of health care services (eg, total risk-adjusted spending per patient) or profiling based on physician characteristics (eg, age, sex, or training).6-15 Because they do not rely on comparisons, direct measures of low-value service use provide an assessment of overuse for all providers, even the most efficient providers,5 and thus may reveal the full potential for strategies to eliminate wasteful care. Direct measures are also likely to discriminate between more and less efficient physicians better than indirect measures. In particular, although characteristics such as physicians’ age, sex, or training may be correlated with practice patterns,6-30 such characteristics may account for only a small proportion of physician-level variation. Also, nonspecific measures of use of health care services do not distinguish between appropriate and inappropriate variation in service use. Thus, direct measurement could better support efforts to identify physicians with cost-effective practices.

Prior studies have explored variation in low-value service use among regions and organizations to understand the potential role of markets and organizations in shaping practice patterns. However, to our knowledge, no study has examined variation in a broad set of low-value services at the physician level.2,3,5,31,32 Evidence of marked physician-level variation in wasteful practices not attributable to patient, organizational, or regional factors would support a role of physician-level determinants in generating overuse. Using Medicare claims and measures of 17 low-value services commonly provided in the primary care setting, we assessed physician-level variation in low-value service use within regions and within provider organizations. We also assessed the extent to which this variation could be explained by observable physician characteristics that may be used to predict physician performance.

Methods
Study Population
Data and Inclusion Criteria

We assessed patient characteristics and health care use from Medicare enrollment and claims files from January 1, 2008, through December 31, 2013, for a random 20% sample of Medicare fee-for-service beneficiaries. We assessed physician characteristics from a database assembled by Doximity, an online social network for US physicians (Doximity, Inc). This database includes not only members of the networking service but also physicians identified via the National Provider Identifier Registry, National Plan and Provider Enumeration System, state licensing boards, specialty societies, and collaborating hospitals and medical schools. The database includes physician characteristics drawn from sources including the National Plan and Provider Enumeration System, the Association of American Medical Colleges faculty roster database, PubMed, ClinicalTrials.gov, the National Institutes of Health Research Portfolio Online Reporting Tools database, and the Centers of Medicare & Medicaid Services Open Payments database. Details of this database and its validation have been published elsewhere.19,33-36 Institutional review board approval was granted by the Harvard University Committee on the Use of Human Subjects. Informed consent was not required for the use of publicly available data, and data were analyzed in 2016 through 2018.

For each measurement year in the 2008-2013 study period, we included beneficiaries who were continuously enrolled in Medicare Parts A and B in that year (while alive for decedents) and in the prior year, were residing in one of the 50 US states or Washington, DC, and had at least 1 of the evaluation and management services that were used to attribute beneficiaries to physicians. We excluded patients attributed to physicians with missing data on key characteristics in the Doximity database (16.4% of patient-year observations) or attributed to physicians without at least 1 attributed patient qualifying for the denominator of each measure of low-value service use (16.7% of remaining patient-year observations).

Patient and Physician Attribution

In each year, each patient was attributed to the primary care physician (physicians with specialty codes indicating general practice, family practice, internal medicine, or geriatric medicine) who accounted for the plurality of the patient’s office visits (Healthcare Common Procedure Coding System codes 92201-99205, 99211-99215, 99241-99245, G0402, G0438, and G0439). Ties were broken based on the plurality of allowed charges for office visits. As in prior work, provider organizations were defined by taxpayer identification numbers.5 Physicians were attributed to the taxpayer identification number accounting for the plurality of the physician’s office visit counts during the study period, with ties broken as above.

Study Variables
Measures of Low-Value Services

The primary outcome of this study was the rate of use of 17 low-value services, reported as the number of services per 100 patients per year. As detailed in prior studies using these measures,2,37 we selected services that produce minimal average clinical benefit in specific clinical situations that could be distinguished from potentially high-value indications with reasonable accuracy using claims and enrollment data. Sources for measure development included the American Board of Internal Medicine Foundation’s Choosing Wisely initiative,38 the US Preventive Services Task Force D recommendations,39 the Canadian Agency for Drugs and Technologies in Health technology assessments,40 and the peer-reviewed medical literature.41 The evidence base for these sources has been detailed previously.2,37 Of note, our study period largely precedes the 2012 expansion of the Choosing Wisely campaign.

As in prior studies,5 when there was discretion in defining measures, we used more specific definitions, which reduce the probability of detecting some low-value services, but also reduce the probability of misclassifying high-value services as low-value. Because this study’s focus was primary care physicians, we restricted our measure set to services whose use could be influenced substantially by a primary care physician’s treatment or referral decisions. Our approach attributed all low-value services received by a patient to the patient’s primary care physician, including some services for which other physicians were responsible. In sensitivity analyses, we therefore repeated analyses with smaller sets of services even more likely to reflect primary care physicians’ decisions.

Table 1 describes each low-value service measure, its operational definition, and the denominator population to which it applies. Operational definitions relied on information from claims, including the presence or timing of procedures, diagnoses, emergency department visits, or hospitalizations, and on information from the annual Master Beneficiary Summary File, such as age and diagnoses recorded in the Chronic Condition Data Warehouse, which draws from Medicare claims since 1999 to describe beneficiaries’ accumulated burden of chronic disease.42 Defining denominator populations for each measure allowed our analysis to account for differences across physicians in the number of patients who could potentially receive certain low-value services. For example, the analysis of low-value preoperative testing only included patients who underwent surgical procedures so that comparisons across physicians would not be confounded by differences in patients’ need for surgery. Specific diagnosis and procedure codes used for each measure are described in eTable 1 in the Supplement.

Beneficiary Covariates

From the Medicare Master Beneficiary Summary File, we assessed age, sex, race/ethnicity, disability as the initial reason for Medicare enrollment, Medicare-Medicaid dual eligibility, and end-stage renal disease. From the Chronic Conditions Data Warehouse, we assessed the presence of 27 chronic conditions. Finally, from the 2007-2011 American Community Survey Summary File, we assessed the following characteristics of the elderly population in each beneficiary’s zip code tabulation area43: median income, proportion of the population below the federal poverty level, proportion with a high school degree, and proportion with a college degree.

Physician Characteristics

From the Doximity database, we assessed physicians’ age, sex, training, academic engagement, and payment from a pharmaceutical or device manufacturer. Age was recorded in years as of July 2015. Training variables included academic degree (doctor of medicine vs doctor of osteopathy), graduation from a foreign medical school, and graduation from a medical school ranked among the top 20 research schools in US News and World Report in 2013.44 Academic engagement variables included rank of academic appointment (full professorship, associate professorship, assistant professorship, or no professorship), number of authored scientific publications, an indicator for authoring any publications, status as principal or subinvestigator of at least 1 registered clinical trial, and status as principal investigator with at least 1 National Institutes of Health grant. In addition to physicians’ amount of payment from pharmaceutical or medical device companies, we included an indicator for any nonzero payment. More information on these variables and their sources has been detailed elsewhere.19,33-36 Finally, using Medicare claims, we estimated Medicare fee-for-service panel size via the attribution methods detailed above (before exclusion of any patients or physicians from the sample), adjusting panel sizes to account for the 20% random sampling.

Statistical Analysis
Physician Variation in Low-Value Service Provision

Data were analyzed from November 2016 through July 2018. Because of sampling error and nonrandom sorting of patients to physicians,45 unadjusted variation in low-value service use across physicians will tend to overstate the amount of variation that is due to differences in physician practice (ie, the amount of variation that would be observed if physicians treated an infinite number of patients and if patients were randomized to physicians). Therefore, we used multilevel models to estimate physician-level variation in rates of low-value service use net of sampling error, with techniques to adjust for patient, regional, and organizational factors.

As in prior work,5 we used a multilevel modeling approach developed by Fay and Herriot.46 This established method has been used to analyze composite measures of health care quality composed of multiple quality elements.47 First, for each low-value service, we calculated scores reflecting each physician’s covariate-adjusted rate of use. The service score was calculated via linear regression modeling of the number of times a beneficiary received the service as a function of beneficiary covariates, indicators for year, hospital referral region fixed effects, and physician fixed effects. The physician fixed effects yielded estimates of each physician’s score for that service. To better account for patient factors that might affect a patient’s likelihood of receiving a low-value service, we limited the samples for these regressions to beneficiaries satisfying the measure’s denominator criteria.

Second, a physician’s composite score was calculated as a weighted sum of the physician’s service-specific scores (eMethods in the Supplement). Third, we fit multilevel models to those composite scores to estimate variation across physicians, adjusted for sampling error. These models produced final estimates of between-physician variation (ie, SD) in rates of low-value services. To facilitate interpretation, we present SDs and their corresponding ratios of 90th to 10th percentiles, calculated based on the properties of normal distributions and physicians’ mean rate of low-value service provision. For example, a 90th:10th percentile ratio of 2.0 would suggest twice the rate of low-value services per patient per year for physicians at the 90th percentile of low-value service use than for physicians at the 10th percentile.

Examining within-region and within-organization variation allowed us to estimate whether substantial variation in physician practices existed even when holding constant all market-level or organization-level factors. The inclusion of hospital referral region fixed effects in this analysis yielded estimates of variation across physicians within the same geographic service area. To estimate within-organization variation, we repeated the analysis with physicians nested within an additional level, provider organizations, in our multilevel model. This model also allowed us to estimate the extent of between-organization variation in low-value services. To gauge the extent to which our estimates of physician-level variation in low-value service provision reflected variation in patients’ clinical characteristics, we conducted a sensitivity analysis without adjustment for chronic conditions.

Proportion of Variation Predicted by Physician Characteristics

To calculate the proportion of variation in low-value service provision predicted by physician characteristics, we compared the magnitude of variation observed in the preceding analyses to the magnitude of variation that would be predicted by differences in physician characteristics alone. This process entailed estimating associations between physician characteristics and low-value service use, predicting each physician’s rate of low-value service use based on the physician’s characteristics, and measuring the variation across physicians in those predicted rates of low-value services (eMethods in the Supplement). Again, we predicted variation within their region and the provider organization. The proportion of low-value service variation attributable to physician characteristics was calculated as the ratio of the variance of these predictions to the variance in observed rates of low-value services from the prior analysis.

Results

Overall, 3 159 834 beneficiaries (58.3% women and 41.7% men; mean [SD] age, 73.2 [11.0] years) served by 41 773 physicians (25.1% women and 74.9% men; mean [SD] age, 48.0 [10.1] years) in 6771 provider organizations distributed across 306 regions were included in this study, with 10 199 293 beneficiary-year observations (Table 2). Physicians’ mean rate of low-value service provision was 33.1 services per 100 beneficiaries per year. Without adjustment for patient, regional, or organizational factors, the SD was 14.4 services per 100 beneficiaries per year, corresponding to a 90th:10th percentile ratio of 3.54.

The Figure displays the distribution of low-value service rates across physicians within the same geographic service (Figure, A) and within the same provider organization (Figure, B), both adjusted for patient characteristics. Within region, the estimated SD was 8.8 services per 100 beneficiaries per year (95% CI, 8.7-8.9), which corresponds to a 90th:10th percentile ratio of 2.03 (95% CI, 2.01-2.06) and a 10th percentile of 21.8 services per 100 beneficiaries per year. Within the same provider organizations, the estimated SD was 6.1 services per 100 beneficiaries per year (95% CI, 6.0-6.2), corresponding to a 90th:10th percentile ratio of 1.61 (95% CI, 1.60-1.63) and a 10th percentile of 25.3 services per 100 beneficiaries per year. The estimated SD in low-value service rates across organizations was 7.7 services per 100 beneficiaries per year. Estimates of variation were not substantially affected by excluding adjustment for patients’ comorbidities (within-region adjusted 90th:10th percentile ratio, 2.13 [10th percentile, 21.1]; within-organization adjusted 90th:10th percentile ratio, 1.66 [10th percentile, 24.9]). Table 3 presents this comparison in tabular form.

The Figure also presents the smaller magnitude of variation in low-value service rates across physicians that would be predicted by differences in physician characteristics. Table 4 presents the estimated associations between physician characteristics and low-value service rates used to calculate the predicted distributions. Within regions, 10 of 15 physician characteristics were predictive of low-value service use. Within organizations, 7 of 15 physician characteristics were predictive of low-value service use. Together, all physician characteristics accounted for 4.4% of within-region physician variation in low-value service provision (Figure, C) and 1.4% of within-organization variation (Figure, D).

Sensitivity analyses using narrower sets of low-value services that were more likely ordered by primary care physicians demonstrated similar results (eFigures 1-2 and eTables 2-3 in the Supplement). For example, for a narrower set of 12 low-value service measures (mean rate, 25.5 services per 100 beneficiaries per year), the adjusted 90th:10th percentile ratio of low-value service rates within region was 2.10 services per 100 beneficiaries per year, with a 10th percentile of 16.4 services per 100 beneficiaries per year; the adjusted 90th:10th percentile ratio of low-value service rates within organization was 1.67 services per 100 beneficiaries per year, with a 10th percentile of 19.1 services per 100 beneficiaries per year (eFigure 1 in the Supplement). Physician characteristics accounted for 4.6% of within-region variance and 1.8% of within-organization variance.

Discussion

In this study of low-value services received by Medicare beneficiaries, we observed substantial variation among primary care physicians, of which a small fraction was attributable to observable physician characteristics. Even within the same provider organization, low-value service use varied substantially among physicians. Adjusted rates of low-value services for patients served by physicians at an organization’s 90th percentile were 61% greater than those for patients served by physicians at the organization’s 10th percentile. Adjustment for patient’s comorbidities had minimal effect on this variation, suggesting that differences in low-value service use between patients served by different physicians were not primarily driven by differences in clinical needs. These findings of extensive physician-level variation not attributable to patient, organizational, or regional factors suggest that physician practice patterns may contribute substantially to overuse. Accordingly, strategies to identify and remediate wasteful practices at the physician level may be effective for provider organizations participating in alternative payment models.

We also observed substantial rates of low-value services even among physicians whose patients received the fewest low-value services. In a typical region, patients served by physicians at the 10th percentile of low-value service use still received more than 20 low-value services per 100 patients annually. Thus, strategies to reduce levels of overuse among all physicians may achieve greater gains than strategies relying on existing variation to identify and focus on physicians exhibiting the most wasteful practice patterns.

Extending prior work,5 we demonstrated substantial variation in rates of low-value services across different provider organizations even after accounting for variation across physicians. Despite the considerable variation we observed across physicians within the same organization, the variation across organizations in the use of low-value services was much greater than would be expected due to chance. This finding is consistent with the hypothesis that provider organizations shape and/or select for the practice patterns of affiliated physicians, which is a premise of organization-focused reforms such as accountable care organizations. Possible mechanisms include differences across organizations in recruitment, training, or compensation of physicians or in practice guidelines, environment, and management of low-value care specifically.

Although we observed associations between several physician characteristics and rates of low-value services, the associations were weak and accounted for a minimal fraction of the physician variation we observed. This finding suggests potential advantages of directly assessing physician performance on measures of low-value care rather than on their personal characteristics. Indeed, only 1 physician characteristic, greater size of patient panel, predicted substantially higher rates of low-value services. This association must be interpreted with caution, in part because our estimates of panel size, based on visits, may have been inflated for physicians more eager to schedule follow-up appointments. Furthermore, because physician characteristics are not randomly distributed among physicians, associations between characteristics and low-value services should not be interpreted causally.

Limitations

Our study had several additional limitations. First, the observational design did not support firm causal conclusions about the amount of low-value service use owing to physicians. If patients were randomized to physicians and physicians were randomized to provider organizations, we may have found different magnitudes of variation across physicians in rates of low-value service use. For example, some proportion of the variation we observed could be attributable to differences in patient characteristics such as preferences regarding the receipt of low-value services. However, prior research has indicated that patient preferences are unlikely to be a strong driver of overuse,5,48,49 and many of the services we studied are not likely to be familiar to patients. Although unmeasured patient clinical characteristics could explain some proportion of the variation we observed, our results were similar with and without adjustment for patient chronic conditions. Furthermore, we isolated physician variation within regions or organizations, accounting for systematic differences in patient populations that we could not observe. Although unmeasured differences in patients across physicians could lead to overestimation of physician variation, the tendency of physicians with similar practice styles to work in similar organizations or regions, (ie, homophily) could lead to underestimation of physician variation.

Second, our study examined only one dimension of quality of care: overuse. The overall value of services delivered by a physician depends on additional dimensions such as diagnostic accuracy and provision of necessary treatments. Furthermore, physicians with high use of the measured services may or may not be consistently high users of other low-value services. Our estimates of variation do not support conclusions regarding the strength of association between physicians’ use of different low-value services.

Third, some high-value services may have been misclassified as low-value in our study. This error would not have induced bias in our estimates if the misclassification were random across physicians. However, bias would have resulted if the misclassification frequency was greater among certain physicians or among physicians with certain characteristics. Fourth, low-value services may have been misattributed to a primary care physician if they were ordered or requested by physicians (eg, specialists) other than the primary care physician. However, sensitivity analyses focusing on tests and studies more likely to be ordered by primary care physicians produced estimates of variation that were similar to our main results. Moreover, even if physicians other than the patients’ attributed primary care physicians were responsible for some services, one could still reasonably interpret the variation we measured as driven by physician-level factors.

Conclusions

New financial incentives for provider organizations to reduce spending have spurred efforts to discourage low-value service use and to identify more and less wasteful organizations. Our study suggests that physician practices may drive much of low-value care and that low-value practices are prevalent even among the least wasteful physicians. Because physician characteristics explain little variation in low-value care, profiling physicians using direct measures of low-value care may be useful in supporting efforts to incentivize more cost-effective practices through targeted retraining of physicians or selective inclusion of physicians in networks and risk contracts.

Back to top
Article Information

Accepted for Publication: August 9, 2018.

Corresponding Author: Aaron L. Schwartz, MD, PhD, Department of Medicine, Brigham and Women’s Hospital, 75 Francis St, Boston, MA 02115 (alschwartz@partners.org).

Published Online: December 3, 2018. doi:10.1001/jamainternmed.2018.5086

Author Contributions: Dr Schwartz had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: Schwartz, McWilliams.

Drafting of the manuscript: Schwartz, Jena.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: All authors.

Obtained funding: Schwartz.

Administrative, technical, or material support: Schwartz.

Supervision: Jena, McWilliams.

Conflict of Interest Disclosures: Drs Schwartz and McWilliams reported being compensated as consultants by the Medicare Payment Advisory Commission in their efforts to measure low-value care in Medicare for research purposes. Dr Jena reports receiving consulting fees unrelated to this work from Pfizer, Hill Rom Services, Bristol-Myers Squibb, Novartis, Amgen, Eli Lilly, Vertex Pharmaceuticals, AstraZeneca, Celgene, Tesaro, Sanofi-Aventis, Biogen, Precision Health Economics, and Analysis Group. No other disclosures were reported.

Funding/Support: This study was supported by grants P01 AG032952 and F30 AG044106-01A1 from the National Institute on Aging, National Institutes of Health.

Role of the Funder/Sponsor: The funder/sponsor had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Centers for Medicare & Medicaid Services. The Medicare Access & CHIP Reauthorization Act of 2015 Quality Payment Program. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/Quality-Payment-Program-MACRA-NPRM-Slides.pdf. 2017. Accessed May 21, 2017.
2.
Schwartz  AL, Landon  BE, Elshaug  AG, Chernew  ME, McWilliams  JM.  Measuring low-value care in Medicare.  JAMA Intern Med. 2014;174(7):1067-1076. doi:10.1001/jamainternmed.2014.1541PubMedGoogle ScholarCrossref
3.
Colla  CH, Morden  NE, Sequist  TD, Schpero  WL, Rosenthal  MB.  Choosing Wisely: prevalence and correlates of low-value health care services in the United States.  J Gen Intern Med. 2015;30(2):221-228. doi:10.1007/s11606-014-3070-zPubMedGoogle ScholarCrossref
4.
Schwartz  AL, Chernew  ME, Landon  BE, McWilliams  JM.  Changes in low-value services in year 1 of the Medicare Pioneer Accountable Care Organization Program.  JAMA Intern Med. 2015;175(11):1815-1825. doi:10.1001/jamainternmed.2015.4525PubMedGoogle ScholarCrossref
5.
Schwartz  AL, Zaslavsky  AM, Landon  BE, Chernew  ME, McWilliams  JM.  Low-value service use in provider organizations.  Health Serv Res. 2018;53(1):87-119. doi:10.1111/1475-6773.12597PubMedGoogle ScholarCrossref
6.
Mehrotra  A, Reid  RO, Adams  JL, Friedberg  MW, McGlynn  EA, Hussey  PS.  Physicians with the least experience have higher cost profiles than do physicians with the most experience.  Health Aff (Millwood). 2012;31(11):2453-2463. doi:10.1377/hlthaff.2011.0252PubMedGoogle ScholarCrossref
7.
O’Neill  L, Kuder  J.  Explaining variation in physician practice patterns and their propensities to recommend services.  Med Care Res Rev. 2005;62(3):339-357. doi:10.1177/1077558705275424PubMedGoogle ScholarCrossref
8.
Meyer  CM, Ladenson  PW, Scharfstein  JA, Danese  MD, Powe  NR.  Evaluation of common problems in primary care: effects of physician, practice, and financial characteristics.  Am J Manag Care. 2000;6(4):457-469.PubMedGoogle Scholar
9.
Pham  HH, Landon  BE, Reschovsky  JD, Wu  B, Schrag  D.  Rapidity and modality of imaging for acute low back pain in elderly patients.  Arch Intern Med. 2009;169(10):972-981. doi:10.1001/archinternmed.2009.78PubMedGoogle ScholarCrossref
10.
Stafford  RS, Misra  B.  Variation in routine electrocardiogram use in academic primary care practice.  Arch Intern Med. 2001;161(19):2351-2355. doi:10.1001/archinte.161.19.2351PubMedGoogle ScholarCrossref
11.
Rosen  MP, Davis  RB, Lesky  LG.  Utilization of outpatient diagnostic imaging: does the physician’s gender play a role?  J Gen Intern Med. 1997;12(7):407-411. doi:10.1046/j.1525-1497.1997.00071.xPubMedGoogle ScholarCrossref
12.
Landon  BE, Reschovsky  J, Reed  M, Blumenthal  D.  Personal, organizational, and market level influences on physicians’ practice patterns: results of a national survey of primary care physicians.  Med Care. 2001;39(8):889-905. doi:10.1097/00005650-200108000-00014PubMedGoogle ScholarCrossref
13.
Chen  YA, Gray  BG, Bandiera  G, MacKinnon  D, Deva  DP.  Variation in the utilization and positivity rates of CT pulmonary angiography among emergency physicians at a tertiary academic emergency department.  Emerg Radiol. 2015;22(3):221-229. doi:10.1007/s10140-014-1265-6PubMedGoogle ScholarCrossref
14.
Obermeyer  Z, Powers  BW, Makar  M, Keating  NL, Cutler  DM.  Physician characteristics strongly predict patient enrollment in hospice.  Health Aff (Millwood). 2015;34(6):993-1000. doi:10.1377/hlthaff.2014.1055PubMedGoogle ScholarCrossref
15.
Doyle  JJ  Jr, Ewer  SM, Wagner  TH.  Returns to physician human capital: evidence from patients randomized to physician teams.  J Health Econ. 2010;29(6):866-882. doi:10.1016/j.jhealeco.2010.08.004PubMedGoogle ScholarCrossref
16.
Choudhry  NK, Fletcher  RH, Soumerai  SB.  Systematic review: the relationship between clinical experience and quality of health care.  Ann Intern Med. 2005;142(4):260-273. doi:10.7326/0003-4819-142-4-200502150-00008PubMedGoogle ScholarCrossref
17.
Pham  HH, Schrag  D, Hargraves  JL, Bach  PB.  Delivery of preventive services to older adults by primary care physicians.  JAMA. 2005;294(4):473-481. doi:10.1001/jama.294.4.473PubMedGoogle ScholarCrossref
18.
Streja  DA, Rabkin  SW.  Factors associated with implementation of preventive care measures in patients with diabetes mellitus.  Arch Intern Med. 1999;159(3):294-302. doi:10.1001/archinte.159.3.294PubMedGoogle ScholarCrossref
19.
Tsugawa  Y, Jena  AB, Figueroa  JF, Orav  EJ, Blumenthal  DM, Jha  AK.  Comparison of hospital mortality and readmission rates for Medicare patients treated by male vs female physicians.  JAMA Intern Med. 2017;177(2):206-213.PubMedGoogle Scholar
20.
Kerfoot  BP, Holmberg  EF, Lawler  EV, Krupat  E, Conlin  PR.  Practitioner-level determinants of inappropriate prostate-specific antigen screening.  Arch Intern Med. 2007;167(13):1367-1372. doi:10.1001/archinte.167.13.1367PubMedGoogle ScholarCrossref
21.
Kim  C, McEwen  LN, Gerzoff  RB,  et al.  Is physician gender associated with the quality of diabetes care?  Diabetes Care. 2005;28(7):1594-1598. doi:10.2337/diacare.28.7.1594PubMedGoogle ScholarCrossref
22.
Berthold  HK, Gouni-Berthold  I, Bestehorn  KP, Böhm  M, Krone  W.  Physician gender is associated with the quality of type 2 diabetes care.  J Intern Med. 2008;264(4):340-350. doi:10.1111/j.1365-2796.2008.01967.xPubMedGoogle ScholarCrossref
23.
Henderson  JT, Weisman  CS.  Physician gender effects on preventive screening and counseling: an analysis of male and female patients’ health care experiences.  Med Care. 2001;39(12):1281-1292. doi:10.1097/00005650-200112000-00004PubMedGoogle ScholarCrossref
24.
Flocke  SA, Gilchrist  V.  Physician and patient gender concordance and the delivery of comprehensive clinical preventive services.  Med Care. 2005;43(5):486-492. doi:10.1097/01.mlr.0000160418.72625.1cPubMedGoogle ScholarCrossref
25.
Asch  DA, Nicholson  S, Srinivas  S, Herrin  J, Epstein  AJ.  Evaluating obstetrical residency programs using patient outcomes.  JAMA. 2009;302(12):1277-1283. doi:10.1001/jama.2009.1356PubMedGoogle ScholarCrossref
26.
Ko  DT, Austin  PC, Chan  BTB, Tu  JV.  Quality of care of international and Canadian medical graduates in acute myocardial infarction.  Arch Intern Med. 2005;165(4):458-463. doi:10.1001/archinte.165.4.458PubMedGoogle ScholarCrossref
27.
Rhee  SO, Lyons  TF, Payne  BC, Moskowitz  SE.  USMGs versus FMGs: are there performance differences in the ambulatory care setting?  Med Care. 1986;24(3):248-258. doi:10.1097/00005650-198603000-00007PubMedGoogle ScholarCrossref
28.
Lurie  N, Slater  J, McGovern  P, Ekstrum  J, Quam  L, Margolis  K.  Preventive care for women: does the sex of the physician matter?  N Engl J Med. 1993;329(7):478-482. doi:10.1056/NEJM199308123290707PubMedGoogle ScholarCrossref
29.
Roter  DL, Hall  JA, Aoki  Y.  Physician gender effects in medical communication: a meta-analytic review.  JAMA. 2002;288(6):756-764. doi:10.1001/jama.288.6.756PubMedGoogle ScholarCrossref
30.
Hodgson  DC, Fuchs  CS, Ayanian  JZ.  Impact of patient and provider characteristics on the treatment and outcomes of colorectal cancer.  J Natl Cancer Inst. 2001;93(7):501-515. doi:10.1093/jnci/93.7.501PubMedGoogle ScholarCrossref
31.
Barnett  ML, Linder  JA, Clark  CR, Sommers  BD.  Low-value medical services in the safety-net population.  JAMA Intern Med. 2017;177(6):829-837. doi:10.1001/jamainternmed.2017.0401PubMedGoogle ScholarCrossref
32.
Mafi  JN, Wee  CC, Davis  RB, Landon  BE.  Association of primary care practice location and ownership with the provision of low-value care in the United States.  JAMA Intern Med. 2017;177(6):838-845. doi:10.1001/jamainternmed.2017.0410PubMedGoogle ScholarCrossref
33.
Jena  AB, Khullar  D, Ho  O, Olenski  AR, Blumenthal  DM.  Sex differences in academic rank in US medical schools in 2014.  JAMA. 2015;314(11):1149-1158. doi:10.1001/jama.2015.10680PubMedGoogle ScholarCrossref
34.
Jena  AB, Olenski  AR, Blumenthal  DM.  Sex differences in physician salary in US public medical schools.  JAMA Intern Med. 2016;176(9):1294-1304. doi:10.1001/jamainternmed.2016.3284PubMedGoogle ScholarCrossref
35.
Tsugawa  Y, Jena  AB, Orav  EJ, Jha  AK.  Quality of care delivered by general internists in US hospitals who graduated from foreign versus US medical schools: observational study.  BMJ. 2017;356:j273. doi:10.1136/bmj.j273PubMedGoogle ScholarCrossref
36.
Tsugawa  Y, Newhouse  JP, Zaslavsky  AM, Blumenthal  DM, Jena  AB.  Physician age and outcomes in elderly patients in hospital in the US: observational study.  BMJ. 2017;357:j1797. doi:10.1136/bmj.j1797PubMedGoogle ScholarCrossref
37.
McWilliams  JM, Hatfield  LA, Chernew  ME, Landon  BE, Schwartz  AL.  Early performance of accountable care organizations in Medicare.  N Engl J Med. 2016;374(24):2357-2366. doi:10.1056/NEJMsa1600142PubMedGoogle ScholarCrossref
38.
Choosing Wisely. Lists of five things physicians and patients should question. http://www.choosingwisely.org/patient-resources/. 2014. Accessed February 16, 2015.
39.
US Preventive Services Task Force. Published recommendations. https://www.uspreventiveservicestaskforce.org/BrowseRec/Index/brouse-recommendations. 2014. Accessed February 16, 2016.
40.
Canadian Agency for Drugs and Technologies in Health. Health technology assessments. https://www.cadth.ca/resources/hta-database-canadian-search-interface. December 23, 2014. Accessed February 16, 2016.
41.
Elshaug  AG, Moss  JR, Littlejohns  P, Karnon  J, Merlin  TL, Hiller  JE.  Identifying existing health care services that do not provide value for money.  Med J Aust. 2009;190(5):269-273.PubMedGoogle Scholar
42.
Chronic Conditions Data Warehouse. https://www.ccwdata.org/. Accessed February 16, 2015.
43.
US Census Bureau. 2009-2011 American Community Survey Summary File. https://www2.census.gov/acs2011_5yr/summaryfile/?sec_ak_reference=18.940a1160.1538683639.9f97ba9. 2012.
44.
US News and World Report.  US News and World Report Best Graduate Schools.  April 2013:50-55.
45.
Adams  JL, Mehrotra  A, Thomas  JW, McGlynn  EA.  Physician cost profiling—reliability and risk of misclassification.  N Engl J Med. 2010;362(11):1014-1021. doi:10.1056/NEJMsa0906323PubMedGoogle ScholarCrossref
46.
Fay  R, Herriot  R.  Estimates of income for small places: an application of James-Stein procedures to census data.  J Am Stat Assoc. 1979;74(366):269-277. doi:10.1080/01621459.1979.10482505Google ScholarCrossref
47.
Agency for Healthcare Research and Quality. Instructions for Analyzing Data from CAHPS Surveys. https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/helpful-resources/analysis/2015-instructions-for-analyzing-data.pdf. Updated June 1, 2017. Accessed October 11, 2018.
48.
Cutler  DM, Skinner  JS, Stern  AD, Wennberg  DE.; National Bureau of Economic Research.  Physician Beliefs and Patient Preferences: A New Look at Regional Variation in Health Care Spending. 2013:19320. http://www.nber.org/papers/w19320. Revised February 2018. Accessed October 11, 2018. doi:10.3386/w19320
49.
Gogineni  K, Shuman  KL, Chinn  D, Gabler  NB, Emanuel  EJ.  Patient demands and requests for cancer tests and treatments.  JAMA Oncol. 2015;1(1):33-39. doi:10.1001/jamaoncol.2014.197PubMedGoogle ScholarCrossref
×