[Skip to Content]
[Skip to Content Landing]
Table.  
Associations Between Evaluation Independence, Findings, and Conclusions
Associations Between Evaluation Independence, Findings, and Conclusions
1.
Nielsen  M, Buelt  L, Patel  K, Nichols  LM. The Patient-Centered Medical Home’s Impact on Cost and Quality, Annual Review of Evidence, 2014-2015. Patient-Centered Primary Care Collaborative website. https://www.pcpcc.org/sites/default/files/resources/The%20Patient-Centered%20Medical%20Home%27s%20Impact%20on%20Cost%20and%20Quality%2C%20Annual%20Review%20of%20Evidence%2C%202014-2015.pdf. Accessed January 2, 2017.
2.
Friedberg  MW.  What do you mean by medical home?  Ann Intern Med. 2016;164(6):444-445.PubMedGoogle ScholarCrossref
3.
Roper  N, Zhang  N, Korenstein  D.  Industry collaboration and randomized clinical trial design and outcomes.  JAMA Intern Med. 2014;174(10):1695-1696.PubMedGoogle ScholarCrossref
4.
Lo  B, Field  MJ, eds.  Institute of Medicine (US) Committee on Conflict of Interest in Medical Research, Education, and Practice. Conflict of Interest in Medical Research, Education, and Practice. Washington, DC: National Academies Press (US); 2009. https://www.ncbi.nlm.nih.gov/books/NBK22942/. Accessed April 7, 2017.
Research Letter
September 2017

Association Between Sponsorship and Findings of Medical Home Evaluations

Author Affiliations
  • 1Division of Geriatrics, Department of Medicine, University of California–San Francisco
  • 2RAND Corporation, Pittsburgh, Pennsylvania
  • 3RAND Corporation, Boston, Massachusetts
  • 4Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
  • 5Department of Medicine, Harvard Medical School, Boston, Massachusetts
JAMA Intern Med. 2017;177(9):1375-1376. doi:10.1001/jamainternmed.2017.3188

Published evaluations of medical home interventions have reported mixed results.1,2 Some evaluators have been employed or funded by the sponsors of the interventions that they evaluated, while others were independent. Relationships between evaluators and sponsors of medical home interventions might influence evaluation designs and decisions about whether to publish evaluation findings.3,4 If such influence exists, evaluators employed or funded by intervention sponsors might be more likely to publish favorable findings (or suppress unfavorable findings), compared with evaluators who are independent—thus potentially biasing the medical home literature toward favorable effect estimates.

Methods

We searched PubMed, Cumulative Index to Nursing and Allied Health Literature, and Cochrane databases through June 2016 using the key words “medical home,” “health-care home,” “comprehensive primary care,” “advanced primary care,” and “patient aligned care team” to identify 45 evaluations of 23 unique medical home interventions2 that compared participating practices with nonparticipating practices on measures of patient experience, technical quality, cost, or utilization of care.

After masking sponsor information, we calculated the proportion of measures on which each article reported statistically significant favorable quantitative results associated with the medical home intervention (eg, higher quality and/or lower costs at P < .05). Two authors (A.O., G.R.M.) independently rated the tone of each article’s conclusion as favorable (better patient experience or quality, lower costs or utilization), neutral (limited, modest, or potential improvements in patient experience or quality; no associations with costs or utilization), or unfavorable (worse patient experience or quality, higher costs, increased utilization) (κ = 0.90). Disagreements were resolved by consensus. We then unmasked sponsor information and categorized evaluations as “not independent” if the intervention sponsor funded the evaluation or employed 1 or more authors; otherwise, evaluations were “independent.” We used Fisher exact tests to assess relationships between evaluation independence, choice of evaluation measures, proportion of statistically significant favorable results, and tone of article conclusion.

Results

Thirty-two of the 45 articles presented findings from nonindependent evaluations (Table). Independent evaluations were less likely than nonindependent evaluations to report patient experience (n = 0 vs n = 10 [31%]; P = .04) and more likely to report technical quality measures (n = 11 [85%] vs n = 16 [50%]; P = .04). The proportion of favorable quantitative results did not statistically significantly differ between independent and nonindependent evaluations. The conclusions of independent evaluations were less likely than nonindependent evaluations to have favorable tone (n = 6 [46%] vs n = 25 [78%]; P = .04). Results were similar when we separately analyzed nonindependence due to evaluation funding and nonindependence due to evaluator employment.

Discussion

The majority of medical home evaluations have been funded by or included employees of the sponsors of the interventions that they evaluated. We found no statistically significant differences between the quantitative results of independent and nonindependent medical home evaluations. However, the tone of narrative conclusions was more likely to be positive when evaluations were nonindependent. Such narrative conclusions matter because medical home evaluations frequently include multiple quantitative measures, allowing authors leeway in crafting a takeaway message when findings are mixed (eg, in deciding how much relative emphasis to put on favorable vs unfavorable findings).

Our study has limitations. We could not analyze evaluations that lacked published sponsorship information; our literature search identified 3 such evaluations. Although article conclusions were rated after blinding sponsor information and with high interrater agreement, these ratings were subjective. The sample size was modest.

We recommend that policy makers and other stakeholders focus on the quantitative findings of medical home evaluations and consider evaluator independence when reading authors’ descriptions of medical home interventions’ overall effectiveness.

Back to top
Article Information

Corresponding Author: Anna Oh, MSN, MPH, RN, Division of Geriatrics, Department of Medicine, University of California–San Francisco, 4150 Clement St, VA181G, San Francisco, CA 94121 (anna.oh@ucsf.edu).

Accepted for Publication: April 29, 2017.

Published Online: July 31, 2017. doi:10.1001/jamainternmed.2017.3188

Author Contributions: Ms Oh and Dr Martsolf had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: Oh, Martsolf.

Drafting of the manuscript: Oh, Martsolf.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Oh, Martsolf.

Obtained funding: Martsolf.

Administrative, technical, or material support: Oh, Martsolf.

Supervision: Friedberg.

Conflict of Interest Disclosures: Dr Martsolf has received funding to study medical home pilots from the Agency for Healthcare Research and Quality. Dr Friedberg has received compensation from the US Department of Veterans Affairs for consultation related to medical home implementation and research support from the Patient-Centered Outcomes Research Institute via subcontract to the National Committee for Quality Assurance. He also has received funding to evaluate medical home pilots from the Commonwealth Fund, the Centers for Medicare & Medicaid Services, the US Air Force, and North Carolina Innovation, LLC. No other disclosures are reported.

References
1.
Nielsen  M, Buelt  L, Patel  K, Nichols  LM. The Patient-Centered Medical Home’s Impact on Cost and Quality, Annual Review of Evidence, 2014-2015. Patient-Centered Primary Care Collaborative website. https://www.pcpcc.org/sites/default/files/resources/The%20Patient-Centered%20Medical%20Home%27s%20Impact%20on%20Cost%20and%20Quality%2C%20Annual%20Review%20of%20Evidence%2C%202014-2015.pdf. Accessed January 2, 2017.
2.
Friedberg  MW.  What do you mean by medical home?  Ann Intern Med. 2016;164(6):444-445.PubMedGoogle ScholarCrossref
3.
Roper  N, Zhang  N, Korenstein  D.  Industry collaboration and randomized clinical trial design and outcomes.  JAMA Intern Med. 2014;174(10):1695-1696.PubMedGoogle ScholarCrossref
4.
Lo  B, Field  MJ, eds.  Institute of Medicine (US) Committee on Conflict of Interest in Medical Research, Education, and Practice. Conflict of Interest in Medical Research, Education, and Practice. Washington, DC: National Academies Press (US); 2009. https://www.ncbi.nlm.nih.gov/books/NBK22942/. Accessed April 7, 2017.
×