[Skip to Navigation]
Sign In
January 6, 2020

Challenges to the Reproducibility of Machine Learning Models in Health Care

Author Affiliations
  • 1Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
  • 2Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts
  • 3Computational Health Informatics Program, Boston Children’s Hospital, Boston, Massachusetts
  • 4Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
  • 5Vector Institute, Toronto, Ontario, Canada
JAMA. 2020;323(4):305-306. doi:10.1001/jama.2019.20866

Reproducibility has been an important and intensely debated topic in science and medicine for the past few decades.1 As the scientific enterprise has grown in scope and complexity, concerns regarding how well new findings can be reproduced and validated across different scientific teams and study populations have emerged. In some instances,2 the failure to replicate numerous previous studies has added to the growing concern that science and biomedicine may be in the midst of a “reproducibility crisis.” Against this backdrop, high-capacity machine learning models are beginning to demonstrate early successes in clinical applications,3 and some have received approval from the US Food and Drug Administration. This new class of clinical prediction tools presents unique challenges and obstacles to reproducibility, which must be carefully considered to ensure that these techniques are valid and deployed safely and effectively.

Add or change institution