[Skip to Navigation]
Sign In
January 6, 2020

Challenges to the Reproducibility of Machine Learning Models in Health Care

Author Affiliations
  • 1Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
  • 2Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts
  • 3Computational Health Informatics Program, Boston Children’s Hospital, Boston, Massachusetts
  • 4Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
  • 5Vector Institute, Toronto, Ontario, Canada
JAMA. 2020;323(4):305-306. doi:10.1001/jama.2019.20866

Reproducibility has been an important and intensely debated topic in science and medicine for the past few decades.1 As the scientific enterprise has grown in scope and complexity, concerns regarding how well new findings can be reproduced and validated across different scientific teams and study populations have emerged. In some instances,2 the failure to replicate numerous previous studies has added to the growing concern that science and biomedicine may be in the midst of a “reproducibility crisis.” Against this backdrop, high-capacity machine learning models are beginning to demonstrate early successes in clinical applications,3 and some have received approval from the US Food and Drug Administration. This new class of clinical prediction tools presents unique challenges and obstacles to reproducibility, which must be carefully considered to ensure that these techniques are valid and deployed safely and effectively.

Add or change institution
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words