[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure.
Methods of Performing Assays by Laboratories Based on Tissue Preparation and Tumor Type
Methods of Performing Assays by Laboratories Based on Tissue Preparation and Tumor Type

A, Percentage of laboratories performing their assays on FFPE tissue preparations exclusively vs laboratories that are inclusive of other types of tissue preparations. B, Percentage of laboratories performing their assays on FDA-approved tumor type (melanoma for BRAF) exclusively vs laboratories that are inclusive of other types of tumors. FDA indicates US Food and Drug Administration; FFPE, formalin-fixed paraffin-embedded; LDT, laboratory-developed tests.

Table 1.  
Acceptable Proficiency Testing Results of FDA Companion Diagnostics vs LDT for BRAF, EGFR, and KRAS
Acceptable Proficiency Testing Results of FDA Companion Diagnostics vs LDT for BRAF, EGFR, and KRAS
Table 2.  
Respondents Reporting That Their Laboratory Performs the Following Preanalytic Steps
Respondents Reporting That Their Laboratory Performs the Following Preanalytic Steps
1.
Evans  JP, Watson  MS.  Genetic testing and FDA regulation: overregulation threatens the emergence of genomic medicine.  JAMA. 2015;313(7):669-670. doi:10.1001/jama.2014.18145PubMedGoogle ScholarCrossref
2.
Ferreira-Gonzalez  A, Emmadi  R, Day  SP,  et al.  Revisiting oversight and regulation of molecular-based laboratory-developed tests: a position statement of the Association for Molecular Pathology.  J Mol Diagn. 2014;16(1):3-6. doi:10.1016/j.jmoldx.2013.10.003PubMedGoogle ScholarCrossref
3.
O’Leary  TJ.  Regulating laboratory-developed tests.  J Mol Diagn. 2014;16(6):595-598. doi:10.1016/j.jmoldx.2014.09.002PubMedGoogle ScholarCrossref
4.
Ratner  M.  FDA pushes for control over laboratory-developed tests.  Nat Biotechnol. 2014;32(9):855. doi:10.1038/nbt0914-855aPubMedGoogle ScholarCrossref
5.
Hwang  TJ, Lehmann  LS, Kesselheim  AS.  Precision medicine and the FDA’s draft guidance on laboratory-developed tests.  Nat Biotechnol. 2015;33(5):449-451. doi:10.1038/nbt.3221PubMedGoogle ScholarCrossref
Brief Report
June 2018

Comparison of Laboratory-Developed Tests and FDA-Approved Assays for BRAF, EGFR, and KRAS Testing

Author Affiliations
  • 1Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
  • 2St Joseph Mercy Hospital, Ann Arbor, Michigan
  • 3University of Nebraska Medical Center, Omaha
  • 4University Health Network, University of Toronto, Ontario, Canada
  • 5University of Texas MD Anderson Cancer Center, Houston
  • 6College of American Pathologists, Northfield, Illinois
  • 7Stanford University, Palo Alto, California
  • 8Columbia University Medical Center, New York, New York
  • 9Yale University School of Medicine, New Haven, Connecticut
  • 10Strong Memorial Hospital, University of Rochester Medical Center, Rochester, New York
  • 11Walter Reed National Military Medical Center, Bethesda, Maryland
JAMA Oncol. 2018;4(6):838-841. doi:10.1001/jamaoncol.2017.4021
Key Points

Question  Are there performance differences between laboratory-developed tests (LDTs) and US Food and Drug Administration–approved companion diagnostics (FDA-CDs [also known as in vitro diagnostics])?

Findings  In 6897 proficiency testing responses, both LDTs and FDA-CDs exceed 97% accuracy combined across all comparable molecular oncology proficiency testing samples. In addition, more than 60% of participants using FDA-CDs report modifying the approved procedure to broaden clinical practice, rendering them LDTs.

Meaning  This study supports the accuracy and comparable performance of LDTs and FDA-CDs and indicates that the majority of laboratories purchasing in vitro diagnostics for FDA-CDs are in fact using them as LDTs.

Abstract

Importance  The debate about the role of the Food and Drug Administration (FDA) in the regulation of laboratory-developed tests (LDTs) has focused attention on the analytical performance of all clinical laboratory testing. This study provides data comparing the performance of LDTs and FDA-approved companion diagnostics (FDA-CDs) in proficiency testing (PT) provided by the College of American Pathologists Molecular Oncology Committee.

Objective  To compare the analytical performance of LDTs and FDA-CDs on well-characterized PT samples and to compare the practice characteristics of laboratories using these assays.

Design, Setting, and Participants  This comparison of PT responses examines the performance of laboratories participating in the College of American Pathologists PT for 3 oncology analytes for which both FDA-CDs and LDTs are used: BRAF, EGFR, and KRAS. A total of 6897 PT responses were included: BRAF (n = 2524; 14 PT samples), EGFR (n = 2216; 11 PT samples), and KRAS (n = 2157, 10 PT samples). US Food and Drug Administration companion diagnostics and LDTs are compared for both accuracy and preanalytic practices of the laboratories.

Main Outcomes and Measures  As per the College of American Pathologists PT standards, results were scored and the percentages of acceptable responses for each analyte were compared. These were also broken down by the specific variants tested, by kit manufacturer for laboratories using commercial reagents, and by preanalytic practices.

Results  From analysis of 6897 PT responses, this study demonstrates that both LDTs and FDA-CDs have excellent performance overall, with both test types exceeding 97% accuracy for all 3 genes (BRAF, EGFR, and KRAS) combined. Rare variant-specific differences did not consistently favor LDTs or FDA-CDs. Additionally, more than 60% of participants using an FDA-CD reported adapting their assay from the approved procedure to allow for a greater breadth of sample types, minimum tumor content, and instrumentation, changing the classification of their assay from FDA-CD to LDT.

Conclusions  This study demonstrates the high degree of accuracy and comparable performance of both LDTs and FDA-CDs for 3 oncology analytes. More significantly, the majority of laboratories using FDA-CDs have modified the scope of their assay to allow for more clinical practice variety, rendering them LDTs. These findings support both the excellent and equivalent performance of both LDTs and FDA-CDs in clinical diagnostic testing.

Introduction

Recent public debate has focused on the regulation of clinical tests.1-5 This includes regulatory practices to ensure the highest quality patient care, with the US Food and Drug Administration (FDA) recommending greater oversight of all in vitro testing. In cancer molecular testing, there are several FDA companion diagnostics (FDA-CDs) sold under the designation of in vitro diagnostics (IVDs). There are many more clinical laboratory-developed tests (LDTs) that are designed, validated, and performed in a single laboratory. Proprietary commercial assays can also fall under the LDT umbrella. This study compares analytical validity of FDA-CDs and LDTs using uniform reference materials from the College of American Pathologists Molecular Oncology Committee.

Methods
Sample Definition and Timeframe

The College of American Pathologists is a well-known provider of external proficiency testing (PT) materials that provide a mechanism for laboratories to fulfill the Clinical Laboratory Improvement Amendments requirement for laboratories to assess the analytical validity of clinical assays during initial development and ongoing clinical use. The College of American Pathologists BRAF PT data from the second half of 2011 through the first half of 2015 were included in this comparison of PT responses (14 samples and 8 mailings). For EGFR, data from the first half of 2013 through the first half of 2015 were examined (11 samples and 5 mailings). For KRAS, data from the second half of 2013 through the first half of 2015 were examined (10 samples and 4 mailings). Additional information detailing PT results categorized as “acceptable,” specific variants assessed, and select qualitative and quantitative preanalytic considerations are available in the Supplement. Based on the participant responses, the testing methodology was categorized as either FDA-CD or LDT. Laboratories using a kit manufactured by a vendor with FDA approval for that kit after the FDA approval date were analyzed as FDA-CDs. All other assays were analyzed as LDTs.

Statistical Analysis

A χ2 test of association compared acceptability across FDA-CDs and LDTs for all tests. For testing acceptability across FDA and LDT by sample type, the Fisher exact test was used owing to counts of fewer than 5 in at least 1 table cell. A significance level of .05 was used.

Results

The overall BRAF College of American Pathologists PT acceptable rate was 96.2% with LDTs achieving a 96.6% acceptable rate while FDA-CD results were significantly lower at 93.0% (P = .002) (Table 1). The main cause of this discrepancy was p.V600K analysis, with LDT acceptable rates at 88.0% while FDA-CDs were 66.1% acceptable (P < .001). For EGFR, LDTs performed slightly less well than the FDA-CDs overall (97.6% acceptability for LDT vs 99.1% for FDA-CDs; P = .03) (Table 1). This discrepancy was driven by the detection of the EGFR p.L861Q mutation (91% for LDTs vs 100% of FDA-CDs; P = .04). For KRAS, there was no significant difference between LDTs and the FDA-CD acceptability rates overall or when the data were broken down by wild-type positions or individual variants.

For all 3 surveys, FDA-CD participants reported using off-label preanalytical practices more than 60% of the time. These off-label practices included accepting unapproved specimen and tumor types (Figure), accepting specimens with lower tumor content than are required for the approved assay, and not quantifying DNA before performing the assay (Table 2). These alterations of the FDA-approved procedure render these tests LDTs.

Discussion

Our primary goal was to compare the accuracy of results between laboratories using FDA-CDs and LDTs. For 1 of 13 variants (BRAF p.V600K), LDTs performed statistically better than FDA-CDs, although 1 of the FDA-CDs is approved only for p.V600E and not other variants at that amino acid or adjacent loci. However, several of these other variants may also respond to BRAF-targeted therapies. For the second variant (EGFR p.L861Q), FDA-CDs performed slightly, but statistically, better than LDTs (Table 1).

This study also compared FDA-CDs and LDTs for preanalytical factors where specifically mentioned in the FDA-CD protocols, including specimen preparations used for testing, pathologist review, DNA quantification, and tissue dissection. The preanalytic questions highlight the fact that many FDA-CD laboratories conduct practices that are not in accord with their FDA-approved methods. Both FDA-CD and LDT laboratories accept a wide range of specimen preparations, as well as tumor types. Although this flexibility is advantageous for patient care, it is important to recognize that the use of specimens other than formalin-fixed paraffin-embedded samples of the specified tumor type for the FDA-CDs is off-label, resulting in reclassification of the assay as an LDT.

Limitations

There are several limitations of this study. First, each PT response was treated as an individual data point rather than grouping the responses by laboratory. Second, all participants using an assay produced by a vendor with an FDA-CD after the FDA approval date were categorized as FDA-CDs although some laboratories may have been using an alternate unapproved kit or had validated it as an LDT. Third, some laboratories failed to provide responses to all of the PT questions. This resulted in inconsistent numbers of data points for many of the variables, although the discrepancies are small and have minimal impact. Finally, the PT did not include questions about all aspects of the practice of each assay. Therefore, this survey cannot determine if there are additional ways that laboratories are using the FDA-CDs off-label.

Conclusions

We find no differences overall between FDA-CDs and LDTs in assay performance for these 3 analytes, with an average of over 97% accuracy from both types of assays more than the 3 surveys, although a technical limitation of one FDA-CD is noted. This study identified alterations from the FDA-approved procedure in greater than 60% of respondents using FDA-CDs, likely to allow for more clinical practice flexibility. Given the overall comparable performance of FDA-CDs and LDTs, as well as the significant off-label use of FDA-CDs, these data question the distinction between FDA-CDs and LDTs from a regulatory standpoint and note the greater clinically relevant applications of LDTs.

Back to top
Article Information

Corresponding Author: Annette S. Kim, MD, PhD, Department of Pathology, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115 (askim@bwh.harvard.edu).

Accepted for Publication: August 29, 2017.

Published Online: December 14, 2017. doi:10.1001/jamaoncol.2017.4021

Author Contributions: Dr Kim and Mr Long had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Kim, Bartley, Bridge, Kamel-Reid, Lazar, Lindeman, Merker, Rai, Rimm, Rothberg, Moncur.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Kim, Bartley, Bridge, Kamel-Reid, Lazar, Lindeman, Long, Merker, Rai, Rimm, Rothberg, Moncur.

Critical revision of the manuscript for important intellectual content: Kim, Bartley, Bridge, Kamel-Reid, Lazar, Lindeman, Merker, Rai, Rimm, Rothberg, Vasalos, Moncur.

Statistical analysis: Kim, Long, Vasalos, Moncur.

Administrative, technical, or material support: Kim, Long, Merker, Vasalos, Moncur.

Study supervision: Kim, Merker, Moncur.

Conflict of Interest Disclosures: Dr Kim works or has worked as a consultant to Aushon Biosciences. Dr Rimm works or has worked as a consultant or advisor to Astra Zeneca, Agendia, Bethyl Labs, Biocept, BMS, Cell Signaling Technology, Merck, OptraScan, Perkin Elmer, and Ultivue; has equity in Metamark Genetics and PixelGear; and receives research funding from Astra Zeneca, Cepheid, Navigate/Novartis, Gilead Sciences, Pierre Fabre, and Perkin Elmer. Dr Rai is the American Association for Clinical Chemistry liaison to the College of American Pathologists Molecular Oncology Committee. Dr Rothberg is the Association for Molecular Pathology liaison to the College of American Pathologists Molecular Oncology Committee. Dr Moncur is employed at Walter Reed National Military Medical Center. The identification of specific products or scientific instrumentation is considered an integral part of the scientific endeavor and does not constitute endorsement or implied endorsement on the part of the authors, the Department of Defense, or any component agency.

Disclaimer: The views expressed in this article are those of the authors and do not reflect the official policy of the Department of Army, Navy, and/or Air Force; Department of Defense; or US Government.

References
1.
Evans  JP, Watson  MS.  Genetic testing and FDA regulation: overregulation threatens the emergence of genomic medicine.  JAMA. 2015;313(7):669-670. doi:10.1001/jama.2014.18145PubMedGoogle ScholarCrossref
2.
Ferreira-Gonzalez  A, Emmadi  R, Day  SP,  et al.  Revisiting oversight and regulation of molecular-based laboratory-developed tests: a position statement of the Association for Molecular Pathology.  J Mol Diagn. 2014;16(1):3-6. doi:10.1016/j.jmoldx.2013.10.003PubMedGoogle ScholarCrossref
3.
O’Leary  TJ.  Regulating laboratory-developed tests.  J Mol Diagn. 2014;16(6):595-598. doi:10.1016/j.jmoldx.2014.09.002PubMedGoogle ScholarCrossref
4.
Ratner  M.  FDA pushes for control over laboratory-developed tests.  Nat Biotechnol. 2014;32(9):855. doi:10.1038/nbt0914-855aPubMedGoogle ScholarCrossref
5.
Hwang  TJ, Lehmann  LS, Kesselheim  AS.  Precision medicine and the FDA’s draft guidance on laboratory-developed tests.  Nat Biotechnol. 2015;33(5):449-451. doi:10.1038/nbt.3221PubMedGoogle ScholarCrossref
×