Evaluation of Journal Registration Policies and Prospective Registration of Randomized Clinical Trials of Nonregulated Health Care Interventions | Medical Journals and Publishing | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Figure.  Flowchart of Included Articles
Flowchart of Included Articles
Table 1.  Journal Registration Policies of 254 Journals That Published Trials
Journal Registration Policies of 254 Journals That Published Trials
Table 2.  Association Between Journal Registration Policies and Journal Characteristics
Association Between Journal Registration Policies and Journal Characteristics
Table 3.  Registration Status of the 953 Trials
Registration Status of the 953 Trials
Table 4.  Association Between Trial Registration Status and Journal and Trial Characteristics for the 953 Trials
Association Between Trial Registration Status and Journal and Trial Characteristics for the 953 Trials
1.
Hartung  DM, Zarin  DA, Guise  J-M, McDonagh  M, Paynter  R, Helfand  M.  Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications.  Ann Intern Med. 2014;160(7):477-483. doi:10.7326/M13-0480PubMedGoogle ScholarCrossref
2.
Zarin  DA, Tse  T, Williams  RJ, Califf  RM, Ide  NC.  The ClinicalTrials.gov results database–update and key issues.  N Engl J Med. 2011;364(9):852-860. doi:10.1056/NEJMsa1012065PubMedGoogle ScholarCrossref
3.
Zarin  DA, Tse  T, Sheehan  J.  The proposed rule for U.S. clinical trial registration and results submission.  N Engl J Med. 2015;372(2):174-180. doi:10.1056/NEJMsr1414226PubMedGoogle ScholarCrossref
4.
Section 801 of the Food and Drug Administration Amendments Act, P.L. No. 110–85, §121 Stat. 823. https://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf. Accessed November 14, 2018.
5.
Food and Drug Administration Modernization Act of 1997, P.L. No. 105-115, §111 Stat. 2296. https://www.gpo.gov/fdsys/pkg/PLAW-105publ115/pdf/PLAW-105publ115.pdf. Accessed November 14, 2018.
6.
US Food and Drug Administration. Investigation new drug (IND) application. https://www.fda.gov/drugs/developmentapprovalprocess/howdrugsaredevelopedandapproved/approvalapplications/investigationalnewdrugindapplication/default.htm. Accessed November 14, 2018.
7.
US Food and Drug Administration. Device advice: investigational device exemption (IDE). https://www.fda.gov/medicaldevices/deviceregulationandguidance/howtomarketyourdevice/investigationaldeviceexemptionide/default.htm. Accessed November 14, 2018.
8.
European Commission. Directive 2001/20/EC of the European parliament and of the council of 4 April 2001 on the approximation of the laws, regulations, and administrative provisions of the member states relating to the implementation of good clinical practice in the conduct of clinical trials on medicinal products for human use. May 2001. http://ec.europa.eu/health//sites/health/files/files/eudralex/vol-1/dir_2001_20/dir_2001_20_en.pdf. Accessed November 14, 2018.
9.
EU Clinical Trials Register. About the EU Clinical Trials Register. https://www.clinicaltrialsregister.eu/about.html. Accessed November 14, 2018.
10.
Dal-Ré  R, Bracken  MB, Ioannidis  JP.  Call to improve transparency of trials of non-regulated interventions.  BMJ. 2015;350:h1323. doi:10.1136/bmj.h1323PubMedGoogle ScholarCrossref
11.
Dal-Ré  R.  Preregistration and publication of nonregulated intervention trials are here to stay (letter commenting Wallach et al 2018, 93, 88-93).  J Clin Epidemiol. 2018;99:167-168. doi:10.1016/j.jclinepi.2018.02.014PubMedGoogle ScholarCrossref
12.
Dal-Ré  R, Ross  JS, Marušić  A.  Compliance with prospective trial registration guidance remained low in high-impact journals and has implications for primary end point reporting.  J Clin Epidemiol. 2016;75:100-107. doi:10.1016/j.jclinepi.2016.01.017PubMedGoogle ScholarCrossref
13.
De Angelis  C, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.  N Engl J Med. 2004;351(12):1250-1251. doi:10.1056/NEJMe048225PubMedGoogle ScholarCrossref
14.
De Angelis  CD, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Is this clinical trial fully registered? a statement from the International Committee of Medical Journal Editors.  N Engl J Med. 2005;352(23):2436-2438. doi:10.1056/NEJMe058127PubMedGoogle ScholarCrossref
15.
International Committee of Medical Journal Editors. Clinical trial registration. http://www.icmje.org/about-icmje/faqs/clinical-trials-registration/. Accessed November 14, 2018.
16.
International Committee of Medical Journal Editors. Journals following the ICMJE recommendations. http://www.icmje.org/journals-following-the-icmje-recommendations/. Accessed November 14, 2018.
17.
Turner  EH, Matthews  AM, Linardatos  E, Tell  RA, Rosenthal  R.  Selective publication of antidepressant trials and its influence on apparent efficacy.  N Engl J Med. 2008;358(3):252-260. doi:10.1056/NEJMsa065779PubMedGoogle ScholarCrossref
18.
Turner  EH, Knoepflmacher  D, Shapley  L.  Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database.  PLoS Med. 2012;9(3):e1001189. doi:10.1371/journal.pmed.1001189PubMedGoogle ScholarCrossref
19.
Dwan  K, Altman  DG, Cresswell  L, Blundell  M, Gamble  CL, Williamson  PR.  Comparison of protocols and registry entries to published reports for randomised controlled trials.  Cochrane Database Syst Rev. 2011;(1):MR000031.PubMedGoogle Scholar
20.
Dwan  K, Altman  DG, Clarke  M,  et al.  Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials.  PLoS Med. 2014;11(6):e1001666. doi:10.1371/journal.pmed.1001666PubMedGoogle ScholarCrossref
21.
Vedula  SS, Bero  L, Scherer  RW, Dickersin  K.  Outcome reporting in industry-sponsored trials of gabapentin for off-label use.  N Engl J Med. 2009;361(20):1963-1971. doi:10.1056/NEJMsa0906126PubMedGoogle ScholarCrossref
22.
Chalmers  I, Glasziou  P.  Avoidable waste in the production and reporting of research evidence.  Obstet Gynecol. 2009;114(6):1341-1345. doi:10.1097/AOG.0b013e3181c3020dPubMedGoogle ScholarCrossref
23.
Moher  D, Glasziou  P, Chalmers  I,  et al.  Increasing value and reducing waste in biomedical research: who’s listening?  Lancet. 2016;387(10027):1573-1586. doi:10.1016/S0140-6736(15)00307-4PubMedGoogle ScholarCrossref
24.
Babu  AS, Veluswamy  SK, Rao  PT, Maiya  AG.  Clinical trial registration in physical therapy journals: a cross-sectional study.  Phys Ther. 2014;94(1):83-90. doi:10.2522/ptj.20120531PubMedGoogle ScholarCrossref
25.
Pinto  RZ, Elkins  MR, Moseley  AM,  et al.  Many randomized trials of physical therapy interventions are not adequately registered: a survey of 200 published trials.  Phys Ther. 2013;93(3):299-309. doi:10.2522/ptj.20120206PubMedGoogle ScholarCrossref
26.
Jull  A, Aye  PS.  Endorsement of the CONSORT guidelines, trial registration, and the quality of reporting randomised controlled trials in leading nursing journals: a cross-sectional analysis.  Int J Nurs Stud. 2015;52(6):1071-1079. doi:10.1016/j.ijnurstu.2014.11.008PubMedGoogle ScholarCrossref
27.
Killeen  S, Sourallous  P, Hunter  IA, Hartley  JE, Grady  HLO.  Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals.  Ann Surg. 2014;259(1):193-196. doi:10.1097/SLA.0b013e318299d00bPubMedGoogle ScholarCrossref
28.
Hardt  JLS, Metzendorf  M-I, Meerpohl  JJ.  Surgical trials and trial registers: a cross-sectional study of randomized controlled trials published in journals requiring trial registration in the author instructions.  Trials. 2013;14:407. doi:10.1186/1745-6215-14-407PubMedGoogle ScholarCrossref
29.
Milette  K, Roseman  M, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review.  J Psychosom Res. 2011;70(3):205-217. doi:10.1016/j.jpsychores.2010.09.015PubMedGoogle ScholarCrossref
30.
Riehm  KE, Azar  M, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: A 5-year follow-up.  J Psychosom Res. 2015;79(1):1-12. doi:10.1016/j.jpsychores.2015.04.010PubMedGoogle ScholarCrossref
31.
Azar  M, Riehm  KE, McKay  D, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials published in the Journal of Consulting and Clinical Psychology PLoS One. 2015;10(11):e0142894. doi:10.1371/journal.pone.0142894PubMedGoogle ScholarCrossref
32.
Cybulski  L, Mayo-Wilson  E, Grant  S.  Improving transparency and reproducibility through registration: the status of intervention trials published in clinical psychology journals.  J Consult Clin Psychol. 2016;84(9):753-767. doi:10.1037/ccp0000115PubMedGoogle ScholarCrossref
33.
Bradley  HA, Rucklidge  JJ, Mulder  RT.  A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials.  Acta Psychiatr Scand. 2017;135(1):65-77. doi:10.1111/acps.12647PubMedGoogle ScholarCrossref
34.
Internet Archive. Wayback Machine. https://archive.org/web/. Accessed October 15, 2018.
35.
Murphy  J, Hashim  NH, O’Connor  P.  Take me back: validating the Wayback Machine.  J Comput Mediat Commun. 2007;13(1):60-75. doi:10.1111/j.1083-6101.2007.00386.xGoogle ScholarCrossref
36.
Chan  AW, Tetzlaff  JM, Gøtzsche  PC,  et al.  SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials.  BMJ. 2013;346:e7586. doi:10.1136/bmj.e7586PubMedGoogle ScholarCrossref
37.
Altman  DG, Moher  D.  Importance of Transparent Reporting of Health Research: Guidelines for Reporting Health Research: A User’s Manual. Hoboken, NJ: John Wiley & Sons; 2014:1-13.
38.
Ioannidis  JPA, Greenland  S, Hlatky  MA,  et al.  Increasing value and reducing waste in research design, conduct, and analysis.  Lancet. 2014;383(9912):166-175. doi:10.1016/S0140-6736(13)62227-8PubMedGoogle ScholarCrossref
39.
Chan  AW, Song  F, Vickers  A,  et al.  Increasing value and reducing waste: addressing inaccessible research.  Lancet. 2014;383(9913):257-266. doi:10.1016/S0140-6736(13)62296-5PubMedGoogle ScholarCrossref
40.
Smyth  RMD, Kirkham  JJ, Jacoby  A, Altman  DG, Gamble  C, Williamson  PR.  Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists.  BMJ. 2011;342:c7153. doi:10.1136/bmj.c7153PubMedGoogle ScholarCrossref
41.
NIH Policy on the Dissemination of NIH-funded Clinical Trial Information. https://grants.nih.gov/policy/clinical-trials/reporting/understanding/nih-policy.htm. Accessed November 14, 2018.
42.
DeVito  NJ, French  L, Goldacre  B.  Noncommercial funders’ policies on trial registration, access to summary results, and individual patient data availability.  JAMA. 2018;319(16):1721-1723. doi:10.1001/jama.2018.2841PubMedGoogle ScholarCrossref
43.
Ioannidis  JPA.  Why most clinical research is not useful.  PLoS Med. 2016;13(6):e1002049. doi:10.1371/journal.pmed.1002049PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    March 11, 2019

    Evaluation of Journal Registration Policies and Prospective Registration of Randomized Clinical Trials of Nonregulated Health Care Interventions

    Author Affiliations
    • 1Lady Davis Institute for Medical Research, Jewish General Hospital, Montreal, Quebec, Canada
    • 2Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Quebec, Canada
    • 3Department of Psychology, McGill University, Montreal, Quebec, Canada
    • 4Library, Concordia University, Montreal, Quebec, Canada
    • 5Studies of Translation, Ethics and Medicine, Biomedical Ethics Unit, McGill University, Montreal, Quebec, Canada
    • 6Respiratory Epidemiology and Clinical Research Unit, McGill University Health Centre, Montreal, Quebec, Canada
    • 7Department of Psychiatry, McGill University, Montreal, Quebec, Canada
    • 8Department of Medicine, McGill University, Montreal, Quebec, Canada
    • 9Department of Educational and Counselling Psychology, McGill University, Montreal, Quebec, Canada
    JAMA Intern Med. 2019;179(5):624-632. doi:10.1001/jamainternmed.2018.8009
    Key Points

    Question  Have specialty journals, such as in the fields of nursing, rehabilitation, or surgery, that publish randomized clinical trials of health care interventions not subject to regulation by the US Food and Drug Administration implemented and enforced prospective trial registration policies?

    Findings  In this study of 953 nonregulated intervention trials published in 254 journals included in the study, only 11% of journals required prospective registration. Only 34% of trials published in journals with a policy were prospectively registered compared with 18% of trials in journals without a policy.

    Meaning  Nonregulated health care intervention trials published in specialty journals are often not prospectively registered, and existing registration policies are rarely enforced;

    Abstract

    Importance  Many interventions that are important to the health care of patients are not subject to regulation by the US Food and Drug Administration (FDA) or comparable regulatory bodies in other nations.

    Objective  To determine whether specialty journals that publish trials of primarily nonregulated health care interventions require prospective registration and whether the prospective registration policies are associated with the publication of prospectively registered trials, trials with adequately registered outcomes, and trials with primary outcomes consistent with the registered primary outcomes.

    Design and Methods  PubMed was searched daily, from March 18, 2016, to September 17, 2016, for nonregulated intervention randomized clinical trials. The search included all journals in the Clarivate Analytics Science Citation Index Expanded categories of behavioral sciences, nursing, nutrition and dietetics, psychology, rehabilitation, and surgery. Trials of interventions not subject to FDA regulation were included. One investigator extracted journal registration policy and trial registration status. Two investigators independently extracted trial registration and publication characteristics.

    Main Outcomes and Measures  For journals, the main outcome was the trial registration policy. For trials, the main outcomes were prospective registration, adequacy of outcome registration, and concordance of registered with published primary outcomes.

    Results  In total, 953 nonregulated intervention trials published in 254 journals were identified. Prospective registration was required for publication by 29 (11.4%) of 254 journals, and an additional 12 journals (4.7%) had conditional date-based requirements. Only 189 (19.8%) of the 953 trials were registered prospectively, including 33 of 98 published in journals with prospective registration policies as compared with 156 of 855 in journals without policies (33.7% vs 18.2%; P = .004). Among the 17 journals that required prospective registration and had at least 2 included trials, none had a prospective registration of more than 50%. In journals with policies, only 3 of 98 trials included primary outcomes consistent with prospectively, adequately registered outcomes, as compared with 34 of 852 trials in journals without policies (3.1% vs 4.0%; P = .62).

    Conclusions and Relevance  Few journals in behavioral sciences or psychology, nursing, nutrition and dietetics, rehabilitation, and surgery require prospective trial registration, and those with existing registration policies rarely enforce them; this finding suggests that strategies for encouraging prospective registration of clinical trials not subject to FDA regulation should be developed and tested.

    Introduction

    In the United States, Food and Drug Administration regulations require prospective registration of most phase 2, 3, and 4 trials of drugs, biologics, and medical devices.1-7 The European Union requires prospective registration of trials of drugs and biologics.8,9 The term nonregulated interventions has been used to refer to health care interventions that are not subject to regulatory oversight and thus not subject to these registration requirements.10-12 Examples of nonregulated health care interventions include surgical procedures, nursing interventions, diets, psychological treatments, and rehabilitation procedures.10-12 Although trials of these interventions may be subject to less scrutiny than trials of drugs and medical devices, some of these interventions pose substantial risks to patients, and some may have important implications for the health care system.

    Since 2005, International Committee of Medical Journal Editors (ICMJE) journals have required that all clinical trials of health care interventions considered for publication be registered prior to participant enrollment.13-15 Many non-ICMJE journals have adopted this policy.16 Prospective registration policies are intended to reduce the risk of bias from nonreporting of trial results or the selective reporting of analyses and outcomes.17-23

    Many nonregulated health care intervention trials are not registered prospectively, even when journal registration requirements are in place.10,24-33 Previous studies on registration policies and practices, however, have focused on retrospective analyses of small groups of journals. One reason is that it is often not possible to determine retrospectively when journal registration policies, if any, were initiated and, thus, to evaluate compliance.

    We conducted a daily search for all publications of randomized clinical trials (RCTs) of nonregulated interventions in journals in the categories of behavioral sciences or psychology, nursing, nutrition and dietetics, rehabilitation, and surgery that were initially listed in PubMed during a 6-month period in 2016. By searching daily, we could examine journal registration policies as close as possible to when the RCTs were submitted. We studied journal registration policies, the associations between these policies and the prospective registration of trials, the registration of primary outcomes and whether this registration was adequate, and the consistency of registered and primary outcomes.

    Methods
    Journals

    Ethics approval was not required for this study. We identified all journals listed in the following categories of the 2014 Clarivate Analytics Science Citation Index Expanded: (1) behavioral sciences, (2) nursing, (3) nutrition and dietetics, (4) psychology, (5) rehabilitation, and (6) surgery, that published at least 1 article listed in PubMed in 2014 or 2015. We compared journal names listed in the Science Citation Index Expanded against those in the National Library of Medicine Catalog and used National Library of Medicine names for this search. The search strategy included 50 behavioral sciences, 107 nursing, 68 nutrition and dietetics, 66 psychology, 57 rehabilitation, and 187 surgery journals, totaling 524 journals given that 11 were in more than 1 category. These 524 journals are listed in eTable 1 in the Supplement.

    Identification of Nonregulated Health Care Intervention RCTs

    We searched PubMed daily, from March 18, 2016, through September 17, 2016, using the names of included journals with the following terms: (random*[Title/Abstract] OR “RCT”[Title/Abstract] NOT (“Drug Therapy”[Mesh] OR “drug therapy”[Subheading] OR “Pharmaceutical Preparations”[Mesh] OR Review[ptyp] OR “meta-analysis”[Title] OR “systematic review”[Title]). More information is provided in eTable 2 in the Supplement.

    Eligible RCTs were trials that evaluated the effects of nonregulated health care interventions on health outcomes.13-15 We excluded trials with a comparison to a drug, biologic, or medical device in any arm as well as publications that stated they reported only secondary outcomes. If results from more than 1 RCT were reported in a single publication, we included only the first RCT in the article.

    We uploaded search results into DistillerSR (Evidence Partners), which was used for all coding procedures. Given the available resources, one investigator assessed titles and abstracts and full-text articles for eligibility.

    Data Extraction

    For each eligible RCT, one investigator extracted journal characteristics, including registration policy, and trial characteristics, including registration status. Two investigators independently extracted trial registration information and published outcomes, and discrepancies were resolved through consensus and consultation with a third investigator, as necessary. All extracted data are available on the Open Science Framework at https://osf.io/jk2sq/.

    Journal Registration Policy

    For each RCT in the study, we reviewed the journal website, including author instructions, within 3 weeks of the daily PubMed search that identified the RCT. Journals were classified as (1) requiring prospective registration; (2) requiring registration with no timing requirement; (3) not requiring registration; or (4) having conditional requirements, typically based on date of trial initiation or journal submission. For RCTs in journals with conditional requirements, we determined whether the requirements applied to the RCT on the basis of dates extracted from the trial publication or, if not specified there, the trial registration.

    To evaluate whether journal policies might have changed between the submission date of a trial and its PubMed listing, we used the website Wayback Machine, which stores archived website snapshots.34,35 For trials with published journal submission dates, if a snapshot was available within 3 months prior to the submission date, we compared the registration policy from the most recent snapshot prior to submission with the policy on the journal website at the time of our PubMed search.

    Prospective Registration

    To determine trial registration status, we followed a procedure used in previous studies.29-31 We first attempted to retrieve registration data from the publication. If the registration data were not reported, we searched ClinicalTrials.gov; the International Standard Randomized Controlled Trials Number registry (http://www.isrctn.com); and registries, if available, from the first and corresponding authors’ countries. For this search, we used corresponding, first, and last author names and key terms. We attempted to match the principal investigator, funding source, intervention, control group, and trial design of the RCT with registration records. If we did not find a registry entry or if we found ambiguity about the match, we contacted the corresponding author once by email.

    To determine if trials were registered prospectively, we compared registration and enrollment dates. We prioritized enrollment dates from publications, but if they were not available, we used dates from trial registrations. We classified trials registered prior to or in the same month enrollment began as registered prospectively.

    Adequate Primary Outcome Registration

    Among RCTs registered prospectively, we recorded whether registrations included a specific measurement of the primary outcome or outcomes (eg, Beck Depression Inventory score, not simply depression), a specific assessment time point (eg, 12-month postrandomization), a specific metric (eg, change from baseline, final value), and a specific method of aggregation (eg, mean, proportion).2,36 We classified registrations as adequate if the registration included a single primary outcome or multiple primary outcomes with a specific measurement and time point of assessment.29-31 We did not require specification of analysis metric or aggregation method for classification of adequacy.

    Trials were classified as adequately registered—single primary outcome if they adequately registered a single primary outcome variable and assessment time point, or multiple primary outcomes or time points, and there was a plan for statistical adjustment for multiple comparisons included in the registration as well as a subsequent publication that included the same primary outcomes or time points with appropriate statistical adjustment. If registrations listed multiple adequately registered primary outcomes, but there was no statistical adjustment for multiple outcomes, the trial was classified as adequately registered—multiple primary outcomes. If there were changes in trial registration records over time, we used the last update prior to participant enrollment.

    Consistency of Registered and Published Outcomes

    For each adequately registered trial, we compared the registered primary outcome with published primary outcome. Trials were classified as consistent if registered and published primary outcome analyses were the same and discrepant if they differed.

    Statistical Analysis

    We present frequencies and percentages of journals with prospective registration requirements; journals with registration requirements with no timing requirement; and journals without a policy, both overall and by journal categories, impact factor, and publishers. For journal policies and prospective registration, we present frequencies and percentages of trials registered prospectively, registered during the trial, registered after the trial, and not registered, both overall and by journal characteristics (registration policy, impact factor, registration policy stratified by impact factor, and publisher) and trial characteristics (category of intervention, country, funding, and sample size). We also present frequencies and percentages of trials with prospectively and adequately registered primary outcomes and with published outcomes consistent with those registered.

    We conducted statistical tests to evaluate whether the presence of a journal prospective registration policy was associated with the prospective registration of trials published in the journal, both overall and stratified by impact factor, prospective and adequate registration of primary outcomes in trials published in the journal, and publication of primary outcomes consistent with prospective and adequately registered primary outcomes. In addition, we tested the association of trial registration with journal impact factor. For all of these analyses, we used generalized estimating equations with a binary outcome and an identity link function. We used an exchangeable correlation structure with robust SEs to account for the correlation between trials within the same journal. Analyses were run using the geepack package in R, version 3.2.3 (R Foundation for Statistical Computing). We used α <.05 to test for statistical significance.

    Results
    Article Selection

    The search retrieved 4930 citations, of which 3332 (67.6%) were excluded after title and abstract review and 645 (13.1%) after full-text review. The remaining 953 RCTs (19.3%) were included (Figure).

    Journal Registration Policy

    The 953 RCTs were published in 254 journals, of which 100 (39.4%) published 1 of these trials in the 6-month study period, 58 (22.8%) published 2, 42 (16.5%) published 3 to 4, 34 (13.4%) published 5 to 10, and 20 (7.9%) published 11 to 44. Of the 254 journals, 29 (11.4%) required prospective trial registration and 12 (4.7%) had conditional policies based on the date of trial initiation or journal submission, as shown in Table 1. Table 2 shows the frequencies and percentages of journal trial registration policies by journal category, impact factor, and publisher.

    Changes in Journal Registration Requirements

    Of the articles and journals, 650 articles (68.2%) from 165 journals (65.0%) included a published submission date. Of these, there was a Wayback Machine archive of author instructions in the 3 months prior to the submission date for 126 articles (13.2%) from 46 journals (18.1%). Author instructions changed between the Wayback Machine archive and our PubMed search date for only 1 article. For that article, the journal Nursing and Health Sciences added a statement encouraging, but not requiring, trial registration. This archived information did not affect the journal’s classification.

    For 3 journals (Disability and Rehabilitation, Journal of Head Trauma Rehabilitation, and Obesity), author instructions changed during the 6-month daily search period. Articles published in Disability and Rehabilitation and Obesity included a submission date, and all included RCTs were submitted prior to the change from not requiring to requiring prospective trial registration; we classified both journals as not requiring registration for all included trials. Articles published in the Journal of Head Trauma Rehabilitation did not provide submission dates, but we confirmed through PubMed listing dates and consultation with the journal editor that all 6 included trials were submitted prior to the change in journal policy. We also classified this journal as not requiring prospective trial registration.

    Prospective Registration

    Of the 953 trials, 189 (19.8%) were registered prospectively, 141 (14.8%) after initiation of participant enrollment and prior to collection of trial outcomes, and 120 (12.6%) after completion of the trial. More than half (496 [52.0%]) were not registered (Table 3; eTable 3 in the Supplement). Of the 141 trials registered after participants were enrolled, 3 (2.1%) were registered within a month and 61 (43.3%) within 6 months.

    As shown in Table 4, 98 (10.3%) of the 953 trials were published in a journal that required prospective registration. Of these RCTs, 33 were registered prospectively, as compared with 156 of 855 trials published in journals without such a requirement (33.7% vs 18.2%; P = .004). Among journals that required prospective registration unconditionally, 17 journals had at least 2 included trials (range of 2 to 14 trials). The percentage of prospectively registered trials was not greater than 50% in any journals. Table 4 also shows prospective registration by impact factor, journal policy stratified by impact factor, publisher, and trial characteristics.

    Adequate Registration of Primary Outcomes

    Of the 953 included trials, 3 (0.3%) were completed or still active in July 2005 and met ICMJE requirements to register by September 2005. These RCTs were not required to register before patients were enrolled. Of the remaining 950 trials, based on the criterion of having a specific measurement and time point, only 85 trials (8.9%) prospectively and adequately registered a single primary outcome (60 [6.3%]) or multiple primary outcomes (25 [2.6%]). This group included 15 of 98 trials that were subjected to prospective registration requirements, as compared with 70 of 852 trials in journals without policies (15.3% vs 8.2%; P = .04).

    Consistency of Registered and Published Outcomes and Journal Policy

    Among the 950 trials that first enrolled participants after July 2005, 37 (3.9%) adequately registered a single primary outcome or multiple primary outcomes and published a consistent outcome or outcomes. This included 35 (58.3%) of 60 trials that registered a single primary outcome and 2 (8.0%) of 25 trials that registered multiple primary outcomes. Among 98 trials subject to a journal policy that required prospective registration, only 3 had adequate registration and a consistent published primary outcome or outcomes, as compared with 34 of 852 not subject to such a policy (3.1% vs 4.0%; P = .62). Of the 48 trials that published primary outcomes discrepant to those registered, 14 (29.2%) registered a single primary outcome but published multiple primary outcomes, 7 (14.6%) registered multiple primary outcomes but published a single primary outcome, 15 (31.3%) registered and published different primary outcomes, and 12 (25.0%) did not identify which outcome or outcomes were primary in the published report.

    Discussion

    We reviewed 953 RCTs of nonregulated health care interventions with initial PubMed listings in a single 6-month period in 2016 that were published in 254 journals in the categories of behavioral sciences, psychology, nursing, nutrition and dietetics, rehabilitation, and surgery. Prospective registration for all submitted articles was required for 29 journals (11.4%), and an additional 12 journals (4.7%) had conditional prospective registration. Of the trials, 189 (19.8%) were registered prospectively; 85 (8.9%) were registered prospectively with an adequately defined primary outcome or outcomes; and 37 (3.9%) published primary outcomes consistent with prospectively, adequately registered outcomes. Prospective registration was about twice as high in journals that required prospective registration as a condition for publication (33 [33.7%]), as compared with journals without such a policy (156 [18.2%]). Our findings suggest, however, that even when prospective registration policies are in place, they are rarely enforced.

    Despite research demonstrating bias and waste owing to misrepresented research results,37-39 our findings suggest that many researchers and journal editors may still be unaware of the importance of trial registration and transparent outcome reporting.40 Research funders and ethics committees are well positioned to effectively enforce registration requirements.10 In the United States, in 2017, the National Institutes of Health began to require preregistration and results reporting of all clinical trials that it funds.41 Other governmental and nongovernmental funding organizations have made similar commitments.1,42 However, the degree to which these requirements for preregistration are enforced or whether the quality of trial registrations is reviewed by funders is not known. A publicly visible report card that would grade journals on their degree of compliance with trial registration requirements and enforcement could encourage needed change.43

    Limitations

    One limitation of this study is that we focused on journals in specialty areas and not other types of journals. We did not collect information on registration of trials of drugs, biologic, and devices that may have been published in the 254 journals in this study. A study that reviewed trials published in 2014 in the 6 highest-impact general medical journals found that 79% of trials of Food and Drug Administration–regulated interventions complied with ICMJE preregistration requirements, as compared with 58% of trials of nonregulated interventions.12 Another limitation is that we used journal policy at the time of trial listing on PubMed rather than at the date of manuscript submission. Our review of internet archives and tracking of journal policies, however, found that few journals had recently changed registration policies; thus, the main findings would not likely be affected by any undetected changes. We did not examine whether included trials reported summary results in clinical trial registries, which would be an important area for future research.

    Conclusions

    Few specialty journals that publish nonregulated intervention trial results have adopted comprehensive trial registration policies. Although these policies are in place, they are rarely enforced. Without universal trial registration, it is difficult to interpret the meaning of trial results because of inherent biases, including selective reporting and publication bias. Journals and other entities with a stake in clinical research, including researchers, funding agencies, ethics review boards, and academic medical centers, should be concerned about these findings. They should make a concerted effort to improve trial registration practices.

    Back to top
    Article Information

    Accepted for Publication: November 23, 2018.

    Corresponding Author: Brett D. Thombs, PhD, Lady Davis Institute for Medical Research, Jewish General Hospital, 4333 Cote Ste Catherine Rd, Montréal, QC H3T 1E4, Canada (brett.thombs@mcgill.ca).

    Published Online: March 11, 2019. doi:10.1001/jamainternmed.2018.8009

    Author Contributions: Ms Azar and Dr Thombs had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Azar, Rice, Benedetti, Thombs.

    Acquisition, analysis, or interpretation of data: Azar, Riehm, Saadat, Sanchez, Chiovitti, Qi, Rice, B. Levis, Fedoruk, A. W. Levis, Kloda, Kimmelman, Thombs.

    Drafting of the manuscript: Azar, Chiovitti, Thombs.

    Critical revision of the manuscript for important intellectual content: Azar, Riehm, Saadat, Sanchez, Qi, Rice, B. Levis, Fedoruk, A. W. Levis, Kloda, Kimmelman, Benedetti, Thombs.

    Statistical analysis: Azar, Benedetti, Thombs.

    Administrative, technical, or material support: Azar, Saadat, Sanchez, Chiovitti, Qi, B. Levis, Fedoruk.

    Supervision: Benedetti, Thombs.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: This study received no funding, but the following authors received salary support or student stipends: Mss Azar and Rice and Mr Levis were supported by Fonds de Recherche Québec—Santé (FRQS) Masters Training Awards; Ms Riehm, by a Canadian Institute of Health Research (CIHR) Masters Research Award; Ms Levis, by a CIHR Doctoral Research Award; and Drs Thombs and Benedetti, by FRQS researcher awards.

    Role of the Funder/Sponsor: The salary and stipend sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: The authors thank Ms. Angie Ye Eun Lee, BSc (medical student at Yonsei University College of Medicine in Seoul, South Korea), for translating articles published in Korean. She received no compensation for her contribution.

    References
    1.
    Hartung  DM, Zarin  DA, Guise  J-M, McDonagh  M, Paynter  R, Helfand  M.  Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications.  Ann Intern Med. 2014;160(7):477-483. doi:10.7326/M13-0480PubMedGoogle ScholarCrossref
    2.
    Zarin  DA, Tse  T, Williams  RJ, Califf  RM, Ide  NC.  The ClinicalTrials.gov results database–update and key issues.  N Engl J Med. 2011;364(9):852-860. doi:10.1056/NEJMsa1012065PubMedGoogle ScholarCrossref
    3.
    Zarin  DA, Tse  T, Sheehan  J.  The proposed rule for U.S. clinical trial registration and results submission.  N Engl J Med. 2015;372(2):174-180. doi:10.1056/NEJMsr1414226PubMedGoogle ScholarCrossref
    4.
    Section 801 of the Food and Drug Administration Amendments Act, P.L. No. 110–85, §121 Stat. 823. https://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf. Accessed November 14, 2018.
    5.
    Food and Drug Administration Modernization Act of 1997, P.L. No. 105-115, §111 Stat. 2296. https://www.gpo.gov/fdsys/pkg/PLAW-105publ115/pdf/PLAW-105publ115.pdf. Accessed November 14, 2018.
    6.
    US Food and Drug Administration. Investigation new drug (IND) application. https://www.fda.gov/drugs/developmentapprovalprocess/howdrugsaredevelopedandapproved/approvalapplications/investigationalnewdrugindapplication/default.htm. Accessed November 14, 2018.
    7.
    US Food and Drug Administration. Device advice: investigational device exemption (IDE). https://www.fda.gov/medicaldevices/deviceregulationandguidance/howtomarketyourdevice/investigationaldeviceexemptionide/default.htm. Accessed November 14, 2018.
    8.
    European Commission. Directive 2001/20/EC of the European parliament and of the council of 4 April 2001 on the approximation of the laws, regulations, and administrative provisions of the member states relating to the implementation of good clinical practice in the conduct of clinical trials on medicinal products for human use. May 2001. http://ec.europa.eu/health//sites/health/files/files/eudralex/vol-1/dir_2001_20/dir_2001_20_en.pdf. Accessed November 14, 2018.
    9.
    EU Clinical Trials Register. About the EU Clinical Trials Register. https://www.clinicaltrialsregister.eu/about.html. Accessed November 14, 2018.
    10.
    Dal-Ré  R, Bracken  MB, Ioannidis  JP.  Call to improve transparency of trials of non-regulated interventions.  BMJ. 2015;350:h1323. doi:10.1136/bmj.h1323PubMedGoogle ScholarCrossref
    11.
    Dal-Ré  R.  Preregistration and publication of nonregulated intervention trials are here to stay (letter commenting Wallach et al 2018, 93, 88-93).  J Clin Epidemiol. 2018;99:167-168. doi:10.1016/j.jclinepi.2018.02.014PubMedGoogle ScholarCrossref
    12.
    Dal-Ré  R, Ross  JS, Marušić  A.  Compliance with prospective trial registration guidance remained low in high-impact journals and has implications for primary end point reporting.  J Clin Epidemiol. 2016;75:100-107. doi:10.1016/j.jclinepi.2016.01.017PubMedGoogle ScholarCrossref
    13.
    De Angelis  C, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.  N Engl J Med. 2004;351(12):1250-1251. doi:10.1056/NEJMe048225PubMedGoogle ScholarCrossref
    14.
    De Angelis  CD, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Is this clinical trial fully registered? a statement from the International Committee of Medical Journal Editors.  N Engl J Med. 2005;352(23):2436-2438. doi:10.1056/NEJMe058127PubMedGoogle ScholarCrossref
    15.
    International Committee of Medical Journal Editors. Clinical trial registration. http://www.icmje.org/about-icmje/faqs/clinical-trials-registration/. Accessed November 14, 2018.
    16.
    International Committee of Medical Journal Editors. Journals following the ICMJE recommendations. http://www.icmje.org/journals-following-the-icmje-recommendations/. Accessed November 14, 2018.
    17.
    Turner  EH, Matthews  AM, Linardatos  E, Tell  RA, Rosenthal  R.  Selective publication of antidepressant trials and its influence on apparent efficacy.  N Engl J Med. 2008;358(3):252-260. doi:10.1056/NEJMsa065779PubMedGoogle ScholarCrossref
    18.
    Turner  EH, Knoepflmacher  D, Shapley  L.  Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database.  PLoS Med. 2012;9(3):e1001189. doi:10.1371/journal.pmed.1001189PubMedGoogle ScholarCrossref
    19.
    Dwan  K, Altman  DG, Cresswell  L, Blundell  M, Gamble  CL, Williamson  PR.  Comparison of protocols and registry entries to published reports for randomised controlled trials.  Cochrane Database Syst Rev. 2011;(1):MR000031.PubMedGoogle Scholar
    20.
    Dwan  K, Altman  DG, Clarke  M,  et al.  Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials.  PLoS Med. 2014;11(6):e1001666. doi:10.1371/journal.pmed.1001666PubMedGoogle ScholarCrossref
    21.
    Vedula  SS, Bero  L, Scherer  RW, Dickersin  K.  Outcome reporting in industry-sponsored trials of gabapentin for off-label use.  N Engl J Med. 2009;361(20):1963-1971. doi:10.1056/NEJMsa0906126PubMedGoogle ScholarCrossref
    22.
    Chalmers  I, Glasziou  P.  Avoidable waste in the production and reporting of research evidence.  Obstet Gynecol. 2009;114(6):1341-1345. doi:10.1097/AOG.0b013e3181c3020dPubMedGoogle ScholarCrossref
    23.
    Moher  D, Glasziou  P, Chalmers  I,  et al.  Increasing value and reducing waste in biomedical research: who’s listening?  Lancet. 2016;387(10027):1573-1586. doi:10.1016/S0140-6736(15)00307-4PubMedGoogle ScholarCrossref
    24.
    Babu  AS, Veluswamy  SK, Rao  PT, Maiya  AG.  Clinical trial registration in physical therapy journals: a cross-sectional study.  Phys Ther. 2014;94(1):83-90. doi:10.2522/ptj.20120531PubMedGoogle ScholarCrossref
    25.
    Pinto  RZ, Elkins  MR, Moseley  AM,  et al.  Many randomized trials of physical therapy interventions are not adequately registered: a survey of 200 published trials.  Phys Ther. 2013;93(3):299-309. doi:10.2522/ptj.20120206PubMedGoogle ScholarCrossref
    26.
    Jull  A, Aye  PS.  Endorsement of the CONSORT guidelines, trial registration, and the quality of reporting randomised controlled trials in leading nursing journals: a cross-sectional analysis.  Int J Nurs Stud. 2015;52(6):1071-1079. doi:10.1016/j.ijnurstu.2014.11.008PubMedGoogle ScholarCrossref
    27.
    Killeen  S, Sourallous  P, Hunter  IA, Hartley  JE, Grady  HLO.  Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals.  Ann Surg. 2014;259(1):193-196. doi:10.1097/SLA.0b013e318299d00bPubMedGoogle ScholarCrossref
    28.
    Hardt  JLS, Metzendorf  M-I, Meerpohl  JJ.  Surgical trials and trial registers: a cross-sectional study of randomized controlled trials published in journals requiring trial registration in the author instructions.  Trials. 2013;14:407. doi:10.1186/1745-6215-14-407PubMedGoogle ScholarCrossref
    29.
    Milette  K, Roseman  M, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review.  J Psychosom Res. 2011;70(3):205-217. doi:10.1016/j.jpsychores.2010.09.015PubMedGoogle ScholarCrossref
    30.
    Riehm  KE, Azar  M, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: A 5-year follow-up.  J Psychosom Res. 2015;79(1):1-12. doi:10.1016/j.jpsychores.2015.04.010PubMedGoogle ScholarCrossref
    31.
    Azar  M, Riehm  KE, McKay  D, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials published in the Journal of Consulting and Clinical Psychology PLoS One. 2015;10(11):e0142894. doi:10.1371/journal.pone.0142894PubMedGoogle ScholarCrossref
    32.
    Cybulski  L, Mayo-Wilson  E, Grant  S.  Improving transparency and reproducibility through registration: the status of intervention trials published in clinical psychology journals.  J Consult Clin Psychol. 2016;84(9):753-767. doi:10.1037/ccp0000115PubMedGoogle ScholarCrossref
    33.
    Bradley  HA, Rucklidge  JJ, Mulder  RT.  A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials.  Acta Psychiatr Scand. 2017;135(1):65-77. doi:10.1111/acps.12647PubMedGoogle ScholarCrossref
    34.
    Internet Archive. Wayback Machine. https://archive.org/web/. Accessed October 15, 2018.
    35.
    Murphy  J, Hashim  NH, O’Connor  P.  Take me back: validating the Wayback Machine.  J Comput Mediat Commun. 2007;13(1):60-75. doi:10.1111/j.1083-6101.2007.00386.xGoogle ScholarCrossref
    36.
    Chan  AW, Tetzlaff  JM, Gøtzsche  PC,  et al.  SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials.  BMJ. 2013;346:e7586. doi:10.1136/bmj.e7586PubMedGoogle ScholarCrossref
    37.
    Altman  DG, Moher  D.  Importance of Transparent Reporting of Health Research: Guidelines for Reporting Health Research: A User’s Manual. Hoboken, NJ: John Wiley & Sons; 2014:1-13.
    38.
    Ioannidis  JPA, Greenland  S, Hlatky  MA,  et al.  Increasing value and reducing waste in research design, conduct, and analysis.  Lancet. 2014;383(9912):166-175. doi:10.1016/S0140-6736(13)62227-8PubMedGoogle ScholarCrossref
    39.
    Chan  AW, Song  F, Vickers  A,  et al.  Increasing value and reducing waste: addressing inaccessible research.  Lancet. 2014;383(9913):257-266. doi:10.1016/S0140-6736(13)62296-5PubMedGoogle ScholarCrossref
    40.
    Smyth  RMD, Kirkham  JJ, Jacoby  A, Altman  DG, Gamble  C, Williamson  PR.  Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists.  BMJ. 2011;342:c7153. doi:10.1136/bmj.c7153PubMedGoogle ScholarCrossref
    41.
    NIH Policy on the Dissemination of NIH-funded Clinical Trial Information. https://grants.nih.gov/policy/clinical-trials/reporting/understanding/nih-policy.htm. Accessed November 14, 2018.
    42.
    DeVito  NJ, French  L, Goldacre  B.  Noncommercial funders’ policies on trial registration, access to summary results, and individual patient data availability.  JAMA. 2018;319(16):1721-1723. doi:10.1001/jama.2018.2841PubMedGoogle ScholarCrossref
    43.
    Ioannidis  JPA.  Why most clinical research is not useful.  PLoS Med. 2016;13(6):e1002049. doi:10.1371/journal.pmed.1002049PubMedGoogle ScholarCrossref
    ×