Evaluation of Journal Registration Policies and Prospective Registration of Randomized Clinical Trials of Nonregulated Health Care Interventions

Key Points

Question  Have specialty journals, such as in the fields of nursing, rehabilitation, or surgery, that publish randomized clinical trials of health care interventions not subject to regulation by the US Food and Drug Administration implemented and enforced prospective trial registration policies?

Findings  In this study of 953 nonregulated intervention trials published in 254 journals included in the study, only 11% of journals required prospective registration. Only 34% of trials published in journals with a policy were prospectively registered compared with 18% of trials in journals without a policy.

Meaning  Nonregulated health care intervention trials published in specialty journals are often not prospectively registered, and existing registration policies are rarely enforced;

Importance  Many interventions that are important to the health care of patients are not subject to regulation by the US Food and Drug Administration (FDA) or comparable regulatory bodies in other nations.

Objective  To determine whether specialty journals that publish trials of primarily nonregulated health care interventions require prospective registration and whether the prospective registration policies are associated with the publication of prospectively registered trials, trials with adequately registered outcomes, and trials with primary outcomes consistent with the registered primary outcomes.

Design and Methods  PubMed was searched daily, from March 18, 2016, to September 17, 2016, for nonregulated intervention randomized clinical trials. The search included all journals in the Clarivate Analytics Science Citation Index Expanded categories of behavioral sciences, nursing, nutrition and dietetics, psychology, rehabilitation, and surgery. Trials of interventions not subject to FDA regulation were included. One investigator extracted journal registration policy and trial registration status. Two investigators independently extracted trial registration and publication characteristics.

Main Outcomes and Measures  For journals, the main outcome was the trial registration policy. For trials, the main outcomes were prospective registration, adequacy of outcome registration, and concordance of registered with published primary outcomes.

Results  In total, 953 nonregulated intervention trials published in 254 journals were identified. Prospective registration was required for publication by 29 (11.4%) of 254 journals, and an additional 12 journals (4.7%) had conditional date-based requirements. Only 189 (19.8%) of the 953 trials were registered prospectively, including 33 of 98 published in journals with prospective registration policies as compared with 156 of 855 in journals without policies (33.7% vs 18.2%; P = .004). Among the 17 journals that required prospective registration and had at least 2 included trials, none had a prospective registration of more than 50%. In journals with policies, only 3 of 98 trials included primary outcomes consistent with prospectively, adequately registered outcomes, as compared with 34 of 852 trials in journals without policies (3.1% vs 4.0%; P = .62).

Conclusions and Relevance  Few journals in behavioral sciences or psychology, nursing, nutrition and dietetics, rehabilitation, and surgery require prospective trial registration, and those with existing registration policies rarely enforce them; this finding suggests that strategies for encouraging prospective registration of clinical trials not subject to FDA regulation should be developed and tested.

In the United States, Food and Drug Administration regulations require prospective registration of most phase 2, 3, and 4 trials of drugs, biologics, and medical devices.^1-7 The European Union requires prospective registration of trials of drugs and biologics.^8,9 The term nonregulated interventions has been used to refer to health care interventions that are not subject to regulatory oversight and thus not subject to these registration requirements.^10-12 Examples of nonregulated health care interventions include surgical procedures, nursing interventions, diets, psychological treatments, and rehabilitation procedures.^10-12 Although trials of these interventions may be subject to less scrutiny than trials of drugs and medical devices, some of these interventions pose substantial risks to patients, and some may have important implications for the health care system.

Since 2005, International Committee of Medical Journal Editors (ICMJE) journals have required that all clinical trials of health care interventions considered for publication be registered prior to participant enrollment.^13-15 Many non-ICMJE journals have adopted this policy.¹⁶ Prospective registration policies are intended to reduce the risk of bias from nonreporting of trial results or the selective reporting of analyses and outcomes.^17-23

Many nonregulated health care intervention trials are not registered prospectively, even when journal registration requirements are in place.^10,24-33 Previous studies on registration policies and practices, however, have focused on retrospective analyses of small groups of journals. One reason is that it is often not possible to determine retrospectively when journal registration policies, if any, were initiated and, thus, to evaluate compliance.

We conducted a daily search for all publications of randomized clinical trials (RCTs) of nonregulated interventions in journals in the categories of behavioral sciences or psychology, nursing, nutrition and dietetics, rehabilitation, and surgery that were initially listed in PubMed during a 6-month period in 2016. By searching daily, we could examine journal registration policies as close as possible to when the RCTs were submitted. We studied journal registration policies, the associations between these policies and the prospective registration of trials, the registration of primary outcomes and whether this registration was adequate, and the consistency of registered and primary outcomes.

Ethics approval was not required for this study. We identified all journals listed in the following categories of the 2014 Clarivate Analytics Science Citation Index Expanded: (1) behavioral sciences, (2) nursing, (3) nutrition and dietetics, (4) psychology, (5) rehabilitation, and (6) surgery, that published at least 1 article listed in PubMed in 2014 or 2015. We compared journal names listed in the Science Citation Index Expanded against those in the National Library of Medicine Catalog and used National Library of Medicine names for this search. The search strategy included 50 behavioral sciences, 107 nursing, 68 nutrition and dietetics, 66 psychology, 57 rehabilitation, and 187 surgery journals, totaling 524 journals given that 11 were in more than 1 category. These 524 journals are listed in eTable 1 in the Supplement.

We searched PubMed daily, from March 18, 2016, through September 17, 2016, using the names of included journals with the following terms: (random*[Title/Abstract] OR “RCT”[Title/Abstract] NOT (“Drug Therapy”[Mesh] OR “drug therapy”[Subheading] OR “Pharmaceutical Preparations”[Mesh] OR Review[ptyp] OR “meta-analysis”[Title] OR “systematic review”[Title]). More information is provided in eTable 2 in the Supplement.

Eligible RCTs were trials that evaluated the effects of nonregulated health care interventions on health outcomes.^13-15 We excluded trials with a comparison to a drug, biologic, or medical device in any arm as well as publications that stated they reported only secondary outcomes. If results from more than 1 RCT were reported in a single publication, we included only the first RCT in the article.

We uploaded search results into DistillerSR (Evidence Partners), which was used for all coding procedures. Given the available resources, one investigator assessed titles and abstracts and full-text articles for eligibility.

For each eligible RCT, one investigator extracted journal characteristics, including registration policy, and trial characteristics, including registration status. Two investigators independently extracted trial registration information and published outcomes, and discrepancies were resolved through consensus and consultation with a third investigator, as necessary. All extracted data are available on the Open Science Framework at https://osf.io/jk2sq/.

For each RCT in the study, we reviewed the journal website, including author instructions, within 3 weeks of the daily PubMed search that identified the RCT. Journals were classified as (1) requiring prospective registration; (2) requiring registration with no timing requirement; (3) not requiring registration; or (4) having conditional requirements, typically based on date of trial initiation or journal submission. For RCTs in journals with conditional requirements, we determined whether the requirements applied to the RCT on the basis of dates extracted from the trial publication or, if not specified there, the trial registration.

To evaluate whether journal policies might have changed between the submission date of a trial and its PubMed listing, we used the website Wayback Machine, which stores archived website snapshots.^34,35 For trials with published journal submission dates, if a snapshot was available within 3 months prior to the submission date, we compared the registration policy from the most recent snapshot prior to submission with the policy on the journal website at the time of our PubMed search.

To determine trial registration status, we followed a procedure used in previous studies.^29-31 We first attempted to retrieve registration data from the publication. If the registration data were not reported, we searched ClinicalTrials.gov; the International Standard Randomized Controlled Trials Number registry (http://www.isrctn.com); and registries, if available, from the first and corresponding authors’ countries. For this search, we used corresponding, first, and last author names and key terms. We attempted to match the principal investigator, funding source, intervention, control group, and trial design of the RCT with registration records. If we did not find a registry entry or if we found ambiguity about the match, we contacted the corresponding author once by email.

To determine if trials were registered prospectively, we compared registration and enrollment dates. We prioritized enrollment dates from publications, but if they were not available, we used dates from trial registrations. We classified trials registered prior to or in the same month enrollment began as registered prospectively.

Among RCTs registered prospectively, we recorded whether registrations included a specific measurement of the primary outcome or outcomes (eg, Beck Depression Inventory score, not simply depression), a specific assessment time point (eg, 12-month postrandomization), a specific metric (eg, change from baseline, final value), and a specific method of aggregation (eg, mean, proportion).^2,36 We classified registrations as adequate if the registration included a single primary outcome or multiple primary outcomes with a specific measurement and time point of assessment.^29-31 We did not require specification of analysis metric or aggregation method for classification of adequacy.

Trials were classified as adequately registered—single primary outcome if they adequately registered a single primary outcome variable and assessment time point, or multiple primary outcomes or time points, and there was a plan for statistical adjustment for multiple comparisons included in the registration as well as a subsequent publication that included the same primary outcomes or time points with appropriate statistical adjustment. If registrations listed multiple adequately registered primary outcomes, but there was no statistical adjustment for multiple outcomes, the trial was classified as adequately registered—multiple primary outcomes. If there were changes in trial registration records over time, we used the last update prior to participant enrollment.

For each adequately registered trial, we compared the registered primary outcome with published primary outcome. Trials were classified as consistent if registered and published primary outcome analyses were the same and discrepant if they differed.

We present frequencies and percentages of journals with prospective registration requirements; journals with registration requirements with no timing requirement; and journals without a policy, both overall and by journal categories, impact factor, and publishers. For journal policies and prospective registration, we present frequencies and percentages of trials registered prospectively, registered during the trial, registered after the trial, and not registered, both overall and by journal characteristics (registration policy, impact factor, registration policy stratified by impact factor, and publisher) and trial characteristics (category of intervention, country, funding, and sample size). We also present frequencies and percentages of trials with prospectively and adequately registered primary outcomes and with published outcomes consistent with those registered.

We conducted statistical tests to evaluate whether the presence of a journal prospective registration policy was associated with the prospective registration of trials published in the journal, both overall and stratified by impact factor, prospective and adequate registration of primary outcomes in trials published in the journal, and publication of primary outcomes consistent with prospective and adequately registered primary outcomes. In addition, we tested the association of trial registration with journal impact factor. For all of these analyses, we used generalized estimating equations with a binary outcome and an identity link function. We used an exchangeable correlation structure with robust SEs to account for the correlation between trials within the same journal. Analyses were run using the geepack package in R, version 3.2.3 (R Foundation for Statistical Computing). We used α <.05 to test for statistical significance.

The search retrieved 4930 citations, of which 3332 (67.6%) were excluded after title and abstract review and 645 (13.1%) after full-text review. The remaining 953 RCTs (19.3%) were included (Figure).

The 953 RCTs were published in 254 journals, of which 100 (39.4%) published 1 of these trials in the 6-month study period, 58 (22.8%) published 2, 42 (16.5%) published 3 to 4, 34 (13.4%) published 5 to 10, and 20 (7.9%) published 11 to 44. Of the 254 journals, 29 (11.4%) required prospective trial registration and 12 (4.7%) had conditional policies based on the date of trial initiation or journal submission, as shown in Table 1. Table 2 shows the frequencies and percentages of journal trial registration policies by journal category, impact factor, and publisher.

Of the articles and journals, 650 articles (68.2%) from 165 journals (65.0%) included a published submission date. Of these, there was a Wayback Machine archive of author instructions in the 3 months prior to the submission date for 126 articles (13.2%) from 46 journals (18.1%). Author instructions changed between the Wayback Machine archive and our PubMed search date for only 1 article. For that article, the journal Nursing and Health Sciences added a statement encouraging, but not requiring, trial registration. This archived information did not affect the journal’s classification.

For 3 journals (Disability and Rehabilitation, Journal of Head Trauma Rehabilitation, and Obesity), author instructions changed during the 6-month daily search period. Articles published in Disability and Rehabilitation and Obesity included a submission date, and all included RCTs were submitted prior to the change from not requiring to requiring prospective trial registration; we classified both journals as not requiring registration for all included trials. Articles published in the Journal of Head Trauma Rehabilitation did not provide submission dates, but we confirmed through PubMed listing dates and consultation with the journal editor that all 6 included trials were submitted prior to the change in journal policy. We also classified this journal as not requiring prospective trial registration.

Of the 953 trials, 189 (19.8%) were registered prospectively, 141 (14.8%) after initiation of participant enrollment and prior to collection of trial outcomes, and 120 (12.6%) after completion of the trial. More than half (496 [52.0%]) were not registered (Table 3; eTable 3 in the Supplement). Of the 141 trials registered after participants were enrolled, 3 (2.1%) were registered within a month and 61 (43.3%) within 6 months.

As shown in Table 4, 98 (10.3%) of the 953 trials were published in a journal that required prospective registration. Of these RCTs, 33 were registered prospectively, as compared with 156 of 855 trials published in journals without such a requirement (33.7% vs 18.2%; P = .004). Among journals that required prospective registration unconditionally, 17 journals had at least 2 included trials (range of 2 to 14 trials). The percentage of prospectively registered trials was not greater than 50% in any journals. Table 4 also shows prospective registration by impact factor, journal policy stratified by impact factor, publisher, and trial characteristics.

Of the 953 included trials, 3 (0.3%) were completed or still active in July 2005 and met ICMJE requirements to register by September 2005. These RCTs were not required to register before patients were enrolled. Of the remaining 950 trials, based on the criterion of having a specific measurement and time point, only 85 trials (8.9%) prospectively and adequately registered a single primary outcome (60 [6.3%]) or multiple primary outcomes (25 [2.6%]). This group included 15 of 98 trials that were subjected to prospective registration requirements, as compared with 70 of 852 trials in journals without policies (15.3% vs 8.2%; P = .04).

Among the 950 trials that first enrolled participants after July 2005, 37 (3.9%) adequately registered a single primary outcome or multiple primary outcomes and published a consistent outcome or outcomes. This included 35 (58.3%) of 60 trials that registered a single primary outcome and 2 (8.0%) of 25 trials that registered multiple primary outcomes. Among 98 trials subject to a journal policy that required prospective registration, only 3 had adequate registration and a consistent published primary outcome or outcomes, as compared with 34 of 852 not subject to such a policy (3.1% vs 4.0%; P = .62). Of the 48 trials that published primary outcomes discrepant to those registered, 14 (29.2%) registered a single primary outcome but published multiple primary outcomes, 7 (14.6%) registered multiple primary outcomes but published a single primary outcome, 15 (31.3%) registered and published different primary outcomes, and 12 (25.0%) did not identify which outcome or outcomes were primary in the published report.

We reviewed 953 RCTs of nonregulated health care interventions with initial PubMed listings in a single 6-month period in 2016 that were published in 254 journals in the categories of behavioral sciences, psychology, nursing, nutrition and dietetics, rehabilitation, and surgery. Prospective registration for all submitted articles was required for 29 journals (11.4%), and an additional 12 journals (4.7%) had conditional prospective registration. Of the trials, 189 (19.8%) were registered prospectively; 85 (8.9%) were registered prospectively with an adequately defined primary outcome or outcomes; and 37 (3.9%) published primary outcomes consistent with prospectively, adequately registered outcomes. Prospective registration was about twice as high in journals that required prospective registration as a condition for publication (33 [33.7%]), as compared with journals without such a policy (156 [18.2%]). Our findings suggest, however, that even when prospective registration policies are in place, they are rarely enforced.

Despite research demonstrating bias and waste owing to misrepresented research results,^37-39 our findings suggest that many researchers and journal editors may still be unaware of the importance of trial registration and transparent outcome reporting.⁴⁰ Research funders and ethics committees are well positioned to effectively enforce registration requirements.¹⁰ In the United States, in 2017, the National Institutes of Health began to require preregistration and results reporting of all clinical trials that it funds.⁴¹ Other governmental and nongovernmental funding organizations have made similar commitments.^1,42 However, the degree to which these requirements for preregistration are enforced or whether the quality of trial registrations is reviewed by funders is not known. A publicly visible report card that would grade journals on their degree of compliance with trial registration requirements and enforcement could encourage needed change.⁴³

One limitation of this study is that we focused on journals in specialty areas and not other types of journals. We did not collect information on registration of trials of drugs, biologic, and devices that may have been published in the 254 journals in this study. A study that reviewed trials published in 2014 in the 6 highest-impact general medical journals found that 79% of trials of Food and Drug Administration–regulated interventions complied with ICMJE preregistration requirements, as compared with 58% of trials of nonregulated interventions.¹² Another limitation is that we used journal policy at the time of trial listing on PubMed rather than at the date of manuscript submission. Our review of internet archives and tracking of journal policies, however, found that few journals had recently changed registration policies; thus, the main findings would not likely be affected by any undetected changes. We did not examine whether included trials reported summary results in clinical trial registries, which would be an important area for future research.

Few specialty journals that publish nonregulated intervention trial results have adopted comprehensive trial registration policies. Although these policies are in place, they are rarely enforced. Without universal trial registration, it is difficult to interpret the meaning of trial results because of inherent biases, including selective reporting and publication bias. Journals and other entities with a stake in clinical research, including researchers, funding agencies, ethics review boards, and academic medical centers, should be concerned about these findings. They should make a concerted effort to improve trial registration practices.

Accepted for Publication: November 23, 2018.

Corresponding Author: Brett D. Thombs, PhD, Lady Davis Institute for Medical Research, Jewish General Hospital, 4333 Cote Ste Catherine Rd, Montréal, QC H3T 1E4, Canada (brett.thombs@mcgill.ca).

Published Online: March 11, 2019. doi:10.1001/jamainternmed.2018.8009

Author Contributions: Ms Azar and Dr Thombs had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Azar, Rice, Benedetti, Thombs.

Acquisition, analysis, or interpretation of data: Azar, Riehm, Saadat, Sanchez, Chiovitti, Qi, Rice, B. Levis, Fedoruk, A. W. Levis, Kloda, Kimmelman, Thombs.

Drafting of the manuscript: Azar, Chiovitti, Thombs.

Critical revision of the manuscript for important intellectual content: Azar, Riehm, Saadat, Sanchez, Qi, Rice, B. Levis, Fedoruk, A. W. Levis, Kloda, Kimmelman, Benedetti, Thombs.

Statistical analysis: Azar, Benedetti, Thombs.

Administrative, technical, or material support: Azar, Saadat, Sanchez, Chiovitti, Qi, B. Levis, Fedoruk.

Supervision: Benedetti, Thombs.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study received no funding, but the following authors received salary support or student stipends: Mss Azar and Rice and Mr Levis were supported by Fonds de Recherche Québec—Santé (FRQS) Masters Training Awards; Ms Riehm, by a Canadian Institute of Health Research (CIHR) Masters Research Award; Ms Levis, by a CIHR Doctoral Research Award; and Drs Thombs and Benedetti, by FRQS researcher awards.

Role of the Funder/Sponsor: The salary and stipend sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: The authors thank Ms. Angie Ye Eun Lee, BSc (medical student at Yonsei University College of Medicine in Seoul, South Korea), for translating articles published in Korean. She received no compensation for her contribution.