Barriers and Facilitating Factors for Conducting Systematic Evidence Assessments in Academic Clinical Trials

Key Points

Question  What are the practices and attitudes of Swiss stakeholders and international funders regarding conducting systematic evidence assessments in academic clinical trials?

Findings  In this qualitative study in which 48 Swiss clinical trial stakeholders and 9 international funders were interviewed, responses varied widely regarding how previous evidence should be summarized and assessed when planning a new clinical trial. A lack of obligation, time, competent support, and financial resources was identified as a barrier for an evidence-based approach.

Meaning  There may be a need for more explicit requirements from funders and ethics committees to clarify the level of comprehensiveness needed in summarizing existing evidence for different types of clinical trials.

Importance  A systematic assessment of existing research should justify the conduct and inform the design of new clinical research but is often lacking. There is little research on the barriers to and factors facilitating systematic evidence assessments.

Objective  To examine the practices and attitudes of Swiss stakeholders and international funders regarding conducting systematic evidence assessments in academic clinical trials.

Design, Setting, and Participants  In this qualitative study, individual semistructured qualitative interviews were conducted between February and August 2020 with 48 Swiss stakeholder groups (27 primary investigators, 9 funders and sponsors, 6 clinical trial support organizations, and 6 ethics committee members) and between January and March 2021 with 9 international funders of clinical trials from North America and Europe with a reputation for requiring systematic evidence synthesis in applications for academic clinical trials.

Main Outcomes and Measures  The main outcomes were practices and attitudes of Swiss stakeholders and international funders regarding conducting systematic evidence assessments in academic clinical trials. Interviews were analyzed using conventional content analysis.

Results  Of the 57 participants, 40 (70.2%) were male. Participants universally acknowledged that a comprehensive understanding of the previous evidence is important but reported wide variation regarding how this should be achieved. Participants reported that the conduct of formal systematic reviews was currently not expected before most clinical trials, but most international funders reported expecting a systematic search for the existing evidence. Whereas time and resources were reported by all participants as barriers to conducting systematic reviews, the Swiss research ecosystem was reported not to be as supportive of a systematic approach compared with international settings.

Conclusions and Relevance  In this qualitative study, Swiss stakeholders and international funders generally agreed that new clinical trials should be justified by a systematic evidence assessment but that barriers on individual, organizational, and political levels kept them from implementing it. More explicit requirements from funders appear to be needed to clarify the required level of comprehensiveness in summarizing existing evidence for different types of clinical trials.

New clinical research requires a systematic assessment of the existing evidence; unnecessary or poorly informed clinical research is costly and unethical, limits the available funding for relevant and well-designed research, and diminishes the public’s trust in science.^1-5 A lack of a systematic assessment of prior research has led to thousands of patients being recruited into clinical trials and inadequately treated with no treatment or inferior control treatment well after the tested intervention was known to be effective.^6-8 Nonsystematic reviews of the literature also often involve bias, with previous studies being selected to justify new research based on strategic considerations and preferences of the investigators, more often citing studies that have supportive, positive, and statistically significant findings vs those that have conflicting, negative, and nonsignificant findings.^9-11 However, previous studies indicate that an evidence-based approach to clinical research remains insufficiently used.^12-17

Previous research regarding the barriers and facilitating factors for using systematic reviews to justify and inform the design of new clinical trials is currently limited (search details are provided in eAppendix 1 in the Supplement). A survey conducted among delegates of the International Clinical Trials Methodology Conference in November 2015 found that time constraints were perceived as the biggest barrier to the use of systematic evidence synthesis when designing a new trial, followed by a belief that the trial was the first in the research area, a belief that previous trials were different from the current trial, financial constraints, and the fact that funders did not require systematic evidence synthesis.¹⁸ However, the response rate of the survey was only 17% (106 of 638 delegates), and 95% of registered delegates were from the UK and Ireland,¹⁸ severely limiting the validity and applicability of the survey.

In Switzerland, the need to support high-quality academic clinical trials has been increasingly recognized; a network of clinical trial support units was initiated in 2007,¹⁹ and the Swiss National Science Foundation has implemented a yearly program for investigator-initiated clinical trials since 2016.²⁰ However, it is currently not known to what extent an evidence-based approach in academic clinical trials in Switzerland is required by funders or implemented in practice. Furthermore, previous research suggests that some international funders currently require a systematic review when applying for financial support for a new clinical trial.^21-24 These funders are typically members of the Ensuring Value in Research Funders’ Forum,²⁵ an organization aiming to help health research funders increase the value of their research. One of the principles endorsed by all members of the forum is that “Research should only be funded if set in the context of one or more existing systematic reviews of what is already known or an otherwise robust demonstration of a research gap.”²⁵ However, there is a lack of research examining what these funders require in practice and their experience implementing an evidence-based approach.

Although the lack of an evidence-based approach in clinical research is well known and there is general agreement about its importance,^12-17 it remains unclear how to best achieve widespread and sustainable implementation. A more thorough understanding of barriers and facilitating factors from the different perspectives of all relevant stakeholders is needed. This study therefore aimed to examine the practices and attitudes of Swiss stakeholders and international funders regarding conducting systematic evidence assessments in academic clinical trials.

For this qualitative study, the study design and data collection did not require approval of an ethics committee according to Articles 1 and 2 of the Federal Act on Research Involving Human Beings in Switzerland. Verbal informed consent was obtained from participants at the start of the interview. The study followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) reporting guideline.²⁶ An extended methods section is provided in eAppendix 2 in the Supplement.

Interviews were primarily conducted by S.M., a male senior researcher in biomedical ethics. One interview was conducted by M.B., a male physician and senior scientist in clinical epidemiology. Both interviewers have long-standing experience with qualitative research in the context of clinical research and evidence-based medicine.^27-34 The interviewers had already had contact with some of the Swiss stakeholders from previous research studies. Otherwise, no relationship existed between the interviewers and the other participants before the study, and participants received limited information about interviewers.

Participants were primarily selected through purposive sampling³⁵ to ensure sample diversity according to predetermined factors (eg, field of expertise). Additional participants were identified using snowball sampling.³⁶ Participants were contacted by email and suitable dates for an interview were found with those willing to participate. The study consisted of 57 participants from 2 different samples. In the first sample, 48 Swiss stakeholders from 4 different groups were recruited: 27 primary investigators, 9 funders and sponsors, 6 clinical trial support organizations, and 6 ethics committee members. In the second sample, 9 international funders from North America and Europe with a reputation for implementing an evidence-based approach for clinical trials were recruited.²⁴ Interviews were conducted between February and August 2020 with Swiss stakeholders and between January and March 2021 with international funders. One participant provided their response in writing via email; the remaining interviews were conducted via a telephone or video call. All interviews were conducted in English. Only the participant and the researcher were present during the interview. A researcher-developed semistructured interview guide was created for each group to guide the discussion (eAppendix 3 in the Supplement). The interview guide was piloted in the first 2 interviews. Because no problems were identified, no further piloting or adaptation of the interview guides was deemed to be necessary. Interviews were audio-recorded and were a mean duration of 29 minutes (range, 12-62 minutes) with Swiss stakeholders and 30 minutes (range, 22-44 minutes) with international funders. After 57 interviews, the topic of data saturation arose and was discussed by the research team.³⁷ It was concluded that no new substantive themes were being expressed by the participants. Transcriptions of the interviews were returned to all participants with an invitation for them to review and send any corrections or clarifications; 8 responses were received with minor corrections to syntax.

Using the interview transcriptions in their original language, S.M. performed conventional content analysis with the assistance of the qualitative software MAXQDA, version 11 (VERBI Software). Analysis commenced while interviews were ongoing and used an iterative approach in which initial codes common across participants as well as those unique to individuals were identified using a process of open coding and developed as the analysis progressed. B.N.-S., L.G.H., and M.B. reviewed the initial analysis to clarify and refine codes, and conversations among the investigators continued until coding differences were resolved and consensus was achieved. Findings are presented as higher- and lower-level categories in a coding frame.

Of the 57 participants, 40 (70.2%) were male. Swiss participants universally acknowledged the importance of having a comprehensive understanding of the previous evidence when designing and justifying a new clinical trial. However, it was reported that most investigators in Switzerland were not currently conducting systematic reviews, and it was estimated that systematic reviews only happened in 10% to 30% of trials. Many participants disagreed that systematic reviews were always necessary before a new clinical trial and reported that the need for a systematic review depended on the type of clinical trial and available evidence. Only a few participants said a systematic review should always be done before a new clinical trial if no systematic review had been recently published on the topic by other researchers (Table 1).

In contrast, international funders reported a clear expectation that academic clinical trial proposals should include some form of systematic evidence synthesis (Table 2). However, participants reported large variations in terms of the type of evidence synthesis required and when it should be required.

The level of comprehensiveness that participants expected in summaries of existing evidence typically depended on the type of clinical trials being funded. A funder of large, late-stage multicenter clinical trials reported that they expected a formal systematic review to be provided; this was seen as important owing to the focus on public funding and accountability in their country, with systematic reviews being a part of opening the scientific process to public scrutiny. However, most funders of smaller, earlier-stage clinical trials reported that they did not mandate formal systematic reviews and only required a systematic search of the literature (including the search strategy and databases used). Nevertheless, 2 international funders who were members of the Ensuring Value in Research Funders’ Forum reported that they did not have explicit requirements regarding evidence assessment and left it to investigators what to provide.

Most funders reported having a 2-step funding process. Although 1 funder required systematic evidence assessments to be provided at the preapplication stage, most funders required it at the full application stage. Participants from the latter group reported that the key reason for requesting the evidence assessment at the second step was the limited space available in preapplications and the desire to reduce the work of the funder and the applicants who would not be funded.

Swiss participants identified 3 key barriers that may explain why systematic reviews are used infrequently.

• The first barrier was investigator perceptions. Participants reported that various views of principal investigators led them to not conduct systematic reviews before a new trial, including the view that they were experts in the field and already knew the relevant research; the view that their study was very innovative, and therefore, a systematic review did not make sense because of the limited evidence available; and the belief that an evidence-based approach could be achieved without a formal systematic review.

• The second barrier was practical challenges. Some practical challenges were also identified by participants as reasons why systematic reviews were not conducted, including systematic reviews being very time consuming; a lack of specific funds, which led to systematic reviews often being done without funding; insufficient personnel with the right expertise to conduct systematic reviews; and a lack of awareness and knowledge of systematic reviews among investigators and funding panels.

• The third barrier was lack of enforcement. Participants also reported that none of the funders of academic clinical trials in Switzerland were currently requiring a systematic review, that the method for reviewing the existing research was left to principal investigators, and that funders relied on the expertise of reviewers to evaluate whether the information presented in applications was sufficient. Participants also reported that ethics committees did not require systematic reviews and that although ethics committees checked whether the literature review conducted was generally sufficient and might ask for more information if the review was clearly inadequate, they ultimately saw this as the responsibility of the principal investigator and outside the task of ethics committees.

International funders did not report perceiving substantial barriers. Investigators were seen to have sufficient awareness and knowledge and generally accepted the funder’s expectations regarding evidence assessment. The practical challenges of time and resources were identified as the main barriers to formal systematic reviews. Because formal systematic reviews were perceived as time consuming and typically unfunded, these participants reported that mandating them could create problems and undermine innovative research. However, a funder of larger clinical trials noted that if an investigator was applying for a substantial sum of money to run a large multicenter clinical trial, they would expect the investigator to have the resources to do this sort of preparatory work.

Many Swiss participants reported that the best way to facilitate systematic evidence assessments was for funders to make them mandatory. Although participants said they wanted such a requirement to be feasible and flexible, many reported that funders were the most suitable stakeholder to enforce evidence assessments and said that knowing what the current evidence is for a research question should be a prerequisite for academic clinical trials, particularly if large amounts of public funds are involved. However, many participants also acknowledged that any such requirement would need to be combined with better support from academic institutions and associated clinical trial support organizations.

In contrast, multiple international funders highlighted how evidence assessment communities had become firmly established in their countries, with sufficient capacity now existing for training, expertise, and collaboration, which had been important in promoting a systematic approach. Moving forward, some funders said that it would be helpful to provide investigators with more explicit requirements to clarify the level of comprehensiveness expected. However, funders identified a need to improve their own knowledge of the different types of evidence assessment (eg, formal systematic reviews vs rapid reviews) and for more research on the advantages and disadvantages of each approach.

To our knowledge, this is the first qualitative study to examine the barriers to and factors facilitating an evidence-based research approach with several stakeholders in the context of academic clinical trials. This study found general agreement among Swiss stakeholders and international funders that new clinical trials should be justified by a systematic evidence assessment, but there seemed to be substantial variation among stakeholders in terms of the expected comprehensiveness and transparency of evidence assessments. Formal systematic reviews are currently not expected and not conducted before most clinical trials, but most international funders expect a systematic search for the existing evidence. Whereas time and resources were reported by all participants as barriers to conducting systematic reviews, the Swiss research ecosystem was reported not to be as supportive of a systematic approach compared with international settings. Moving toward a more evidence-based research approach in academic clinical trials requires changes on multiple levels.

On an individual level, we identified a lack of awareness and knowledge of the importance of systematic reviews as well as a lack of skills to conduct systematic reviews. Providing training and integrating courses into the curricula of life sciences to build capacity in evidence-based research as well as more support from academic institutions (eg, information specialists in libraries where systematic literature searches are conducted) could help to overcome these barriers.^38,39 In addition, it seems necessary to raise awareness within the scientific community that expert opinion and narrative reviews are not enough to implement evidence-based research. They have a high risk of being distorted by confirmation bias—the tendency to search and interpret information in a way that supports one’s own beliefs—and therefore are not sufficient to foster transparent evidence-based research.⁴⁰

At an organizational level, resource constraints (eg, time and money) often hinder the conduct of comprehensive systematic reviews. Conducting a comprehensive systematic review can take up to 1 to 2 years.⁴¹ Rapid reviews, “a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting specific methods to produce evidence for stakeholders in a resource-efficient manner,”⁴² have emerged as pragmatic alternatives to systematic reviews and could enable primary researchers to sustainably adopt an evidence-based approach.^43,44 Conducting a systematic review is (if at all) a prerequisite during the application stage, meaning that the update or conduct of a systematic review is often not funded. Thus, researchers may opt to apply for a clinical trial grant without putting unpaid resources into the production of a systematic review. Funders could consider financially supporting the production of systematic reviews or providing grant application funds. Although this might cost money initially, it could save money and resources by avoiding research waste in the long run.

On a political level, the lack of funders and ethics committees requiring systematic reviews was perceived by the study participants as a barrier. Making systematic evidence assessments mandatory for funding and ethics approval emerged as the most promising factor to facilitate implementation of evidence-based research. According to the National Institutes of Health Research in the UK, specifically requiring systematic reviews to justify the need for a new large multicenter trial can increase their use to nearly 100%.¹⁷ For smaller, earlier-stage clinical trials, funders could also explicitly clarify the requirements with respect to a systematic evidence assessment (eg, requesting at least a documented search strategy with searched databases). Ethics committees have also been called on to more vigorously advocate for an evidence-based approach.⁴⁵

This study fills a research gap because, to our knowledge, other studies generally investigated the “natural history of conducting and reporting clinical trials” in interviews, with trialists only touching on “insufficient reference to previous research” when planning a new trial,⁴⁶ or they specifically focused on a survey of methodologists on their opinions about a conditional trial design framework using network meta-analysis for the planning of new trials.⁴⁷ The current study contributes to the findings of the preliminary survey of methodologists by Clayton et al¹⁸ by including perceptions of various stakeholders outside the UK and Ireland and by conducting in-depth interviews, which resulted in a more thorough understanding of barriers to and factors facilitating an evidence-based approach when designing a new trial.

This study has limitations. First, this qualitative study did not use a random sample of stakeholders. However, we included a range of experts who had direct experience with systematic evidence assessment in academic clinical trials, which makes it likely that this study captured key aspects of a multisided issue and provided applicable results. Second, a bias might have existed toward the reporting of socially desirable attitudes.⁴⁸ Given that the results were critical of current practice, we believe such a bias is limited. Third, with the exception of funders, all participants were from Switzerland, compromising the generalizability of the findings to some degree. However, the findings are generally in line with existing research,¹⁸ and probing current practices of funders from different countries that have the reputation of endorsing an evidence-based approach and contrasting their views with Swiss stakeholders has yielded rich findings about knowledge gaps and insufficient clarity of funder requirements regarding evidence assessments. There appear to be a number of common issues across countries, which likely make our findings of interest to many countries.

In this qualitative study, there was general agreement among Swiss stakeholders and international funders that new clinical trials should be justified by a systematic evidence assessment. However, investigators reported that barriers on individual, organizational, and political levels still regularly kept them from implementing these assessments. In their role as gatekeepers, funding agencies and ethics committees are in a position to enforce an evidence-based research approach by making it mandatory for new clinical trials.^17,45 In addition, universities should train students and researchers in evidence-based methods and raise awareness of the importance of a systematic and transparent approach to justify new trials.

Accepted for Publication: October 4, 2021.

Published: November 30, 2021. doi:10.1001/jamanetworkopen.2021.36577

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 McLennan S et al. JAMA Network Open.

Corresponding Author: Matthias Briel, MD, PhD, Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University of Basel and University Hospital Basel, Spitalstrasse 12, 4031 Basel, Switzerland (matthias.briel@usb.ch).

Author Contributions: Drs McLennan and Briel had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: McLennan, Briel.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: McLennan, Nussbaumer-Streit, Briel.

Critical revision of the manuscript for important intellectual content: All authors.

Obtained funding: Hemkens, Briel.

Administrative, technical, or material support: McLennan, Nussbaumer-Streit, Briel.

Supervision: Briel.

Conflict of Interest Disclosures: Dr Hemkens reported receiving a grant from the Swiss National Science Foundation during the conduct of the study. Dr Briel reported receiving grants from the Swiss National Science Foundation during the conduct of the study. No other disclosures were reported.

Funding/Support: This study was supported by project grant IZCOZ0_198082/1 from the Swiss National Science Foundation.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.