Preferred Reporting Items for a Systematic Review and Meta-analysis of Individual Participant Data: The PRISMA-IPD Statement

Importance  Systematic reviews and meta-analyses of individual participant data (IPD) aim to collect, check, and reanalyze individual-level data from all studies addressing a particular research question and are therefore considered a gold standard approach to evidence synthesis. They are likely to be used with increasing frequency as current initiatives to share clinical trial data gain momentum and may be particularly important in reviewing controversial therapeutic areas.

Objective  To develop PRISMA-IPD as a stand-alone extension to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement, tailored to the specific requirements of reporting systematic reviews and meta-analyses of IPD. Although developed primarily for reviews of randomized trials, many items will apply in other contexts, including reviews of diagnosis and prognosis.

Design  Development of PRISMA-IPD followed the EQUATOR Network framework guidance and used the existing standard PRISMA Statement as a starting point to draft additional relevant material. A web-based survey informed discussion at an international workshop that included researchers, clinicians, methodologists experienced in conducting systematic reviews and meta-analyses of IPD, and journal editors. The statement was drafted and iterative refinements were made by the project, advisory, and development groups. The PRISMA-IPD Development Group reached agreement on the PRISMA-IPD checklist and flow diagram by consensus.

Findings  Compared with standard PRISMA, the PRISMA-IPD checklist includes 3 new items that address (1) methods of checking the integrity of the IPD (such as pattern of randomization, data consistency, baseline imbalance, and missing data), (2) reporting any important issues that emerge, and (3) exploring variation (such as whether certain types of individual benefit more from the intervention than others). A further additional item was created by reorganization of standard PRISMA items relating to interpreting results. Wording was modified in 23 items to reflect the IPD approach.

Conclusions and Relevance  PRISMA-IPD provides guidelines for reporting systematic reviews and meta-analyses of IPD.

Systematic reviews and meta-analyses of individual participant data (IPD) aim to identify, appraise, and summarize the evidence from multiple studies addressing the same research question or topic. Unlike most systematic reviews, they do not rely on aggregate data extracted from journal publications. Rather, the original data on each individual participant are sought from each eligible study. These data typically include characteristics such as age or stage of disease, the intervention or exposure being investigated, and follow-up data on outcomes and events.

“Participant” is used to describe the unit of analysis because most commonly this is an individual person. However, it may apply equally to other units of analysis, such as a school, primary care practice, or hospital in a cluster randomized trial, or an individual body part. What is important is the availability of raw unit-level data rather than aggregate-level data extracted from a report.

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement published in 2009,¹ which includes a 27-item checklist and flow diagram, was developed principally for systematic reviews and meta-analyses of randomized trials that use aggregate data, generally extracted from published reports. It therefore does not cover some important aspects of the IPD approach to systematic review and meta-analysis, particularly the methods used to obtain, check, and synthesize the IPD, and to handle studies for which IPD were not available. PRISMA-IPD was developed to address these issues.

PRISMA-IPD was developed based on the methodological framework for guideline development published by the EQUATOR Network.² The standard PRISMA Statement¹ was used as a starting point, and initial work to adapt and build on this was led by a steering group aided by a small project group.

The project group conducted a review of how systematic reviews and IPD meta-analyses are currently reported to update previous evaluations and guidance in this area.^3,4 An initial draft adaptation of the standard PRISMA checklist was then prepared by the steering and project groups. This formed the basis of an electronic questionnaire distributed to 95 members of the Cochrane IPD Meta-analysis Methods Group, 88 members of the Society for Research Synthesis Methods, 4 members of both, and 4 additional individuals invited to help develop the extension. Recipients were asked to send the questionnaire to others who might be interested. Links to the survey were placed on the websites of the Cochrane IPD Meta-analysis Methods Group, the Centre for Reviews and Dissemination (CRD), and the Systematic Reviews journal, as well as on the Cochrane Facebook page. The CRD also advertised the survey using Twitter. The survey remained open for 14 days.

The questionnaire sought feedback on the appropriateness of standard PRISMA Statement items to IPD and on suggested changes and additions. It included 4 items with suggested wording changes (1, 3, 6, 18), 8 items with additional proposed elements (11, 14, 15, 16, 17, 19, 20, 25), and 1 item with both (10). Proposed wording changes were rated “appropriate” or “other change required,” and suggested additional components were rated as “not required,” “possible,” “desirable,” or “essential.” Fourteen items had no suggested changes at that stage (2, 4, 5, 7, 8, 9, 12, 13, 21, 22, 23, 24, 26, 27).

A 1-day international workshop was convened in York, United Kingdom, in March 2013. The 26 participants included systematic reviewers with experience in IPD synthesis (n = 21), clinicians (n = 6), methodologists (n = 20), and journal editors (n = 10). Survey feedback was presented, and each PRISMA item was discussed in detail and agreement on inclusion and wording reached by consensus. The checklist prepared during the workshop was then circulated for further comment to participants and to 3 people unable to attend. It continued to be refined iteratively by the steering group and subsequently the wider development group.

Fifty-three responses to the questionnaire were received, a 28% response rate based on direct invitations (numbers of respondents not invited directly are unknown). Of 38 respondents who answered demographic questions, there were 28 systematic reviewers, 9 health care practitioners, 1 policy maker, 27 methodologists, 13 statisticians, and 3 publishers. Twenty-seven reported membership of the Cochrane Collaboration and 11 indicated that they were not members. Twelve reported no practical experience with IPD systematic reviews, whereas 5 had completed 1, 10 had completed between 2 and 5, and 11 had completed more than 5 such reviews.

Respondents supported suggested wording changes for 4 items (>80% scored as appropriate) and required further change for 1 item (31% scored as appropriate). Suggested additions concerning data checking, prespecification, statistical analysis, study variables sought, numbers of participants, IPD-based description of clinical characteristics, reasons for nonprovision of IPD, how studies with and without IPD were analyzed together, and the effect of unavailable trials and missing IPD were supported (>70%). Providing particular data-checking details and production of forest plots for all analyses were not supported (<70%). These results were used only as a starting point for deliberation at the workshop.

At the workshop 25 checklist items were modified during discussion, much of which centered on striking a balance between being consistent with the standard PRISMA Statement and covering all aspects of reporting pertinent to IPD. Iterative modification and refinement of almost all items by the steering and wider development groups continued until September 2014.

The Table presents the final PRISMA-IPD checklist adapted and extended from the original PRISMA Statement, as agreed by consensus of those involved in its development. This includes provision for additional information required to describe adequately the IPD approach or where some rewording provides clarity, particularly in the context of IPD. PRISMA-IPD contains 23 items in which the wording has been modified and 3 new items on methods of checking data integrity (A1), on methods of exploring variation (A2), and on reporting any important issues identified as a result of data checking (A3). The other additional item (A4) has been created as a consequence of rearranging items relating to interpretation of results.

To clarify differences, eTable 1 in the Supplement presents the PRISMA and PRISMA-IPD checklists side by side. eTable 2 in the Supplement provides examples from previously published work illustrating how reports may be suitably worded. The Figure presents a modified version of the PRISMA flow diagram, which may also be downloaded from the PRISMA (http://www.prisma-statement.org) and EQUATOR (http://www.equator-network.org) websites, as may the checklist.

Systematic reviews and meta-analyses of individual participant data have been recognized as a gold standard approach from the early days of systematic review.^5,6 They offer many advantages over analyses that use aggregate data extracted from publications.^4,7 These include the potential to avoid bias arising from the absence of unpublished studies and unreported outcomes,^7,8 checking and transforming data to common scores or measures and standardizing analyses across studies,⁷ and the possibility of handling missing data within studies more appropriately.⁹ Individual-level information enables more flexible and robust analyses than are possible with aggregate study results, including the ability to deal appropriately with time-to-event and longitudinal data. IPD meta-analyses also may enhance evidence synthesis more widely; for example, they may help quantify potentially causal associations from multiple observational studies⁹ or develop risk prediction models.^10,11

With an established history in reviews of interventions in cancer¹² and cardiovascular disease,¹³ the IPD approach is being used increasingly⁴ and across a broadening range of health care areas. However, it represents a minority of the systematic reviews undertaken,^3,4 perhaps owing to the time and resources required to build collaborations of study investigators to share data and agree on analyses to be performed. If IPD becomes more readily available as a result of current initiatives aiming to make provision of clinical trial data for research purposes a legal, regulatory, or ethical requirement,^14,15 it is likely that in the future more systematic reviews will access and analyze IPD. This will likely include synthesis of IPD released in controversial areas where transparent, complete, and high-quality reporting is essential.

As with all areas of research, systematic reviews and meta-analyses of IPD could be better reported,¹⁶ making it easier for readers to understand, critique, and implement findings. Standard PRISMA guidelines are geared toward systematic reviews based on aggregate data and so lack reference to some important aspects of the IPD approach. The PRISMA-IPD extension was therefore developed to provide a framework for full and transparent reporting of methods used in the collection, checking, and meta-analysis of IPD.

Based on the PRISMA extension for abstracts¹⁷ tailored to the IPD approach, a structured abstract (item 2) should include important details of methods and results. Although journal format and word limits may make it difficult to include all the outlined information, as much relevant information as possible should be succinctly summarized.

It may be useful to explain briefly the benefits that the IPD approach brings to the review, as well as the specific rationale for the research question being addressed (item 3). This may highlight inadequacies of any existing systematic reviews of aggregate data and may be particularly important in areas in which the approach is not yet well established and the target audience is not familiar with its strengths. Providing such information may also be helpful in persuading readers of the robustness and consequent value of findings, which may help encourage their uptake and use in practice, including within clinical guidelines.¹⁸ For clarity, it is always helpful for reviews that update or build on previously published reviews to explicitly make the link with prior versions.

Access to IPD both permits more flexible and powerful analysis and provides an opportunity for data manipulation. Thus, the production of, and adherence to, a protocol that includes a detailed analysis plan is perhaps even more important than for a standard systematic review. Deviation from the planned analyses may be necessary and may even improve on what was intended. However, transparency is important, for example in stating which reported analyses were preplanned and which were primary and secondary outcomes. Such statements can be supported by referral to the protocol. Because systematic review protocols are increasingly being registered (eg, in PROSPERO [http://www.crd.york.ac.uk/PROSPERO/]) and published formally, a recommendation to include a citation or link to registration records and formal published protocols has been added (item 5). It also may be helpful to include a copy of the protocol and data request forms as an appendix to the published report.

Inclusion criteria are generally developed as for a standard systematic review. However, the IPD approach provides an opportunity to use only a subset of the enrolled population within a study. For example, in a review of pediatric research interventions it may be possible to include the IPD from just the children enrolled in a study that recruited both adults and children. Where any such additional inclusion/exclusion criteria are applied at the participant level, they should be reported (item 6). As well as ensuring transparency, doing so helps readers avoid confusion if the number of included participants differs markedly from the number reported in the original study publication.

Direct contact and collaboration with study investigators, including enlisting their help in identifying eligible studies, is a key feature of systematic reviews and meta-analyses of IPD. Item 7 was therefore extended to include additional means of identifying studies. These can be particularly important for identifying data not published at the time of the systematic review. Item 17 has also been extended to include additional information on seeking IPD from the original studies.

Data extraction does not usually apply to those studies for which IPD are obtained. The checklist therefore includes a new element under data collection processes (item 10) to capture how IPD were obtained and managed, and item 11 now includes reporting the methods of standardizing or redefining the IPD received.¹⁹ New items on methods of exploring data integrity (item A1) and reporting data integrity (item A3) have also been added, reflecting the importance of data checking and correcting any inaccuracies or errors in the IPD supplied.

Risk of bias assessments for included studies (item 12) and presentation of findings (item 19) have been extended to consider direct investigation of the IPD. For example, there might be less concern about potential bias associated with envelope randomization if checking the IPD shows that treatment allocation to study groups is balanced over time, provides reassurance that envelopes have been used in the planned sequence with none discarded, and that there are no important imbalances in patient characteristics across allocation groups. Conversely, it may be necessary to highlight concerns revealed only by the IPD. For example, if checking IPD randomization sequence reveals an alternating pattern of allocation for a trial that reported apparently sound methods of randomization, this should be noted. Obtaining the full study protocol and direct contact with the participating study investigators also may provide additional information to inform assessment of risk of bias.

An important aspect of validity is the completeness and representativeness of the data collected. It is therefore important to provide information on the numbers of studies and participants for which IPD were sought and obtained (item 17), to report whether there is potential risk of bias associated with nonavailability of IPD (items 15 and 22), and to compare results from analyses that include and exclude studies for which IPD were not available (item 23). For the latter, 1-stage and 2-stage meta-analysis models (discussed below) can be used to combine aggregate data (from studies not providing IPD) with the available IPD,²⁰ and both sources of evidence can be distinctly displayed on forest and funnel plots.¹⁶ This allows the effect of non-IPD studies on meta-analysis conclusions to be quantified and transparently displayed.

IPD enables more flexible and potentially more powerful statistical analyses than are possible with aggregate data. However, it also creates potential for “data dredging,” whereby reviewers explore numerous outcomes and subgroups to find those that yield interesting results. This made it important to add a recommendation to report all outcomes analyzed and whether these were prespecified (item 13) and to record all subgroup analyses conducted and whether these were prespecified (item A2).

Methods of synthesis are not always reported fully or well.^3,4 Item 14 has been extended to list aspects that should be addressed in analyses of clinical trial data. A variety of analytic models can be used including (1) those that first generate estimates of effectiveness (aggregate data) for each study separately and then combine these summary statistics using standard meta-analysis methods (commonly termed a 2-stage approach), and (2) those that estimate the overall meta-analytic effect from all data in all studies simultaneously (commonly termed a 1-stage approach). If a 1-stage modeling approach is used, it is important that the clustering of patients within studies is taken into account and that the data set is not analyzed simplistically as a single “mega trial.”²¹ Care should be taken to ensure that the model selection process is described adequately and that full specifications are provided. Results should be presented with a nonstatistical audience in mind. For example, coefficients from a 1-stage logistic or Cox regression model relate to a log odds ratio or a log hazard ratio, respectively; converting and reporting these as odds ratios or hazard ratios makes them easier to understand.

It may be helpful to present results as both relative and absolute differences (which depends on relative differences and on baseline risk) between, for example, interventions from randomized trials. A large relative benefit may be of little practical importance if the underlying risk and hence absolute improvement is small. Furthermore, because baseline event rates can differ substantially between different types of individual, the same relative effect may translate to different absolute improvements, even when the relative effect of intervention is the same. It therefore can be helpful to present information on relative and absolute differences according to a series of differing baseline risks.^22,23

Forest plots enable readers to examine combined estimates, inconsistency across studies, and the precision of individual studies. In common with the standard PRISMA Statement the display of forest plots for key outcomes is advocated, irrespective of the type of approach to statistical analysis.

A major motivation for adopting the IPD approach is the ability to explore between-study heterogeneity and participant-level variation in treatment response. The latter is particularly important, as it allows analyses to explore whether there are any particular types (subgroups) of participants who benefit differentially from the intervention under investigation.²⁴ This is reflected through an additional element in stating objectives (item 4) relating to presentation of subgroup hypotheses, a new item (item A2) to describe methods, and an additional element relating to presenting results (item 21). When reporting such analyses it is vital to state whether there is any clear statistical evidence of a difference in outcome by participant characteristics. As well as being of clinical relevance, any such variation may explain heterogeneity in results between trials.

Some PRISMA-IPD additions are relevant to all systematic reviews. These include statement of any subgroup hypotheses (item 4); citation of published protocols (item 5); description of how the information to be collected (or extracted) was chosen (item 11); statement of all comparisons and outcomes addressed and whether these were primary outcomes (item 13); statement of whether analyses were prespecified (items 13, 16, and 21); reporting of the number of participants for which data were available (item 17); and description of interventions (item 18). The PRISMA group may wish to consider some of these for inclusion in future versions of the standard PRISMA Statement.

PRISMA-IPD is intended to apply to IPD meta-analysis primarily within the context of systematic review of randomized trials. It has been developed largely from experience of undertaking reviews of studies of the effects of health care interventions, where the approach was established in the late 1980s.^25,26 Most examples are drawn from this literature, and it is anticipated that the checklist will be used mainly in this context. However, much is also relevant to other areas in which IPD synthesis is gaining popularity, including systematic reviews of diagnostic,²⁷ prognostic,^10,28,29 observational,^30,31 causal,³² or animal^33,34 studies. Although not designed specifically for prospective meta-analyses, in which study investigators decide in advance to formally combine individual-level data from each of their studies in a larger meta-analytic synthesis,³⁵ many checklist items will apply. Because systematic reviews and meta-analyses of IPD are often substantial projects, it may not always be possible to address all items in detail within journal article word limits. In this case, further information should be made available as supplementary material.

The PRISMA-IPD checklist should be largely self-explanatory and represents the minimum amount of information that should be reported to provide a full and transparent account of how the review was conducted. It will sometimes be necessary to include further information not covered by PRISMA-IPD items to deal with nonstandard issues that occurred during the review process and convey nuances of findings. Syntheses of study designs other than randomized trials may require further or different information. Authors and peer reviewers are encouraged to use the checklist to improve reporting and journal editors to include it in their endorsement of PRISMA.

Development of the PRISMA-IPD Statement (as for the standard PRISMA Statement) was evidence-based where possible and was otherwise based on opinions gathered from persons with relevant expertise and experience. Response to the survey was limited (53 respondents [28% response rate]). This may reflect it being open for only a short time (necessitated by a fixed workshop date) as well as its specialist focus. However, as its stated purpose was to provide a starting point for discussion at the workshop, this is not considered a major limitation. The checklist has not been formally evaluated prior to proposed implementation. Whether use of PRISMA-IPD will improve reporting quality requires evaluation in future research.

PRISMA-IPD provides guidelines for reporting systematic reviews and meta-analyses of IPD. Future research is needed to determine whether this approach will lead to improved reporting of this type of research.

Corresponding Author: Lesley A. Stewart, PhD, Centre for Reviews and Dissemination, University of York, York, UK YO10 5DD.

Author Contributions: Dr L. A. Stewart had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: L. A. Stewart, Clarke, Rovers, Riley, G. Stewart, Tierney.

Acquisition, analysis, or interpretation of data: L. A. Stewart, Clarke, Rovers, Riley, Simmonds, G. Stewart, Tierney.

Drafting of the manuscript: L. A. Stewart, Clarke, Rovers, Riley, Simmonds, G. Stewart, Tierney.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Simmonds.

Obtained funding: L. A. Stewart.

Administrative, technical, or material support: Simmonds, G. Stewart.

Study supervision: L. A. Stewart, Clarke, Rovers, Riley, Tierney.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr L. A. Stewart reported that she is employed as Director of the Centre for Reviews and Dissemination at the University of York and is in receipt of research funding to carry out systematic reviews and meta-analyses; has previously received research funding to explore and develop methods for systematic reviews and meta-analysis of IPD; is co-convenor of the Cochrane Collaboration IPD Meta-analysis Methods group; and has previously published on and advocated IPD approaches to systematic review. Dr Clarke reported that he is co-convenor of the Cochrane Collaboration IPD Meta-analysis Methods group and is involved in the conduct of IPD analyses. Dr Rovers reported that she is a co-convenor of the Cochrane Collaboration IPD Meta-analysis Group and has undertaken several IPD reviews. Dr Riley reported that he has conducted and published IPD meta-analyses and related methodological research and guidance on the IPD approach to systematic review and receives funding for these activities. Dr Tierney reported that she is co-convenor of the Cochrane Collaboration IPD Methods Group; has conducted and published IPD meta-analyses, related methodological research, and guidance on the IPD approach to systematic review; and receives funding for these activities. No other disclosures were reported.

Members of the PRISMA-IPD Development Group:Project Group: Mark Simmonds, Gavin Stewart, Lesley Stewart. Steering Group: Mike Clarke, Richard Riley, Maroeska Rovers, Lesley Stewart (chair), Jayne Tierney. PRISMA-IPD Development Group (additional to project and steering groups): Doug Altman (Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom), Lisa Askie (NHMRC Clinical Trials Centre, University of Sydney, Australia), Sarah Burdett (MRC Clinical Trials Unit, UCL, London United Kingdom), Lelia Duley (University of Nottingham, United Kingdom), Jonathan Emberson (Clinical Trial Service Unit & Epidemiological Studies Unit, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom), Francois Gueffier (Faculty of Medicine, University of Lyon, Lyon, France), Amanda Kerr (Clinical Trial Service Unit & Epidemiological Studies Unit [CTSU], Nuffield Department of Population Health, Oxford University, Oxford, United Kingdom), Karmela Krleža-Jerić (Ottawa Group, IMPACT, and Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia), Toby Lasserson (Cochrane Editorial Unit, Cochrane Collaboration, London, United Kingdom), David Moher (Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada), Karel Moons (Julius Center For Health Sciences and Primary Care, UMC Utrecht, the Netherlands), Bob Phillips (Centre for Reviews and Dissemination, University of York, York, United Kingdom), Pascal Piedbois (Centre Paul Strauss, Strasbourg, France), Jean-Pierre Pignon (Meta-analysis unit, Gustave Roussy, Villejuif, France), Christopher H. Schmid (Department of Biostatistics and Center for Evidence Based Medicine, Brown University School of Public Health, Providence Rhode Island), Catrin Tudur Smith (Department of Biostatistics and North West Hub for Trials Methodology Research, University of Liverpool, Liverpool, United Kingdom), Claire Vale (MRC Clinical Trials Unit, UCL, London, United Kingdom), Emma Veitch (PLOS ONE, Cambridge, United Kingdom), Keith Wheatley (Cancer Research UK Clinical Trials Unit, University of Birmingham, Birmingham, United Kingdom).

Doug Altman is a member of the PRISMA group and author of the PRISMA Statement and is also a member of the EQUATOR Steering Group. David Moher is a member of the PRISMA group and author of the PRISMA Statement, a member of the EQUATOR Steering Group, and is funded through a University Research Chair. Toby Lasserson is an employee of the Cochrane Collaboration and a member of the group developing Cochrane’s own set of standards for conducting and reporting intervention reviews (MECIR) and for its plain language summaries (PLEACS).

Funding/Support: The PRISMA-IPD workshop was funded through a National Institute for Health Research (NIHR) Senior Investigator personal award (Dr L. Stewart). Preparatory work was partially supported by the same award and by the MRC as part of the MRC-National Institute for Health Research Methodology Research Programme (grant ID 88053).

Role of Funders/Sponsors: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The views expressed are those of the authors and not necessarily those of the Cochrane Collaboration, MRC, National Health Service, the NIHR, or the Department of Health.

Additional Contributions: We are grateful to Alison Tonks, MB, ChB (BMJ), and Iain Hrynaszkiewicz, MA (Nature Publishing Group & Palgrave Macmillan), who attended and participated in discussion and debate at the workshop, and to Marc Buyse, ScD (IDDI), who commented on the draft statement. We thank Alison Smith, MSc, and Dean Langan, MSc, for undertaking data extraction for the review of current individual patient data (IPD) publishing practice; Vanda Castle and Jackie Richmond for workshop organization; and John Jackson, BSc, and Vanda Castle for editorial assistance, all of whom are or were members of staff at the Centre for Reviews and Dissemination, University of York. Thanks also to Larysa Rydzewska (MRC Clinical Trials Unit, University College, London) for liaising with the Cochrane IPD Meta-analysis Methods Group, providing editorial assistance, and sourcing example material. None of these individuals received compensation for their contributions.

Correction: This article was corrected online on May 22, 2015, to correct errors in the text and Figure.