A Checklist to Elevate the Science of Surgical Database Research

Each year, JAMA Surgery receives hundreds of submissions that retrospectively analyze large surgical databases. Although many of these attempt to shed light on new and important questions, most do not get published. A majority of submissions are not even sent out for peer review because they have clear flaws in the data analytic techniques or they attempt to address a research question that cannot be adequately answered with the proposed data set. Of those that are sent out for peer review, many are recommended to be rejected by expert peer reviewers as they find major methodological flaws in the use of these otherwise powerful data sets. Articles that are published frequently come from a select group of investigators who have developed a mastery of specific data sets and the analytic techniques required to truly harness their potential.

To help more and more investigators develop the skills needed to appropriately use the increasing number of large surgical data sets available, the editors of JAMA Surgery have commissioned this current series of statistical methodology articles. The series is aimed at providing a short, practical guide for academic surgeons and researchers in the use of the most widely available surgical data sets that can be used across the research continuum, from conceptualization to peer-reviewed publication. To achieve this, JAMA Surgery is pleased to partner with the Surgical Outcomes Club (http://www.surgicaloutcomesclub.com) to publish a series that will be instrumental in elevating the science used in surgical outcomes research.

This 13-part series provides a succinct overview of the 11 most widely used data sets^1-11 (Box 1), their specific features, strengths, limitations, and some important statistical considerations. In addition, we present a 10-item checklist (Box 2) in this Editorial that authors can use to ensure that they have covered what is “at minimum” expected from a manuscript that uses 1 of these databases. Finally, we support this series with an Editorial¹² by our biostatistician colleagues, who provide more in-depth information on statistical methodologies mentioned in the practical guides as well as potential pitfalls that need to be avoided. To ensure that these guides are truly practical and relevant, we have leveraged our partnership as the official journal of the Surgical Outcomes Club to develop a 3-person authorship team that includes (1) a surgeon investigator who is a senior member of the Surgical Outcomes Club with extensive experience using that particular data set; (2) a member of the JAMA Surgery Editorial Board who commonly reviews such manuscripts; and (3) a JAMA Surgery biostatistician who is routinely consulted to knowledgeably evaluate the methods for these types of papers (in some cases, the JAMA Surgery board member is also an expert methodologist, obviating the need for a biostatistician). This authorship strategy has ensured that each guide is presented in terms that are relevant to surgeons, even if they do not have previous experience with the biostatistics or the data set involved and includes basic information required to prepare a manuscript for the rigorous JAMA Surgery peer review process.

To help authors improve the quality of their submissions, we have developed a 10-item checklist (Box 2). The first item in our checklist encourages authors to pursue hypothesis-driven science. Defining a solid research question is key to translating a problem into an operational hypothesis. The FINER (Feasible, Interesting, Novel, Ethical, Relevant) criteria or the PICO (Patient, Population, or Problem; Intervention, Prognostic Factor, or Exposure; Comparison or Intervention; Outcome) format can help develop a meaningful research question.^13,14 Adequately defining the population of interest lays a solid groundwork for the interpretation, applicability, and generalizability of the research findings. We understand that in many cases, authors may be using these large databases for “hypothesis-generating” research. That is of course acceptable, but one must start with a solid research question to conduct a meaningful research project that will generate important hypotheses from the large data sets that can then be further studied with translational or prospective approaches. Some authors ask if it is acceptable to try and see what they can find in a data set that they may have access to without a real research question. This is never acceptable.

Second, we remind authors to seek approval or an exemption from an institutional review board and to properly document and comply with applicable data use agreements. These are often overlooked, but compliance with applicable rules are necessary for patient privacy and a variety of important reasons. Third, a thorough literature review will assist in making sure the best database is selected to answer research questions and to make sure the research question has not been previously answered. Fourth, we encourage authors to invest enough time early on to get to know the database, confirm that it has the appropriate variables, and understand methodological considerations to make sure this is the best data set available for the study. Fifth, a clear definition of the inclusion and exclusion criteria, as well as outcome variables, is necessary for reviewers and readers to understand the population under study. This also helps facilitate data query and extraction of a complete and useful data set.

Another important aspect of working with databases is the need to identify potential confounders or covariates and use risk adjustment to minimize bias. Given the observational nature of data in these surgical registries, 1 approach to do this is to create a directed acyclic graph,¹⁵ which will allow a visual depiction of the potential association being explored along with the covariates and confounders that need to be kept in mind or accounted for while studying the association. Please refer to the Editorial by Kaji et al¹² for further details. Authors should also avoid use of causal language when describing the results of these observational studies. Seventh, authors must account for any updates or significant changes to the variables of interest over time as this might jeopardize comparison between and across years (for example, in the National Cancer Database, the definition of sentinel lymph node biopsy for breast and melanoma has changed during the last 10 years, and this must be accounted for). Eighth, authors are encouraged to identify if competing risks exist in outcomes.¹⁶ For example, if authors are studying complication rates 30 days after surgery, one must account for patients who may have already died and are not at risk for developing these complications. Ninth, authors must ensure that any data issues, such as missing data, are openly discussed in a clear, cohesive, and replicable way. Authors must lay out any data limitations, how they were addressed, and measures taken to reduce their impact (eg, sensitivity analyses, multiple imputation¹⁷ for missing data). Finally, as our last item in the checklist, we encourage authors to clearly state a take-home message. It is best to communicate how the study advances the science, addresses gaps in knowledge, highlights further research opportunities, and discusses important policy or clinical implications of the work.

We recommend that authors use this checklist, the practical guide for their chosen data set, and the statistical tips for analyzing data sets as a 3-part series to consult before submission of their manuscript. We hope that by following these simple guides, authors can benefit from the collective wisdom of so many colleagues who have successfully completed similar analyses in the past. We look forward to the opportunity to publish analytically advanced studies and hope that these guides will help elevate the science of surgical database research.

Corresponding Author: Adil H. Haider, MD, MPH, Center for Surgery and Public Health, Department of Surgery, Brigham and Women’s Hospital, 1620 Tremont St, Ste 4-020, Boston, MA 02120 (ahhaider@bwh.harvard.edu).

Published Online: April 4, 2018. doi:10.1001/jamasurg.2018.0628

Conflict of Interest Disclosures: Dr Haider reports receiving grants from the Henry M. Jackson Foundation of the Department of Defense, the Orthopaedic Research and Education Foundation, and the National Institutes of Health, and nonfinancial research supports from the Centers for Medicare and Medicaid Services Office of Minority Health. Dr Bilimoria was the president of the Surgical Outcomes Club from 2016 to 2017. No other disclosures were reported.

Funding/Support: This work is supported by the Henry M. Jackson Foundation for the Advancement of Military Medicine of the Department of Defense (Dr Haider).

Role of the Funder/Sponsor: The funder had no role in the preparation, review, or approval of the manuscript and decision to submit the manuscript for publication.