[Skip to Navigation]
Sign In
November 25, 2020

Same Data, Opposite Results?A Call to Improve Surgical Database Research

Author Affiliations
  • 1Department of Surgery, David Geffen School of Medicine, University of California, Los Angeles
JAMA Surg. 2021;156(3):219-220. doi:10.1001/jamasurg.2020.4991
Add or change institution
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    4 Comments for this article
    Two missing aspects of register based research in the paper: ”Same Data, Opposite results?
    Ingmar Naslund, MD, PhD | Department of Surgery. Faculty of Medicine and Health, Örebro University, Örebro, Sweden
    The paper “Same Data, Opposite Results?” by CP Childers and M-Maggard-Gibbons recently published in JAMA-Surgery (nov 25th, 2020) is an important paper not only for the researcher using register databases but also for the reader of such studies 1. There are however two important aspects missing in the paper.
    First, databases can be validated and/or complemented by cross running with other databases. At least in Scandinavia there are high quality registers covering the whole population with different specific purposes which by legal and ethical rules can be available for research. E.g. among many examples, by combining the Scandinavian Obesity Surgery
    Registry, SOReg, with the Swedish Birth Registry we could show a decreased risk of birth defects in babies after mothers with previous gastric bypass surgery compared to non-operated mothers with obesity 2. Drug use, re-hospitalization, cancer, mortality, socio-economic background are other areas available for deepening the clinical research. This is further described in a recent validation of the register 3
    Second, high quality registers can also be used to perform randomized controlled studies, so called registry-based RCTs (R-RCTs). These studies have the potential to offer higher enrolment in short time to a reduced effort and to a much lower economic cost than traditional RCTs. Furthermore, the studies can be conducted in standard clinical practise also offering higher generalizability 4. An example is the study of closing the mesenteric defects in laparoscopic gastric bypass 5. Further, the experimental results of this RCT could also be compared with the concomitant “real-world” situation of those outside the RCT by the national quality register, SOReg 6.
    Ingmar Näslund, MD, PhD 1
    Johan Ottosson MD PhD1
    Erik Stenberg MD, PhD1
    (JO director, IN and ES assistant-directors of SOReg)
    1.Department of surgery, Faculty of Medicine and Health, Örebro University, Örebro, Sweden

    1. Childers CP, Maggard-Gibbons M. Same Data, Opposite Results?: A Call to Improve Surgical Database Research. JAMA Surg. 2020 Online ahead-of-print.
    2. Neovius M, Pasternak B, Näslund I, Söderling J, Johansson K, Stephansson O. Association of Maternal Gastric Bypass Surgery With Offspring Birth Defects. JAMA. 2019;322(15):1515-1517.
    3. Sundbom M, Näslund I, Näslund E, Ottosson J. High acquisition rate and internal validity in the Scandinavian Obesity Surgery Registry. Surg Obes Relat Dis. 2020.
    4. Lauer MS, D'Agostino RB, Sr. The randomized registry trial--the next disruptive technology in clinical research? N Engl J Med. 2013;369(17):1579-1581.
    5. Stenberg E, Szabo E, Agren G, et al. Closure of mesenteric defects in laparoscopic gastric bypass: a multicentre, randomised, parallel, open-label trial. Lancet. 2016;387(10026):1397-1404.
    6. Stenberg E, Szabo E, Ottosson J, Naslund I. Outcomes of laparoscopic gastric bypass in a randomized clinical trial compared with a concurrent national database. Br J Surg. 2017;104(5):562-569.
    Re: Two missing aspects of register based research in the paper: ”Same Data, Opposite results?
    Christopher Childers, MD, PhD | UCLA
    We thank Dr. Naslund for their thoughtful comments. Our piece highlights how easy it is for database research to "go wrong" and one may easily find themselves doubtful of all database research by the end of the article. Dr. Naslund correctly reminds us of the promise and opportunities of database research. Not only are many individuals and groups performing high quality database research that meets and exceeds the quality standards we outline, but database research also affords a number of unique opportunities. Linking multiple databases can be challenging, but the breadth of questions that can be answered and the ability to have more granular data, easily justify the added upfront effort. Many linked databases are available, such as SEER-Medicare or NHIS-MEPS, and unique combinations can be created using novel linkage methods. Dr. Naslund also highlights one of the greatest strengths of database research which is the ability to enable large pragmatic randomized trials. Indeed, landmark trials in surgery, such as the FIRST trial, would not have been possible without registry data. Our article highlights what can go wrong, but Dr. Naslund reminds us of what can go right!
    Statistical Research Using Improved Surgical Databases
    Michael McAleer, PhD(Econometrics),Queen's | Asia University, Taiwan
    To the Editor:

    The pointed, challenging, and informative viewpoint by Childers and Maggard-Gibbons (2020) on improving research on surgical databases is highly topical.

    However, the viewpoint is not particularly different from valid critiques of empirical research in many disciplines in the sciences or social sciences.

    Moreover, there is little point in improving any databases if the statistical analysis has flaws that can easily be eliminated.

    It is well known that estimating different models on the same data set can lead to different results, regardless of whether one model is a special case of the other or

    This is illustrated perfectly in the Table, where two sets of variables are used, with neither set being a special case of the other.

    The paper by Fields, Lu, Palenzuela et al. (2019) simply uses many more variables than those in Turner, Jung and Scarborough (2019).

    On the other hand, Turner, Jung and Scarborough (2019) use some variables that are not used in the paper by Fields, Lu, Palenzuela et al. (2019), so neither is a special case of the other.

    Moreover, with different sample sizes, 11475 versus 10357, and different primary outcomes, the outcomes are virtually guaranteed to be different, both quantitatively and statistically;

    Critical analysis of surgical databases is important, as is critical analysis of empirical analysis, especially when improvements and simple explanations are readily available.
    Improving Surgical Database Research Beyond Analysis
    Colin Sue-Chue-Lam, MD | University of Toronto
    Authors conducting database research face many potential analytic pitfalls. These are well-illustrated by Childers and Maggard-Gibbons, who highlight two papers answering the same question using the same database but arriving at contradictory conclusions.(1) We hope to highlight two additional pitfalls in surgical database research.

    First, researchers should differentiate between associational and causal questions.(2) The authors rightly note that observational data are prone to selection and confounding bias. This is true for causal questions, such as whether retrieval bags for appendectomy reduce infection. However, researchers using databases to answer associational questions (‘which factors predict retrieval bag use?’) need not control
    for confounders to obtain unbiased measures of association.(2)

    To address this concern, we recommend that authors clearly state the aim of their study. Associational and causal aims are fundamentally different in their intent, analysis, and interpretation. Where unclear, reviewers and editors could work with authors to ensure that all aspects of a study are consistent with its stated aims. Reporting guidelines could be amended to encourage unambiguous reporting of study intent.

    A second issue is that existing databases are often poorly suited to new research questions. In database research, the onus is predominantly on individual researchers to recognize the limitations of their data. Researchers should discuss limitations in manuscripts, but in some cases it may be most appropriate to abstain from conducting analyses that are unlikely to be useful and even potentially misleading.

    Addressing this second concern is more complex than the first. Prospective studies typically entail great investments of time and money, and academics face significant pressure to publish.(3) The barriers to conducting prospective research might be lowered, for example by incorporating routinely collected data into clinical trials.4 Academics have for decades called on institutions to rely less on publication volume as a performance metric, potentially by capping the number of publications considered for promotion.(3)

    The upheaval of the COVID-19 pandemic may catalyze these kinds of deep change in surgical research. We have seen prospective research be rapidly mobilized to answer pressing questions with appropriate political will and funding. At the same time, important conversations are resurfacing about how simplistic evaluations of publication volume can perpetuate gender inequities in academia when the responsibility for care work falls disproportionately to women.(5) Greater rigor must certainly be brought to the analysis of database research, but there is perhaps no better moment to challenge these more upstream determinants of surgical research quality.

    Colin Sue-Chue-Lam MD
    Matthew P. Guttman MD
    Hala Muaddi MD MSc
    Division of General Surgery, University of Toronto

    1. Childers CP, Maggard-Gibbons M. Same Data, Opposite Results? A Call to Improve Surgical Database Research. JAMA Surg. 2020.
    2. Hernán MA. The C-word: Scientific euphemisms do not improve causal inference from observational data. Am J Public Health. 2018;108(5):616-619.
    3. Angell M. Publish or perish: A proposal. Ann Intern Med. 1986;104(2):261-262.
    4. Hemkens L, Contopoulos-Ioannidis D, Ioannidis J. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ. 2016;188(8):E158-E164.
    5. Kibbe MR. Consequences ofthe COVID-19 Pandemic on Manuscript Submissions by Women. JAMA Surg. 2020;155(9):803-804.