In A, it is unclear whether there is a fascial defect or just bulging.
It is unclear whether there is mesh.
eTable 1. Patient Characteristics in Patients With Postoperative CT Scan Versus No Postoperative CT Scan
eTable 2. Reasons for CT Scan
eTable 3. Rate of Agreement Among 9 Blinded Reviewers in Interpretation of 100 CT Scans
eTable 4. Reasons for Disagreement and Proposed Solutions
Customize your JAMA Network experience by selecting one or more topics from the list below.
Holihan JL, Karanjawala B, Ko A, et al. Use of Computed Tomography in Diagnosing Ventral Hernia Recurrence: A Blinded, Prospective, Multispecialty Evaluation. JAMA Surg. 2016;151(1):7–13. doi:10.1001/jamasurg.2015.2580
Physical examination misses up to one-third of ventral hernia recurrences seen on radiologic imaging. However, tests such as computed tomographic (CT) imaging are subject to interpretation and require validation of interobserver reliability.
To determine the interobserver reliability of CT scans for detecting a ventral hernia recurrence among surgeons and radiologists. We hypothesized there would be significant disagreement in the diagnosis of a ventral hernia recurrence among different observers. Our secondary aim was to determine reasons for disagreement in the interpretation of CT scans.
Design, Setting, and Participants
One hundred patients who underwent ventral hernia repair from 2010-2011 at an academic health care center with a postoperative CT scan were randomly selected from a larger cohort. This study was conducted from July 2014 to March 2015. Prospective assessment of the presence or absence of a recurrent ventral hernia on CT scans was compared among 9 blinded reviewers and the radiology report. Five reviewers (consensus group) met to discuss all CT scans with disagreement. The discussion was assessed for keywords and key concepts. The remaining 4 reviewers (validation group) read the consensus group recommendations and reassessed the CT scans. Pre- and post-review κ were calculated; the post-review assessments were compared with clinical examination findings.
Main Outcomes and Measures
Interobserver reliability in determining hernia recurrence radiographically.
Of 100 CT scans, there was disagreement among all 9 reviewers and the radiology report on the presence/absence of a ventral hernia in 73 cases (κ = 0.44; 95% CI, 0.35-0.54; P < .001). Following discussion among the consensus group, there remained disagreement in 10 cases (κ = 0.91; 95% CI, 0.83-0.95; P < .001). Among the validation group, the κ value had a slight improvement from 0.21 (95% CI, 0.12-0.33) to 0.34 (95% CI, 0.23-0.46) (P < .001) after reviewing the consensus group proposals. There was disagreement between clinical examination and the consensus group assessment of CT scans on the presence/absence of a ventral hernia in 25 cases. The concepts most frequently discussed were the absence of an accepted definition for a radiographic ventral hernia and differentiating pseudorecurrence from recurrence.
Conclusions and Relevance
Owing to the high interobserver variability, CT scan was not associated with reliable diagnosing in ventral hernia recurrence. Consensus guidelines and improved communication between surgeon and radiologist may decrease interobserver variability.
Ventral hernia repair is a common procedure performed by general surgeons, and recurrence is an important quality measure.1-3 Historically, physical examination was considered the accepted method for detecting recurrences in clinical practice and research reporting. However, with the increasing use of imaging techniques, physical examination is beginning to be considered inadequate to accurately diagnose a ventral hernia.4 Factors that impact the ability to diagnose a recurrence on physical examination alone include a small defect, large body habitus, or pseudorecurrence, defined as a bulge not related to a recurrence (eg, seroma, mesh, or tissue eventration/bulging beyond the boundaries of the fascia).5,6 Previous unblinded studies suggested that physical examination misses up to 31% of ventral hernias seen on radiographic imaging and concluded that surgeon physical examination is inferior to computed tomographic (CT) scan for the diagnosis of ventral hernias.4 In addition, dictated radiology reports compared with unblinded reads by a single surgeon have reported a difference of 21% in diagnosing a ventral hernia.7 Misreporting of recurrences may result in inappropriate management decisions and inaccurate benchmarking of surgical outcomes.
The diagnosis of a ventral hernia, regardless of symptoms, is important to make. Hernias are repaired not only for symptoms (such as pain, functional limitation, or cosmesis), but also for prevention of progression of size, development of symptoms, and acute presentation. Whether an asymptomatic ventral hernia should be repaired remains unclear and largely uninvestigated. With inguinal hernias, 2 randomized trials noted that most asymptomatic/oligosymptomatic hernias eventually require surgery owing to progression of symptoms, increase in size, or acute presentation.8,9 In addition, previous studies have suggested that the rate of emergency ventral hernia repair is increasing, and many of these patients do not have a prior diagnosis of a ventral hernia.10,11
Currently, there is no accepted standard of care to diagnose a ventral hernia. Options for diagnosis include clinical examination, CT scan or other imaging modality, or surgery (laparotomy or diagnostic laparoscopy). Surgery has the risk for missing hernia recurrence during laparoscopy (incomplete exposure of the peritoneum or fascia, or hernia recurrence with only preperitoneal fat but no peritoneal protrusion through the defect) or during laparotomy (incomplete exposure of the entire incision). In addition, it is not ethically feasible to operate solely to diagnose a hernia recurrence. For this reason, CT scan and clinical examination are relied on heavily for the diagnosis of a ventral hernia recurrence.
The purpose of this study was to determine the interobserver reliability of CT scans for detecting a ventral hernia recurrence in patients who had a prior ventral hernia repair among different readers, including surgeons and radiologists, who were actively evaluating for a hernia on CT scan. We hypothesized that there would be significant disagreement in the diagnosis of a ventral hernia recurrence between surgeons and radiologists as well as disagreement between readers of the same specialty. Our secondary aim was to determine reasons for disagreement in the interpretation of CT scans.
Following institutional review board approval from the University of Texas Health Science Center, a database of 6 institutions was retrospectively analyzed to identify patients who had undergone ventral hernia repair. Patient consent was waived because the medical record review was retrospective and it was not feasible to obtain consent. A total of 1207 patients were identified from 2010-2011. Of these, 308 (25.5%) had received a postoperative CT scan and 899 (74.5%) had not. An analysis was performed using this database to determine whether there were differences between the group with and without postoperative CT scans. All variables were defined by National Surgical Quality Improvement Program or European Hernia Society definitions whenever possible; all other variables have been previously defined and reported.12-15 To compare patients with and without a postoperative CT scan, univariate analyses were performed using the χ2 test for categorical variables and unpaired 2-tailed t test or Wilcoxon rank-sum test for continuous variables. Normal data were reported as mean and standard deviation, while all other data were reported as median and range.
All patients who had a ventral hernia repair at a single institution were assessed (n = 126 patients at a single institution with a postoperative CT scan and 404 without a postoperative CT scan). Of these patients, 100 patients who had received a postoperative CT scan were randomly selected using a random number generator, and these patients formed the cohort for our analysis, which was conducted from July 2014 to March 2015. The reason for ordering each CT scan was recorded. To determine correlation to physical examination, patient medical records were reviewed for a documented hernia recurrence on at least 1 physical examination following surgery. Finally, the number of patients who underwent abdominal surgery following their postoperative CT scan was recorded, and the presence or absence of a ventral hernia recurrence on reoperation was determined.
Nine independent and blinded reviewers (3 surgeons and 6 radiologists) evaluated the CT scans and reported the presence or absence of a ventral hernia. In current practice, many surgeons rely on their own interpretation of a CT scan to determine whether a patient has a hernia recurrence because they are able to have a focused evaluation of the abdominal wall, understand the previous procedure performed, and clinically correlate their findings. Because of this, we opted to include both radiology and surgery specialists. Two of the surgeons were staff referral physicians for complex ventral hernia repair, while 1 was a staff general surgeon. Four of the radiologists were staff radiologists specializing in body imaging, 1 a staff radiologist, and the other a chief radiology resident with an interest in body imaging. In addition, the official radiology report in each patient’s medical records was assessed for the mention of a ventral hernia. Scans were considered to have disagreement if 1 or more of the reviewers had a different conclusion than the others.
Following the initial evaluation, a qualitative analysis was performed. Five reviewers, known as the consensus group (3 surgeons and 2 radiologists), met to discuss the cases that had less than 100% agreement. Scans were reviewed, an open discussion ensued, and the reviewers voted in a blinded fashion (ie, reviewers were blinded to the votes of the other reviewers as well as to the clinical history of the patient) on the presence or absence of a ventral hernia. In cases where there was disagreement, relevant clinical data—such as the use of mesh—were provided. Notes of the discussion were reviewed and edited by 5 reviewers and the study coordinator. All reviewers abstracted keywords and key concepts. Based on the discussions, the 5 reviewers developed proposals to address the areas of disagreement. The remaining 4 reviewers (4 radiologists), referred to as the validation group, read the proposals from the discussion and reevaluated the 100 CT scans to assess whether the proposals changed their opinion regarding the presence or absence of a ventral hernia on CT scan.
The κ statistic was calculated to report on interobserver reliability for CT scans for the following groups both prediscussion and postdiscussion: (1) all 9 reviewers and the CT report, (2) all 9 reviewers, (3) between surgeons, (4) between radiologists, (5) the consensus group, and (6) the validation group. The κ statistic calculates the magnitude of agreement between observers above and beyond that expected by chance alone.16 A κ of 0 suggests no agreement between observers, while a κ of 1 demonstrates perfect agreement. A κ of 0 to 0.50 suggests agreement by chance. For this study, we considered good agreement to be a κ>0.90.17,18 A significant P value of κ implies that the agreement between observers exceeds that expected under the null hypothesis of random ratings.18
Coding and condensing data from the qualitative discussion into categories was performed using latent content analysis, applying inductive coding methods.19,20 A list of categories, or coding frame, was developed and transcribed from the discussion notes.
Of 1207 patients who underwent ventral hernia repair from a multi-institutional database, 308 (25.5%) had a postoperative CT scan. There were few differences between the patients who had a postoperative CT scan and those who did not (eTable 1 in the Supplement). Patients with a postoperative CT scan were slightly more likely to have chronic obstructive pulmonary disease and undergo laparoscopic surgery than those without a postoperative CT scan. Of those with a postoperative CT scan, scans were performed a median of 375 days (range, 2-1711 days) following ventral hernia repair. Of the 100 patients included in this study, 61% had a postoperative CT scan for concerns related to their hernia; 39% of patients had a CT scan for reasons unrelated to their hernia repair (eTable 2 in the Supplement).
Based on blinded review of the 100 CT scans, there was significant disagreement in the number of recurrences reported by the surgeons (35%, 45%, and 37%), radiologists (47%, 48%, 40%, 85%, 69%, and 24%), and radiology report (35%). There was great variation in the number of reviewers who diagnosed a ventral hernia recurrence on each CT scan (eTable 3 in the Supplement). Lack of consensus agreement occurred in 73 scans between all 9 reviewers plus the CT scan report. Pre-review κ was 0.44 (P < .001) (Table 1).
After discussion and reassessment among the consensus group, no consensus agreement could be reached on 10 cases. Post-review κ was 0.91 (P < .001) (Table 1). There were several common themes responsible for disagreements, including poorly defined terminology, misinterpretation of CT scans, and poor communication between surgeons and radiologists (Table 2; eTable 4 in the Supplement). After review of the proposals in Table 2 and eTable 4 in the Supplement, the validation group had slight improvement in reliability; however, there was disagreement in 60 cases and κ was 0.34 (P < .001).
The presence or absence of radiographic recurrence was determined only when consensus was reached between all 5 reviewers in the consensus group after the second review. Using this definition, there was a radiologic recurrence in 44 cases and no recurrence in 46 cases. No consensus was reached in 10 cases. There was disagreement between the clinical examination and 5-reviewer consensus on CT scan in 25 cases with a κ value of 0.44 (P < .001) (Table 2). Eighteen patients underwent an abdominal reoperation following their CT scan. Of these patients, 14 (77.8%) had a hernia noted during reoperation (Table 2).
The diagnosis of a ventral hernia or its recurrence remains challenging, and no accepted diagnostic standard of care exists. Computed tomographic scan is increasingly being recommended as the standard for the diagnosis of a ventral hernia recurrence.21,22 However, in this blinded, multispecialty study, the reliability of CT scans for diagnosing the presence or absence of a ventral hernia following prior surgical repair was poor, with initial disagreement among reviewers in 73% of cases. While some disagreement could be overcome by providing further clinical information, consensus could not be achieved in 10 cases. In addition, although the agreement of the validation group improved after reviewing the proposals by the consensus group, the improvement was nominal.
The poor interobserver reliability of CT scans for diagnosing ventral hernia recurrences has important implications in clinical care and hernia research. Hernia recurrence is a commonly used outcome in clinical patient care as well as in hernia research, and using CT only, examination only, or both to determine recurrence could affect conclusions. Depending on the reviewer, using CT scan to diagnose ventral hernia recurrence may lead to over- or underdiagnosis. This over or under diagnosis could result in unnecessary surgical procedures or missed opportunities for the prevention of hernia complications, such as incarcerated or strangulated bowel.
Most disagreements in this study stemmed from the lack of a clear definition of a ventral hernia on CT scan. A commonly used definition for a ventral hernia in surgery and radiology is the abnormal protrusion of an organ or tissue through the anterior abdominal wall fascia.23 However, this becomes less straightforward when trying to decipher whether a recurrence has occurred; the accepted definition does not differentiate the more subtle differences between recurrence, eventration, and an unclosed fascial defect bridged with mesh (Figure 1 and Figure 2). In addition, one-fifth of initial disagreements occurred owing to errors in interpreting the scan, primarily in identifying or recognizing a hernia. Often, hernias distant from the midline or from the original site of repair were overlooked, leading to a significant number of false-negatives. A systematic and standardized approach to reading CT scans for ventral hernias needs to be developed and disseminated. Finally, many disagreements were resolved simply through communication between clinicians and radiologists. For example, disagreements over mesh eventration vs recurrence were resolved by determining whether mesh had been placed during surgery. Providing reasons for scans may improve the radiologist’s ability to differentiate other diagnoses from ventral hernia recurrences. From this study, it is clear that a standardized definition of a ventral hernia recurrence on CT scan must be developed. However, while a consensus statement provides the framework to identify key areas of concern, it fails to provide the adequate details to improve the reliability of interpreting CT scans for the diagnosis of ventral hernia recurrence. The consensus group had considerably better agreement after the second review than the validation group, which is likely attributed to the fact that the consensus group had an in-person discussion over the scans, while the validation group did not meet in person. This confirms that communication between surgeon, the CT scan–ordering physician, and radiologist needs to be improved.
While physical examination remains an important part of ventral hernia diagnosis, it is not always adequate for the detection of a ventral hernia. Particularly in this country’s population, obesity and complex surgical history can mask a hernia recurrence, necessitating the use of radiologic imaging to detect a recurrence. In this study, one-half of CT scan–detected hernias as determined by the consensus group were not noted on physical examination, while almost all hernias diagnosed clinically were seen on CT scan (Table 3). The clinical significance of hernias that are seen solely on CT scan without physical findings is unknown, particularly because CT scans are often obtained for symptoms and the inability of the physician to identify a definite hernia.24-26 Given that the risk for incarceration or strangulation of clinically detectable hernias is low (estimated 1%-2% per year; range, 0%-20%), one might postulate that the risk for complications from radiologically detected hernias would also be low.27-32 Repair of hernias only detectable on radiologic imaging may result in the unnecessary use of resources, increased risk for complications, and increased stress on patients.33 In addition, without the correlation of intraoperative findings, it is unclear whether these hernias only seen on imaging are actually present. Currently, there is insufficient evidence to recommend routine CT scan for evaluation of all suspected ventral hernias or of all patients who have undergone repair. The clinical significance and optimal management of hernias identified only on imaging remain to be determined. Currently, our practice is to recommend CT scan to diagnose a ventral hernia recurrence when physical examination is uncertain only in the presence of symptoms.
In 4 cases, CT scans concluded that no hernia was present, while physical examination diagnosed a ventral hernia. Of these, 2 of the CT scans were done to evaluate for a hernia recurrence, and no reoperation was ever performed. In 1 of these cases, the CT scan was done for a wound infection at the repair site, and a subsequent unrelated reoperation (for acute cholecystitis) did not reveal a hernia recurrence. Finally, in the fourth case, a CT scan was done for reasons unrelated to the hernia, and no reoperation was ever performed. In these rare cases, CT scan may have prevented an unnecessary surgery. Thus, there may be use to obtaining a CT scan in patients with examination findings consistent with a recurrence to confirm the presence of a hernia. However, given that this scenario only occurred rarely, the cost-effectiveness of such a strategy needs to be carefully evaluated.
This study had several limitations. First, it was performed at a single academic institution. The applicability of these results to other centers and other clinicians needs to be validated. For example, the patient population was on average obese, with a body mass index (calculated as weight in kilograms divided by height in meters squared) of more than 30, which may have affected the ability of the clinician to detect a recurrence on examination. Second, there is no gold standard for the diagnosis of a ventral hernia, thus, the sensitivity, specificity, and accuracy of CT scans could not be calculated. Furthermore, only 100 scans were reviewed, which is a small sample size. To our knowledge, this is the first study published examining the subject of the interrater reliability of diagnosing ventral hernia recurrence on CT scan. Future studies should include more surgeons and radiologists and more CT scans of various patient groups (including patients with or without clinical hernia following ventral hernia repair). In addition, hernia recurrence was not correlated with intraoperative findings because it is not ethically feasible to reoperate solely for diagnostic purposes. In addition, reoperation still runs the risk for missing a ventral hernia recurrence. Next, there was selection bias, as only patients who received a postoperative CT scan were evaluated; however, there did not appear to be major differences between these patients and patients who did not have a postoperative CT scan. Furthermore, the clinical significance of hernias noted on examination and CT scan is unclear. Patient-centered outcomes were not measured and correlated with the findings. In addition, it is unknown whether the finding of a hernia recurrence on CT scan influenced the clinician examining the patient for a hernia recurrence, and the interobserver reliability among clinicians for detecting a hernia recurrence on physical examination is unknown. In this study, a single physical examination was used to determine the presence of a hernia on examination.
The reliability of the interpretation of CT scans for detecting recurrence after ventral hernia repair is poor. Standardized definitions of a radiographic hernia recurrence and improved communication between surgeons and radiologists may enhance the reliability of diagnosing hernia recurrences on CT scans. The clinical significance of hernias noted only on CT scan and not on clinical examination remains to be established because it is unclear what percentage of these are true hernias, symptomatic, or will develop future symptoms (increasing size, pain, or incarceration). Further studies are needed to assess (1) the impact of standardized definitions for radiographic recurrence on the reporting of outcomes and clinical practice and (2) the short-term and long-term clinical significance of hernias seen on CT but not detected on physical examination.
Corresponding Author: Julie L. Holihan, MD, Department of Surgery, University of Texas Health Science Center at Houston, 6431 Fannin St, MSB 5.254, Houston, TX 77030 (email@example.com).
Accepted for Publication: May 20, 2015.
Published Online: September 23, 2015. doi:10.1001/jamasurg.2015.2580.
Author Contributions: Drs Holihan and Liang had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Holihan, Karanjawala, Askenasy, Alawadi, Kao, Liang.
Acquisition, analysis, or interpretation of data: Holihan, Karanjawala, Ko, Askenasy, Matta, Gharbaoui, Hasapes, Tammisetti, Thupili, Bondre, Flores-Gonzales, Liang.
Drafting of the manuscript: Holihan, Matta, Alawadi, Flores-Gonzales, Liang.
Critical revision of the manuscript for important intellectual content: Holihan, Karanjawala, Ko, Askenasy, Gharbaoui, Hasapes, Tammisetti, Thupili, Alawadi, Bondre, Kao, Liang.
Statistical analysis: Holihan, Flores-Gonzales, Liang.
Administrative, technical, or material support: Askenasy, Matta, Hasapes, Flores-Gonzales, Liang.
Study supervision: Askenasy, Matta, Kao, Liang.
Conflict of Interest Disclosures: None reported.
Funding/Support: This work was supported by the Center for Clinical and Translational Sciences, which is funded by the National Institutes of Health Clinical and Translational Awards UL1 TR000371 and KL2 TR000370 from the National Center for Advancing Translational Sciences.
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.