Assessing the Accuracy and Readability of Online Health Information for Patients With Pancreatic Cancer

Importance  The degree to which patients are empowered by written educational materials depends on the text’s readability level and the accuracy of the information provided. The association of a website’s affiliation or focus on treatment modality with its readability and accuracy has yet to be thoroughly elucidated.

Objective  To compare the readability and accuracy of patient-oriented online resources for pancreatic cancer by treatment modality and website affiliation.

Design  An online search of 50 websites discussing 5 pancreatic cancer treatment modalities (alternative therapy, chemotherapy, clinical trials, radiation therapy, and surgery) was conducted. The website’s affiliation was identified. Readability was measured by 9 standardized tests, and accuracy was assessed by an expert panel.

Main Outcomes and Measures  Nine standardized tests were used to compute the median readability level of each website. The median readability scores were compared among treatment modality and affiliation categories. Accuracy was determined by an expert panel consisting of 2 medical specialists and 2 surgical specialists. The 4 raters independently evaluated all websites belonging to the 5 treatment modalities (a score of 1 indicates that <25% of the information is accurate, a score of 2 indicates that 26%-50% of the information is accurate, a score of 3 indicates that 51%-75% of the information is accurate, a score of 4 indicates that 76%-99% of the information is accurate, and a score of 5 indicates that 100% of the information is accurate).

Results  The 50 evaluated websites differed in readability and accuracy based on the focus of the treatment modality and the website’s affiliation. Websites discussing surgery (with a median readability level of 13.7 and an interquartile range [IQR] of 11.9-15.6) were easier to read than those discussing radiotherapy (median readability level, 15.2 [IQR, 13.0-17.0]) (P = .003) and clinical trials (median readability level, 15.2 [IQR, 12.8-17.0]) (P = .002). Websites of nonprofit organizations (median readability level, 12.9 [IQR, 11.2-15.0]) were easier to read than media (median readability level, 16.0 [IQR, 13.4-17.0]) (P < .001) and academic (median readability level, 14.8 [IQR, 12.9-17.0]) (P < .001) websites. Privately owned websites (median readability level, 14.0 [IQR, 12.1-16.1]) were easier to read than media websites (P = .001). Among treatment modalities, alternative therapy websites exhibited the lowest accuracy scores (median accuracy score, 2 [IQR, 1-4]) (P < .001). Nonprofit (median accuracy score, 4 [IQR, 4-5]), government (median accuracy score, 5 [IQR, 4-5]), and academic (median accuracy score, 4 [IQR, 3.5-5]) websites were more accurate than privately owned (median accuracy score, 3.5 [IQR, 1.5-4]) and media (median accuracy score, 4 [IQR, 2-4]) websites (P < .004). Websites with higher accuracy were more difficult to read than websites with lower accuracy.

Conclusions and Relevance  Online information on pancreatic cancer overestimates the reading ability of the overall population and lacks accurate information about alternative therapy. In the absence of quality control on the Internet, physicians should provide guidance to patients in the selection of online resources with readable and accurate information.

Patients increasingly use the Internet to learn more about diseases and treatment options.^1-4 Online medical information is accessible and relatively inexpensive and empowers patients to make decisions.^5,6 It is often used in lieu of formal medical consultations, particularly among patients with financial limitations.^4,7-9 However, online materials may be too difficult for the average patient to comprehend.^1,6 Furthermore, the accuracy of online information can vary widely. Online information may be biased, motivated by underlying commercial interests, taken out of context, or incomplete. This has the potential to lead to unfavorable downstream effects. Particularly for complex and aggressive diseases such as pancreatic cancer, misinformation or perceptions of dismal outcomes can hinder patients from seeking appropriate care.¹⁰

Compounding this problem is the fact that reading materials are rarely written at the recommended sixth-grade reading level.^11-15 A patient’s health literacy or ability to comprehend written health information can impact clinical outcomes. Elderly patients with low health literacy are at greater risk of all-cause mortality.¹⁶ Patients with low health literacy and advanced colorectal cancer are also less likely to receive standard-of-care chemotherapy.¹⁷

It has been reported that patients believe searching the Internet empowers them to make decisions about their health.^4,7,18-20 However, it has been found that online information varies widely in terms of quality.² Patients must have accurate and understandable information to participate in the shared decision-making process.^14,15 It is important to study whether available online health information is accurate and written at an appropriate reading level. This study assesses the readability, a surrogate for reading level, and accuracy of websites discussing various treatment options for pancreatic cancer.

Box Section Ref ID

This study was approved by the institutional review board at the Dana-Farber/Harvard Cancer Center in Boston, Massachusetts. Online searches using Google, the search engine most commonly used in the United States and throughout the world,²¹ were limited to 1 location (the United States) and 1 language (English). Data were collected during May 2014 using the following search terms, which are also the 5 treatment modalities: (1) pancreas cancer alternative therapies, (2) pancreas cancer chemotherapy, (3) pancreas cancer clinical trials, (4) pancreas cancer radiotherapy, and (5) pancreas cancer surgery. Websites had to meet the following eligibility criteria: they were designed for public consumption and not for health care professionals; they were specific to pancreatic cancer treatment, at least 100 words in length, and written in English. Websites meeting the following criteria were excluded: general newspaper reports that did not discuss treatment information, access restricted by subscriptions or fees, duplicate websites, and websites presenting videos only. Websites were classified into 5 affiliation categories using the following method: (1) websites with “.org” domains that were not affiliated with academic hospitals were treated as nonprofit organizations; (2) websites with “.edu” domains were treated as academic websites; (3) websites with “.gov” domains were treated as government websites; (4) websites with news portals were treated as communication media websites; and (5) websites that did not disclose affiliation or that disclosed affiliation to a private holder were treated as private. Affiliation was confirmed by use of the WHOis.net database.²²

The first 10 eligible websites appearing for each treatment category were analyzed for accuracy and readability. Accuracy was defined as the degree of concordance of the information provided with the best evidence or with generally accepted medical practice.² This includes providing critical facts to describe a topic and the absence of erroneous or misleading information.²³ Readability level was defined as the number of years of education required to comprehend written information. It is assessed based on word and sentence length as surrogates of text complexity.²⁴ The evaluation of completeness is questionable among Internet resources. Websites are often purposely designed to address 1 specific topic in particular. This allows for accuracy, but not completeness, to be assessed.^2,18,23 For these reasons, completeness was not evaluated in this study.

Readability was analyzed using the Readability Studio Professional Edition version 2012.1 software (Oleander Software Ltd). Nine standardized tests were used: the Coleman-Liau Index, the Flesch-Kincaid Grade Level, the FORCAST Readability Formula, the Fry Readability Graph, the Gunning Fog Index, the New Dale-Chall Readability Formula, the New Fog Count, the Raygor Readability Estimate, and the Simple Measure of Gobbledygook Readability Formula.²⁴ The use of multiple formulae has been found to improve the reliability and accuracy of readability scores.²⁴ The 9 tests were used to compute the median readability level of each website. The median readability scores were compared among treatment modality and affiliation categories. Table 1 shows the education equivalence of readability levels.

Accuracy was determined by an expert panel consisting of 2 medical specialists (oncology and radiation oncology) and 2 surgical specialists. The 4 raters independently evaluated all websites belonging to the 5 treatment modalities. Website identifiers were removed to allow for blind assessment. The raters were instructed to evaluate the accuracy of the websites with regard to describing the therapy, the eligible population, the goals of therapy, and the adverse effects associated with therapy. Raters were instructed to identify accurate and inaccurate statements in each website, and to assign the level of accuracy of the website based on the proportion of accurate statements made in the text vs the total number of statements. They were encouraged to document their observations and opinions with regard to each website. These observations were treated independently, along with the remainder of the data acquired. Each expert independently scored the 50 websites (eAppendix in the Supplement) using the following 5-point scoring system adapted from Dy et al²⁵: a score of 1 indicates that less than 25% of the information is accurate, a score of 2 indicates that 26% to 50% of the information is accurate, a score of 3 indicates that 51% to 75% of the information is accurate, a score of 4 indicates that 76% to 99% of the information is accurate, and a score of 5 indicates that 100% of the information is accurate. The maximum score of 5 was only granted when a website contained fully accurate information.

Scores were analyzed for interrater correlation for absolute agreement. Accuracy was treated as an ordinal variable. Comparisons were made among treatment modality and website affiliation using this 5-point scoring system.

Data were treated as nonparametric. Readability level was treated as a continuous variable and analyzed by use of the Kruskal-Wallis test. Accuracy level was treated as an ordinal variable and analyzed with the Kruskal-Wallis test. Pairwise comparisons were performed using the Wilcoxon rank sum test for readability and accuracy level. Statistical significance was set at P ≤ .05 for the main comparisons, and the Bonferroni correction was applied for pairwise comparisons with statistical significance set at P ≤ .005. Small Stata 13 software (StataCorp) was used for all analysis.

All 5 treatment modalities had median readability levels higher than the recommended sixth-grade reading level. Readability level ranged widely between 3.8 and 19.0 (Table 2). The median readability level was statistically different among the 5 treatment modalities (14.2 for alternative therapies, 14.5 for chemotherapy, 15.2 for clinical trials and radiotherapy, and 13.7 for surgery [P = .03]). Websites discussing surgery (with a median readability level of 13.7 and an interquartile range [IQR] of 11.9-15.6) were easier to read than websites discussing clinical trials (median readability level, 15.2 [IQR, 12.8-17.0]) (P = .002) and radiotherapy (median readability level, 15.2 [IQR, 13.0-17.0]) (P = .003) (Table 2). No statistically significant differences were found among other categories.

Interrater correlation for absolute agreement was 0.836 (95% CI, 0.746-0.899). Median accuracy scores ranged between 2 and 5, and were statistically different among treatment groups (P < .001) (Table 3). Websites on alternative therapies had the lowest accuracy (median accuracy score, 2 [IQR, 1-4]), reaching statistical significance when compared with websites on chemotherapy (median accuracy score, 4 [IQR, 3-5]) (P < .001), clinical trials (median accuracy score, 5 [IQR, 4-5]) (P < .001), and radiotherapy (median accuracy score, 4 [IQR, 4-5]) (P < .001). Websites on clinical trials were found to be more accurate than websites on chemotherapy (P = .003) and surgery (median accuracy score, 4 [IQR, 3-4]) (P < .001). A sensitivity analysis weighting the 2 surgeons’ accuracy ratings was performed owing to concerns of bias when rating websites of different modalities. Reported findings remained significant (P < .001).

According to the raters’ observations accrued via the methods already described, there were concerns about websites by treatment modality. Several alternative therapy websites described topics using anecdotes and suggested therapies and imaging techniques that were not standard practice as if they were standard practice. Several radiotherapy and chemotherapy websites contained inaccuracies in their descriptions of the risks and adverse effects of therapy. Several radiotherapy websites omitted information considered by the experts to be fundamental to the description of the treatment, timing/indication for therapy, and/or treatment goals. Some of the chemotherapy websites provided outdated information, and some of surgery websites used outdated descriptions of the surgical intervention.

Websites with an accuracy score of 5 had the highest median readability (16 [IQR, 14-17]). The readability levels of websites having accuracy scores of 4, 3, 2, and 1 were 14.4 (IQR, 13.0-16.1), 14.25 (IQR, 13.3-15.6), 14.9 (IQR, 13.3-16.7), and 14 (IQR, 14-16), respectively. The readability levels were different among the websites with different accuracy scores (P = .01). Significant differences in reading ease were found when websites with an accuracy score of 5 were compared with those with an accuracy score of 1 (P = .003), 3 (P = .003), and 4 (P = .002).

Median readability levels are shown in Table 4. There was a statistically significant difference in readability levels among the affiliation categories (P < .001). Websites owned by nonprofit organizations (median readability level, 12.9 [IQR, 11.2-15.0]) were easier to read than websites owned by the media (median readability level, 16.0 [IQR, 13.4-17.0]) and academic hospitals (median readability level, 14.8 [IQR, 12.9-17.0]) (both P < .001). Websites owned by the media were more difficult to read than websites owned by private holders (median readability level, 14.0 [IQR, 12.1-16.1]) (P = .001).

There were statistical differences in the median accuracy scores among websites by affiliation (P < .001) (Table 5). Nonprofit (4 [IQR, 4-5]), academic (4 [IQR, 3.5-5]), and government websites (5 [IQR, 4-5]) were more accurate than media (4 [IQR, 2-4]) and privately owned websites (3.5 [IQR, 1.5-4]) (P = .004).

No association was found between website affiliation and the treatment modality discussed (P = .18).

Internet use has increased dramatically in the past 2 decades. Over 80% of the US population uses the Internet,²⁶ including 53% of people older than 65 years of age.²⁷ Forty percent of Americans use the Internet to search for health information.²⁸

Online health resources have the potential to become powerful tools to help patients become better informed and to enable them to have meaningful discussions with health care professionals.^5,6,14,29 However, unguided health-related searches by patients may have a negative effect on patient care, decision making, and outcomes.^{3,5-8,19,20,30,31}

A recent report⁴ found that only 55% of those who searched the Internet for health inquiries ultimately pursued professional health care because of financial limitations. Among those who search the Internet, 92% believe that health searches empower them to make health care decisions, and 70% are strongly influenced by the Internet when choosing a course of treatment.^4,7,18-20 However, half of surveyed oncologists report that Internet searches have a negative effect on their patients owing to the inaccuracy of content and information overload.^20,30

According to the 2014 population-level survey data, 99% of the US population has completed 5 to 6 years of formal education.³² The American Medical Association therefore recommends a sixth-grade reading level for patient-oriented education materials targeting the general population. Within our sample, the median readability level of all website categories was higher than recommended, requiring at least 13 years of education to be comprehended. Only 58% of the adult US population has attained this level of education (Table 1). These data indicate that online information on pancreatic cancer is geared to more educated groups. The general population and vulnerable groups with particularly low health literacy will likely struggle to understand this information. This is consistent with Friedman et al,³³ who studied online resources on colon, breast, and prostate cancer. Like Friedman et al,³³ we found that government-affiliated websites were written at a collegiate reading level, targeting the most educated 29% of the population (Table 1). Unlike Friedman et al,³³ we found university-based websites were harder to read; they exhibited the second highest median readability level after media-owned websites.

Interestingly, this study found that accuracy was associated with a difficult reading level. This illustrates one of the challenges incurred in the creation of accurate, yet understandable online information about a complex disease and its treatment options. This limitation may be at least in part overcome by conveying information in visual forms, which are known to increase attention, improve comprehension, and encourage searching for more information, especially among patients with low literacy skills.^34,35 The design of educational materials of the highest accuracy should involve the continuous evaluation of the readability level to ensure that patients’ comprehension will not be compromised. When there are known high readability levels, visual aids should be used to aid the reader.

Overall, websites discussing treatment modalities for pancreatic cancer were reasonably accurate, except for those discussing alternative therapies. Alternative therapy websites contained more noticeable inaccuracies and contained seemingly incorrect information, as reported by the raters. Low accuracy in online resources discussing alternative therapies has been documented before.¹ Previous reports have shown the association of the domain “.org” with more relevant and unbiased information.¹ This domain has been reported to acknowledge areas of uncertainty, address treatment choices more thoroughly, and provide more support for shared decision making compared with the “.com,” “.edu,” and “.gov” domains.³³ Although the acquisition of an “.org” domain is not restricted, such domains are mostly used by nonprofit organizations, whereas the “.com” domain is preferred by private companies; the “.gov” and “.edu” domains are restricted to government entities and postsecondary institutions, respectively. In our sample, nonprofit, government, and academic websites were found to be largely accurate. Martin-Facklam et al³⁶ reported that the absence of a financial interest and the inclusion of citations for scientific facts were associated with higher accuracy. Websites in these categories (ie, nonprofit organizations, government, and academic) may be more likely to comply with an expected educational goal and to be patient-centered.⁷ On the other hand, websites of private holders and communication media may be more diverse in terms of quality. For example, it has been reported that media resources attract the reader’s attention by focusing on controversy, which created confusion among readers.³⁷ Some websites may be genuinely committed to the patient’s well-being, whereas others will present biased information influenced by commercial or audience interests.

Notably, none of the 10 websites discussing alternative therapies had an academic or government affiliation. Four of the websites were privately owned, 4 belonged to the media, and 2 were for nonprofit organizations. Conversely, the websites discussing clinical trials (7 websites) and radiotherapy (5 websites) were academic websites of high accuracy. This implies that accurate sources of online information tend to overlook the role of alternative therapies in pancreatic cancer. Consequently, the information gap is dominated by websites of lower accuracy. Government and academic websites should be comprehensive in providing accurate information about all treatment modalities.

Patients may benefit from recommendations by physicians of websites pertaining to their disease. A survey of 330 patients showed that 62.1% of respondents believe their physicians should recommend specific websites to learn more about their health condition.³⁸ In particular, patients who are most vulnerable to poor cancer outcomes (ie, those with low socioeconomic status, low education levels, and weak support networks) might be most likely to benefit from guidance in the selection of sources of alternate information. Online resources could represent a powerful instrument for health care professionals to strengthen patient-physician relationships and to stimulate active patient participation in the decision-making process.

There are several limitations to this study. Readability level does not assess the role of previous knowledge, motivation to learn, illustrations, use of active voice, use of examples, and organization of text in relation to comprehension.^24,39 In addition, standard readability tests may be inherently limited by the challenge of analyzing complex medical conditions with inherently longer words. This may lead to higher reading level results regardless of the effort to write at an accessible reading level. Future studies are need to evaluate how reading ease, assessed through widely used readability tests, correlates with subjective patient comprehension of complex diseases such as pancreatic cancer.

Based on the modern multidisciplinary approach to the treatment of pancreatic cancer, we assumed that the 4 raters were experts in evaluating the online educational materials discussing the 5 treatment options. When evaluating accuracy, they compared information with the best available evidence on current practice. To reduce subjectivity in scoring, we determined that websites should be 100% accurate; if inaccuracies were present, the raters scored the websites in terms of the proportion of the inaccuracies contained. The adaptation of printed materials for scoring systems would not have been ideal. We are unaware of the existence of validated scoring systems to evaluate accuracy of patient-targeted online resources addressing therapy options; however, we believe that our scoring system was adapted to answer our primary research question. Our study did not analyze “completeness” as a distinct entity. In Internet resources, the evaluation of completeness is questionable given that websites may be designed to address one specific topic, in particular, and still be accurate, although not complete.^2,18 We did not assess the impact of graphic tools or diagrams used to complement and summarize information. The number of websites evaluated is small but was selected to match Internet use practices.

Finally, non-English websites were excluded from this study. Owing to the growing number of non-English–speaking patients, studying and improving online educational materials in other languages would be important. Despite these limitations, this is the first study to our knowledge to evaluate the readability and accuracy of online resources discussing pancreatic cancer treatment.

Two conclusions arise from this study. We demonstrated that online information on pancreatic cancer overestimates the reading ability of the overall population. This may have its greatest impact among those with low literacy and limited access to health care. We also documented appreciable differences among websites discussing treatment modalities and among website affiliations. Nonprofit, academic, and government websites had the highest accuracy, particularly websites relating to clinical trials and radiotherapy. In the absence of an Internet librarian, health care professionals should acknowledge that online information on aggressive diseases such as pancreatic cancer could be misleading and potentially harmful, and, thus, they should assume an active role in the evaluation and recommendation of online resources, selecting readable and accurate online resources for their patients, as an instrument to empower patients in the shared decision-making process.

Accepted for Publication: March 2, 2016.

Corresponding Author: Tara S. Kent, MD, Pancreas and Liver Institute, Department of General Surgery, Beth Israel Deaconess Medical Center, 330 Brookline Ave, Stoneman Bldg, 9th Floor, Boston, MA 02215-5400 (tkent@bidmc.harvard.edu).

Published Online: May 4, 2016. doi:10.1001/jamasurg.2016.0730.

Author Contributions: Drs Kent and Storino had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Storino, Castillo-Angeles, Watkins, Vargas, Kent.

Acquisition, analysis, or interpretation of data: Storino, Castillo-Angeles, Vargas, Mancias, Bullock, Demirjian, Moser, Kent.

Drafting of the manuscript: Storino, Watkins, Moser, Kent.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Storino, Vargas.

Obtained funding: Storino, Watkins, Kent.

Administrative, technical, or material support: Castillo-Angeles, Watkins, Vargas, Demirjian, Moser, Kent.

Study supervision: Watkins, Vargas, Mancias, Demirjian, Moser, Kent.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study is supported by the Alliance of Families Fighting Pancreatic Cancer and the Griffith Family Foundation.

Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The opinions expressed are those of the authors and not necessarily those of the Alliance of Families Fighting Pancreatic Cancer.

Previous Presentation: This paper was presented at the Annual Meeting of the Americas Hepato-Pancreato-Biliary Association; March 14, 2015; Miami Beach, Florida.