[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Table 1.  
Data on Readability Levels and Education Equivalency
Data on Readability Levels and Education Equivalency
Table 2.  
Readability Level of Online Resources by Treatment Modality
Readability Level of Online Resources by Treatment Modality
Table 3.  
Accuracy of Online Resources by Treatment Modality
Accuracy of Online Resources by Treatment Modality
Table 4.  
Readability Level by Website Affiliation
Readability Level by Website Affiliation
Table 5.  
Accuracy Scores by Website Affiliation
Accuracy Scores by Website Affiliation
1.
Scarton  LA, Del Fiol  G, Treitler-Zeng  Q.  Completeness, accuracy, and presentation of information on interactions between prescription drugs and alternative medicines: an internet review.  Stud Health Technol Inform. 2013;192:841-845.PubMed
2.
Eysenbach  G, Powell  J, Kuss  O, Sa  ER.  Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review.  JAMA. 2002;287(20):2691-2700.PubMedArticle
3.
Weymann  N, Härter  M, Dirmaier  J.  Quality of online information on type 2 diabetes: a cross-sectional study.  Health Promot Int. 2015;30(4):821-831.PubMedArticle
4.
Choi  N.  Relationship between health service use and health information technology use among older adults: analysis of the US National Health Interview Survey.  J Med Internet Res. 2011;13(2):e33.PubMedArticle
5.
Mathur  S, Shanti  N, Brkaric  M,  et al.  Surfing for scoliosis: the quality of information available on the Internet.  Spine (Phila Pa 1976). 2005;30(23):2695-2700.PubMedArticle
6.
Tian  C, Champlin  S, Mackert  M, Lazard  A, Agrawal  D.  Readability, suitability, and health content assessment of web-based patient education materials on colorectal cancer screening.  Gastrointest Endosc. 2014;80(2):284-290.PubMedArticle
7.
Bailey  MA, Coughlin  PA, Sohrabi  S,  et al.  Quality and readability of online patient information for abdominal aortic aneurysms.  J Vasc Surg. 2012;56(1):21-26.PubMedArticle
8.
Shon  J, Musen  MA.  The low availability of metadata elements for evaluating the quality of medical information on the World Wide Web.  Proc AMIA Symp. 1999:945-949. PubMed
9.
Burgard  SA, Hawkins  JM.  Race/Ethnicity, educational attainment, and foregone health care in the United States in the 2007-2009 recession.  Am J Public Health. 2014;104(2):e134-e140.PubMedArticle
10.
Bilimoria  KY, Bentrem  DJ, Ko  CY, Stewart  AK, Winchester  DP, Talamonti  MS.  National failure to operate on early stage pancreatic cancer.  Ann Surg. 2007;246(2):173-180.PubMedArticle
11.
Mishoe  SC.  Consumer health care information on the internet: does the public benefit?  Respir Care. 2008;53(10):1285-1286.PubMed
12.
Weiss  B.  Health Literacy and Patient Safety: Help Patients Understand. 2nd ed. Chicago, IL: American Medical Association Foundation/American Medical Association; 2007.
13.
Sand-Jecklin  K.  The impact of medical terminology on readability of patient education materials.  J Community Health Nurs. 2007;24(2):119-129.PubMedArticle
14.
Weih  M, Reinhold  A, Richter-Schmidinger  T, Sulimma  AK, Klein  H, Kornhuber  J.  Unsuitable readability levels of patient information pertaining to dementia and related diseases: a comparative analysis.  Int Psychogeriatr. 2008;20(6):1116-1123. PubMedArticle
15.
Weintraub  D, Maliski  SL, Fink  A, Choe  S, Litwin  MS.  Suitability of prostate cancer education materials: applying a standardized assessment tool to currently available materials.  Patient Educ Couns. 2004;55(2):275-280.PubMedArticle
16.
Baker  DW, Wolf  MS, Feinglass  J, Thompson  JA, Gazmararian  JA, Huang  J.  Health literacy and mortality among elderly persons.  Arch Intern Med. 2007;167(14):1503-1509.PubMedArticle
17.
Busch  EL, Martin  C, DeWalt  DA, Sandler  RS.  Functional health literacy, chemotherapy decisions, and outcomes among a colorectal cancer cohort.  Cancer Control. 2015;22(1):95-101. PubMed
18.
Eysenbach  G.  The impact of the Internet on cancer outcomes.  CA Cancer J Clin. 2003;53(6):356-371.PubMedArticle
19.
Sullivan  TB, Anderson  JT, Ahn  UM, Ahn  NU.  Can Internet information on vertebroplasty be a reliable means of patient self-education?  Clin Orthop Relat Res. 2014;472(5):1597-1604.PubMedArticle
20.
Helft  PR, Hlubocky  F, Daugherty  CK.  American oncologists’ views of internet use by cancer patients: a mail survey of American Society of Clinical Oncology members.  J Clin Oncol. 2003;21(5):942-947.PubMedArticle
21.
Competitive intelligence. Alexa website. http://www.alexa.com/siteinfo/google.com. Accessed April 5, 2016.
22.
WHOis.net Lookup. WHOis.net website. https://www.whois.net/. Accessed September 15, 2015.
23.
Walling  AM, Maliski  S, Bogorad  A, Litwin  MS.  Assessment of content completeness and accuracy of prostate cancer patient education materials.  Patient Educ Couns. 2004;54(3):337-343.PubMedArticle
24.
Mumford  ME.  A descriptive study of the readability of patient information leaflets designed by nurses.  J Adv Nurs. 1997;26(5):985-991.PubMedArticle
25.
Dy  CJ, Taylor  SA, Patel  RM, McCarthy  MM, Roberts  TR, Daluiski  A.  Does the quality, accuracy, and readability of information about lateral epicondylitis on the internet vary with the search term used?  Hand (N Y). 2012;7(4):420-425.PubMedArticle
26.
Internet users (per 100 people). The World Bank website. http://data.worldbank.org/indicator/IT.NET.USER.P2. Accessed January 30, 2015.
27.
Internet use associated with better decision making in older adults. HealthCetera website. http://www.healthmediapolicy.com/2013/11/25/internet-use-associated-with-better-decision-making-in-older-adults/. Published November 25, 2013. Accessed April 5, 2016.
28.
Tsai  J, Rosenheck  RA.  Use of the internet and an online personal health record system by US veterans: comparison of Veterans Affairs mental health service users and other veterans nationally.  J Am Med Inform Assoc. 2012;19(6):1089-1094.PubMedArticle
29.
Magnezi  R, Grosberg  D, Novikov  I, Ziv  A, Shani  M, Freedman  LS.  Characteristics of patients seeking health information online via social health networks versus general Internet sites: a comparative study.  Inform Health Soc Care. 2015;40(2):125-138.PubMedArticle
30.
Joseph-Williams  N, Elwyn  G, Edwards  A.  Knowledge is not power for patients: a systematic review and thematic synthesis of patient-reported barriers and facilitators to shared decision making.  Patient Educ Couns. 2014;94(3):291-309.PubMedArticle
31.
McKearney  TC, McKearney  RM.  The quality and accuracy of internet information on the subject of ear tubes.  Int J Pediatr Otorhinolaryngol. 2013;77(6):894-897.PubMedArticle
32.
Current Population Survey Data on Educational Attainment. CPS 2015. US Census Bureau website. https://www.census.gov/hhes/socdemo/education/data/cps/index.html. Accessed April 5, 2016.
33.
Friedman  DB, Hoffman-Goetz  L, Arocha  JF.  Health literacy and the World Wide Web: comparing the readability of leading incident cancers on the Internet.  Med Inform Internet Med. 2006;31(1):67-87.PubMedArticle
34.
Houts  PS, Doak  CC, Doak  LG, Loscalzo  MJ.  The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence.  Patient Educ Couns. 2006;61(2):173-190.PubMedArticle
35.
Shaffer  VA, Owens  J, Zikmund-Fisher  BJ.  The effect of patient narratives on information search in a web-based breast cancer decision aid: an eye-tracking study.  J Med Internet Res. 2013;15(12):e273.PubMedArticle
36.
Martin-Facklam  M, Kostrzewa  M, Schubert  F, Gasse  C, Haefeli  WE.  Quality markers of drug information on the Internet: an evaluation of sites about St. John’s wort.  Am J Med. 2002;113(9):740-745.PubMedArticle
37.
Weeks  BE, Friedenberg  LM, Southwell  BG, Slater  JS.  Behavioral consequences of conflict-oriented health news coverage: the 2009 mammography guideline controversy and online information seeking.  Health Commun. 2012;27(2):158-166.PubMedArticle
38.
Centers for Disease Control and Prevention (CDC). Simply put: a guide for creating easy-to-understand materials. Atlanta, GA: US Dept of Health and Human Services/CDC; 2009.
39.
Davis  TC, Arnold  C, Berkel  HJ, Nandy  I, Jackson  RH, Glass  J.  Knowledge and attitude on screening mammography among low-literate, low-income women.  Cancer. 1996;78(9):1912-1920.PubMedArticle
Views 649
Citations 0
Original Investigation
September 2016

Assessing the Accuracy and Readability of Online Health Information for Patients With Pancreatic Cancer

Author Affiliations
  • 1Pancreas and Liver Institute, Department of General Surgery, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, Massachusetts
  • 2Division of Hepatobiliary and Pancreas Surgery, University of California, Irvine Medical Center, Orange
JAMA Surg. 2016;151(9):831-837. doi:10.1001/jamasurg.2016.0730
Abstract

Importance  The degree to which patients are empowered by written educational materials depends on the text’s readability level and the accuracy of the information provided. The association of a website’s affiliation or focus on treatment modality with its readability and accuracy has yet to be thoroughly elucidated.

Objective  To compare the readability and accuracy of patient-oriented online resources for pancreatic cancer by treatment modality and website affiliation.

Design  An online search of 50 websites discussing 5 pancreatic cancer treatment modalities (alternative therapy, chemotherapy, clinical trials, radiation therapy, and surgery) was conducted. The website’s affiliation was identified. Readability was measured by 9 standardized tests, and accuracy was assessed by an expert panel.

Main Outcomes and Measures  Nine standardized tests were used to compute the median readability level of each website. The median readability scores were compared among treatment modality and affiliation categories. Accuracy was determined by an expert panel consisting of 2 medical specialists and 2 surgical specialists. The 4 raters independently evaluated all websites belonging to the 5 treatment modalities (a score of 1 indicates that <25% of the information is accurate, a score of 2 indicates that 26%-50% of the information is accurate, a score of 3 indicates that 51%-75% of the information is accurate, a score of 4 indicates that 76%-99% of the information is accurate, and a score of 5 indicates that 100% of the information is accurate).

Results  The 50 evaluated websites differed in readability and accuracy based on the focus of the treatment modality and the website’s affiliation. Websites discussing surgery (with a median readability level of 13.7 and an interquartile range [IQR] of 11.9-15.6) were easier to read than those discussing radiotherapy (median readability level, 15.2 [IQR, 13.0-17.0]) (P = .003) and clinical trials (median readability level, 15.2 [IQR, 12.8-17.0]) (P = .002). Websites of nonprofit organizations (median readability level, 12.9 [IQR, 11.2-15.0]) were easier to read than media (median readability level, 16.0 [IQR, 13.4-17.0]) (P < .001) and academic (median readability level, 14.8 [IQR, 12.9-17.0]) (P < .001) websites. Privately owned websites (median readability level, 14.0 [IQR, 12.1-16.1]) were easier to read than media websites (P = .001). Among treatment modalities, alternative therapy websites exhibited the lowest accuracy scores (median accuracy score, 2 [IQR, 1-4]) (P < .001). Nonprofit (median accuracy score, 4 [IQR, 4-5]), government (median accuracy score, 5 [IQR, 4-5]), and academic (median accuracy score, 4 [IQR, 3.5-5]) websites were more accurate than privately owned (median accuracy score, 3.5 [IQR, 1.5-4]) and media (median accuracy score, 4 [IQR, 2-4]) websites (P < .004). Websites with higher accuracy were more difficult to read than websites with lower accuracy.

Conclusions and Relevance  Online information on pancreatic cancer overestimates the reading ability of the overall population and lacks accurate information about alternative therapy. In the absence of quality control on the Internet, physicians should provide guidance to patients in the selection of online resources with readable and accurate information.

Introduction

Patients increasingly use the Internet to learn more about diseases and treatment options.14 Online medical information is accessible and relatively inexpensive and empowers patients to make decisions.5,6 It is often used in lieu of formal medical consultations, particularly among patients with financial limitations.4,79 However, online materials may be too difficult for the average patient to comprehend.1,6 Furthermore, the accuracy of online information can vary widely. Online information may be biased, motivated by underlying commercial interests, taken out of context, or incomplete. This has the potential to lead to unfavorable downstream effects. Particularly for complex and aggressive diseases such as pancreatic cancer, misinformation or perceptions of dismal outcomes can hinder patients from seeking appropriate care.10

Compounding this problem is the fact that reading materials are rarely written at the recommended sixth-grade reading level.1115 A patient’s health literacy or ability to comprehend written health information can impact clinical outcomes. Elderly patients with low health literacy are at greater risk of all-cause mortality.16 Patients with low health literacy and advanced colorectal cancer are also less likely to receive standard-of-care chemotherapy.17

It has been reported that patients believe searching the Internet empowers them to make decisions about their health.4,7,1820 However, it has been found that online information varies widely in terms of quality.2 Patients must have accurate and understandable information to participate in the shared decision-making process.14,15 It is important to study whether available online health information is accurate and written at an appropriate reading level. This study assesses the readability, a surrogate for reading level, and accuracy of websites discussing various treatment options for pancreatic cancer.

Box Section Ref ID

Key Points

  • Question Are there differences in the accuracy and readability of online resources discussing pancreatic cancer?

  • Findings In this study of 50 websites on pancreatic cancer treatment options, all of the websites overestimated the reading ability of the intended population. Most of the websites, except for those discussing alternative therapies for pancreatic cancer, were accurate. Nonprofit, academic, and government websites presented more accurate information than media and privately owned websites.

  • Meaning Physician guidance is critical in the selection of appropriate online resources to empower patients’ decision making through receipt of readable and accurate information.

Methods
Data Collection

This study was approved by the institutional review board at the Dana-Farber/Harvard Cancer Center in Boston, Massachusetts. Online searches using Google, the search engine most commonly used in the United States and throughout the world,21 were limited to 1 location (the United States) and 1 language (English). Data were collected during May 2014 using the following search terms, which are also the 5 treatment modalities: (1) pancreas cancer alternative therapies, (2) pancreas cancer chemotherapy, (3) pancreas cancer clinical trials, (4) pancreas cancer radiotherapy, and (5) pancreas cancer surgery. Websites had to meet the following eligibility criteria: they were designed for public consumption and not for health care professionals; they were specific to pancreatic cancer treatment, at least 100 words in length, and written in English. Websites meeting the following criteria were excluded: general newspaper reports that did not discuss treatment information, access restricted by subscriptions or fees, duplicate websites, and websites presenting videos only. Websites were classified into 5 affiliation categories using the following method: (1) websites with “.org” domains that were not affiliated with academic hospitals were treated as nonprofit organizations; (2) websites with “.edu” domains were treated as academic websites; (3) websites with “.gov” domains were treated as government websites; (4) websites with news portals were treated as communication media websites; and (5) websites that did not disclose affiliation or that disclosed affiliation to a private holder were treated as private. Affiliation was confirmed by use of the WHOis.net database.22

Analysis and Definition of Terms

The first 10 eligible websites appearing for each treatment category were analyzed for accuracy and readability. Accuracy was defined as the degree of concordance of the information provided with the best evidence or with generally accepted medical practice.2 This includes providing critical facts to describe a topic and the absence of erroneous or misleading information.23 Readability level was defined as the number of years of education required to comprehend written information. It is assessed based on word and sentence length as surrogates of text complexity.24 The evaluation of completeness is questionable among Internet resources. Websites are often purposely designed to address 1 specific topic in particular. This allows for accuracy, but not completeness, to be assessed.2,18,23 For these reasons, completeness was not evaluated in this study.

Assessment of Readability

Readability was analyzed using the Readability Studio Professional Edition version 2012.1 software (Oleander Software Ltd). Nine standardized tests were used: the Coleman-Liau Index, the Flesch-Kincaid Grade Level, the FORCAST Readability Formula, the Fry Readability Graph, the Gunning Fog Index, the New Dale-Chall Readability Formula, the New Fog Count, the Raygor Readability Estimate, and the Simple Measure of Gobbledygook Readability Formula.24 The use of multiple formulae has been found to improve the reliability and accuracy of readability scores.24 The 9 tests were used to compute the median readability level of each website. The median readability scores were compared among treatment modality and affiliation categories. Table 1 shows the education equivalence of readability levels.

Assessment of Accuracy

Accuracy was determined by an expert panel consisting of 2 medical specialists (oncology and radiation oncology) and 2 surgical specialists. The 4 raters independently evaluated all websites belonging to the 5 treatment modalities. Website identifiers were removed to allow for blind assessment. The raters were instructed to evaluate the accuracy of the websites with regard to describing the therapy, the eligible population, the goals of therapy, and the adverse effects associated with therapy. Raters were instructed to identify accurate and inaccurate statements in each website, and to assign the level of accuracy of the website based on the proportion of accurate statements made in the text vs the total number of statements. They were encouraged to document their observations and opinions with regard to each website. These observations were treated independently, along with the remainder of the data acquired. Each expert independently scored the 50 websites (eAppendix in the Supplement) using the following 5-point scoring system adapted from Dy et al25: a score of 1 indicates that less than 25% of the information is accurate, a score of 2 indicates that 26% to 50% of the information is accurate, a score of 3 indicates that 51% to 75% of the information is accurate, a score of 4 indicates that 76% to 99% of the information is accurate, and a score of 5 indicates that 100% of the information is accurate. The maximum score of 5 was only granted when a website contained fully accurate information.

Scores were analyzed for interrater correlation for absolute agreement. Accuracy was treated as an ordinal variable. Comparisons were made among treatment modality and website affiliation using this 5-point scoring system.

Statistical Analysis

Data were treated as nonparametric. Readability level was treated as a continuous variable and analyzed by use of the Kruskal-Wallis test. Accuracy level was treated as an ordinal variable and analyzed with the Kruskal-Wallis test. Pairwise comparisons were performed using the Wilcoxon rank sum test for readability and accuracy level. Statistical significance was set at P ≤ .05 for the main comparisons, and the Bonferroni correction was applied for pairwise comparisons with statistical significance set at P ≤ .005. Small Stata 13 software (StataCorp) was used for all analysis.

Results
Readability and Treatment Modality

All 5 treatment modalities had median readability levels higher than the recommended sixth-grade reading level. Readability level ranged widely between 3.8 and 19.0 (Table 2). The median readability level was statistically different among the 5 treatment modalities (14.2 for alternative therapies, 14.5 for chemotherapy, 15.2 for clinical trials and radiotherapy, and 13.7 for surgery [P = .03]). Websites discussing surgery (with a median readability level of 13.7 and an interquartile range [IQR] of 11.9-15.6) were easier to read than websites discussing clinical trials (median readability level, 15.2 [IQR, 12.8-17.0]) (P = .002) and radiotherapy (median readability level, 15.2 [IQR, 13.0-17.0]) (P = .003) (Table 2). No statistically significant differences were found among other categories.

Accuracy and Treatment Modality

Interrater correlation for absolute agreement was 0.836 (95% CI, 0.746-0.899). Median accuracy scores ranged between 2 and 5, and were statistically different among treatment groups (P < .001) (Table 3). Websites on alternative therapies had the lowest accuracy (median accuracy score, 2 [IQR, 1-4]), reaching statistical significance when compared with websites on chemotherapy (median accuracy score, 4 [IQR, 3-5]) (P < .001), clinical trials (median accuracy score, 5 [IQR, 4-5]) (P < .001), and radiotherapy (median accuracy score, 4 [IQR, 4-5]) (P < .001). Websites on clinical trials were found to be more accurate than websites on chemotherapy (P = .003) and surgery (median accuracy score, 4 [IQR, 3-4]) (P < .001). A sensitivity analysis weighting the 2 surgeons’ accuracy ratings was performed owing to concerns of bias when rating websites of different modalities. Reported findings remained significant (P < .001).

According to the raters’ observations accrued via the methods already described, there were concerns about websites by treatment modality. Several alternative therapy websites described topics using anecdotes and suggested therapies and imaging techniques that were not standard practice as if they were standard practice. Several radiotherapy and chemotherapy websites contained inaccuracies in their descriptions of the risks and adverse effects of therapy. Several radiotherapy websites omitted information considered by the experts to be fundamental to the description of the treatment, timing/indication for therapy, and/or treatment goals. Some of the chemotherapy websites provided outdated information, and some of surgery websites used outdated descriptions of the surgical intervention.

Readability Level and Accuracy Score

Websites with an accuracy score of 5 had the highest median readability (16 [IQR, 14-17]). The readability levels of websites having accuracy scores of 4, 3, 2, and 1 were 14.4 (IQR, 13.0-16.1), 14.25 (IQR, 13.3-15.6), 14.9 (IQR, 13.3-16.7), and 14 (IQR, 14-16), respectively. The readability levels were different among the websites with different accuracy scores (P = .01). Significant differences in reading ease were found when websites with an accuracy score of 5 were compared with those with an accuracy score of 1 (P = .003), 3 (P = .003), and 4 (P = .002).

Readability Level and Website Affiliation

Median readability levels are shown in Table 4. There was a statistically significant difference in readability levels among the affiliation categories (P < .001). Websites owned by nonprofit organizations (median readability level, 12.9 [IQR, 11.2-15.0]) were easier to read than websites owned by the media (median readability level, 16.0 [IQR, 13.4-17.0]) and academic hospitals (median readability level, 14.8 [IQR, 12.9-17.0]) (both P < .001). Websites owned by the media were more difficult to read than websites owned by private holders (median readability level, 14.0 [IQR, 12.1-16.1]) (P = .001).

Accuracy and Website Affiliation

There were statistical differences in the median accuracy scores among websites by affiliation (P < .001) (Table 5). Nonprofit (4 [IQR, 4-5]), academic (4 [IQR, 3.5-5]), and government websites (5 [IQR, 4-5]) were more accurate than media (4 [IQR, 2-4]) and privately owned websites (3.5 [IQR, 1.5-4]) (P = .004).

Treatment Modality and Website Affiliation

No association was found between website affiliation and the treatment modality discussed (P = .18).

Discussion

Internet use has increased dramatically in the past 2 decades. Over 80% of the US population uses the Internet,26 including 53% of people older than 65 years of age.27 Forty percent of Americans use the Internet to search for health information.28

Online health resources have the potential to become powerful tools to help patients become better informed and to enable them to have meaningful discussions with health care professionals.5,6,14,29 However, unguided health-related searches by patients may have a negative effect on patient care, decision making, and outcomes.3,58,19,20,30,31

A recent report4 found that only 55% of those who searched the Internet for health inquiries ultimately pursued professional health care because of financial limitations. Among those who search the Internet, 92% believe that health searches empower them to make health care decisions, and 70% are strongly influenced by the Internet when choosing a course of treatment.4,7,1820 However, half of surveyed oncologists report that Internet searches have a negative effect on their patients owing to the inaccuracy of content and information overload.20,30

According to the 2014 population-level survey data, 99% of the US population has completed 5 to 6 years of formal education.32 The American Medical Association therefore recommends a sixth-grade reading level for patient-oriented education materials targeting the general population. Within our sample, the median readability level of all website categories was higher than recommended, requiring at least 13 years of education to be comprehended. Only 58% of the adult US population has attained this level of education (Table 1). These data indicate that online information on pancreatic cancer is geared to more educated groups. The general population and vulnerable groups with particularly low health literacy will likely struggle to understand this information. This is consistent with Friedman et al,33 who studied online resources on colon, breast, and prostate cancer. Like Friedman et al,33 we found that government-affiliated websites were written at a collegiate reading level, targeting the most educated 29% of the population (Table 1). Unlike Friedman et al,33 we found university-based websites were harder to read; they exhibited the second highest median readability level after media-owned websites.

Interestingly, this study found that accuracy was associated with a difficult reading level. This illustrates one of the challenges incurred in the creation of accurate, yet understandable online information about a complex disease and its treatment options. This limitation may be at least in part overcome by conveying information in visual forms, which are known to increase attention, improve comprehension, and encourage searching for more information, especially among patients with low literacy skills.34,35 The design of educational materials of the highest accuracy should involve the continuous evaluation of the readability level to ensure that patients’ comprehension will not be compromised. When there are known high readability levels, visual aids should be used to aid the reader.

Overall, websites discussing treatment modalities for pancreatic cancer were reasonably accurate, except for those discussing alternative therapies. Alternative therapy websites contained more noticeable inaccuracies and contained seemingly incorrect information, as reported by the raters. Low accuracy in online resources discussing alternative therapies has been documented before.1 Previous reports have shown the association of the domain “.org” with more relevant and unbiased information.1 This domain has been reported to acknowledge areas of uncertainty, address treatment choices more thoroughly, and provide more support for shared decision making compared with the “.com,” “.edu,” and “.gov” domains.33 Although the acquisition of an “.org” domain is not restricted, such domains are mostly used by nonprofit organizations, whereas the “.com” domain is preferred by private companies; the “.gov” and “.edu” domains are restricted to government entities and postsecondary institutions, respectively. In our sample, nonprofit, government, and academic websites were found to be largely accurate. Martin-Facklam et al36 reported that the absence of a financial interest and the inclusion of citations for scientific facts were associated with higher accuracy. Websites in these categories (ie, nonprofit organizations, government, and academic) may be more likely to comply with an expected educational goal and to be patient-centered.7 On the other hand, websites of private holders and communication media may be more diverse in terms of quality. For example, it has been reported that media resources attract the reader’s attention by focusing on controversy, which created confusion among readers.37 Some websites may be genuinely committed to the patient’s well-being, whereas others will present biased information influenced by commercial or audience interests.

Notably, none of the 10 websites discussing alternative therapies had an academic or government affiliation. Four of the websites were privately owned, 4 belonged to the media, and 2 were for nonprofit organizations. Conversely, the websites discussing clinical trials (7 websites) and radiotherapy (5 websites) were academic websites of high accuracy. This implies that accurate sources of online information tend to overlook the role of alternative therapies in pancreatic cancer. Consequently, the information gap is dominated by websites of lower accuracy. Government and academic websites should be comprehensive in providing accurate information about all treatment modalities.

Patients may benefit from recommendations by physicians of websites pertaining to their disease. A survey of 330 patients showed that 62.1% of respondents believe their physicians should recommend specific websites to learn more about their health condition.38 In particular, patients who are most vulnerable to poor cancer outcomes (ie, those with low socioeconomic status, low education levels, and weak support networks) might be most likely to benefit from guidance in the selection of sources of alternate information. Online resources could represent a powerful instrument for health care professionals to strengthen patient-physician relationships and to stimulate active patient participation in the decision-making process.

There are several limitations to this study. Readability level does not assess the role of previous knowledge, motivation to learn, illustrations, use of active voice, use of examples, and organization of text in relation to comprehension.24,39 In addition, standard readability tests may be inherently limited by the challenge of analyzing complex medical conditions with inherently longer words. This may lead to higher reading level results regardless of the effort to write at an accessible reading level. Future studies are need to evaluate how reading ease, assessed through widely used readability tests, correlates with subjective patient comprehension of complex diseases such as pancreatic cancer.

Based on the modern multidisciplinary approach to the treatment of pancreatic cancer, we assumed that the 4 raters were experts in evaluating the online educational materials discussing the 5 treatment options. When evaluating accuracy, they compared information with the best available evidence on current practice. To reduce subjectivity in scoring, we determined that websites should be 100% accurate; if inaccuracies were present, the raters scored the websites in terms of the proportion of the inaccuracies contained. The adaptation of printed materials for scoring systems would not have been ideal. We are unaware of the existence of validated scoring systems to evaluate accuracy of patient-targeted online resources addressing therapy options; however, we believe that our scoring system was adapted to answer our primary research question. Our study did not analyze “completeness” as a distinct entity. In Internet resources, the evaluation of completeness is questionable given that websites may be designed to address one specific topic, in particular, and still be accurate, although not complete.2,18 We did not assess the impact of graphic tools or diagrams used to complement and summarize information. The number of websites evaluated is small but was selected to match Internet use practices.

Finally, non-English websites were excluded from this study. Owing to the growing number of non-English–speaking patients, studying and improving online educational materials in other languages would be important. Despite these limitations, this is the first study to our knowledge to evaluate the readability and accuracy of online resources discussing pancreatic cancer treatment.

Conclusions

Two conclusions arise from this study. We demonstrated that online information on pancreatic cancer overestimates the reading ability of the overall population. This may have its greatest impact among those with low literacy and limited access to health care. We also documented appreciable differences among websites discussing treatment modalities and among website affiliations. Nonprofit, academic, and government websites had the highest accuracy, particularly websites relating to clinical trials and radiotherapy. In the absence of an Internet librarian, health care professionals should acknowledge that online information on aggressive diseases such as pancreatic cancer could be misleading and potentially harmful, and, thus, they should assume an active role in the evaluation and recommendation of online resources, selecting readable and accurate online resources for their patients, as an instrument to empower patients in the shared decision-making process.

Back to top
Article Information

Accepted for Publication: March 2, 2016.

Corresponding Author: Tara S. Kent, MD, Pancreas and Liver Institute, Department of General Surgery, Beth Israel Deaconess Medical Center, 330 Brookline Ave, Stoneman Bldg, 9th Floor, Boston, MA 02215-5400 (tkent@bidmc.harvard.edu).

Published Online: May 4, 2016. doi:10.1001/jamasurg.2016.0730.

Author Contributions: Drs Kent and Storino had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Storino, Castillo-Angeles, Watkins, Vargas, Kent.

Acquisition, analysis, or interpretation of data: Storino, Castillo-Angeles, Vargas, Mancias, Bullock, Demirjian, Moser, Kent.

Drafting of the manuscript: Storino, Watkins, Moser, Kent.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Storino, Vargas.

Obtained funding: Storino, Watkins, Kent.

Administrative, technical, or material support: Castillo-Angeles, Watkins, Vargas, Demirjian, Moser, Kent.

Study supervision: Watkins, Vargas, Mancias, Demirjian, Moser, Kent.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study is supported by the Alliance of Families Fighting Pancreatic Cancer and the Griffith Family Foundation.

Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The opinions expressed are those of the authors and not necessarily those of the Alliance of Families Fighting Pancreatic Cancer.

Previous Presentation: This paper was presented at the Annual Meeting of the Americas Hepato-Pancreato-Biliary Association; March 14, 2015; Miami Beach, Florida.

References
1.
Scarton  LA, Del Fiol  G, Treitler-Zeng  Q.  Completeness, accuracy, and presentation of information on interactions between prescription drugs and alternative medicines: an internet review.  Stud Health Technol Inform. 2013;192:841-845.PubMed
2.
Eysenbach  G, Powell  J, Kuss  O, Sa  ER.  Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review.  JAMA. 2002;287(20):2691-2700.PubMedArticle
3.
Weymann  N, Härter  M, Dirmaier  J.  Quality of online information on type 2 diabetes: a cross-sectional study.  Health Promot Int. 2015;30(4):821-831.PubMedArticle
4.
Choi  N.  Relationship between health service use and health information technology use among older adults: analysis of the US National Health Interview Survey.  J Med Internet Res. 2011;13(2):e33.PubMedArticle
5.
Mathur  S, Shanti  N, Brkaric  M,  et al.  Surfing for scoliosis: the quality of information available on the Internet.  Spine (Phila Pa 1976). 2005;30(23):2695-2700.PubMedArticle
6.
Tian  C, Champlin  S, Mackert  M, Lazard  A, Agrawal  D.  Readability, suitability, and health content assessment of web-based patient education materials on colorectal cancer screening.  Gastrointest Endosc. 2014;80(2):284-290.PubMedArticle
7.
Bailey  MA, Coughlin  PA, Sohrabi  S,  et al.  Quality and readability of online patient information for abdominal aortic aneurysms.  J Vasc Surg. 2012;56(1):21-26.PubMedArticle
8.
Shon  J, Musen  MA.  The low availability of metadata elements for evaluating the quality of medical information on the World Wide Web.  Proc AMIA Symp. 1999:945-949. PubMed
9.
Burgard  SA, Hawkins  JM.  Race/Ethnicity, educational attainment, and foregone health care in the United States in the 2007-2009 recession.  Am J Public Health. 2014;104(2):e134-e140.PubMedArticle
10.
Bilimoria  KY, Bentrem  DJ, Ko  CY, Stewart  AK, Winchester  DP, Talamonti  MS.  National failure to operate on early stage pancreatic cancer.  Ann Surg. 2007;246(2):173-180.PubMedArticle
11.
Mishoe  SC.  Consumer health care information on the internet: does the public benefit?  Respir Care. 2008;53(10):1285-1286.PubMed
12.
Weiss  B.  Health Literacy and Patient Safety: Help Patients Understand. 2nd ed. Chicago, IL: American Medical Association Foundation/American Medical Association; 2007.
13.
Sand-Jecklin  K.  The impact of medical terminology on readability of patient education materials.  J Community Health Nurs. 2007;24(2):119-129.PubMedArticle
14.
Weih  M, Reinhold  A, Richter-Schmidinger  T, Sulimma  AK, Klein  H, Kornhuber  J.  Unsuitable readability levels of patient information pertaining to dementia and related diseases: a comparative analysis.  Int Psychogeriatr. 2008;20(6):1116-1123. PubMedArticle
15.
Weintraub  D, Maliski  SL, Fink  A, Choe  S, Litwin  MS.  Suitability of prostate cancer education materials: applying a standardized assessment tool to currently available materials.  Patient Educ Couns. 2004;55(2):275-280.PubMedArticle
16.
Baker  DW, Wolf  MS, Feinglass  J, Thompson  JA, Gazmararian  JA, Huang  J.  Health literacy and mortality among elderly persons.  Arch Intern Med. 2007;167(14):1503-1509.PubMedArticle
17.
Busch  EL, Martin  C, DeWalt  DA, Sandler  RS.  Functional health literacy, chemotherapy decisions, and outcomes among a colorectal cancer cohort.  Cancer Control. 2015;22(1):95-101. PubMed
18.
Eysenbach  G.  The impact of the Internet on cancer outcomes.  CA Cancer J Clin. 2003;53(6):356-371.PubMedArticle
19.
Sullivan  TB, Anderson  JT, Ahn  UM, Ahn  NU.  Can Internet information on vertebroplasty be a reliable means of patient self-education?  Clin Orthop Relat Res. 2014;472(5):1597-1604.PubMedArticle
20.
Helft  PR, Hlubocky  F, Daugherty  CK.  American oncologists’ views of internet use by cancer patients: a mail survey of American Society of Clinical Oncology members.  J Clin Oncol. 2003;21(5):942-947.PubMedArticle
21.
Competitive intelligence. Alexa website. http://www.alexa.com/siteinfo/google.com. Accessed April 5, 2016.
22.
WHOis.net Lookup. WHOis.net website. https://www.whois.net/. Accessed September 15, 2015.
23.
Walling  AM, Maliski  S, Bogorad  A, Litwin  MS.  Assessment of content completeness and accuracy of prostate cancer patient education materials.  Patient Educ Couns. 2004;54(3):337-343.PubMedArticle
24.
Mumford  ME.  A descriptive study of the readability of patient information leaflets designed by nurses.  J Adv Nurs. 1997;26(5):985-991.PubMedArticle
25.
Dy  CJ, Taylor  SA, Patel  RM, McCarthy  MM, Roberts  TR, Daluiski  A.  Does the quality, accuracy, and readability of information about lateral epicondylitis on the internet vary with the search term used?  Hand (N Y). 2012;7(4):420-425.PubMedArticle
26.
Internet users (per 100 people). The World Bank website. http://data.worldbank.org/indicator/IT.NET.USER.P2. Accessed January 30, 2015.
27.
Internet use associated with better decision making in older adults. HealthCetera website. http://www.healthmediapolicy.com/2013/11/25/internet-use-associated-with-better-decision-making-in-older-adults/. Published November 25, 2013. Accessed April 5, 2016.
28.
Tsai  J, Rosenheck  RA.  Use of the internet and an online personal health record system by US veterans: comparison of Veterans Affairs mental health service users and other veterans nationally.  J Am Med Inform Assoc. 2012;19(6):1089-1094.PubMedArticle
29.
Magnezi  R, Grosberg  D, Novikov  I, Ziv  A, Shani  M, Freedman  LS.  Characteristics of patients seeking health information online via social health networks versus general Internet sites: a comparative study.  Inform Health Soc Care. 2015;40(2):125-138.PubMedArticle
30.
Joseph-Williams  N, Elwyn  G, Edwards  A.  Knowledge is not power for patients: a systematic review and thematic synthesis of patient-reported barriers and facilitators to shared decision making.  Patient Educ Couns. 2014;94(3):291-309.PubMedArticle
31.
McKearney  TC, McKearney  RM.  The quality and accuracy of internet information on the subject of ear tubes.  Int J Pediatr Otorhinolaryngol. 2013;77(6):894-897.PubMedArticle
32.
Current Population Survey Data on Educational Attainment. CPS 2015. US Census Bureau website. https://www.census.gov/hhes/socdemo/education/data/cps/index.html. Accessed April 5, 2016.
33.
Friedman  DB, Hoffman-Goetz  L, Arocha  JF.  Health literacy and the World Wide Web: comparing the readability of leading incident cancers on the Internet.  Med Inform Internet Med. 2006;31(1):67-87.PubMedArticle
34.
Houts  PS, Doak  CC, Doak  LG, Loscalzo  MJ.  The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence.  Patient Educ Couns. 2006;61(2):173-190.PubMedArticle
35.
Shaffer  VA, Owens  J, Zikmund-Fisher  BJ.  The effect of patient narratives on information search in a web-based breast cancer decision aid: an eye-tracking study.  J Med Internet Res. 2013;15(12):e273.PubMedArticle
36.
Martin-Facklam  M, Kostrzewa  M, Schubert  F, Gasse  C, Haefeli  WE.  Quality markers of drug information on the Internet: an evaluation of sites about St. John’s wort.  Am J Med. 2002;113(9):740-745.PubMedArticle
37.
Weeks  BE, Friedenberg  LM, Southwell  BG, Slater  JS.  Behavioral consequences of conflict-oriented health news coverage: the 2009 mammography guideline controversy and online information seeking.  Health Commun. 2012;27(2):158-166.PubMedArticle
38.
Centers for Disease Control and Prevention (CDC). Simply put: a guide for creating easy-to-understand materials. Atlanta, GA: US Dept of Health and Human Services/CDC; 2009.
39.
Davis  TC, Arnold  C, Berkel  HJ, Nandy  I, Jackson  RH, Glass  J.  Knowledge and attitude on screening mammography among low-literate, low-income women.  Cancer. 1996;78(9):1912-1920.PubMedArticle
×