The Readability of Pediatric Patient Education Materials on the World Wide Web

Background  Literacy is a national and international problem. Studies have shown the readability of adult and pediatric patient education materials to be too high for average adults. Materials should be written at the 8th-grade level or lower.

Objective  To determine the general readability of pediatric patient education materials designed for adults on the World Wide Web (WWW).

Materials and Methods  GeneralPediatrics.com (http://www.generalpediatrics.com) is a digital library serving the medical information needs of pediatric health care providers, patients, and families. Documents from 100 different authoritative Web sites designed for laypersons were evaluated using a built-in computer software readability formula (Flesch Reading Ease and Flesch-Kincaid reading levels) and hand calculation methods (Fry Formula and SMOG methods). Analysis of variance and paired t tests determined significance.

Results  Eighty-nine documents constituted the final sample; they covered a wide spectrum of pediatric topics. The overall Flesch Reading Ease score was 57.0. The overall mean Fry Formula was 12.0 (12th grade, 0 months of schooling) and SMOG was 12.2. The overall Flesch-Kincaid grade level was significantly lower (P<.0001), at a mean of 7.1, when compared with the other 2 methods. All author and institution groups had an average reading level above 10.6 by the Fry Formula and SMOG methods.

Conclusions  Pediatric patient education materials on the WWW are not written at an appropriate reading level for the average adult. We propose that a practical reading level and how it was determined be included on all patient education materials on the WWW for general guidance in material selection. We discuss suggestions for improved readability of patient education materials.

LITERACY IS a national and international problem. Recent studies have shown that almost one quarter of the adult population in the United States and Canada are functionally illiterate (reading level below the fifth grade); an additional one quarter have poor reading and comprehension skills. Other countries' statistics are almost as bleak.^1,2

In the United States, the average reading level is eighth to ninth grade. By comparison, the average newspaper is written at the 9th- to 12th-grade level.³ Studies have shown that literacy levels are approximately 3 to 5 grades below the highest grade of schooling completed.⁴ Patient education materials should be aimed at the eighth-grade level or below³; however, most patient education materials are still written at the 10th-grade level or higher.^5-11 Studies of pediatric patient education materials in outpatient clinics, emergency departments, and Women, Infant, and Children program clinics have found that the reading levels were consistently higher than recommended.^2,12-15 A previous study of the readability of general patient education materials on the World Wide Web (WWW) revealed the mean reading level at almost the 10th-grade level.¹⁶

Pediatricians and other pediatric health care providers often use written patient education materials in their practices. Material sources are varied, including those written or modified by pediatricians or their staff members. Therefore, pediatricians not only need to be aware of the materials' accuracy, but also should have some knowledge about the readability of the materials to ensure that accurate information can be comprehended. The goal of this study was to determine the general readability of pediatric patient education materials designed for adults on the WWW.

GeneralPediatrics.com (http://www.generalpediatrics.com) is a digital library serving the medical information needs of pediatric health care providers and patients. As of October 2000, it contained 1994 hyperlinks to 439 authoritative Web sites. The digital library links to pediatric health information on 390 common problems faced by pediatric health care providers and patients. Although the digital library does not provide the health information itself, it links to other authoritative information using the following criteria of displaying the author(s), institution(s), currency of the information, sponsor(s), and references.^17-24 The criteria did not include readability (http://www.generalpediatrics.com/Mission.html). The Common Problems for Patients and Families page (http://www.generalpediatrics.com/CommonProbLay.html) has information on 204 common problems, with 1122 hyperlinks to 236 different Web sites. The linked content is written for adolescents and adults and not for younger children.

A researcher (P.K.) systematically evaluated every 10th link and its corresponding document on the Common Problems for Patients and Families page in March 2000. To obtain the broadest view of the patient education materials on the WWW, only 1 patient education document was evaluated from an individual Web site (eg, 1 document from the American Academy of Pediatrics, 1 document from the Centers for Disease Control and Prevention). If during the systematic evaluation, another link from a previously used Web site was chosen, this link was skipped over and the next 10th link was evaluated. Similarly, if the link led to a document that was not appropriate for this evaluation (eg, a table of contents), then the next 10th link was used instead. One hundred Web sites and their corresponding documents were evaluated. Each document's title and information was copied into Microsoft 98 (Microsoft Corp, Redmond, Wash). The documents' Flesch Reading Ease score and Flesch-Kincaid reading level were generated using the built-in software in Microsoft 98 and entered into a Microsoft Excel spreadsheet. The same documents were printed and readability levels were determined by hand calculation by a second researcher (J.J.-W.) using the Fry Formula and SMOG methods.^25,26 A third researcher (D.M.D.) also hand calculated 10% of the documents and found the reading levels to be accurate. The Fry and SMOG readability levels were entered into the Microsoft Excel spreadsheet. Means were obtained for the Flesch Reading Ease scores only because its results are not directly comparable with the other 3 readability scores. Statistical evaluation of the Flesch-Kincaid, Fry, and SMOG reading levels, using analysis of variance and paired t tests, were calculated using the built-in statistical software in Microsoft Excel. A predetermined value of P<.05 was used for significance.

There are approximately 40 different readability formulas, all of which will give a reasonably accurate grade level (generally plus or minus 1 grade level with 68% confidence, an acceptable standard in the field).³ These 2 readability levels were chosen because they are built into Microsoft Word and can be easily obtained by users of this common software program. The Flesch Reading Ease score and Flesch-Kincaid reading level are 2 of the most widely used and validated systems for scoring readability.^27,28 The Flesch Reading Ease ranges from 0 to 100, with a lower score being more difficult to read than a higher score. It uses sentence length and polysyllabic words to determine difficulty and does not give a grade level. It is a standard used by most of the insurance industry for consumer documents and contracts. A score of 70 or above is described as "easy" and is written at the grade school level. A score of 60 to 70 is described as "standard" and is written at approximately at the high school level. A score of 60 or below is described as "difficult" (Table 1). The Flesch-Kincaid score is only valid between US grades 3 and 12 and cannot differentiate information that is easier or more difficult than these levels. It uses mean sentence and word length to determine reading level. The Fry Formula is a hand-calculation method that is valid between the 1st- and the 17th-grade levels and is the method for patient education materials promoted by experts in the field.³ It uses average numbers of sentences and syllables in 100 word passages to determine the reading level. The SMOG is another hand calculation method used commonly for health information. It is valid from grades 3 through 19, but maybe less accurate below the sixth grade. The SMOG uses the number of polysyllabic words in 10 sentence passages to determine reading level. In general, improved readability has been shown for low numbers of words per sentence, characters per word, syllables per word, and percentage of passive sentences.³ Illustrations and tables also improve the readability.

Data concerning a document's Internet domain, author(s), and institutional affiliation were also collected. A document's author may be an individual, group, or the institution itself. For example, a document written by Jane Doe, MD, for the XYZ Professional Society of America was coded into the "physician" author and "professional society" institution groups. If no specific individual(s) authored the document, the document was coded into "professional society" for both the author and institution groups.

One hundred documents from 100 different Web sites were evaluated. Eleven documents were not further evaluated because their Flesch-Kincaid reading level was below 3.0 or above 12.0; therefore, those reading levels were not valid. There were 5 documents that were below the 3.0 grade level and 6 documents that were above the 12.0 grade level. Eighty-nine documents composed the final sample. The documents that were evaluated covered a wide spectrum of pediatric topics (Table 2). The documents came from the domains of .org (n = 32), .edu (n = 20), .com (n = 19), .gov (n = 10), .net (n = 4), and international domains (n = 4). The majority of the documents specifically noted that they were designed for laypersons (n = 71). Some documents contained illustrations (n = 11) and tables (n = 8). No documents had their reading level noted.

The overall mean for the Flesch Reading Ease was 57.0 (described as "fairly difficult" [range, 52.1-65.1]). The overall means for Flesch-Kincaid reading level was 7.1 (range, 3.5-10.9); for the Fry Formula, 12.0 (range, 6.0-17.0); and for the SMOG method, 12.2 (range, 7.0-17.0). The Flesch-Kincaid reading level was statistically different from the Fry Formula and SMOG methods (P<.0001), and the Fry Formula and SMOG methods were not statistically different from one another (P = .44). Documents with illustrations and tables did not have substantively different mean reading levels (data not shown).

Table 3 shows the mean reading levels based on the author and the institution of the document. The Flesch Reading Ease scores for all groups were in the fairly difficult to standard range. The Flesch-Kincaid mean reading level was below the eighth-grade level for all author and institutional groups. In contrast, no author or institutional group had a mean reading level below 10.6 for the Fry Formula and SMOG methods. In general, the Flesch-Kincaid reading levels were 4 to 5 grade levels lower than for the Fry Formula and SMOG methods. The Flesch-Kincaid reading level was again statistically different from the Fry Formula and SMOG methods for each author and institutional group (all groups, P<.01). The Fry Formula and SMOG methods were not statistically different for each author and institutional group (P = .15 to P>.99)

The goal of this study was to determine the general readability of pediatric patient education materials designed for adults on the WWW. This study is unfortunately consistent with previous studies, which showed that the readability of patient care information in both print and the WWW are written at too high a level for most adults. This has significant implications, as adults are the main group health care providers are educating on how to take care of themselves and their children. Using the Flesch Reading Ease, the patient education materials on the WWW were written at the fairly difficult level. By the Fry Formula and SMOG methods (the readability formulas recommended for patient education materials), the materials are written at the 12th-grade level, with no author or institutional group being less than the 10.6 grade level. The Flesch-Kincaid computer method did find the same pediatric education materials to be written at the 7.1 level overall, substantially below the other readability formulas. This is most likely because of some of the inherent problems using readability formulas, including those generated via computers, since each formula uses slightly different mathematical methods to calculate a readability level. Mailloux et al²⁹ compared 4 software readability programs and found a variation of 1.3 grade levels between the programs, with Microsoft Word having the largest variance (approximately 3 grade levels).

Pediatricians and other pediatric health care providers have the opportunity to improve the readability of their patient education materials in the following ways: (1) Pediatricians should be aware that the readability of the materials they are giving parents is probably too high. (2) When these patient education materials are written or modified, the provider can use their well-honed professional skills with children to express the information using noncomplex language. (3) Computer formulas may be helpful during the writing and editing process to obtain an initial "ballpark" readability level. We suggest using the Flesch Reading Ease score, as it shows less variability and its interpretations are more consistent with the hand-calculation methods. We suggest that final documents should have the readability level reanalyzed using the Fry Formula or SMOG method, which are easily obtained through the WWW.^30,31 (4) The readability level should be noted on the patient education materials for general guidance in material selection. This study found that none of the patient education materials had reading levels noted. An individual patient education document may still not be written at the appropriate level for an individual adult. The simple addition of a reading level on this material would make it easier for busy health care providers and patients to choose more appropriate patient education materials. Although some providers may be concerned that good readers will feel talked down to by instruction that has a low reading level, it has been shown that adults at all reading levels prefer and learn better with easy-to-read materials.^3,32 Suitability of reading materials depends on many factors, but the reading level is usually the main criterion predicting patient comprehension of the material.³ Adapting the Suitability Assessment of Materials instrument³ and the Flesch Reading Ease scores, the following designations could be used: (1) easy reading (Flesch Reading Ease, >80 [<6th-grade level]); (2) adequate or standard reading (Flesch Reading Ease, 60-80 [6th- to 8th-grade level]); (3) and difficult reading (Flesch Reading Ease, <60 [≥9th-grade level]).

As standards develop for consumer health information on the WWW, we propose that a reading level and how it was determined also be included, again as a general guide to material selection. Writing patient education materials for print literature can be challenging and the WWW presents its own additional challenges. Many resources on improving the readability of patient education materials and designing materials for the WWW are available, including the tips described in Table 4.^30,31,33-35 The materials should use short sentences, be written in the active voice, and give examples when appropriate. The materials should also use large easy-to-read type and contain illustrations if appropriate. Definitions and glossaries help readers with new vocabulary. Pediatricians should remember that these patient education materials may be printed by users and therefore are dual-purpose materials (ie, both print and WWW). There are also methods of determining an individual's reading ability in the clinic setting, suitability of materials,³ and readability formulas for languages other than English³⁶; however, these discussions are beyond the scope of this article.

This study is limited in that it only evaluated 100 documents on the WWW, which has an estimated 50 million pages of health-related information.³⁷ We attempted to evaluate a wide range of pediatric information from numerous sources. Other limitations of the study are that the patient education materials were written in English only, and do not address the patient education needs of younger children. The researchers originally choose the hyperlinks in the digital library using criteria that did not include readability, but there still could be an unrecognized systematic bias. It is less likely that document selection bias occurred because only 1 document from a single Web site was selected and a variety of different types of Web sites were evaluated. The numbers in some of the author and institution groups are small and therefore may be subject to sampling error.

Finally, readability formulas themselves may underestimate the difficulty of medical information, because unfamiliar terminology increases the difficulty. The typical adult has little exposure to medical terminology; often the meaning has to be determined anew because of the lack of associations and references. Therefore, even short words increase the difficulty but are scored as easy to read in the formulas. The comprehension level needed to understand medical information might be even higher than that determined by the readability formulas used.^3,16 Using definitions or glossaries can help this common problem.

Pediatric patient education materials on the WWW do not meet the guideline of being written below the eighth-grade level. We propose all patient education materials on the WWW also include a practical reading level and how it was determined to guide material selection. Computer-generated reading levels could be used as an initial measure of readability, but patient education materials should be reevaluated using another method.

Accepted for publication February 2, 2001.

Supported in part by the Robert Wood Johnson Foundation Generalist Faculty Scholars Grant, Princeton, NJ (Dr D'Alessandro).

We thank Leonard Doak, PE, and Cecilia Doak, MPH, for their helpful insights into readability formulas.

Corresponding author: Donna M. D'Alessandro, MD, Department of Pediatrics, Children's Hospital of Iowa, 200 Hawkins Dr, Iowa City, IA 52242-1009 (e-mail: donna-dalessandro@uiowa.edu).