Comparison of Readability of Official Public Health Information About COVID-19 on Websites of International Agencies and the Governments of 15 Countries

Containment strategies for the coronavirus disease 2019 (COVID-19) pandemic have required broad public compliance, yet complex, contradictory, and false information proliferates.¹ The American Medical Association (AMA), National Institutes of Health (NIH), and Centers for Disease Control and Prevention (CDC) recommend that medical information for the public be written at no higher than an eighth-grade reading level.² We evaluated the readability of online information about COVID-19 provided by government and public health agencies and departments.

For this cross-sectional study, between April 1, 2020, and April 5, 2020, we reviewed 18 websites, including 3 public health agency sites and 15 official government sites of countries with 5000 or more confirmed cases as of April 5 and with guidelines written in English. We identified pages about COVID-19 intended for a general audience, such as lists of frequently asked questions and fact sheets, and extracted the content into text files. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

Readability grade levels based on 5 formulas (Flesch-Kincaid grade level [FKGL]; Simple Measure of Gobbledygook; Gunning Fog Index; Ford, Caylor, Sticht formula; and Coleman-Liau Index) were calculated using Readability Studio Professional, version 2019.3 (Oleander Software). Measures of syntactic complexity, including mean length of clause and dependent clauses per T-unit, were computed using the L2 Syntactic Complexity Analyzer, version 3.3.3.³

We evaluated web pages against grade level recommendations of the AMA, NIH, and CDC; the CDC pages were evaluated using their health literacy guidelines (a reading level of grade 8, 1 to 2 syllables per word, 8 to 10 words per sentence, and substitution of “everyday” synonyms for 121 difficult terms related to public health).^2,4,5 Sample passages from the websites with varying FKGL are provided in the eAppendix in the Supplement. State literacy data were obtained from a previous survey.⁶

Statistical calculations were performed using Stata, version 13 (StataCorp). A Wilcoxon rank-sum test was used for 2-sample comparisons, and correlation was assessed by Spearman r. Statistical significance was defined as P < .05.

Among all 18 websites evaluated, information about COVID-19 uniformly exceeded the recommended reading level of grades 6 through 8 (Table). All pages (n = 149) drawn from the websites scored above 8.0 by at least 1 metric, and 141 (95%) scored above 8.0 by all 5 metrics. A total of 145 pages (97%) exceeded the syntactic complexity typical for text written at a grade 8 level (mean length of clause, 8.0; dependent clauses per T-unit, 0.2).

Across all CDC pages, the median FKGL was 11.0 (interquartile range [IQR], 3.2; range, 5.4-16.6). Median syllables per word was 1.7 (IQR, 0.2; range, 1.3-2.0), median words per sentence was 15.6 (IQR, 3.2; range, 8.2-31.5), and 67 pages (99%) used at least 1 difficult term. Median number of difficult terms used was 11.0 (IQR, 10.5; range, 0-50).

The FKGL was above 8.0 for every state (Figure). Compared with the CDC, states (median, 24.5; IQR, 19.0; range, 4.0-53.0) used significantly more difficult terms (P < .001). Use of difficult terms by states was correlated with FKGL (Spearman r = 0.36; 95% CI, 0.09-0.58; P = .01). Nine of the 10 states with the highest illiteracy rates had information written above a grade 10 level.⁶

We found that official information about COVID-19 exceeded the recommended reading level, exhibited complex syntax, and used technical terminology. The significant difference in use of difficult terms between the CDC and state resources may reflect the influence of federal oversight mandating government communication that is understandable to the public. Limitations included the focus on text, with no evaluation of multimedia communication, and lack of data about actual comprehension or relevant outcomes such as adherence to mitigation strategies.

Nonadherence to readability standards may have a greater influence in communities with lower health literacy, potentially exacerbating the disparate effects of the pandemic. As such, efforts should focus on the urgent development of plain-language COVID-19 resources that conform to established guidelines for clear communication and are more accessible to all audiences.

Accepted for Publication: July 14, 2020.

Published: August 18, 2020. doi:10.1001/jamanetworkopen.2020.18033

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Mishra V et al. JAMA Network Open.

Corresponding Author: Joseph P. Dexter, PhD, Neukom Institute for Computational Science, Dartmouth College, 27 N Main St, Hanover, NH 03755 (joseph.p.dexter@dartmouth.edu).

Author Contributions: Drs Mishra and Dexter had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Both authors.

Acquisition, analysis, or interpretation of data: Both authors.

Drafting of the manuscript: Both authors.

Critical revision of the manuscript for important intellectual content: Both authors.

Statistical analysis: Both authors.

Obtained funding: Dexter.

Supervision: Dexter.

Conflict of Interest Disclosures: Dr Dexter reported receiving grants from the American Council of Learned Societies and from the Neukom Institute for Computational Science during the conduct of the study. No other disclosures were reported.

Funding/Support: This work was supported by a Digital Extension Grant from the American Council of Learned Societies and a Neukom Fellowship.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: Pramit Chaudhuri, PhD (University of Texas at Austin), provided helpful discussions and comments on the manuscript, and Anish Chattopadhyay (AIIMS Patna), Cuiwei Guo, MBA (Northeastern University), and Vasudha Mishra (AIIMS Patna) assisted with data collection and curation. None of these individuals received compensation for this work.