Assessment of Diagnosis and Triage in Validated Case Vignettes Among Nonphysicians Before and After Internet Search | Acute Coronary Syndromes | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Diagnosis, Triage, and Anxiety Before and After an Internet Search, by Severity of Case
Diagnosis, Triage, and Anxiety Before and After an Internet Search, by Severity of Case

Error bars represent 95% CI. For panels A and B, the y-axis does not begin at 0.

Figure 2.  Diagnostic Accuracy Before and After Internet Search, by Severity of Case
Diagnostic Accuracy Before and After Internet Search, by Severity of Case
Figure 3.  Anchoring or Flipping on Original Diagnosis and Triage
Anchoring or Flipping on Original Diagnosis and Triage

The intersection of blue and orange color blocks represents the percentage of patients who flipped their original diagnosis (A) or triage (B).

Table 1.  Participant Characteristics
Participant Characteristics
Table 2.  Associations of Sociodemographic Variables With Any Correct Diagnosis and Dichotomized Triage Before and After Internet Search
Associations of Sociodemographic Variables With Any Correct Diagnosis and Dichotomized Triage Before and After Internet Search
1.
Wyatt  JC.  Fifty million people use computerised self triage.   BMJ. 2015;351:h3727. doi:10.1136/bmj.h3727 PubMedGoogle ScholarCrossref
2.
Morahan-Martin  JM.  How internet users find, evaluate, and use online health information: a cross-cultural review.   Cyberpsychol Behav. 2004;7(5):497-510. doi:10.1089/cpb.2004.7.497 PubMedGoogle ScholarCrossref
3.
Schoenherr  GP, White  RW. Interactions between health searchers and search engines. In:  SIGIR '14: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM Press; 2014:143-152. doi:10.1145/2600428.2609602
4.
Loos  A.  Cyberchondria: too much information for the health anxious patient?   J Consum Health Internet. 2013;17(4):439-445. doi:10.1080/15398285.2013.833452 Google ScholarCrossref
5.
Luger  TM, Houston  TK, Suls  J.  Older adult experience of online diagnosis: results from a scenario-based think-aloud protocol.   J Med Internet Res. 2014;16(1):e16. doi:10.2196/jmir.2924 PubMedGoogle Scholar
6.
North  F, Ward  WJ, Varkey  P, Tulledge-Scheitel  SM.  Should you search the internet for information about your acute symptom?   Telemed J E Health. 2012;18(3):213-218. doi:10.1089/tmj.2011.0127PubMedGoogle ScholarCrossref
7.
Weller  C. Dr. Google breeds hypochondria by scaring people into thinking the worst. Medical Daily. Published May 7, 2015. Accessed March 2, 2020. https://www.medicaldaily.com/dr-google-breeds-hypochondria-scaring-people-thinking-worst-332316
8.
Larimer  S. Can this ad campaign get people in Belgium to stop Googling their symptoms? The Washington Post. Published November 11, 2014. Accessed January 2, 2019. https://www.washingtonpost.com/news/to-your-health/wp/2014/11/11/can-this-ad-campaign-get-people-in-belgium-to-stop-googling-their-symptoms/
9.
Bates N. Don’t google it. Accessed January 24, 2020. https://vimeo.com/115097884
10.
Fiksdal  AS, Kumbamu  A, Jadhav  AS,  et al.  Evaluating the process of online health information searching: a qualitative approach to exploring consumer perspectives.   J Med Internet Res. 2014;16(10):e224. doi:10.2196/jmir.3341 PubMedGoogle Scholar
11.
Hu  Y, Haake  J.  Search your way to an accurate diagnosis: predictors of internet-based diagnosis accuracy.   Atl J Commun. 2010;18(2):79-88. doi:10.1080/15456870903554916 Google ScholarCrossref
12.
Tang  H, Ng  JHK.  Googling for a diagnosis—use of Google as a diagnostic aid: internet based study.   BMJ. 2006;333(7579):1143-1145. doi:10.1136/bmj.39003.640567.AE PubMedGoogle ScholarCrossref
13.
Semigran  HL, Linder  JA, Gidengil  C, Mehrotra  A.  Evaluation of symptom checkers for self diagnosis and triage: audit study.   BMJ. 2015;351:h3480. doi:10.1136/bmj.h3480 PubMedGoogle ScholarCrossref
14.
Semigran  HL, Levine  DM, Nundy  S, Mehrotra  A.  Comparison of physician and computer diagnostic accuracy.   JAMA Intern Med. 2016;176(12):1860-1861. doi:10.1001/jamainternmed.2016.6001 PubMedGoogle ScholarCrossref
15.
Kroenke  K, Spitzer  RL, Williams  JB, Monahan  PO, Löwe  B.  Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection.   Ann Intern Med. 2007;146(5):317-325. doi:10.7326/0003-4819-146-5-200703060-00004 PubMedGoogle ScholarCrossref
16.
Goldman  L, Kirtane  AJ.  Triage of patients with acute chest pain and possible cardiac ischemia: the elusive search for diagnostic perfection.   Ann Intern Med. 2003;139(12):987-995. doi:10.7326/0003-4819-139-12-200312160-00008PubMedGoogle ScholarCrossref
17.
Pew Research Center. The internet and health. Published February 12, 2013. Accessed January 2, 2019. https://www.pewresearch.org/internet/2013/02/12/the-internet-and-health/
18.
Martin  SS, Quaye  E, Schultz  S,  et al.  A randomized controlled trial of online symptom searching to inform patient generated differential diagnoses.   NPJ Digit Med. 2019;2(1):110. doi:10.1038/s41746-019-0183-0 PubMedGoogle ScholarCrossref
19.
Wang  J, Ashvetiya  T, Quaye  E, Parakh  K, Martin  SS.  Online health searches and their perceived effects on patients and patient-clinician relationships: a systematic review.   Am J Med. 2018;131(10):1250.e1-1250.e10. doi:10.1016/j.amjmed.2018.04.019 PubMedGoogle ScholarCrossref
21.
Daraz  L, Morrow  AS, Ponce  OJ,  et al.  Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet.   J Gen Intern Med. 2019;34(9):1884-1891. doi:10.1007/s11606-019-05109-0 PubMedGoogle ScholarCrossref
22.
Winn  AN, Somai  M, Fergestrom  N, Crotty  BH.  Association of use of online symptom checkers with patients’ plans for seeking care.   JAMA Netw Open. 2019;2(12):e1918561. doi:10.1001/jamanetworkopen.2019.18561 PubMedGoogle Scholar
23.
American Society of Anesthesiologists – Pain medicine; five things physicians and patients should question. Released January 21, 2014. Accessed February 2, 2016. https://www.choosingwisely.org/societies/american-society-of-anesthesiologists-pain-medicine/
24.
Chambers  D, Cantrell  AJ, Johnson  M,  et al.  Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review.   BMJ Open. 2019;9(8):e027743. doi:10.1136/bmjopen-2018-027743 PubMedGoogle Scholar
25.
Fraser  H, Coiera  E, Wong  D.  Safety of patient-facing digital symptom checkers.   Lancet. 2018;392(10161):2263-2264. doi:10.1016/S0140-6736(18)32819-8 PubMedGoogle ScholarCrossref
26.
Garrett  RK.  Politically motivated reinforcement seeking: reframing the selective exposure debate.   J Commun. 2009;59(4):676-699. doi:10.1111/j.1460-2466.2009.01452.x Google ScholarCrossref
27.
Bertakis  KD, Azari  R, Helms  LJ, Callahan  EJ, Robbins  JA.  Gender differences in the utilization of health care services.   J Fam Pract. 2000;49(2):147-152.PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 8,939
    Citations 0
    Original Investigation
    Health Informatics
    March 29, 2021

    Assessment of Diagnosis and Triage in Validated Case Vignettes Among Nonphysicians Before and After Internet Search

    Author Affiliations
    • 1Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, Massachusetts
    • 2Harvard Medical School, Boston, Massachusetts
    • 3Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
    • 4Division of General Medicine and Primary Care, Beth Israel Deaconess Medical Center, Boston, Massachusetts
    JAMA Netw Open. 2021;4(3):e213287. doi:10.1001/jamanetworkopen.2021.3287
    Key Points

    Question  Is there an association between an internet search for health information and improved accuracy in diagnosis and triage among nonphysicians?

    Findings  In this survey study of 5000 US adults who were asked to assess validated case vignettes, small improvements in diagnostic accuracy were found after an internet search for health information, but no difference in triage accuracy was observed. Adults 40 years or older, women, and those with poor health status were superior at diagnosis.

    Meaning  Results of this study suggest that, contrary to concerns of its harmfulness, an internet search was associated with modest improvements in diagnosis but had no association with triage.

    Abstract

    Importance  When confronted with new medical symptoms, many people turn to the internet to understand why they are ill as well as whether and where they should get care. Such searches may be harmful because they may facilitate misdiagnosis and inappropriate triage.

    Objective  To empirically measure the association of an internet search for health information with diagnosis, triage, and anxiety by laypeople.

    Design, Setting, and Participants  This survey study used a nationally representative sample of US adults who were recruited through an online platform between April 1, 2019, and April 15, 2019. A total of 48 validated case vignettes of both common (eg, viral illness) and severe (eg, heart attack) conditions were used. Participants were asked to relay their diagnosis, triage, and anxiety regarding 1 of these cases before and after searching the internet for health information.

    Exposures  Short, validated case vignettes written at or below the sixth-grade reading level randomly assigned to participants.

    Main Outcomes and Measures  Correct diagnosis, correct triage, and flipping (changing) or anchoring (not changing) diagnosis and triage decisions were the main outcomes. Multivariable modeling was performed to identify patient factors associated with correct triage and diagnosis.

    Results  Of the 5000 participants, 2549 were female (51.0%), 3819 were White (76.4%), and the mean (SD) age was 45.0 (16.9) years. Mean internet search time was 12.1 (95% CI, 10.7-13.5) minutes per case. No difference in triage accuracy was found before and after search (74.5% vs 74.1%; difference, −0.4 [95% CI, −1.4 to 0.6]; P = .06), but improved diagnostic accuracy was found (49.8% vs 54.0%; difference, 4.2% [95% CI, 3.1%-5.3%]; P < .001). Most participants (4254 [85.1%]) were anchored on their diagnosis. Of the 14.9% of participants (n = 746) who flipped their diagnosis, 9.6% (n = 478) flipped from incorrect to correct and 5.4% (n = 268) flipped from correct to incorrect. The following groups had an increased rate of correct diagnosis: adults 40 years or older (eg, 40-49 years: 5.1 [95% CI, 0.8-9.4] percentage points better than those aged <30 years; P = .02), women (9.4 [95% CI, 6.8-12.0] percentage points better than men; P < .001), and those with perceived poor health status (16.3 [95% CI, 6.9-25.6] percentage points better than those with excellent status; P = .001) and with more than 2 chronic diseases (6.8 [95% CI, 1.5-12.1] percentage points better than those with 0 conditions; P = .01).

    Conclusions and Relevance  This study found that an internet search for health information was associated with small increases in diagnostic accuracy but not with triage accuracy.

    Introduction

    Each day, millions of people worldwide who are confronted with new medical symptoms turn to the internet before seeking care to understand why they are ill, whether they should get care, and where they should get care.1,2 The value of performing an internet search for health purposes is controversial, with concerns that it leads to inaccurate diagnosis, inappropriate triage (ie, choosing the right location to seek care), and increased anxiety (cyberchondria).3-7 An internet search may lead people to low-quality health information that might hurt their choice of whether to get care or to alarming content that might easily overwhelm or confuse people. Some governments have even launched Don’t Google It advertising campaigns to urge their residents to not use the internet to search their health concerns.8,9

    Despite its ubiquitous use, the benefits and harms of an internet search for health information are poorly understood. Previous research has been largely limited to observational studies of internet search behavior and may lack a criterion standard.2,10-12 In this study, we sought to empirically measure the association of an internet search with diagnosis, triage, and anxiety by presenting laypeople with a clinical vignette and assessing the accuracy of their decisions before and after searching the internet.

    Methods

    We performed a before-after survey study with a national sample of internet users in the United States. Participants reviewed a simple case vignette and relayed their presumed diagnosis, triage, and anxiety regarding the case. Next, participants were asked to use the internet to search for information about the case and relay their updated diagnosis, triage, and anxiety. This study design emulated how a person typically interacts with the internet: encountering information, forming a preliminary conclusion, and then reforming a conclusion after searching the internet. We enrolled participants between April 1, 2019, and April 15, 2019, and conducted no follow-up. Participants provided written informed consent by clicking on the accept button in the online survey after reading a description of the study and its risks and benefits. The protocol was approved by the Harvard Medical School Institutional Review Board. We followed the American Association for Public Opinion Research (AAPOR) reporting guideline.

    Setting and Participants

    We recruited people using Toluna, a company specializing in online surveys for research, marketing, and business intelligence. Toluna uses a multifaceted approach to recruit a nationally representative sample of survey respondents. We requested a representative cohort by sex, age, and census region. Toluna also deploys multiple quality assurance strategies as respondents complete a survey to ensure high-quality data. People were eligible for inclusion in this survey if they were 18 years or older and resided in the United States. Participants earned compensation for their time through Toluna’s standard reimbursement system.

    Separately from the online survey participants, we enrolled a convenience sample of 21 attending primary care physicians at Harvard Medical School to validate the case vignettes. Each physician received a $20 gift card for their time.

    The correct diagnosis and triage category for each vignette were first ascertained by the 2 of us (both general internists) and used as the criterion standard. We gave the primary care physicians the 48 clinical vignettes and asked for their triage and diagnosis on the basis of only their clinical experience. The physicians did not use the internet.

    Intervention and Outcomes

    Building on prior work that evaluated online symptom checker tools,13,14 we created 48 case vignettes that included a chief complaint followed by additional pertinent details (eTable 1 in the Supplement). Each vignette was fewer than 50 words and written at or below a sixth-grade reading level. Twelve vignettes were written for each of 4 triage categories: emergent cases, same-day cases, 1-week cases, and self-care cases. We included both common (eg, viral illness) and severe (eg, heart attack) conditions but not those with highly obscure presentations.

    Before beginning the principal task, respondents were asked to report on several sociodemographic variables (Table 1). Respondents were randomly assigned to 1 of the 48 vignettes and given instructions to “please read the following health problem, and imagine it were happening to your close family member.” After reviewing the vignette, participants selected from the following triage options that they deemed the best: (1) let the health issue get better on its own; the issue most likely does not require seeing a doctor; (2) try to see a doctor within a week; the issue likely will not get better on its own, but it is also not an emergency; (3) try to see a doctor within a day; the issue is urgent, but it is not an emergency; or (4) call 911 or go directly to the emergency department; the issue requires immediate attention.

    Next, via free-text response to the following question, participants listed the conditions in order of likelihood: “What do you think are the three most likely medical diseases or diagnoses that could be causing this health problem?” Using a Likert scale (not at all, slightly, moderately, highly, or extremely), they selected an emotional response to this question: “If you had a family member experience the health problem described, how nervous/anxious/on edge would you be?”15 In addition, they were asked to rate their confidence in their responses using the same Likert scale.

    Participants were then asked to use the internet in any way they believed to be useful to find the correct diagnosis and triage option for the health problem in the same vignette. We measured the time spent searching in minutes. After the internet search, respondents reported the triage and diagnosis that they selected and, using the same Likert scale, ranked their level of anxiety and confidence in their response as well as their perceived difficulty in finding useful information, their trust in the information they found, and the kinds of websites that they deemed to be most helpful. We categorized websites as a search engine, health specialty site (for example, WebMD), general information site (for example, Wikipedia), social network site (for example, Facebook), news site, forum, or other.

    Statistical Analysis

    We presented descriptive data with counts and percentages or means and 95% CIs, as appropriate. Because of the survey design, there were no missing data. Free-text diagnoses were manually reviewed and scored. Consistent with prior work,13,14 we measured diagnostic accuracy in 2 ways: whether the respondent’s first selected diagnosis was correct (first correct) and whether any of the respondent’s 3 diagnoses were correct (any correct). In this article, we present the analyses of any-correct diagnosis, whereas eTable 2 in the Supplement presents first-correct analyses.

    We used 2 methods to assess the accuracy of the triage: whether the respondent’s selected triage was exactly correct (exact) and whether the respondent’s selected triage matched a dichotomized triage variable of emergent or same-day cases vs 1-week or self-care cases (dichotomized). In the main analyses, we present a dichotomized triage as we observed some inconsistency among physicians in triage with emergent vs same-day cases16; the exact triage analyses are included in eTable 2 in the Supplement.

    To identify whether acuity was associated with diagnosis or triage, we performed subgroup analysis by acuity. We assessed flipping (changing the answer) and anchoring (not changing the answer) for both diagnosis and triage decision.

    For a bivariate comparison of diagnosis and triage, we used the McNemar test, and for anxiety and confidence, we used a paired, 2-tailed t test. Diagnosis and triage stratified by participant characteristics are shown in eTables 3 and 4 in the Supplement. For multivariable analyses, we used general estimating equations that accounted for repeated measures. We present marginal effects for associations for values of the independent variable of interest while adjusting for all covariates in Table 1. Adjusted odds ratios are presented in eTable 5 in the Supplement.

    We considered 2-sided P < .05 to be significant. We performed all analyses in SAS, version 9.4 (SAS Institute Inc), and Stata, version 15 (StataCorp LLC).

    Results

    We enrolled 5000 participants with a mean (SD) age of 45.0 (16.9) years. Of these participants, 2549 were female (51.0%), 3819 were White (76.4%), 2484 were privately insured (49.7%), 736 had poor or fair perceived health (14.7%), and 2203 had at least 1 chronic disease (44.1%) (Table 1). Most participants (3963 [79.3%]) reported having primary care. Respondents reported a mean of 2.1 (95% CI, 2.0-2.1) physician visits and 0.3 (95% CI, 0.3 to 0.3) emergency department visits in the past 6 months.

    The 21 primary care physicians who validated the correct responses from the clinical vignettes included 10 women (47.6%) and 11 men (52.4%) with a mean duration of practice of 12.4 years. The physicians reported the correct triage in 91.2% (95% CI, 89.2%-93.2%) of vignettes and the correct diagnosis in 95.7% (95% CI, 94.3%-97.0%) of vignettes.

    Before Internet Search

    Before conducting an internet search, respondents differed in diagnosis, triage, and anxiety across triage categories (Figure 1; eTable 6 in the Supplement). Rates of correct diagnosis for emergent cases (40.4%; 95% CI, 37.8%-43.2%) were significantly lower than for self-care cases (67.4%; 95% CI, 64.8%-70.0%) (Figure 1A). The opposite was true for triage, where rates of correct triage for emergent cases (87.0%; 95% CI, 85.2%-88.9%) were significantly higher than for self-care cases (69.3%; 95% CI, 66.7%-71.8%) (Figure 1B). As the case acuity became more serious, respondents reported more anxiety (on a 5-point scale: self-care cases, 2.6 of 5; emergent cases, 3.9 of 5) (Figure 1C).

    After Internet Search

    Respondents spent a mean internet search time of 12.1 (95% CI, 10.7-13.5) minutes per case. There was a significant increase in diagnostic accuracy observed before vs after the internet search (49.8% vs 54.0%; difference, 4.2% [95% CI, 3.1%-5.3%]; P < .001) (Figure 1A). Improvements in diagnostic accuracy were observed across all triage categories: emergent (3.1%; 95% CI, 1.0%-5.3%; P = .004), same-day (3.5%; 5% CI, 1.5%-5.6%; P < .001), 1-week (6.4%; 95% CI, 4.1%-8.7%; P < .001), and self-care (3.7%; 95% CI, 1.7%-5.8%; P < .001) cases (Figure 2; eTable 6 in the Supplement).

    No difference in triage accuracy was observed between before and after the internet search across all cases (74.5% vs 74.1%; difference, −0.4 [95% CI, −1.4 to 0.6]; P = .06) or among any level of case acuity (Figure 1B; eTable 6 in the Supplement). Anxiety also did not change from before to after search (3.3 of 5 points vs 3.3 of 5 points) (Figure 1C; eTable 6 in the Supplement), nor did participants’ confidence in their responses (3.8 of 5 points vs 3.8 of 5 points) (eTable 6 in the Supplement).

    Participants reported that, in general, it was slightly difficult to find useful information on the internet and they moderately trusted the information found (eTable 7 in the Supplement). They noted that the most helpful sources of information were search engines (48.2% [n = 2411]), followed by health specialty sites (42.9% [n = 2145]). A small proportion of respondents (1.5% [n = 73]) rated social network sites as most helpful.

    Anchoring and Flipping

    Most respondents were anchored on their original diagnosis (4254 [85.1%]) or triage (4360 [87.2%]) (Figure 3; eFigure in the Supplement). A small proportion (14.9% [n = 746]) flipped their diagnosis after the internet search: 9.6% (n = 478) changed from incorrect to correct diagnosis, whereas 5.4% (n = 268) changed from correct to incorrect diagnosis. Similarly, 12.8% of respondents (n = 640) flipped their triage decision after the internet search, with roughly similar percentages in both directions: 6.6% (n = 329) changed from correct to incorrect triage, whereas 6.2% (n = 311) changed from incorrect to correct triage.

    Factors Associated With Correct Diagnosis and Triage

    In multivariable modeling, characteristics that were associated with an increased rate of correct diagnosis were age of 40 years or older (40-49 years: 5.1 [95% CI, 0.8-9.4] percentage points more than for those <30 years [P = .02]; 50-59 years: 6.6 [95% CI, 1.8-11.3] percentage points more than for those <30 years [P = .006]), female sex (9.4 [95% CI, 6.8-12.0] percentage points more than for male sex; P < .001), White race/ethnicity (9.6 [95% CI, 4.9-14.4] percentage points more than for Black race/ethnicity [P < .001]; 6.4 [95% CI, 0.9-11.8] percentage points more than for Hispanic race/ethnicity [P = .02]; and 14.7 [95% CI, 9.3-20.0] percentage points more than for Asian race/ethnicity [P < .001]), uninsured status (9.1 [95% CI, 3.2-15.0] percentage points more than for Medicare; P = .003), perceived poor health status (16.3 [95% CI, 6.9-25.6] percentage points higher than for those with excellent status; P = .001), and more than 2 chronic diseases (6.8 [95% CI, 1.5-12.1] percentage points higher than for those with 0 conditions; P = .01) (Table 2).

    Factors associated with correct triage were less consistent. Characteristics that were significantly associated with an increased rate of correct triage were age of 50-59 years (5.9 [95% CI, 1.8-10.0] percentage points more than those <30 years; P = .005), female sex (4.5 [95% CI, 2.3-6.8] percentage points more than for male sex; P < .001), and White race/ethnicity (9.7 [95% CI, 5.2-14.2] percentage points more than for Black race/ethnicity; P < .001).

    Discussion

    In a given year in the US, almost two-thirds of adults use the internet to search for health information and roughly one-third of adults have used the internet specifically for self-diagnosis, trying to discover an underlying cause to a health problem that they or their family members may have.17 Among a nationally representative sample of survey participants asked to diagnose and select a triage for a clinical vignette, we observed that the use of the internet was associated with modest but significant improvements in diagnosis, but we observed no association with triage or anxiety. Although the perceived harm of an internet search for health information may be unfounded, the potential benefits are also currently minimal.

    This work builds on others in the literature. Similar to the present study, research by Martin and colleagues18 demonstrated that performing an internet search among patients in an emergency department waiting room was not associated with increased anxiety. Wang and colleagues19 also noted that observational studies reported mixed results with respect to the association between an internet search and patient anxiety.

    Results of this survey study challenge the common belief among clinicians and policy makers that using the internet to search for health information is harmful. We found that performing an internet search was associated with improved diagnosis. One potential reason for this disconnect is that over time, search engines have tried to direct people to higher-quality health information. For example, several search engines have their own built-in health information curated by major medical centers,20 and in this study, almost half of the respondents believed such information was the most helpful. In this study, only a small percentage of respondents used social media or forums, which may have a lower quality of information. We also found that performing an internet search was not associated with selection of a more aggressive triage option or with increased anxiety. That is, we found no evidence of the hypothetical scenario of patients believing they were having a heart attack and calling 911 in response.

    These results could be framed quite differently. Although it was associated with no harm, any benefit of an internet search was small. A recent systematic review found an overall low level of quality of online health information, which might explain the modest association.21 Websites or applications specifically designed to help people diagnose and triage themselves may be more helpful. Previous evaluations of older tools (called symptom checkers) demonstrated that their performance was mixed, but newer tools use artificial intelligence and may be more beneficial.13,14,22-25

    Another explanation for the small before-after changes that we observed is anchoring. Only a small fraction of respondents changed their diagnosis or triage decision after the internet search. Consistent with the theory of reinforcement seeking, internet searchers may simply look for information to justify their initial decision rather than being open to all recommendations.26

    With or without an internet search, roughly three-quarters of participants were able to identify the severity of the situation and when to seek care. Participants were more accurate when a case was more severe: almost 9 in 10 emergent cases were triaged appropriately. These results were reassuring, although it is important to recognize that an incorrect triage occurred in 1 in 10 emergent cases. How this rate of triage inaccuracy compares with the triage inaccuracy rate for nurse triage lines and how it is associated with patient outcome remain unclear.

    In contrast to triage, we found that participants were much worse at diagnosis, responding correctly only about half the time. Sociodemographic differences in diagnostic accuracy were observed, with adults older than 40 years, women, and those with more health care experience performing significantly better. Consistent with previous work demonstrating that baseline knowledge was associated with improved diagnostic accuracy,11 results of the present study implied that lived experience (lower perceived health status, more comorbidities, and older age) seemed to assist participants with triage and diagnosis. Lived experience may also explain better performance by women because they, in general, experience more health care and may make more decisions for their family to seek out care.27

    Limitations

    This study has limitations. First, we tested responses using simulated cases. Survey participants may respond differently or perform an internet search differently if they themselves or their family members were actually experiencing the symptoms. However, the anxiety levels reported by respondents increased as case acuity increased, suggesting that they were internalizing the cases appropriately. Participants also appeared to take the task seriously, spending about 12 minutes per case, approximately 3 times longer than what other studies have reported for similar in-person tasks for personally experienced symptoms.18 Second, the study has generalizability concerns given that the sample was obtained through an online platform and was fully conducted online. However, the sample was representative of the national population in terms of sex, age, and census region. The online platform also ensured that participants used the internet and allowed us to leverage its built-in quality controls, which may not have been available with in-person testing.

    Third, the study may have lacked power in some of the variables to detect statistically significant differences, although this does not change the overall findings. Fourth, the vignette validation method relied on a convenience sample of physicians, and the physicians were not entirely in agreement with our choices. However, we selected multiple experienced physicians, and the accuracy rates among the physicians were more than 90% for all of the vignettes.

    Conclusions

    We found that, among survey participants, using the internet to search for health information was associated with small increases in diagnostic accuracy. However, we observed no association between an internet search for health information and triage accuracy.

    Back to top
    Article Information

    Accepted for Publication: February 5, 2021.

    Published: March 29, 2021. doi:10.1001/jamanetworkopen.2021.3287

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Levine DM et al. JAMA Network Open.

    Corresponding Author: David M. Levine, MD, MPH, MA, Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, 1620 Tremont St, 3rd Floor, Boston, MA 02120 (dmlevine@bwh.harvard.edu).

    Author Contributions: Dr Levine had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: All authors.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Levine.

    Critical revision of the manuscript for important intellectual content: Mehrotra.

    Statistical analysis: Levine.

    Obtained funding: Mehrotra.

    Administrative, technical, or material support: Levine.

    Supervision: Mehrotra.

    Conflict of Interest Disclosures: Dr Levine reported receiving grants from Biofourmis and IBM for PI-initiated studies outside the submitted work. No other disclosures were reported.

    Funding/Support: This research was supported by an unrestricted gift to Harvard Medical School from Mel Hall.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Wyatt  JC.  Fifty million people use computerised self triage.   BMJ. 2015;351:h3727. doi:10.1136/bmj.h3727 PubMedGoogle ScholarCrossref
    2.
    Morahan-Martin  JM.  How internet users find, evaluate, and use online health information: a cross-cultural review.   Cyberpsychol Behav. 2004;7(5):497-510. doi:10.1089/cpb.2004.7.497 PubMedGoogle ScholarCrossref
    3.
    Schoenherr  GP, White  RW. Interactions between health searchers and search engines. In:  SIGIR '14: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM Press; 2014:143-152. doi:10.1145/2600428.2609602
    4.
    Loos  A.  Cyberchondria: too much information for the health anxious patient?   J Consum Health Internet. 2013;17(4):439-445. doi:10.1080/15398285.2013.833452 Google ScholarCrossref
    5.
    Luger  TM, Houston  TK, Suls  J.  Older adult experience of online diagnosis: results from a scenario-based think-aloud protocol.   J Med Internet Res. 2014;16(1):e16. doi:10.2196/jmir.2924 PubMedGoogle Scholar
    6.
    North  F, Ward  WJ, Varkey  P, Tulledge-Scheitel  SM.  Should you search the internet for information about your acute symptom?   Telemed J E Health. 2012;18(3):213-218. doi:10.1089/tmj.2011.0127PubMedGoogle ScholarCrossref
    7.
    Weller  C. Dr. Google breeds hypochondria by scaring people into thinking the worst. Medical Daily. Published May 7, 2015. Accessed March 2, 2020. https://www.medicaldaily.com/dr-google-breeds-hypochondria-scaring-people-thinking-worst-332316
    8.
    Larimer  S. Can this ad campaign get people in Belgium to stop Googling their symptoms? The Washington Post. Published November 11, 2014. Accessed January 2, 2019. https://www.washingtonpost.com/news/to-your-health/wp/2014/11/11/can-this-ad-campaign-get-people-in-belgium-to-stop-googling-their-symptoms/
    9.
    Bates N. Don’t google it. Accessed January 24, 2020. https://vimeo.com/115097884
    10.
    Fiksdal  AS, Kumbamu  A, Jadhav  AS,  et al.  Evaluating the process of online health information searching: a qualitative approach to exploring consumer perspectives.   J Med Internet Res. 2014;16(10):e224. doi:10.2196/jmir.3341 PubMedGoogle Scholar
    11.
    Hu  Y, Haake  J.  Search your way to an accurate diagnosis: predictors of internet-based diagnosis accuracy.   Atl J Commun. 2010;18(2):79-88. doi:10.1080/15456870903554916 Google ScholarCrossref
    12.
    Tang  H, Ng  JHK.  Googling for a diagnosis—use of Google as a diagnostic aid: internet based study.   BMJ. 2006;333(7579):1143-1145. doi:10.1136/bmj.39003.640567.AE PubMedGoogle ScholarCrossref
    13.
    Semigran  HL, Linder  JA, Gidengil  C, Mehrotra  A.  Evaluation of symptom checkers for self diagnosis and triage: audit study.   BMJ. 2015;351:h3480. doi:10.1136/bmj.h3480 PubMedGoogle ScholarCrossref
    14.
    Semigran  HL, Levine  DM, Nundy  S, Mehrotra  A.  Comparison of physician and computer diagnostic accuracy.   JAMA Intern Med. 2016;176(12):1860-1861. doi:10.1001/jamainternmed.2016.6001 PubMedGoogle ScholarCrossref
    15.
    Kroenke  K, Spitzer  RL, Williams  JB, Monahan  PO, Löwe  B.  Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection.   Ann Intern Med. 2007;146(5):317-325. doi:10.7326/0003-4819-146-5-200703060-00004 PubMedGoogle ScholarCrossref
    16.
    Goldman  L, Kirtane  AJ.  Triage of patients with acute chest pain and possible cardiac ischemia: the elusive search for diagnostic perfection.   Ann Intern Med. 2003;139(12):987-995. doi:10.7326/0003-4819-139-12-200312160-00008PubMedGoogle ScholarCrossref
    17.
    Pew Research Center. The internet and health. Published February 12, 2013. Accessed January 2, 2019. https://www.pewresearch.org/internet/2013/02/12/the-internet-and-health/
    18.
    Martin  SS, Quaye  E, Schultz  S,  et al.  A randomized controlled trial of online symptom searching to inform patient generated differential diagnoses.   NPJ Digit Med. 2019;2(1):110. doi:10.1038/s41746-019-0183-0 PubMedGoogle ScholarCrossref
    19.
    Wang  J, Ashvetiya  T, Quaye  E, Parakh  K, Martin  SS.  Online health searches and their perceived effects on patients and patient-clinician relationships: a systematic review.   Am J Med. 2018;131(10):1250.e1-1250.e10. doi:10.1016/j.amjmed.2018.04.019 PubMedGoogle ScholarCrossref
    21.
    Daraz  L, Morrow  AS, Ponce  OJ,  et al.  Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet.   J Gen Intern Med. 2019;34(9):1884-1891. doi:10.1007/s11606-019-05109-0 PubMedGoogle ScholarCrossref
    22.
    Winn  AN, Somai  M, Fergestrom  N, Crotty  BH.  Association of use of online symptom checkers with patients’ plans for seeking care.   JAMA Netw Open. 2019;2(12):e1918561. doi:10.1001/jamanetworkopen.2019.18561 PubMedGoogle Scholar
    23.
    American Society of Anesthesiologists – Pain medicine; five things physicians and patients should question. Released January 21, 2014. Accessed February 2, 2016. https://www.choosingwisely.org/societies/american-society-of-anesthesiologists-pain-medicine/
    24.
    Chambers  D, Cantrell  AJ, Johnson  M,  et al.  Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review.   BMJ Open. 2019;9(8):e027743. doi:10.1136/bmjopen-2018-027743 PubMedGoogle Scholar
    25.
    Fraser  H, Coiera  E, Wong  D.  Safety of patient-facing digital symptom checkers.   Lancet. 2018;392(10161):2263-2264. doi:10.1016/S0140-6736(18)32819-8 PubMedGoogle ScholarCrossref
    26.
    Garrett  RK.  Politically motivated reinforcement seeking: reframing the selective exposure debate.   J Commun. 2009;59(4):676-699. doi:10.1111/j.1460-2466.2009.01452.x Google ScholarCrossref
    27.
    Bertakis  KD, Azari  R, Helms  LJ, Callahan  EJ, Robbins  JA.  Gender differences in the utilization of health care services.   J Fam Pract. 2000;49(2):147-152.PubMedGoogle Scholar
    ×