Association of the Implementation of a Standardized Thyroid Ultrasonography Reporting Program With Documentation of Nodule Characteristics | Endocrinology | JAMA Otolaryngology–Head & Neck Surgery | JAMA Network
[Skip to Navigation]
Sign In
Figure.  Barrier Analysis and Fishbone Diagram Before the Plan, Do, Study, Act
Barrier Analysis and Fishbone Diagram Before the Plan, Do, Study, Act

ACR TI-RADS indicates American College of Radiology Thyroid Imaging Reporting and Data System.

Table 1.  Study Timeline With Proposed Interventions Corresponding to Each Respective PDSA Cycle
Study Timeline With Proposed Interventions Corresponding to Each Respective PDSA Cycle
Table 2.  Summary of Results
Summary of Results
1.
Ezzat  S, Sarti  DA, Cain  DR, Braunstein  GD.  Thyroid incidentalomas: prevalence by palpation and ultrasonography.   Arch Intern Med. 1994;154(16):1838-1840. doi:10.1001/archinte.1994.00420160075010 PubMedGoogle ScholarCrossref
2.
Griffin  AS, Mitsky  J, Rawal  U, Bronner  AJ, Tessler  FN, Hoang  JK.  Improved quality of thyroid ultrasound reports after implementation of the ACR Thyroid Imaging Reporting and Data System nodule lexicon and risk stratification system.   J Am Coll Radiol. 2018;15(5):743-748. doi:10.1016/j.jacr.2018.01.024 PubMedGoogle ScholarCrossref
3.
Smith-Bindman  R, Lebda  P, Feldstein  VA,  et al.  Risk of thyroid cancer based on thyroid ultrasound imaging characteristics: results of a population-based study.   JAMA Intern Med. 2013;173(19):1788-1796. doi:10.1001/jamainternmed.2013.9245 PubMedGoogle ScholarCrossref
4.
Davies  L, Ouellette  M, Hunter  M, Welch  HG.  The increasing incidence of small thyroid cancers: where are the cases coming from?   Laryngoscope. 2010;120(12):2446-2451. doi:10.1002/lary.21076 PubMedGoogle ScholarCrossref
5.
Hoang  JK, Langer  JE, Middleton  WD,  et al.  Managing incidental thyroid nodules detected on imaging: white paper of the ACR Incidental Thyroid Findings Committee.   J Am Coll Radiol. 2015;12(2):143-150. doi:10.1016/j.jacr.2014.09.038 PubMedGoogle ScholarCrossref
6.
Youserm  DM, Huang  T, Loevner  LA, Langlotz  CP.  Clinical and economic impact of incidental thyroid lesions found with CT and MR.   AJNR Am J Neuroradiol. 1997;18(8):1423-1428.PubMedGoogle Scholar
7.
Wong  CKH, Liu  X, Lang  BHH.  Cost-effectiveness of fine-needle aspiration cytology (FNAC) and watchful observation for incidental thyroid nodules.   J Endocrinol Invest. 2020;43(11):1645-1654. doi:10.1007/s40618-020-01254-0 PubMedGoogle ScholarCrossref
8.
Ho  AS, Davies  L, Nixon  IJ,  et al.  Increasing diagnosis of subclinical thyroid cancers leads to spurious improvements in survival rates.   Cancer. 2015;121(11):1793-1799. doi:10.1002/cncr.29289 PubMedGoogle ScholarCrossref
9.
Haugen  BR, Alexander  EK, Bible  KC,  et al.  American Thyroid Association Management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer.   Thyroid. 2015;2016. doi:10.1089/thy.2015.0020PubMedGoogle Scholar
10.
Tessler  FN, Middleton  WD, Grant  EG,  et al.  ACR Thyroid Imaging, Reporting and Data System (TI-RADS): white paper of the ACR TI-RADS Committee.   J Am Coll Radiol. 2017;14(5):587-595. doi:10.1016/j.jacr.2017.01.046 PubMedGoogle ScholarCrossref
11.
Grant  EG, Tessler  FN, Hoang  JK,  et al.  Thyroid ultrasound reporting lexicon: white paper of the ACR Thyroid Imaging, Reporting and Data System (TIRADS) committee.   J Am Coll Radiol. 2015;12(12, pt A):1272-1279. doi:10.1016/j.jacr.2015.07.011 PubMedGoogle ScholarCrossref
12.
Russ  G, Bonnema  SJ, Erdogan  MF, Durante  C, Ngu  R, Leenhardt  L.  European Thyroid Association guidelines for ultrasound malignancy risk stratification of thyroid nodules in adults: the EU-TIRADS.   Eur Thyroid J. 2017;6(5):225-237. doi:10.1159/000478927 PubMedGoogle ScholarCrossref
13.
Gharib  H, Papini  E, Garber  JR,  et al; AACE/ACE/AME Task Force on Thyroid Nodules.  American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules—2016 update.   Endocr Pract. 2016;22(5):622-639. doi:10.4158/EP161208.GL PubMedGoogle Scholar
14.
Shin  JH, Baek  JH, Chung  J,  et al; Korean Society of Thyroid Radiology (KSThR) and Korean Society of Radiology.  Ultrasonography diagnosis and imaging-based management of thyroid nodules: revised Korean Society of Thyroid Radiology consensus statement and recommendations.   Korean J Radiol. 2016;17(3):370-395. doi:10.3348/kjr.2016.17.3.370 PubMedGoogle ScholarCrossref
15.
Mistry  R, Hillyar  C, Nibber  A, Sooriyamoorthy  T, Kumar  N.  Ultrasound classification of thyroid nodules: a systematic review.   Cureus. 2020;12(3):e7239. doi:10.7759/cureus.7239 PubMedGoogle Scholar
16.
Castellana  M, Castellana  C, Treglia  G,  et al.  Performance of five ultrasound risk stratification systems in selecting thyroid nodules for FNA.   J Clin Endocrinol Metab. 2020;105(5):1-11. doi:10.1210/clinem/dgz170 PubMedGoogle ScholarCrossref
17.
Lauria Pantano  A, Maddaloni  E, Briganti  SI,  et al.  Differences between ATA, AACE/ACE/AME and ACR TI-RADS ultrasound classifications performance in identifying cytological high-risk thyroid nodules.   Eur J Endocrinol. 2018;178(6):595-603. doi:10.1530/EJE-18-0083 PubMedGoogle ScholarCrossref
18.
Grani  G, Lamartina  L, Ascoli  V,  et al.  Reducing the number of unnecessary thyroid biopsies while improving diagnostic accuracy: toward the “right” TIRADS.   J Clin Endocrinol Metab. 2019;104(1):95-102. doi:10.1210/jc.2018-01674 PubMedGoogle ScholarCrossref
19.
Hoang  JK, Middleton  WD, Farjat  AE,  et al.  Reduction in thyroid nodule biopsies and improved accuracy with American College of Radiology Thyroid Imaging Reporting and Data System.   Radiology. 2018;287(1):185-193. doi:10.1148/radiol.2018172572 PubMedGoogle ScholarCrossref
20.
Wu  XL, Du  JR, Wang  H,  et al.  Comparison and preliminary discussion of the reasons for the differences in diagnostic performance and unnecessary FNA biopsies between the ACR TIRADS and 2015 ATA guidelines.   Endocrine. 2019;65(1):121-131. doi:10.1007/s12020-019-01886-0 PubMedGoogle ScholarCrossref
21.
Solorzano  CC, Carneiro  DM, Ramirez  M, Lee  TM, Irvin  GL. Surgeon-performed ultrasound in the management of thyroid malignancy.  Am Surg. 2004;70(7):576-582.
22.
Gu WX, Tan CS, Ho TWT. Surgeon-performed ultrasound-guided fine-needle aspiration cytology (SP-US-FNAC) shortens time for diagnosis of thyroid nodules.  Ann Acad Med Singap. 2014;43(6):320-324.
23.
Lee  CY, Snyder  SK, Lairmore  TC, Dupont  SC, Jupiter  DC.  Utility of surgeon-performed ultrasound assessment of the lateral neck for metastatic papillary thyroid cancer.   J Oncol. 2012;2012:973124. doi:10.1155/2012/973124 PubMedGoogle Scholar
24.
Robitschek  J, Straub  M, Wirtz  E, Klem  C, Sniezek  J.  Diagnostic efficacy of surgeon-performed ultrasound-guided fine needle aspiration: a randomized controlled trial.   Otolaryngol Head Neck Surg. 2010;142(3):306-309. doi:10.1016/j.otohns.2009.11.011 PubMedGoogle ScholarCrossref
25.
Ahn  D, Kim  H, Sohn  JH, Choi  JH, Na  KJ.  Surgeon-performed ultrasound-guided fine-needle aspiration cytology of head and neck mass lesions: sampling adequacy and diagnostic accuracy.   Ann Surg Oncol. 2015;22(4):1360-1365. doi:10.1245/s10434-014-4119-2 PubMedGoogle ScholarCrossref
26.
Rozycki  GS.  Surgeon-performed ultrasound: its use in clinical practice.   Ann Surg. 1998;228(1):16-28. doi:10.1097/00000658-199807000-00004 PubMedGoogle ScholarCrossref
27.
Hamill  C, Ellis  PK, Johnston  PC.  Point of care thyroid ultrasound (POCUS) in endocrine outpatients: a pilot study.   Ulster Med J. 2020;89(1):21-24.PubMedGoogle Scholar
28.
Karkada  M, Costa  AF, Imran  SA,  et al.  Incomplete thyroid ultrasound reports for patients with thyroid nodules: implications regarding risk assessment and management.   AJR Am J Roentgenol. 2018;211(6):1348-1353. doi:10.2214/AJR.18.20056 PubMedGoogle ScholarCrossref
29.
Koseoglu Atilla  FD, Ozgen Saydam  B, Erarslan  NA,  et al.  Does the ACR TI-RADS scoring allow us to safely avoid unnecessary thyroid biopsy? single center analysis in a large cohort.   Endocrine. 2018;61(3):398-402. doi:10.1007/s12020-018-1620-6 PubMedGoogle ScholarCrossref
30.
Ogrinc  G, Davies  L, Goodman  D, Batalden  P, Davidoff  F, Stevens  D.  SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process.   BMJ Qual Saf. 2016;25(12):986-992. doi:10.1136/bmjqs-2015-004411 PubMedGoogle ScholarCrossref
31.
Penfold  RB, Zhang  F.  Use of interrupted time series analysis in evaluating health care quality improvements.   Acad Pediatr. 2013;13(6)(suppl):S38-S44. doi:10.1016/j.acap.2013.08.002 PubMedGoogle ScholarCrossref
32.
Shrestha  M, Crothers  BA, Burch  HB.  The impact of thyroid nodule size on the risk of malignancy and accuracy of fine-needle aspiration: a 10-year study from a single institution.   Thyroid. 2012;22(12):1251-1256. doi:10.1089/thy.2012.0265 PubMedGoogle ScholarCrossref
33.
Cavallo  A, Johnson  DN, White  MG,  et al.  Thyroid nodule size at ultrasound as a predictor of malignancy and final pathologic size.   Thyroid. 2017;27(5):641-650. doi:10.1089/thy.2016.0336 PubMedGoogle ScholarCrossref
34.
Kamran  SC, Marqusee  E, Kim  MI,  et al.  Thyroid nodule size and prediction of cancer.   J Clin Endocrinol Metab. 2013;98(2):564-570. doi:10.1210/jc.2012-2968 PubMedGoogle ScholarCrossref
35.
Dobruch-Sobczak  K, Migda  B, Krauze  A,  et al.  Prospective analysis of inter-observer and intra-observer variability in multi ultrasound descriptor assessment of thyroid nodules.   J Ultrason. 2019;19(78):198-206. doi:10.15557/JoU.2019.0030 PubMedGoogle ScholarCrossref
36.
Moon  WJ, Jung  SL, Lee  JH,  et al; Thyroid Study Group, Korean Society of Neuro- and Head and Neck Radiology.  Benign and malignant thyroid nodules: US differentiation—multicenter retrospective study.   Radiology. 2008;247(3):762-770. doi:10.1148/radiol.2473070944 PubMedGoogle ScholarCrossref
37.
Ren  J, Liu  B, Zhang  LL,  et al.  A taller-than-wide shape is a good predictor of papillary thyroid carcinoma in small solid nodules.   J Ultrasound Med. 2015;34(1):19-26. doi:10.7863/ultra.34.1.19 PubMedGoogle ScholarCrossref
38.
Kumbhar  SS, O’Malley  RB, Robinson  TJ,  et al.  Why thyroid surgeons are frustrated with radiologists: lessons learned from pre- and postoperative US.   Radiographics. 2016;36(7):2141-2153. doi:10.1148/rg.2016150250 PubMedGoogle ScholarCrossref
39.
Leis  JA, Shojania  KG.  A primer on PDSA: executing Plan-Do-Study-Act cycles in practice, not just in name.   BMJ Qual Saf. 2017;26(7):572-577. doi:10.1136/bmjqs-2016-006245 PubMedGoogle ScholarCrossref
40.
Rideout  M, Held  M, Holmes  AV.  The didactic makeover: keep it short, active, relevant.   Pediatrics. 2016;138(1):e20160751. doi:10.1542/peds.2016-0751 PubMedGoogle Scholar
41.
Semler  MW, Keriwala  RD, Clune  JK,  et al.  A randomized trial comparing didactics, demonstration, and simulation for teaching teamwork to medical residents.   Ann Am Thorac Soc. 2015;12(4):512-519. doi:10.1513/AnnalsATS.201501-030OC PubMedGoogle ScholarCrossref
42.
Middleton  WD, Teefey  SA, Reading  CC,  et al.  Comparison of performance characteristics of American College of Radiology TI-RADS, Korean Society of Thyroid Radiology TIRADS, and American Thyroid Association guidelines.   AJR Am J Roentgenol. 2018;210(5):1148-1154. doi:10.2214/AJR.17.18822 PubMedGoogle ScholarCrossref
43.
Toma  M, Dreischulte  T, Gray  NM, Campbell  D, Guthrie  B.  Balancing measures or a balanced accounting of improvement impact: a qualitative analysis of individual and focus group interviews with improvement experts in Scotland.   BMJ Qual Saf. 2018;27(7):547-556. doi:10.1136/bmjqs-2017-006554 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    February 11, 2021

    Association of the Implementation of a Standardized Thyroid Ultrasonography Reporting Program With Documentation of Nodule Characteristics

    Author Affiliations
    • 1Department of Otolaryngology–Head and Neck Surgery, University of Toronto, Toronto, Ontario, Canada
    • 2Department of Otolaryngology–Head and Neck Surgery, Department of Surgery, The Scarborough Hospital, Toronto, Ontario, Canada
    • 3Department of Otolaryngology–Head and Neck Surgery, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
    • 4Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
    • 5Department of Otolaryngology–Head and Neck Surgery, Sinai Health System, Toronto, Ontario, Canada
    • 6Department of Otolaryngology–Head and Neck Surgery/Surgical Oncology, University Health Network, Toronto, Ontario, Canada
    JAMA Otolaryngol Head Neck Surg. 2021;147(4):343-349. doi:10.1001/jamaoto.2020.5233
    Key Points

    Question  How can the documentation rate of American College of Radiology Thyroid Imaging Reporting and Data System thyroid nodule ultrasonographic characteristics in a high-volume tertiary head and neck endocrine clinic be improved?

    Findings  In this quality improvement study that included 229 patients who underwent thyroid ultrasonography, documentation of thyroid nodule characteristics increased from 34% at baseline to 90% during 3 Plan, Do, Study, Act cycles, a result that was statistically significant.

    Meaning  This study suggests that a comprehensive thyroid ultrasonography documentation system that includes a standardized reporting form can substantially improve clinic-wide documentation of thyroid nodule characteristics.

    Abstract

    Importance  Although most thyroid nodules are benign, the potential for malignant neoplasms is associated with unnecessary workup in the form of imaging, fine-needle aspiration, and diagnostic surgery. The American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) is commonly used to assess the malignant neoplasm risk potential of thyroid nodules imaged by ultrasonography. However, standardized reporting of ACR TI-RADS descriptors is inconsistent.

    Objective  To increase the documentation rate of ACR TI-RADS thyroid nodule characteristics to 80% in 18 months.

    Design, Setting, and Participants  This prospective interrupted time series quality improvement study was conducted from December 1, 2018, to March 31, 2020, at a tertiary outpatient head and neck clinic among 229 patients who had at least 1 documented thyroid nodule identified on bedside clinic ultrasonography. Data analysis was performed throughout the entire study period because this was a quality improvement study with iterative small cycle changes; final analysis of the data was performed in April 2020.

    Main Outcomes and Measures  The primary outcome was the documentation rates of 6 ACR TI-RADS ultrasonographic descriptors. Secondary outcomes included nodule fine-needle aspiration biopsy rate and physician-reported clinic flow efficiency.

    Results  A total of 229 patients had at least 1 documented thyroid nodule and were included in the analysis. Size was the most frequently documented nodule characteristic (72 of 74 [97.3%]) at baseline, followed by echogenic foci (31 of 74 [41.9%]), composition (23 of 74 [31.1%]), echogenicity (17 of 74 [23.0%]), margin (6 of 74 [8.1%]), and shape (1 of 74 [1.4%]). After 3 Plan, Do, Study, Act (PDSA) cycles, the final intervention consisted of a standardized ultrasonography reporting form and educational initiatives for surgical trainees. After the third PDSA cycle (n = 36), reporting of nodule size, echogenic foci, and composition increased to 100%. Similarly, reporting of echogenicity (34 of 36 [94.4%]), shape (28 of 36 [77.8%]), and margin (25 of 36 [69.4%]) all increased. This represented a cumulative 90.3% documentation rate (195 of 216), a 56.5% increase from baseline (95% CI, 50.0%-61.9%). The standardized reporting form was used in 83.3% of eligible thyroid ultrasonography cases (30 of 36) after PDSA cycle 3, demonstrating good fidelity of implementation. There were no unintended consequences associated with clinic workflow, as a balancing measure, reported by staff surgeons.

    Conclusions and Relevance  This study suggests that implementation of an ACR TI-RADS–based reporting form in conjunction with educational initiatives improved documentation of ultrasonographic thyroid nodule characteristics, potentially allowing for improved bedside risk stratification and communication among clinicians.

    Introduction

    Investigation of thyroid nodules represents a clinical dilemma for physicians. The increase in use of thyroid ultrasonography by primary care practitioners, radiologists, and endocrine clinicians has led to the discovery of an abundance of asymptomatic thyroid nodules, which occur in up to 67% of adults.1 Although 85% to 98% of thyroid nodules are benign, the potential for malignant neoplasms is associated with increased unnecessary workup in the form of imaging, fine-needle aspiration (FNA), and diagnostic surgery.2-4 This presents the well-described dilemma of overinvestigation, which is potentially associated with increased patient morbidity without improving mortality rates from thyroid cancer, coupled with an increased systemwide cost burden.5-8 To address this dilemma, many organizations have described systems to facilitate communication and decision-making around thyroid nodule workup, based on ultrasonographic findings. These include the American Thyroid Association management guidelines,9 the American College of Radiology (ACR) Thyroid Imaging Reporting and Data System (TI-RADS),10,11 and the European Thyroid Association guidelines,12 among others.13,14 Comparison of these guidelines reveals that the ACR TI-RADS performs best in identifying cytologically high-risk nodules in adult patients.15-17 In addition, use of the ACR TI-RADS has been shown to improve interdisciplinary communication about thyroid nodules imaged by ultrasonography2 and reduce the rate of subsequent unnecessary FNA biopsies.18-20

    Recent advances in bedside ultrasonography have allowed surgeons to obtain point-of-care imaging for thyroid nodules,21-26 expediting patient care by endocrinologists and surgeons in a cost-effective manner. Such point-of-care thyroid imaging by nonradiologists may be equal in quality to reports from radiologists when a standardized reporting system is used.27 However, despite the presence of useful clinical guidelines, the reporting of thyroid nodule characteristics remains a nonstandardized process in most clinical settings. Several studies have revealed widespread underreporting of clinically important nodule characteristics in thyroid ultrasonography reports, causing discrepancy among clinicians regarding the risk of malignant neoplasms and the need for biopsy.28 To better integrate the ACR TI-RADS or other guidelines into the clinical workflow, standardized reporting systems have been used to characterize both benign and malignant thyroid nodules.2,18,29

    Sunnybrook Health Sciences Centre is a 1300-bed tertiary care hospital in Toronto, Ontario, Canada. It houses a large-volume head and neck endocrine service that performs a high number of clinic thyroid ultrasonographic procedures. From December 1, 2018, to January 31, 2019, the documentation rate of ACR TI-RADS ultrasonography descriptors was approximately 34% based on preliminary review.

    The evidence-based utility of the ACR TI-RADS for assessing thyroid nodules14,15 led us to develop a quality improvement initiative aimed at increasing the documentation rate of ACR TI-RADS ultrasonographic descriptors. This system has been demonstrated to improve bedside assessment of malignant neoplasm risk potential, promote interdisciplinary communication regarding next steps in management, and avoid unnecessary further testing.2,15-20

    First, we aimed to characterize the baseline reporting rate of ACR TI-RADS ultrasonographic characteristics of thyroid nodules. After this, the primary objective was to improve bundle reporting of all 6 ACR TI-RADS ultrasonographic characteristics of thyroid nodules to 80% in 18 months, a threshold targeted a priori, as assessed by a multidisciplinary working group. Our secondary objective was to evaluate if our interventions were associated with any changes to the rate of thyroid nodule FNA biopsy offered or clinic flow and efficiency.

    Methods
    Study Design and Setting

    A prospective interrupted time series quality improvement study was undertaken from December 1, 2018, to March 31, 2020. Ethics approval was obtained from the Sunnybrook Health Sciences Centre Research Ethics Board, which considered this project a quality improvement initiative and waived patient consent. Study design and reporting of findings followed the principles detailed in the Standards in Quality Improvement Reporting Excellence (SQUIRE) 2.0 guidelines for prospective interrupted time series quality improvement studies.30

    Sunnybrook Health Sciences Centre is one of the main academic hospitals within the University of Toronto’s Otolaryngology–Head and Neck Surgery training program. This center houses 3 otolaryngologists with subspecializations in head and neck endocrine surgery, who perform a high volume of bedside thyroid ultrasonographies and biopsies. These faculty train more than 20 residents and fellows annually. These contextual factors helped identify interventions for increasing thyroid ultrasonography reporting rates via Plan, Do, Study, Act (PDSA) cycles.

    PDSA Cycle 1: Baseline Metrics, Stakeholder Engagement, and Barrier Analysis

    Baseline data were collected from December 1, 2018, to January 31, 2019, to assess the degree of underdocumentation of thyroid nodule characteristics on ultrasonography. Given the low rate, a multidisciplinary working group was convened in February 2019. Representation from key stakeholders included 5 otolaryngology–head and neck surgery residents, 13 staff head and neck endocrine surgeons, 1 clinic nurse, and 2 administrative staff members. Each group member elicited feedback from colleagues regarding barriers to thyroid ultrasonography documentation. During 2 multidisciplinary stakeholder meetings, barriers to complete reporting of thyroid ultrasonographic descriptors were identified (Figure). These included (1) time constraints for surgical staff and trainees, (2) lack of knowledge among trainees surrounding thyroid ultrasonographic characterization systems, and (3) practice variation within the multiple clinical sites. Some surgeons used their own nonstandardized forms for documentation, while others simply dictated their ultrasonography findings. Using the barrier analysis, the working group identified several strategies and implemented them after a consensus discussion of feasibility and potential effectiveness.

    The initial intervention focused on education-based initiatives to improve knowledge among trainees and normalize documentation of ACR TI-RADS ultrasonographic characteristics. These interventions were designed to (1) provide education about different reporting systems for thyroid nodule ultrasonography, (2) identify potential barriers among clinicians in reporting these data, and (3) improve engagement and awareness of thyroid ultrasonography documentation standards within the clinic. A resident lead was assigned to remind trainees and staff surgeons about the ACR TI-RADS descriptors once weekly prior to clinics. During this time, results from the baseline data collection were also disseminated among the team. In addition, a resident-led didactic teaching session about thyroid nodule sonography was held at Sunnybrook Health Sciences Centre in February 2019. Finally, an outline of the ACR TI-RADS criteria was posted in each clinic ultrasonography room (Table 1). The effectiveness of these initial interventions was evaluated in March 2019.

    PDSA Cycle 2

    After modest improvements with the initial intervention, a second quality improvement meeting was held. At this meeting, it was thought that the didactic initiatives did not adequately improve the ultrasonographic reporting rates. Furthermore, the current improvements were unlikely to be sustainable, particularly with the ongoing rotations of clinical teams. To address these critiques, the quality improvement team proposed the use of a computer-based or print-based standardized reporting form that was based on the ACR TI-RADS descriptors (Table 1). This second intervention stemmed from the change concepts of improving workflow and changing the work environment, and was designed to (1) act as an educational aid for junior trainees who were less comfortable with thyroid ultrasonography, (2) provide a structured platform for standardized assessments and serial monitoring of thyroid nodules, and (3) act as a valuable reminder for clinicians to document thyroid nodule descriptors using either the printed form, or within their clinical dictations. This form was developed by a subcommittee composed of 2 residents and a staff surgeon (from a separate academic site). An initial draft was disseminated among the working group in April 2019. After 6 rounds of usability testing and revisions, the standardized form was finalized in October 2019 and implemented for clinical use in November 2019 (eFigure in the Supplement). After a short 2-week adjustment period, the overall effectiveness of this second intervention was evaluated from December 1, 2019, to January 31, 2020.

    PDSA Cycle 3

    After substantial improvement in documentation rates, a third PDSA cycle was initiated owing to low use of the standardized form. Although reporting rates increased significantly, they did not meet the targeted 80% reporting threshold. To address these issues, a meeting was held with the quality improvement team, where barriers surrounding form accessibility, storage protocols, and trainee education were identified. This meeting led to interventions that included placing forms in visible and easily accessible locations, sending out electronic messages reminding administrative staff to include the reporting form in patient medical records, and incorporating the standardized reporting form with routine bedside ultrasonography teaching (Table 1). The effectiveness of these interventions was evaluated in March 2020.

    Study of the Interventions

    Interventions were assessed using iterative PDSA cycles. Reporting rates were assessed at the end of each month by retrospective medical record review, at which time discussion and refinement of the PDSA implementation measures were conducted.

    Measures and Statistical Analysis

    Statistical analysis was performed throughout the entire study period because this was a quality improvement study with iterative small cycle changes; final analysis of the data was performed in April 2020. The primary outcome was documentation rates of ultrasonographic features of thyroid nodules, based on the 6 ACR TI-RADS descriptors. A cumulative completeness score was calculated. Secondary outcomes included use rates of the implemented reporting form (fidelity measure) and thyroid biopsy rates (rate of a biopsy being offered to the patient). We collected data from December 1, 2018, to March 30, 2020, using an interrupted time series design.31 The preintervention period was from December 1, 2018, to January 31, 2019. Postintervention periods were from March 1 to 30, 2019, December 1, 2019, to January 31, 2020, and March 1 to 30, 2020. Statistical process control was considered, but ultimately not used owing to incompatibility with clinic flow as well as a lack of appropriate resources. At the conclusion of the final PDSA cycle, a post hoc questionnaire was given to all 3 attending head and neck endocrine surgeons within the study clinic to evaluate potential unintended consequences on clinic workflow (Box). Effect size metrics and 95% CIs around the effect size were used to describe the magnitude of the difference and strength of association for compared variables. Analysis was performed using SPSS, version 26.0 (SPSS Inc).

    Box Section Ref ID
    Box.

    Questionnaire Delivered to Attending Head and Neck Endocrine Surgeons After the Final PDSA Cycle

    Post hoc questionnaire
    1. Has this initiative’s interventions (including the reporting form) impacted your clinic workflow in a negative way? If yes, how so?

    2. Does it take longer to see patients and get through your clinic because of the changes made?

    3. Do you think that you are more selective in who you offer biopsy because of the changes made (including the reporting form)?

    4. Overall, do you think that the reporting form is beneficial to your practice?

    Abbreviation: PDSA, Plan, Do, Study, Act.

    Results

    During the study period, a total of 278 patients underwent clinic thyroid ultrasonography. Of these, 229 patients had at least 1 documented thyroid nodule and were therefore included in the analysis. Seventy-four patients were included in the baseline cohort. Size was the most frequently documented nodule characteristic (72 of 74 [97.3%]), followed by echogenic foci (31 of 74 [41.9%]), composition (23 of 74 [31.1%]), echogenicity (17 of 74 [23.0%]), margin (6 of 74 [8.1%]), and shape (1 of 74 [1.4%]) (Table 2). Cumulatively, this represented a 33.8% documentation rate (150 of 444). In addition, the baseline rate of nodule FNA biopsy offered was 35.1% (26 of 74).

    After PDSA cycle 1, the cumulative documentation rate increased by 6.8% (95% CI, −0.7% to 14.5%) (Table 2). The rate of nodule FNA biopsy offered was 25.6% (10 of 39). This increase resulted in the development of a standardized thyroid ultrasonography reporting form (eFigure in the Supplement), which was subsequently implemented and studied in PDSA cycle 2. Implementation of the form increased the cumulative documentation rate by 29.0% (95% CI, 21.3%-36.2%), despite the form being used in only 53.3% of eligible cases (43 of 80). The rate of nodule FNA biopsy offered during PDSA cycle 2 was 26.3% (21 of 80).

    In PDSA cycle 3, the low use of the standardized reporting form was addressed by incorporating the form with routine bedside ultrasonography teaching and placing the form in a receptacle adjacent to the ultrasonography machine. These changes improved reporting of nodule size, echogenic foci, and composition to 100% (Table 2). Similar increases were seen in reporting of echogenicity (34 of 36 [94.4%]), shape (28 of 36 [77.8%]), and margin (25 of 36 [69.4%]). This represented a 90.3% documentation rate (195 of 216), a 56.5% absolute increase from baseline (95% CI, 50.0%-61.9%). After the third PDSA cycle, the form was used in 83.3% (30 of 36) of eligible thyroid ultrasonography cases (absolute increase of 29.5%; 95% CI, 11.1%-43.6%), showing a significant improvement in the fidelity of our implementation. Overall, there was a modest reduction in FNA biopsies offered between baseline (26 of 74 [35.1%]) and postintervention rates (44 of 155 [28.4%]), an absolute difference of 6.8% (95% CI, −5.7% to 19.8%). The post hoc survey directed to Sunnybrook head and neck endocrine surgeons had a 100% response rate (3 of 3). All surgeons agreed that the interventions did not negatively affect their clinic workflow and had no association with clinic duration (balancing measures). In addition, 2 of the 3 surgeons believed that they could better assess which patients would benefit from biopsy of their thyroid nodule. Finally, all 3 surgeons viewed the reporting form as beneficial to their practice.

    Discussion

    During an 18-month period, this quality improvement initiative used repeated stakeholder engagement and ongoing evaluation of interventions to improve reporting of ACR TI-RADS ultrasonographic characteristics of thyroid nodules from 33.8% to 90.3%. Analysis of baseline data revealed that only size was reliably documented when assessing thyroid nodules by ultrasonography. The association of thyroid size with malignant neoplasm risk remains unclear, with some studies finding an inverse association32 and others demonstrating a 2.0-cm threshold after which malignant neoplasm risk is reduced.33,34 Despite this, size is the most readily assessed characteristic by ultrasonography, and is often the focus of patients during clinical assessments. The ACR TI-RADS descriptors, including echogenic foci, composition, and echogenicity, were reported less than half the time. Margin and shape were initially reported in less than 10% of thyroid ultrasonography cases. After the final PDSA cycle, margin and shape showed the largest percentage improvement in reporting but were still the least frequently reported ACR TI-RADS descriptors. These findings were similar to initiatives aimed at improving ultrasonography reporting among radiologists.2 Ultrasonographic characterization of thyroid nodule margins is considered more technically challenging, particularly if the surrounding thyroid gland is heterogeneous or nodule borders overlap.35 Assessment of margins is thus subject to a high degree of interobserver variability, causing reduced reporting frequency.36 For the ACR TI-RADS, thyroid nodule shape reporting is a binary output and, while it is highly specific for thyroid cancer, it lacks overall sensitivity.36-38 We also found reduced reporting of thyroid nodule shape, particularly for large thyroid nodules.

    The interventions implemented within each PDSA cycle aimed to address the multiple barriers we had identified in the pre-PDSA cycle barrier analysis (Figure). Learnings from prior PDSA cycles led to the subsequent intervention changes tested in the next PDSA cycle (authentic PDSA).39 Overarching themes addressed included surgical trainee factors, attending surgeon factors, clinic policy, and clinic environment factors. Our initial education-based intervention was the easiest to implement, but was expectedly the least effective. Didactic teaching shows minimal information retention40 and no improvement in clinical performance.41 Our second intervention was to implement a standardized reporting form, with minimal instruction to clinicians and administrative staff, which allowed only passive uptake by clinicians and relied on them remembering to use the form during thyroid ultrasonography, resulting in poor fidelity of implementation. When we improved visibility of the form, reminded staff to include the forms within patient medical records, and introduced the form during bedside ultrasonography teaching, the reporting rates of ACR TI-RADS descriptors significantly increased. We associate these findings with improved workflow integration of the standardized reporting form, making it easier for the surgeon to use without much or any additional work, and increased overall acceptance of the initiative. As some of the surgeons were already using a nonstandardized form for documentation prior to this study, this intervention was met with approval.

    Improved documentation of thyroid nodule descriptors is often associated with a reduction in overall FNA biopsy rate, resulting in reduced patient morbidity.18-20,28,42 In our study, there was a modest reduction in FNA biopsies offered between baseline and PDSA cycle 3, in line with previous literature. More important, our study may have been underpowered to detect statistical significance. Our poststudy survey revealed that the benefits associated with the initiative did not come at the cost of perceived clinic workflow. All 3 attending staff surgeons surveyed did not believe that the interventions were associated with increased clinic duration time. In addition, each staff surgeon viewed the reporting form as a useful adjunct for documentation. Although subjective, these balancing measures indicate that there were no overt undesirable consequences of the initiative, which is important in understanding trade-offs.43 Such ongoing engagement of key stakeholders is critical to the ongoing success and sustainability of this initiative, beyond the scope of this study.

    Limitations

    There are limitations in our study design that merit discussion. First, single-center studies may reduce generalizability and require further evaluation. Our findings are in keeping with prior literature reports but may be more readily implemented in an academic otolaryngology practice that is familiar with quality-based initiatives. Second, the outcomes after our final PDSA cycle were assessed after 1 month of implementation. Longer follow-up and repeated data collection are required to assess whether the observed changes are durable. Ongoing assessment of the interventions herein is imperative to ensure sustainability of the desired outcomes. Third, it is unclear whether our interventions improved patient understanding of their individual malignant neoplasm risk or was associated with their decision-making process in surgical management. It is unknown whether implementation of a standardized reporting form improved clinical outcomes of patients with thyroid nodules. Fourth, studies without a rigorously defined control group may allow for external factors to unexpectedly influence the observed results, which is an inherent potential bias in interrupted time series study designs. Use of statistical process control as part of the methods would have addressed this concern and allowed us to measure secular trends. The inability to directly associate outcomes in reporting and documentation with our respective interventions is a limitation of this study. Despite this, the temporal association between the observed outcomes with our intervention, as well as our repeated stakeholder engagement, are all supportive of the interventions. Multimodal programs composed of implementation of standardized reporting forms with educational initiatives and improvements in workflow represent a low-cost intervention that improves documentation rate among clinicians. This study was undertaken within a specific hospital system and, while the general learnings from this study may be applicable to other centers, documentation standards may differ based on jurisdiction and location of practice.

    As point-of-care ultrasonography becomes more commonly used, appropriate documentation standards are necessary to ensure that patient safety is prioritized. The methodological concepts demonstrated in this study may be extended to other medical uses of point-of-care ultrasonography, with the goal of optimizing documentation and improving quality of care.

    Conclusions

    This quality improvement study used repeated stakeholder engagement in the setting of 3 PDSA cycles to improve documentation of ACR TI-RADS ultrasonographic thyroid nodule characteristics in an outpatient clinic. Through educational initiatives, a standardized reporting form, and enhanced training for all individuals involved with patient care, documentation rates of ACR TI-RADS criteria increased. Such improved levels of documentation allow for improved bedside risk stratification and communication among clinicians.

    Back to top
    Article Information

    Accepted for Publication: November 28, 2020.

    Published Online: February 11, 2021. doi:10.1001/jamaoto.2020.5233

    Corresponding Author: Eric Monteiro, MD, MSc, Department of Otolaryngology–Head and Neck Surgery, University of Toronto, 600 University Ave, Ste 401, Toronto, ON M5G 1X5, Canada (eric.monteiro@sinaihealth.ca).

    Author Contributions: Drs Hamour and Monteiro had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Hamour, Yang, Lee, Wu, Ziai, Singh, Eskander, Sahovaler, Witterick, Vescan, Freeman, de Almeida, Chepeha, Irish, Enepekides, Monteiro.

    Acquisition, analysis, or interpretation of data: Hamour, Yang, Lee, Wu, Ziai, Eskander, Higgins, de Almeida, Goldstein, Gilbert, Irish, Monteiro.

    Drafting of the manuscript: Hamour, Yang, Lee, Wu, Ziai, Freeman, de Almeida, Irish, Monteiro.

    Critical revision of the manuscript for important intellectual content: Hamour, Yang, Lee, Wu, Ziai, Singh, Eskander, Sahovaler, Higgins, Witterick, Vescan, Goldstein, Gilbert, Chepeha, Irish, Enepekides, Monteiro.

    Statistical analysis: Hamour, Yang, Lee, Wu, Ziai, Eskander.

    Obtained funding: Wu, Ziai, Monteiro.

    Administrative, technical, or material support: Wu, Singh, Eskander, Higgins, Witterick, Vescan, de Almeida, Gilbert, Chepeha, Enepekides, Monteiro.

    Supervision: Higgins, Witterick, Freeman, Irish, Monteiro.

    Conflict of Interest Disclosures: Dr Eskander reported receiving research funds from Merck and serving as a consultant for Bristol Myers Squibb. No other disclosures were reported.

    Meeting Presentation: This work was presented at the 2020 Canadian Society of Otolaryngology–Head and Neck Surgery Annual Meeting; November 1, 2020; Toronto, Ontario, Canada.

    Additional Contributions: We would like to acknowledge Nuzhat Sheikh and Elena Gennaro for their tireless efforts in helping us implement the study interventions within the otolaryngology clinic at Sunnybrook Hospital. They were not compensated for their contributions.

    References
    1.
    Ezzat  S, Sarti  DA, Cain  DR, Braunstein  GD.  Thyroid incidentalomas: prevalence by palpation and ultrasonography.   Arch Intern Med. 1994;154(16):1838-1840. doi:10.1001/archinte.1994.00420160075010 PubMedGoogle ScholarCrossref
    2.
    Griffin  AS, Mitsky  J, Rawal  U, Bronner  AJ, Tessler  FN, Hoang  JK.  Improved quality of thyroid ultrasound reports after implementation of the ACR Thyroid Imaging Reporting and Data System nodule lexicon and risk stratification system.   J Am Coll Radiol. 2018;15(5):743-748. doi:10.1016/j.jacr.2018.01.024 PubMedGoogle ScholarCrossref
    3.
    Smith-Bindman  R, Lebda  P, Feldstein  VA,  et al.  Risk of thyroid cancer based on thyroid ultrasound imaging characteristics: results of a population-based study.   JAMA Intern Med. 2013;173(19):1788-1796. doi:10.1001/jamainternmed.2013.9245 PubMedGoogle ScholarCrossref
    4.
    Davies  L, Ouellette  M, Hunter  M, Welch  HG.  The increasing incidence of small thyroid cancers: where are the cases coming from?   Laryngoscope. 2010;120(12):2446-2451. doi:10.1002/lary.21076 PubMedGoogle ScholarCrossref
    5.
    Hoang  JK, Langer  JE, Middleton  WD,  et al.  Managing incidental thyroid nodules detected on imaging: white paper of the ACR Incidental Thyroid Findings Committee.   J Am Coll Radiol. 2015;12(2):143-150. doi:10.1016/j.jacr.2014.09.038 PubMedGoogle ScholarCrossref
    6.
    Youserm  DM, Huang  T, Loevner  LA, Langlotz  CP.  Clinical and economic impact of incidental thyroid lesions found with CT and MR.   AJNR Am J Neuroradiol. 1997;18(8):1423-1428.PubMedGoogle Scholar
    7.
    Wong  CKH, Liu  X, Lang  BHH.  Cost-effectiveness of fine-needle aspiration cytology (FNAC) and watchful observation for incidental thyroid nodules.   J Endocrinol Invest. 2020;43(11):1645-1654. doi:10.1007/s40618-020-01254-0 PubMedGoogle ScholarCrossref
    8.
    Ho  AS, Davies  L, Nixon  IJ,  et al.  Increasing diagnosis of subclinical thyroid cancers leads to spurious improvements in survival rates.   Cancer. 2015;121(11):1793-1799. doi:10.1002/cncr.29289 PubMedGoogle ScholarCrossref
    9.
    Haugen  BR, Alexander  EK, Bible  KC,  et al.  American Thyroid Association Management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer.   Thyroid. 2015;2016. doi:10.1089/thy.2015.0020PubMedGoogle Scholar
    10.
    Tessler  FN, Middleton  WD, Grant  EG,  et al.  ACR Thyroid Imaging, Reporting and Data System (TI-RADS): white paper of the ACR TI-RADS Committee.   J Am Coll Radiol. 2017;14(5):587-595. doi:10.1016/j.jacr.2017.01.046 PubMedGoogle ScholarCrossref
    11.
    Grant  EG, Tessler  FN, Hoang  JK,  et al.  Thyroid ultrasound reporting lexicon: white paper of the ACR Thyroid Imaging, Reporting and Data System (TIRADS) committee.   J Am Coll Radiol. 2015;12(12, pt A):1272-1279. doi:10.1016/j.jacr.2015.07.011 PubMedGoogle ScholarCrossref
    12.
    Russ  G, Bonnema  SJ, Erdogan  MF, Durante  C, Ngu  R, Leenhardt  L.  European Thyroid Association guidelines for ultrasound malignancy risk stratification of thyroid nodules in adults: the EU-TIRADS.   Eur Thyroid J. 2017;6(5):225-237. doi:10.1159/000478927 PubMedGoogle ScholarCrossref
    13.
    Gharib  H, Papini  E, Garber  JR,  et al; AACE/ACE/AME Task Force on Thyroid Nodules.  American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules—2016 update.   Endocr Pract. 2016;22(5):622-639. doi:10.4158/EP161208.GL PubMedGoogle Scholar
    14.
    Shin  JH, Baek  JH, Chung  J,  et al; Korean Society of Thyroid Radiology (KSThR) and Korean Society of Radiology.  Ultrasonography diagnosis and imaging-based management of thyroid nodules: revised Korean Society of Thyroid Radiology consensus statement and recommendations.   Korean J Radiol. 2016;17(3):370-395. doi:10.3348/kjr.2016.17.3.370 PubMedGoogle ScholarCrossref
    15.
    Mistry  R, Hillyar  C, Nibber  A, Sooriyamoorthy  T, Kumar  N.  Ultrasound classification of thyroid nodules: a systematic review.   Cureus. 2020;12(3):e7239. doi:10.7759/cureus.7239 PubMedGoogle Scholar
    16.
    Castellana  M, Castellana  C, Treglia  G,  et al.  Performance of five ultrasound risk stratification systems in selecting thyroid nodules for FNA.   J Clin Endocrinol Metab. 2020;105(5):1-11. doi:10.1210/clinem/dgz170 PubMedGoogle ScholarCrossref
    17.
    Lauria Pantano  A, Maddaloni  E, Briganti  SI,  et al.  Differences between ATA, AACE/ACE/AME and ACR TI-RADS ultrasound classifications performance in identifying cytological high-risk thyroid nodules.   Eur J Endocrinol. 2018;178(6):595-603. doi:10.1530/EJE-18-0083 PubMedGoogle ScholarCrossref
    18.
    Grani  G, Lamartina  L, Ascoli  V,  et al.  Reducing the number of unnecessary thyroid biopsies while improving diagnostic accuracy: toward the “right” TIRADS.   J Clin Endocrinol Metab. 2019;104(1):95-102. doi:10.1210/jc.2018-01674 PubMedGoogle ScholarCrossref
    19.
    Hoang  JK, Middleton  WD, Farjat  AE,  et al.  Reduction in thyroid nodule biopsies and improved accuracy with American College of Radiology Thyroid Imaging Reporting and Data System.   Radiology. 2018;287(1):185-193. doi:10.1148/radiol.2018172572 PubMedGoogle ScholarCrossref
    20.
    Wu  XL, Du  JR, Wang  H,  et al.  Comparison and preliminary discussion of the reasons for the differences in diagnostic performance and unnecessary FNA biopsies between the ACR TIRADS and 2015 ATA guidelines.   Endocrine. 2019;65(1):121-131. doi:10.1007/s12020-019-01886-0 PubMedGoogle ScholarCrossref
    21.
    Solorzano  CC, Carneiro  DM, Ramirez  M, Lee  TM, Irvin  GL. Surgeon-performed ultrasound in the management of thyroid malignancy.  Am Surg. 2004;70(7):576-582.
    22.
    Gu WX, Tan CS, Ho TWT. Surgeon-performed ultrasound-guided fine-needle aspiration cytology (SP-US-FNAC) shortens time for diagnosis of thyroid nodules.  Ann Acad Med Singap. 2014;43(6):320-324.
    23.
    Lee  CY, Snyder  SK, Lairmore  TC, Dupont  SC, Jupiter  DC.  Utility of surgeon-performed ultrasound assessment of the lateral neck for metastatic papillary thyroid cancer.   J Oncol. 2012;2012:973124. doi:10.1155/2012/973124 PubMedGoogle Scholar
    24.
    Robitschek  J, Straub  M, Wirtz  E, Klem  C, Sniezek  J.  Diagnostic efficacy of surgeon-performed ultrasound-guided fine needle aspiration: a randomized controlled trial.   Otolaryngol Head Neck Surg. 2010;142(3):306-309. doi:10.1016/j.otohns.2009.11.011 PubMedGoogle ScholarCrossref
    25.
    Ahn  D, Kim  H, Sohn  JH, Choi  JH, Na  KJ.  Surgeon-performed ultrasound-guided fine-needle aspiration cytology of head and neck mass lesions: sampling adequacy and diagnostic accuracy.   Ann Surg Oncol. 2015;22(4):1360-1365. doi:10.1245/s10434-014-4119-2 PubMedGoogle ScholarCrossref
    26.
    Rozycki  GS.  Surgeon-performed ultrasound: its use in clinical practice.   Ann Surg. 1998;228(1):16-28. doi:10.1097/00000658-199807000-00004 PubMedGoogle ScholarCrossref
    27.
    Hamill  C, Ellis  PK, Johnston  PC.  Point of care thyroid ultrasound (POCUS) in endocrine outpatients: a pilot study.   Ulster Med J. 2020;89(1):21-24.PubMedGoogle Scholar
    28.
    Karkada  M, Costa  AF, Imran  SA,  et al.  Incomplete thyroid ultrasound reports for patients with thyroid nodules: implications regarding risk assessment and management.   AJR Am J Roentgenol. 2018;211(6):1348-1353. doi:10.2214/AJR.18.20056 PubMedGoogle ScholarCrossref
    29.
    Koseoglu Atilla  FD, Ozgen Saydam  B, Erarslan  NA,  et al.  Does the ACR TI-RADS scoring allow us to safely avoid unnecessary thyroid biopsy? single center analysis in a large cohort.   Endocrine. 2018;61(3):398-402. doi:10.1007/s12020-018-1620-6 PubMedGoogle ScholarCrossref
    30.
    Ogrinc  G, Davies  L, Goodman  D, Batalden  P, Davidoff  F, Stevens  D.  SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process.   BMJ Qual Saf. 2016;25(12):986-992. doi:10.1136/bmjqs-2015-004411 PubMedGoogle ScholarCrossref
    31.
    Penfold  RB, Zhang  F.  Use of interrupted time series analysis in evaluating health care quality improvements.   Acad Pediatr. 2013;13(6)(suppl):S38-S44. doi:10.1016/j.acap.2013.08.002 PubMedGoogle ScholarCrossref
    32.
    Shrestha  M, Crothers  BA, Burch  HB.  The impact of thyroid nodule size on the risk of malignancy and accuracy of fine-needle aspiration: a 10-year study from a single institution.   Thyroid. 2012;22(12):1251-1256. doi:10.1089/thy.2012.0265 PubMedGoogle ScholarCrossref
    33.
    Cavallo  A, Johnson  DN, White  MG,  et al.  Thyroid nodule size at ultrasound as a predictor of malignancy and final pathologic size.   Thyroid. 2017;27(5):641-650. doi:10.1089/thy.2016.0336 PubMedGoogle ScholarCrossref
    34.
    Kamran  SC, Marqusee  E, Kim  MI,  et al.  Thyroid nodule size and prediction of cancer.   J Clin Endocrinol Metab. 2013;98(2):564-570. doi:10.1210/jc.2012-2968 PubMedGoogle ScholarCrossref
    35.
    Dobruch-Sobczak  K, Migda  B, Krauze  A,  et al.  Prospective analysis of inter-observer and intra-observer variability in multi ultrasound descriptor assessment of thyroid nodules.   J Ultrason. 2019;19(78):198-206. doi:10.15557/JoU.2019.0030 PubMedGoogle ScholarCrossref
    36.
    Moon  WJ, Jung  SL, Lee  JH,  et al; Thyroid Study Group, Korean Society of Neuro- and Head and Neck Radiology.  Benign and malignant thyroid nodules: US differentiation—multicenter retrospective study.   Radiology. 2008;247(3):762-770. doi:10.1148/radiol.2473070944 PubMedGoogle ScholarCrossref
    37.
    Ren  J, Liu  B, Zhang  LL,  et al.  A taller-than-wide shape is a good predictor of papillary thyroid carcinoma in small solid nodules.   J Ultrasound Med. 2015;34(1):19-26. doi:10.7863/ultra.34.1.19 PubMedGoogle ScholarCrossref
    38.
    Kumbhar  SS, O’Malley  RB, Robinson  TJ,  et al.  Why thyroid surgeons are frustrated with radiologists: lessons learned from pre- and postoperative US.   Radiographics. 2016;36(7):2141-2153. doi:10.1148/rg.2016150250 PubMedGoogle ScholarCrossref
    39.
    Leis  JA, Shojania  KG.  A primer on PDSA: executing Plan-Do-Study-Act cycles in practice, not just in name.   BMJ Qual Saf. 2017;26(7):572-577. doi:10.1136/bmjqs-2016-006245 PubMedGoogle ScholarCrossref
    40.
    Rideout  M, Held  M, Holmes  AV.  The didactic makeover: keep it short, active, relevant.   Pediatrics. 2016;138(1):e20160751. doi:10.1542/peds.2016-0751 PubMedGoogle Scholar
    41.
    Semler  MW, Keriwala  RD, Clune  JK,  et al.  A randomized trial comparing didactics, demonstration, and simulation for teaching teamwork to medical residents.   Ann Am Thorac Soc. 2015;12(4):512-519. doi:10.1513/AnnalsATS.201501-030OC PubMedGoogle ScholarCrossref
    42.
    Middleton  WD, Teefey  SA, Reading  CC,  et al.  Comparison of performance characteristics of American College of Radiology TI-RADS, Korean Society of Thyroid Radiology TIRADS, and American Thyroid Association guidelines.   AJR Am J Roentgenol. 2018;210(5):1148-1154. doi:10.2214/AJR.17.18822 PubMedGoogle ScholarCrossref
    43.
    Toma  M, Dreischulte  T, Gray  NM, Campbell  D, Guthrie  B.  Balancing measures or a balanced accounting of improvement impact: a qualitative analysis of individual and focus group interviews with improvement experts in Scotland.   BMJ Qual Saf. 2018;27(7):547-556. doi:10.1136/bmjqs-2017-006554 PubMedGoogle ScholarCrossref
    ×