Comparison of an Artificial Intelligence–Enabled Patient Decision Aid vs Educational Material on Decision Quality, Shared Decision-Making, Patient Experience, and Functional Outcomes in Adults With Knee Osteoarthritis: A Randomized Clinical Trial | Clinical Decision Support | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Visual Abstract. Comparison of an Artificial Intelligence–Enabled Patient Decision Aid Versus Educational Material on Decision Quality, Shared Decision-Making, Patient Experience, and Functional Outcomes in Adults With Knee Osteoarthritis: A Randomized Clinical Trial
Comparison of an Artificial Intelligence–Enabled Patient Decision Aid Versus Educational Material on Decision Quality, Shared Decision-Making, Patient Experience, and Functional Outcomes in Adults With Knee Osteoarthritis: A Randomized Clinical Trial
Figure 1.  Schematic of Components of Decision Aid and Data Inputs for Artificial Intelligence/Machine Learning (AI/ML) Platform to Generate Personalized Outcomes
Schematic of Components of Decision Aid and Data Inputs for Artificial Intelligence/Machine Learning (AI/ML) Platform to Generate Personalized Outcomes

Schematic visualizing separate components (modules) of the decision aid provided to control and intervention group patients, including data inputs required for the AI/ML platform to generate the personalized outcomes report. BMI indicates body mass index (calculated as weight in kilograms divided by height in meters squared); ED, emergency department; KOOS JR, Knee Injury and Osteoarthritis Outcome Score for Joint Replacement; PROMIS Global-10, Patient Reported Outcome Measurement Instrumentation System Global 10.

aBenefit defined as likelihood of experiencing at least a minimal clinically important difference in functional outcome; risk defined as the likelihood of experiencing no change in condition or being worse off after undergoing surgery.

bComplication rate defined as estimated complication rate due to joint infection within 90 days, pulmonary embolism or death within 30 days, and pneumonia, sepsis, or acute myocardial infarction within 7 days. Likelihood of improvement in stiffness, pain, and quality of life based on KOOS JR metrics.

Figure 2.  Schematic of Patient Flow and Surgeon Flow Including Demographic, Clinical, and Process Factors and Outcome Measurements at Preconsultation, Postconsultation, and 4-to-6–Month Follow-up
Schematic of Patient Flow and Surgeon Flow Including Demographic, Clinical, and Process Factors and Outcome Measurements at Preconsultation, Postconsultation, and 4-to-6–Month Follow-up

Schematic visualizing a step-by-step patient flow and the surgeon flow throughout the study, the timepoints at which demographic, clinical, process, and patient outcome measures are collected. Abbreviations: BMI, body mass index (calculated as weight in kilograms divided by height in meters squared); ED, emergency department; GAD-2/GAD-7, Generalized Anxiety Disorder Questionnaires; K-DQI, Knee Decision Quality Instrument; KOOS JR, Knee Injury and Osteoarthritis Outcomes Survey Joint Replacement; NRS-C, Numerical Rating Scale for Satisfaction; PHQ-2/PHQ-9, Patient Health Questionnaires; PROMIS Global-10, Patient Reported Outcome Measurement Information System Global-10; TKR, total knee replacement.

Figure 3.  Study Flow Diagram
Study Flow Diagram

Abbreviations: BMI indicates body mass index (calculated as weight in kilograms divided by height in meters squared); OA, osteoarthritis; PROMs, patient-reported outcome measurements.

Table 1.  Descriptive Statistics of Patient Demographic and Other Characteristics by Study Group
Descriptive Statistics of Patient Demographic and Other Characteristics by Study Group
Table 2.  Descriptive Statistics for Primary and Secondary Outcome Measures
Descriptive Statistics for Primary and Secondary Outcome Measures
1.
Deshpande  BR, Katz  JN, Solomon  DH,  et al.  Number of persons with symptomatic knee osteoarthritis in the US: impact of race and ethnicity, age, sex, and obesity.   Arthritis Care Res (Hoboken). 2016;68(12):1743-1750. doi:10.1002/acr.22897PubMedGoogle ScholarCrossref
2.
Wallace  IJ, Worthington  S, Felson  DT, Jurmain  RD, Wren  KT, Maijanen  H,  et al.  Knee osteoarthritis has doubled in prevalence since the mid-20th century.   Proc Natl Acad Sci USA. 2017;114(35):9332–9336. doi:10.1073/pnas.1703856114Google ScholarCrossref
3.
Hootman  JM, Helmick  CG, Barbour  KE, Theis  KA, Boring  MA.  Updated projected prevalence of self-reported doctor-diagnosed arthritis and arthritis-attributable activity limitation among US adults, 2015-2040.   Arthritis Rheumatol. 2016;68(7):1582-1587. doi:10.1002/art.39692PubMedGoogle ScholarCrossref
4.
Hootman  JM, Helmick  CG, Brady  TJ.  A public health approach to addressing arthritis in older adults: the most common cause of disability.   Am J Public Health. 2012;102(3):426-433. doi:10.2105/AJPH.2011.300423PubMedGoogle ScholarCrossref
5.
Wright  EA, Katz  JN, Cisternas  MG, Kessler  CL, Wagenseller  A, Losina  E.  Impact of knee osteoarthritis on health care resource utilization in a US population-based national sample.   Med Care. 2010;48(9):785-791. doi:10.1097/MLR.0b013e3181e419b1PubMedGoogle ScholarCrossref
6.
American Academy of Orthopaedic Surgeons. American Academy of Orthopaedic Surgeons OA Knee Recommendations. Accessed November 28, 2019. https://www.aaos.org/quality/quality-programs/lower-extremity-programs/osteoarthritis-of-the-knee/
7.
Elwyn  G, Frosch  D, Rollnick  S.  Dual equipoise shared decision making: definitions for decision and behaviour support interventions.   Implement Sci. 2009;4:75. doi:10.1186/1748-5908-4-75PubMedGoogle ScholarCrossref
8.
Bryant  D, Bednarski  E, Gafni  A.  Incorporating patient preferences into orthopaedic practice: should the orthopaedic encounter change?   Injury. 2006;37(4):328-334. doi:10.1016/j.injury.2006.01.030PubMedGoogle ScholarCrossref
9.
Barry  MJ, Edgman-Levitan  S.  Shared decision making—pinnacle of patient-centered care.   N Engl J Med. 2012;366(9):780-781. doi:10.1056/NEJMp1109283PubMedGoogle ScholarCrossref
10.
Rabi  DM, Kunneman  M, Montori  VM.  When guidelines recommend shared decision-making.   JAMA. 2020;323(14):1345-1346. doi:10.1001/jama.2020.1525PubMedGoogle ScholarCrossref
11.
Bozic  KJ, Chiu  V.  Emerging ideas: shared decision making in patients with osteoarthritis of the hip and knee.   Clin Orthop Relat Res. 2011;469(7):2081-2085. doi:10.1007/s11999-010-1740-7PubMedGoogle ScholarCrossref
12.
Christensen  DL, Dickens  JF, Freedman  B,  et al; MOTION Group.  Patient-reported outcomes in orthopaedics.   J Bone Joint Surg Am. 2018;100(5):436-442. doi:10.2106/JBJS.17.00608PubMedGoogle ScholarCrossref
13.
Bartlett  SJ, Ahmed  S.  Montreal Accord on patient-reported outcomes (PROs) use series–paper 1: introduction.   J Clin Epidemiol. 2017;89:114-118. doi:10.1016/j.jclinepi.2017.04.012PubMedGoogle ScholarCrossref
14.
Ayers  DC.  Implementation of patient-reported outcome measures in total knee arthroplasty.   J Am Acad Orthop Surg. 2017;25(suppl 1):S48-S50. doi:10.5435/JAAOS-D-16-00631PubMedGoogle ScholarCrossref
15.
Noonan  VK, Lyddiatt  A, Ware  P,  et al.  Montreal Accord on patient-reported outcomes (PROs) use series–paper 3: patient-reported outcomes can facilitate shared decision-making and guide self-management.   J Clin Epidemiol. 2017;89:125-135. doi:10.1016/j.jclinepi.2017.04.017PubMedGoogle ScholarCrossref
16.
Bingham  CO  III, Noonan  VK, Auger  C, Feldman  DE, Ahmed  S, Bartlett  SJ.  Montreal Accord on patient-reported outcomes (PROs) use series–paper 4: patient-reported outcomes can inform clinical decision making in chronic care.   J Clin Epidemiol. 2017;89:136-141. doi:10.1016/j.jclinepi.2017.04.014PubMedGoogle ScholarCrossref
17.
Lavallee  DC, Chenok  KE, Love  RM,  et al.  Incorporating patient-reported outcomes into health care to engage patients and enhance care.   Health Aff (Millwood). 2016;35(4):575-582. doi:10.1377/hlthaff.2015.1362PubMedGoogle ScholarCrossref
18.
Berliner  JL, Brodke  DJ, Chan  V, SooHoo  NF, Bozic  KJ.  Can preoperative patient-reported outcome measures be used to predict meaningful improvement in function after TKA?   Clin Orthop Relat Res. 2017;475(1):149-157. doi:10.1007/s11999-016-4770-yPubMedGoogle ScholarCrossref
19.
Moor  JH.  The Dartmouth College Artificial Intelligence Conference: the next fifty years.   AI Mag. 2006;27:87–91. doi:10.1609/aimag.v27i4.1911Google Scholar
20.
Wang  S, Summers  RM.  Machine learning and radiology.   Med Image Anal. 2012;16(5):933-951. doi:10.1016/j.media.2012.02.005PubMedGoogle ScholarCrossref
21.
Obermeyer  Z, Emanuel  EJ.  Predicting the future—big data, machine learning, and clinical medicine.   N Engl J Med. 2016;375(13):1216-1219. doi:10.1056/NEJMp1606181PubMedGoogle ScholarCrossref
22.
Bannuru  RR, Osani  MC, Vaysbrot  EE,  et al.  OARSI guidelines for the non-surgical management of knee, hip, and polyarticular osteoarthritis.   Osteoarthritis Cartilage. 2019;27(11):1578-1589. doi:10.1016/j.joca.2019.06.011PubMedGoogle ScholarCrossref
23.
Lee  VS, Kawamoto  K, Hess  R,  et al.  Implementation of a value-driven outcomes program to identify high variability in clinical costs and outcomes and association with reduced cost and improved quality.   JAMA. 2016;316(10):1061-1072. doi:10.1001/jama.2016.12226PubMedGoogle ScholarCrossref
24.
Sepucha  KR, Stacey  D, Clay  CF,  et al.  Decision quality instrument for treatment of hip and knee osteoarthritis: a psychometric evaluation.   BMC Musculoskelet Disord. 2011;12:149. doi:10.1186/1471-2474-12-149PubMedGoogle ScholarCrossref
25.
Elwyn  G, Barr  PJ, Grande  SW, Thompson  R, Walsh  T, Ozanne  EM.  Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters.   Patient Educ Couns. 2013;93(1):102-107. doi:10.1016/j.pec.2013.05.009PubMedGoogle ScholarCrossref
26.
Jayakumar  P, Teunis  T, Vranceanu  A-M, Lamb  S, Ring  D, Gwilym  S.  Relationship between magnitude of limitations and patient experience during recovery from upper-extremity fracture.   JB JS Open Access. 2019;4(3):e0002.1-7. doi:10.2106/JBJS.OA.19.00002PubMedGoogle Scholar
27.
Lyman  S, Lee  Y-Y, Franklin  PD, Li  W, Cross  MB, Padgett  DE.  Validation of the KOOS, JR: a short-form knee arthroplasty outcomes survey.   Clin Orthop Relat Res. 2016;474(6):1461-1471. doi:10.1007/s11999-016-4719-1PubMedGoogle ScholarCrossref
28.
Hays  RD, Bjorner  JB, Revicki  DA, Spritzer  KL, Cella  D.  Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items.   Qual Life Res. 2009;18(7):873-880. doi:10.1007/s11136-009-9496-9PubMedGoogle ScholarCrossref
29.
Skapinakis  P.  The 2-item generalized anxiety disorder scale had high sensitivity and specificity for detecting GAD in primary care.   Evid Based Med. 2007;12(5):149. doi:10.1136/ebm.12.5.149PubMedGoogle ScholarCrossref
30.
Spitzer  RL, Kroenke  K, Williams  JBW, Löwe  B.  A brief measure for assessing generalized anxiety disorder: the GAD-7.   Arch Intern Med. 2006;166(10):1092-1097. doi:10.1001/archinte.166.10.1092PubMedGoogle ScholarCrossref
31.
Kroenke  K, Spitzer  RL, Williams  JBW.  The Patient Health Questionnaire-2: validity of a two-item depression screener.   Med Care. 2003;41(11):1284-1292. doi:10.1097/01.MLR.0000093487.78664.3CPubMedGoogle ScholarCrossref
32.
Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.   J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
33.
Harris  PA, Taylor  R, Thielke  R, Payne  J, Gonzalez  N, Conde  JG.  Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.   J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010PubMedGoogle ScholarCrossref
34.
Hochberg  Y.  A sharper Bonferroni procedure for multiple tests of significance.   Biometrika. 1988;75(4):800–2. doi:10.1093/biomet/75.4.800Google ScholarCrossref
35.
Kurtz  SM, Lau  E, Ong  K, Zhao  K, Kelly  M, Bozic  KJ.  Future young patient demand for primary and revision joint replacement: national projections from 2010 to 2030.   Clin Orthop Relat Res. 2009;467(10):2606-2612. doi:10.1007/s11999-009-0834-6PubMedGoogle ScholarCrossref
36.
Sloan  M, Premkumar  A, Sheth  NP.  Projected volume of primary total joint arthroplasty in the U.S., 2014 to 2030.   J Bone Joint Surg Am. 2018;100(17):1455-1460. doi:10.2106/JBJS.17.01617PubMedGoogle ScholarCrossref
37.
Williams  SN, Wolford  ML, Bercovitz  A.  Hospitalization for total knee replacement among inpatients aged 45 and over: United States, 2000-2010.   NCHS Data Brief. 2015;(210):1-8.PubMedGoogle Scholar
38.
Tew  M, Dalziel  K, Clarke  P, Smith  A, Choong  PF, Dowsey  M.  Patient-reported outcome measures (PROMs): can they be used to guide patient-centered care and optimize outcomes in total knee replacement?   Qual Life Res. 2020;29(12):3273-3283. doi:10.1007/s11136-020-02577-4Google ScholarCrossref
39.
Franklin  PD, Zheng  H, Bond  C, Lavallee  DC.  Translating clinical and patient-reported data to tailored shared decision reports with predictive analytics for knee and hip arthritis.   Qual Life Res. Online. June 2020:1-8. doi:10.1007/s11136-020-02557-8Google Scholar
40.
Mancuso  CA, Sculco  TP, Wickiewicz  TL,  et al.  Patients’ expectations of knee surgery.   J Bone Joint Surg Am. 2001;83(7):1005-1012. doi:10.2106/00004623-200107000-00005PubMedGoogle ScholarCrossref
41.
Bozic  KJ, Belkora  J, Chan  V,  et al.  Shared decision making in patients with osteoarthritis of the hip and knee: results of a randomized controlled trial.   J Bone Joint Surg Am. 2013;95(18):1633-1639. doi:10.2106/JBJS.M.00004PubMedGoogle ScholarCrossref
42.
Stacey  D, Taljaard  M, Dervin  G,  et al.  Impact of patient decision aids on appropriate and timely access to hip or knee arthroplasty for osteoarthritis: a randomized controlled trial.   Osteoarthritis Cartilage. 2016;24(1):99-107. doi:10.1016/j.joca.2015.07.024PubMedGoogle ScholarCrossref
43.
Sepucha  K, Atlas  SJ, Chang  Y,  et al.  Patient decision aids improve decision quality and patient experience and reduce surgical rates in routine orthopaedic care: a prospective cohort study.   J Bone Joint Surg Am. 2017;99(15):1253-1260. doi:10.2106/JBJS.16.01045PubMedGoogle ScholarCrossref
44.
Sepucha  K, Bedair  H, Yu  L,  et al.  Decision support strategies for hip and knee osteoarthritis: less is more: a randomized comparative effectiveness trial (DECIDE-OA Study).   J Bone Joint Surg Am. 2019;101(18):1645-1653. doi:10.2106/JBJS.19.00004PubMedGoogle ScholarCrossref
45.
Gooberman-Hill  R, Sansom  A, Sanders  CM,  et al.  Unstated factors in orthopaedic decision-making: a qualitative study.   BMC Musculoskelet Disord. 2010;11(1):213. doi:10.1186/1471-2474-11-213PubMedGoogle ScholarCrossref
46.
Youm  J, Chan  V, Belkora  J, Bozic  KJ.  Impact of socioeconomic factors on informed decision making and treatment choice in patients with hip and knee OA.   J Arthroplasty. 2015;30(2):171-175. doi:10.1016/j.arth.2014.09.006PubMedGoogle ScholarCrossref
47.
Shue  J, Karia  RJ, Cardone  D, Samuels  J, Shah  M, Slover  JD.  A randomized controlled trial of two distinct shared decision-making aids for hip and knee osteoarthritis in an ethnically diverse patient population.   Value Health. 2016;19(4):487-493. doi:10.1016/j.jval.2016.01.006PubMedGoogle ScholarCrossref
48.
Jain  D, Bendich  I, Nguyen  LL,  et al.  Do patient expectations influence patient-reported outcomes and satisfaction in total hip arthroplasty? a prospective, multicenter study.   J Arthroplasty. 2017;32(11):3322-3327. doi:10.1016/j.arth.2017.06.017PubMedGoogle ScholarCrossref
49.
Stevens  KN, Nadarajah  V, Jauregui  JJ,  et al.  Preoperative expectations of patients undergoing knee surgery.   J Knee Surg. 2019. doi:10.1055/s-0039-1698805PubMedGoogle Scholar
50.
du Long  J, Hageman  M, Vuijk  D, Rakic  A, Haverkamp  D.  Facing the decision about the treatment of hip or knee osteoarthritis: what are patients’ needs?   Knee Surg Sports Traumatol Arthrosc. 2016;24(5):1710-1716. doi:10.1007/s00167-016-3993-5PubMedGoogle ScholarCrossref
51.
Riddle  DL, Sando  T, Tarver  T,  et al.  Shared decision making applied to knee arthroplasty: a systematic review of randomized trials.   Arthritis Care Res (Hoboken). 2020. doi:10.1002/acr.24240PubMedGoogle Scholar
52.
Lützner  C, Postler  A, Beyer  F, Kirschner  S, Lützner  J.  Fulfillment of expectations influence patient satisfaction 5 years after total knee arthroplasty.   Knee Surg Sports Traumatol Arthrosc. 2019;27(7):2061-2070. doi:10.1007/s00167-018-5320-9PubMedGoogle ScholarCrossref
53.
Filardo  G, Roffi  A, Merli  G,  et al.  Patients control preferences and results in knee arthroplasty.   Knee Surg Sports Traumatol Arthrosc. 2017;25(2):552-558. doi:10.1007/s00167-016-4405-6PubMedGoogle ScholarCrossref
54.
Jayadev  C, Khan  T, Coulter  A, Beard  DJ, Price  AJ.  Patient decision aids in knee replacement surgery.   Knee. 2012;19(6):746-750. doi:10.1016/j.knee.2012.02.001PubMedGoogle ScholarCrossref
55.
Sansom  A, Donovan  J, Sanders  C,  et al.  Routes to total joint replacement surgery: patients’ and clinicians’ perceptions of need.   Arthritis Care Res (Hoboken). 2010;62(9):1252-1257. doi:10.1002/acr.20218PubMedGoogle ScholarCrossref
56.
Smith  S, Alvand  A, Locock  L,  et al.  Partial or total knee replacement? identifying patients’ information needs on knee replacement surgery: a qualitative study to inform a decision aid.   Qual Life Res. 2020;29(4):999-1011. doi:10.1007/s11136-019-02381-9PubMedGoogle ScholarCrossref
57.
Sepucha  K, Feibelmann  S, Chang  Y,  et al.  Factors associated with the quality of patients’ surgical decisions for treatment of hip and knee osteoarthritis.   J Am Coll Surg. 2013;217(4):694-701. doi:10.1016/j.jamcollsurg.2013.06.002PubMedGoogle ScholarCrossref
58.
Bansback  N, Trenaman  L, MacDonald  KV,  et al.  An individualized patient-reported outcome measure (PROM) based patient decision aid and surgeon report for patients considering total knee arthroplasty: protocol for a pragmatic randomized controlled trial.   BMC Musculoskelet Disord. 2019;20(1):89. doi:10.1186/s12891-019-2434-2PubMedGoogle ScholarCrossref
59.
Spatz  ES, Krumholz  HM, Moulton  BW.  Prime time for shared decision making.   JAMA. 2017;317(13):1309-1310. doi:10.1001/jama.2017.0616PubMedGoogle ScholarCrossref
60.
Utrillas-Compaired  A, De la Torre-Escuredo  BJ, Tebar-Martínez  AJ, Asúnsolo-Del Barco  Á.  Does preoperative psychologic distress influence pain, function, and quality of life after TKA?   Clin Orthop Relat Res. 2014;472(8):2457-2465. doi:10.1007/s11999-014-3570-5PubMedGoogle ScholarCrossref
61.
Huber  M, Kurz  C, Leidl  R.  Predicting patient-reported outcomes following hip and knee replacement surgery using supervised machine learning.   BMC Med Inform Decis Mak. 2019;19(1):3. doi:10.1186/s12911-018-0731-6Google ScholarCrossref
62.
Cronström  A, Dahlberg  LE, Nero  H, Hammarlund  CS.  “I was considering surgery because I believed that was how it was treated”: a qualitative study on willingness for joint surgery after completion of a digital management program for osteoarthritis.   Osteoarthritis Cartilage. 2019;27(7):1026-1032. doi:10.1016/j.joca.2019.04.004PubMedGoogle ScholarCrossref
63.
Cronström  A, Nero  H, Dahlberg  LE.  Factors associated with patients’ willingness to consider joint surgery after completion of a digital osteoarthritis treatment program: a prospective cohort study.   Arthritis Care Res (Hoboken). 2019;71(9):1194-1201. doi:10.1002/acr.23772PubMedGoogle ScholarCrossref
64.
Arterburn  D, Wellman  R, Westbrook  E,  et al.  Introducing decision aids at Group Health was linked to sharply lower hip and knee surgery rates and costs.   Health Aff (Millwood). 2012;31(9):2094-2104. doi:10.1377/hlthaff.2011.0686PubMedGoogle ScholarCrossref
65.
Trenaman  L, Stacey  D, Bryan  S,  et al.  Decision aids for patients considering total joint replacement: a cost-effectiveness analysis alongside a randomised controlled trial.   Osteoarthritis Cartilage. 2017;25(10):1615-1622. doi:10.1016/j.joca.2017.05.022PubMedGoogle ScholarCrossref
66.
Riddle  DL, Perera  RA.  Appropriateness and total knee arthroplasty: an examination of the American Academy of Orthopaedic Surgeons appropriateness rating system.   Osteoarthritis Cartilage. 2017;25(12):1994-1998. doi:10.1016/j.joca.2017.08.018PubMedGoogle ScholarCrossref
67.
Riddle  DL, Ghomrawi  H, Jiranek  WA, Dumenci  L, Perera  RA, Escobar  A.  Appropriateness criteria for total knee arthroplasty: additional comments and considerations.   J Bone Joint Surg Am. 2018;100(4):e22. doi:10.2106/JBJS.17.00405Google Scholar
68.
Riddle  DL, Jiranek  WA, Hayes  CW.  Use of a validated algorithm to judge the appropriateness of total knee arthroplasty in the United States: a multicenter longitudinal cohort study.   Arthritis Rheumatol. 2014;66(8):2134-2143. doi:10.1002/art.38685PubMedGoogle ScholarCrossref
69.
Bumpass  DB, Nunley  RM.  Assessing the value of a total joint replacement.   Curr Rev Musculoskelet Med. 2012;5(4):274-282. doi:10.1007/s12178-012-9139-6PubMedGoogle ScholarCrossref
70.
Katz  JN, Winter  AR, Hawker  G.  Measures of the appropriateness of elective orthopaedic joint and spine procedures.   J Bone Joint Surg Am. 2017;99(4):e15. doi:10.2106/JBJS.16.00473PubMedGoogle Scholar
71.
American Academy of Orthopaedic Surgeons. Appropriate use criteria. Accessed November 15,2020. http://www.orthoguidelines.org/go/auc/auc.cfm?auc_id=224986
72.
Ghomrawi  HMK, Lee  LY-Y, Nwachukwu  BU,  et al.  Preoperative expectations associated with postoperative dissatisfaction after total knee arthroplasty: a cohort study.   J Am Acad Orthop Surg. 2020;28(4):e145-e150. doi:10.5435/JAAOS-D-18-00785PubMedGoogle ScholarCrossref
73.
Parikh  RB, Obermeyer  Z, Navathe  AS.  Regulation of predictive analytics in medicine.   Science. 2019;363(6429):810-812. doi:10.1126/science.aaw0029PubMedGoogle ScholarCrossref
74.
Loftus  TJ, Filiberto  AC, Balch  J,  et al.  Intelligent, autonomous machines in surgery.   J Surg Res. 2020;253:92-99. doi:10.1016/j.jss.2020.03.046PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 8,607
    Citations 0
    Original Investigation
    Orthopedics
    February 18, 2021

    Comparison of an Artificial Intelligence–Enabled Patient Decision Aid vs Educational Material on Decision Quality, Shared Decision-Making, Patient Experience, and Functional Outcomes in Adults With Knee Osteoarthritis: A Randomized Clinical Trial

    Author Affiliations
    • 1Dell Medical School at the University of Texas at Austin, Austin
    • 2University of Cincinnati College of Medicine, Cincinnati, Ohio
    • 3Chicago Medical School, North Chicago, Illinois
    • 4Harbor-UCLA Medical Center, West Carson, California
    JAMA Netw Open. 2021;4(2):e2037107. doi:10.1001/jamanetworkopen.2020.37107
    Visual Abstract. Comparison of an Artificial Intelligence–Enabled Patient Decision Aid Versus Educational Material on Decision Quality, Shared Decision-Making, Patient Experience, and Functional Outcomes in Adults With Knee Osteoarthritis: A Randomized Clinical Trial
    Comparison of an Artificial Intelligence–Enabled Patient Decision Aid Versus Educational Material on Decision Quality, Shared Decision-Making, Patient Experience, and Functional Outcomes in Adults With Knee Osteoarthritis: A Randomized Clinical Trial
    Key Points

    Question  How does an artificial intelligence (AI)–enabled decision aid generated using patient-reported outcome measurements compare with education only on decision quality, patient satisfaction, and functional outcomes among individuals with knee osteoarthritis considering total knee replacement?

    Findings  This randomized clinical trial of 129 patients demonstrated statistically significant improvement in decision quality, level of shared decision-making, patient satisfaction, and functional outcomes in patients using an AI-enabled decision aid.

    Meaning  These findings suggest that AI-enabled decision aids incorporating patient-reported outcome measurement data provide a personalized, data-driven approach to shared decision-making for the surgical management of knee osteoarthritis.

    Abstract

    Importance  Decision aids can help inform appropriate selection of total knee replacement (TKR) for advanced knee osteoarthritis (OA). However, few decision aids combine patient education, preference assessment, and artificial intelligence (AI) using patient-reported outcome measurement data to generate personalized estimations of outcomes to augment shared decision-making (SDM).

    Objective  To assess the effect of an AI-enabled patient decision aid that includes education, preference assessment, and personalized outcome estimations (using patient-reported outcome measurements) on decision quality, patient experience, functional outcomes, and process-level outcomes among individuals with advanced knee OA considering TKR in comparison with education only.

    Design, Setting, and Participants  This randomized clinical trial at a single US academic orthopedic practice included 129 new adult patients presenting for OA-related knee pain from March 2019 to January 2020. Data were analyzed from April to May 2020.

    Intervention  Patients were randomized into a group that received a decision aid including patient education, preference assessment, and personalized outcome estimations (intervention group) or a group receiving educational material only (control group) alongside usual care.

    Main Outcomes and Measures  The primary outcome was decision quality, measured using the Knee OA Decision Quality Instrument (K-DQI). Secondary outcomes were collaborative decision-making (assessed using the CollaboRATE survey), patient satisfaction with consultation (using a numerical rating scale), Knee Injury and Osteoarthritis Outcome Score Joint Replacement (KOOS JR) score, consultation time, TKR rate, and treatment concordance.

    Results  A total of 69 patients in the intervention group (46 [67%] women) and 60 patients in the control group (37 [62%] women) were included in the analysis. The intervention group showed better decisional quality (K-DQI mean difference, 20.0%; SE, 3.02; 95% CI, 14.2%-26.1%; P < .001), collaborative decision-making (CollaboRATE, 8 of 69 [12%] vs 28 of 60 [47%] patients below median; P < .001), satisfaction (numerical rating scale, 9 of 65 [14%] vs 19 of 58 [33%] patients below median; P = .01), and improved functional outcomes at 4 to 6 months (mean [SE] KOOS JR, 4.9 [2.24] points higher in intervention group; 95% CI, 0.8-9.0 points; P = .02). The intervention did not significantly affect consultation time (mean [SE] difference, 2.23 [2.18] minutes; P = .31), TKR rates (16 of 69 [23%] vs 7 of 60 [12%] patients; P = .11), or treatment concordance (58 of 69 [84%] vs 44 of 60 [73%] patients; P = .19).

    Conclusions and Relevance  In this randomized clinical trial, an AI-enabled decision aid significantly improved decision quality, level of SDM, satisfaction, and physical limitations without significantly impacting consultation times, TKR rates, or treatment concordance in patients with knee OA considering TKR. Decision aids using a personalized, data-driven approach can enhance SDM in the management of knee OA.

    Trial Registration  ClinicalTrials.gov Identifier: NCT03956004

    Introduction

    Osteoarthritis (OA) of the knee has increased in prevalence and now represents a major public health concern and driver of health care spending.1-5 Treatments for knee OA range from activity modification, weight loss, physical therapy, and oral analgesics to joint injections and joint replacement surgery.6 The presence of multiple treatment options highlights the preference-sensitive nature of OA management and an opportunity for shared decision-making (SDM).7,8 SDM is a concept that integrates effective communication and clinician-patient relationship building to understand patient preferences, values, and needs, with the transfer of knowledge regarding treatments, risks, benefits, and alternatives prior to making informed decisions.9,10 Simultaneously, there is growing interest in incorporating patient-reported outcome measurements (PROMs) in the decision-making process.11,12 PROMs quantify physical, emotional, and social aspects of health from the patient’s perspective.13,14 These tools have revolutionized patient outcomes research and are increasingly used at the point-of-care for clinical decision support.9,15-17

    Baseline PROM scores can estimate postoperative outcomes when measured against scoring thresholds indicating whether patients are more or less likely to experience clinically meaningful improvement after total knee replacement (TKR)—a procedure consistently providing pain relief, functional restoration, and quality of life (QoL) improvement for advanced OA.14,18 This function of PROMs can be augmented by artificial intelligence (AI) and machine learning to synthesize complex relationships within large data sets.19-21 Combining the analytical power of machine learning with clinical and patient-generated data can provide personalized estimations of health outcomes and minimize guesswork during decision-making. We sought to evaluate an AI-enabled patient decision aid (Joint Insights, OM1) delivering patient education, an interactive preferences assessment, and personalized outcome reports generated by a machine learning algorithm using a large national data set.

    The primary objective of this study was to evaluate how an AI-enabled decision aid (intervention group) affected decision quality for patients with knee OA considering TKR compared with the provision of digital patient education and usual care alone (control group). Secondarily, this study quantified differences between intervention and control groups on the patient’s perspective of SDM during the clinical encounter, consultation satisfaction, change in functional outcome, consultation duration, TKR rates, and treatment concordance.

    Methods
    Trial Design

    We performed a parallel randomized clinical trial with a 1:1 allocation between cohorts at a musculoskeletal integrated practice unit (IPU) in an academic center in the US serving a diverse population. Two orthopedic surgeons work with a coordinated multidisciplinary team—including an advanced practice health professional, physical therapist, behavioral therapy–trained social worker, and nutritionist—in a collocated outpatient facility. Services include structured exercise programs, imaging, joint injections, weight loss counseling, dietary advice, social support, smoking and alcohol cessation, behavioral therapies, pain management, and surgery where appropriate.22 PROMs are collected prior to or on arrival in clinic and at follow-up time points as a standard of care.23 The study was reviewed and approved by the University of Texas at Austin, Dell Medical School institutional review board, and verbal informed consent was obtained from participants. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline. The trial protocol appears in Supplement 1.

    Participants

    A total of 129 patients referred with presumptive knee OA and candidacy for primary TKR were recruited between March 2019 and January 2020. Study subjects were identified during the preclinic team meeting (ie, team huddle). Adult patients aged between 45 and 89 years, fluent in English or Spanish, with body mass index (BMI; calculated as weight in kilograms divided by height in meters squared) between 20 and 46, a primary diagnosis of advanced knee OA (radiographic Kellgren-Lawrence [KL] grade 3 or 4 where grade 0 represents no OA, grade 4 severe OA), baseline Knee Injury and Osteoarthritis Outcome Score for Joint Replacement (KOOS JR) between 0 and 85, able to provide informed consent, and medically fit for TKR were included. Age, BMI, and KOOS JR limits were set based on the estimation model. Exclusion criteria disqualified patients with knee problems not primarily related to OA (eg, trauma, inflammatory arthropathy), advanced OA affecting 1 or both knees with need for care of a different joint problem, prior OA management by an orthopedic specialist, and/or any prior lower extremity total joint replacement.

    Intervention

    The intervention was an AI-enabled patient decision aid incorporating 3 modules (ie, education, preferences, and personalized outcomes) within a software platform (Figure 1). The education module incorporated an overview of the natural history of OA, evidence-based nationally recognized treatment recommendations,6,22 comparisons of treatment alternatives, risks and benefits, and a knowledge test developed using guidance from the surgeon authors (K.J.B., J.A., P.J.). The preferences module included ratings for desired levels of pain relief, commitment to postoperative recovery, and willingness to accept surgical risk on a continuum of nonoperative to operative care. Patients also rated whether they gained sufficient knowledge, their self-awareness of their preferences, and their confidence with the level of support received during decision-making. The outcomes module included a personalized report incorporating estimated probabilities of benefits, risks (complications), and likelihood of improvement in joint pain, stiffness, and QoL following TKR, alongside a summary of preferences and education modules. All content was available in English or Spanish, and the decision aid was deployed after a period of familiarization, testing within the clinical setting, and discussions about its fidelity between the clinical team and the company.

    Outcomes

    The primary outcome was the decision process score of the knee decision quality instrument (K-DQI) questions 3.1 through 3.5 (eFigure in Supplement 2) (Figure 2).24 Secondary outcomes included the level of SDM (assessed using the CollaboRATE survey),25 patient satisfaction with the consultation (numerical rating scale [NRS]),26 condition-specific symptoms and functional limitations (KOOS JR),27 duration of consultation in minutes, TKR rates (proportion of patients undergoing surgery), and treatment concordance (K-DQI question 1.6) (eFigure in Supplement 2).24 All outcomes were assessed at the end of the clinical visit except KOOS JR and treatment concordance, which were assessed prior to the consultation and again at a follow-up appointment 4 to 6 months from initial consultation or date of TKR, as applicable. No changes to trial outcomes were made after study commencement.

    Procedures

    All patients completed baseline PROMs after registration that included KOOS JR assessment,27 a Patient Reported Outcome Measurement Instrumentation System (PROMIS) Global-10 questionnaire,28 Generalized Anxiety Disorder screener (GAD-2)29 and full measure (GAD-7)30 as indicated, and the Patient Health Questionnaire screener (PHQ-2)31 and full measure (PHQ-9)32 as indicated. After escort to the clinic room, patients met the research assistant who provided study information and obtained verbal informed consent. We randomized eligible patients to the intervention group or control group using the Randomization Module in REDCap (Research Electronic Data Capture), which also housed our study data.33 Demographic characteristics and ethnicity were captured from electronic health records. Ethnicity was classified by patients during clinic registration with preset options; consideration of patient race/ethnicity was included to ensure the study population reflected our patient demographic characteristics and represented minority populations.

    Stratified block randomization was used to allot equal numbers of patients from each surgeon’s clinic to each treatment group. An independent project manager performed the random allocation sequence setup with guidance from institutional biostatisticians. Group assignment was revealed after consent and enrollment to research assistants, participants, and surgeons ahead of the consultation. Following randomization, all patients completed a baseline study survey capturing demographic characteristics, clinical factors, and process measures.

    The control group received the education module and usual care while those randomized to the intervention group received the education and preferences module before receiving the report from the outcomes module (Figure 2). Patients reviewed the decision aid modules independently prior to consultations following a brief introduction by the researcher who periodically checked in with the patient and remained available to answer any questions. Patients received the personalized report at the same time as the surgeon. Both parties had a chance to review the report prior to the consultation, in which surgeons walked patients through the metrics as part of the discussion. We recorded consultation time by stopwatch, marking surgeon entry and exit from the patient room for the decision-making discussion. Patients completed a final set of questionnaires (K-DQI, CollaboRATE, and NRS) before leaving the clinic, and KOOS JR and treatment concordance were completed at the 4-to-6–month follow-up appointment.

    Statistical Analysis

    All primary and secondary outcome measures were examined for distributional properties; data were analyzed from April to May 2020 using Stata statistical software version 16 (StataCorp). Measures from the K-DQI, KOOS JR, and consultation duration were treated as continuous outcomes; NRS and total CollaboRATE scores as ordinal outcomes; and treatment concordance and TKR surgery as binary measures. In testing for intervention effectiveness, assumptions of unequal variance across both groups were checked for continuous measures and, if violated, Satterthwaite adjustments were reported for independent sample t tests (ie, K-DQI, consultation time). We fitted linear mixed effect models with random intercepts at the subject level to test the interaction of group assignment with time for KOOS JR. We conducted Mann-Whitney U tests to assess differences between groups for ordinal measures (ie, CollaboRATE, NRS) and performed Fisher exact tests to evaluate differing surgical rates and treatment concordance. To control for inflations of the type I errors rate, we set α = .05 for the primary outcome measures in 2-tailed tests and did the same for secondary outcome measures. The Hochberg-Y procedure was applied to correct individual test α levels in both groups of outcomes to maintain a familywise type I error rate of 0.05 for each.34

    Power analysis indicated that 130 patients would yield 99% power to detect a comprehensive set of minimum meaningful group differences in K-DQI (15%) and CollaboRATE total scale (2 points), with 63% and 90% power to detect a 7-point or 9-point pre- to postintervention difference in KOOS JR scores, respectively, with α = .017 under a Bonferroni correction (0.05/3). These 3 measures were considered the most directly clinically relevant measures to SDM, decision aids, and patient outcomes in our study.

    Results

    Sixty-nine intervention group patients (46 [67%] women) and 60 control group patients (37 [62%] women) were included in the final analysis. Table 1 provides descriptive statistics and counts for demographic and clinical characteristics.

    Participant Flow

    During the study period (March 2019 to July 2020, including follow-up), 705 new patients with joint pain presented to the IPU. A total of 560 patients (79.4%) were excluded on initial screening because of exclusion criteria (543 patients [77.0%]) or declining participation (17 patients [2.4%]) (Figure 3). The remaining 145 (20.6%) were randomized into the control group (69 patients) or intervention group (76 patients). In the control group, 9 patients were excluded because of failure to complete postconsultation surveys or follow-up KOOS JR assessment, shifting care to hip or low back pain treatment, or study withdrawal. In the intervention group, 7 patients were excluded after failing to complete postconsultation surveys, transfer of care to another service, or not completing follow-up KOOS JR assessment. There were no harms or unintended effects in either group.

    Outcomes and Estimation

    Table 2 provides descriptive statistics and counts for primary and secondary outcome measures. Patients in the intervention group showed better decisional quality (K-DQI, mean difference, 20.0%; SE, 3.0; 95% CI, 14.2%-26.1%; P < .0001). Ordinal ratings for level of SDM and satisfaction were highly skewed and consistent with improved outcomes in the intervention group using nonparametric tests. More control patients had scores lower than the CollaboRATE median (also the maximum on the measure) than in the intervention group (28 of 60 [47%] vs 8 of 69 [12%]; P < .001). Similarly, 19 of 58 patients (33%) had scores lower than the median value of 10 (also the maximum on the measure) for consultation satisfaction in the control group while 9 of 65 (14%) had scores lower than the median in the intervention group (P = .01). Greater improvement in functional outcomes (KOOS JR) was shown from baseline to 4 to 6 months at follow-up (mean [SE], 4.9 [2.1] points higher in intervention group than control group; 95% CI, 0.8-9.0 points; P = .02). The intervention group did not experience longer consultation times (mean difference, 2.23 minutes; SE, 2.18). Finally, differential rates of TKR and differential treatment concordance did not reach statistical significance.

    To assess whether greater TKR rates in the intervention group relative to control group accounted for greater improvement in KOOS JR during the follow-up (although already shown not to be statistically significant), we adjusted for TKR effect in the linear mixed effects model. The relative improvement in KOOS JR remained statistically significant for those in the intervention group (mean [SE], 6.42 [2.31] points higher in intervention group than control group; 95% CI, 1.8-10.9 points; P = .02).

    Discussion

    TKR—among the most common surgical procedures in the US—is a high stakes encounter for advanced knee OA demanding careful selection.35-37 The promise of PROMs to improve SDM through enhanced patient engagement has been recognized.15,38,39 We developed an AI-enabled decision aid incorporating PROMs, patient education, preference assessment, and personalized estimations of clinical outcomes that improved decision quality, SDM, patient satisfaction, and functional outcomes compared with education and usual care.

    Our study joins others that have demonstrated improvements in decision quality, levels of SDM, and greater patient satisfaction in care for knee OA when using decision aids.41-44 Our decision aid produced positive outcomes for individuals from a range of backgrounds, including those experiencing unemployment and those with limited resources. Thus, populations that have been shown to receive less support in making informed care decisions45-47 may also benefit from this tool. Notably, the relatively low number of patients declining participation indicates that diverse patient populations, including the underserved, also want to participate in their health decisions. Furthermore, both groups in this study had similar characteristics, although commercial insurance was slightly more common in the intervention group despite randomization. For all individuals with advanced knee OA, this decision aid may better address patient expectations around symptom relief, improvement in physical function, and psychosocial wellness, as well as provide clarity around fears, attitudes, and the risks and benefits of surgery.40,45,48-51 Expectation management, patient engagement, and patient-perceived control over decisions synergistically yield optimal outcomes and experiences for those with OA.52-54 Decision aids addressing patient preferences may equip teams of health professionals to align with these expectations.55 Further, the ability of patients to appreciate multiple treatment options and the dynamic nature of their condition is supported by visual elements of our decision aid, including scales for preferences and quantified variability in outcomes based on modifiable personal and clinical factors. Patients using the tool may feel more involved, informed, and in control of the decision-making process; this warrants further study.56,57

    Notably, few decision aids have demonstrated the positive impact on functional outcomes in knee OA.15,58 The findings in our study could be explained by greater patient engagement in those experiencing the full decision aid, which promoted improvements in their ability to perform physical activities. Improvement in PROMs is paramount as payers and policy makers aim to shift toward using patient-centered metrics in value-based payment reforms.59 Notably, mental health affects PROMs, with preoperative psychological distress affecting pain, function, and QoL following TKR.60 PROMIS Global mental health scores were incorporated in the algorithm, and raw scores were made available for enriching the surgeon-patient discussion.

    The combination of PROMs with AI analytics for decision support has also been observed in a 2020 study39 that demonstrated the feasibility of a web-based tool providing an estimated outcomes report using previsit PROMs plus clinical risk factors to facilitate SDM for patients with hip and knee OA. While large-scale implementation of this decision aid was achieved, inconsistencies remained around whether the report was reviewed during the decision-making consultation—in contrast to our trial, in which review by both patient and surgeon was mandatory in the intervention group. In another study,61 machine learning was shown to estimate postoperative PROMs using preoperative visual analogue scores for pain, Q scores, and clinical factors.

    Other studies have shown surgeons experience more efficient use of their time with shorter consultations using decision aids.41 Our study found no significant change in consultation time in either direction despite the inclusion of a joint review of the personalized outcomes report during the consultation.

    In relation to TKR use, some authors suggest that informed patients opt for more conservative treatments.62,63 Indeed, reduced surgical rates have been demonstrated.42,64,65 However, we advise caution in relying on these tools to steer patients toward less invasive treatment options; our previous work exhibited no differential impact on TKR use between decision aid users and nonusers.41 Interestingly, while outside our target population, those with less severe OA may also benefit from experiencing similar analytics, given that in these cases decision aids are likely to reveal (and underline) less favorable outcomes from undergoing surgery. Advanced decision support may lead patients toward evidence-based nonoperative strategies and surgeons toward safe and judicious use of TKR. This may be a powerful asset as volume and expenditure for TKR increase.11,66-71 More informed patients with more realistic preoperative expectations are less likely to be dissatisfied with the results of their operations, and therefore may use fewer resources (eg, extended physical therapy or pain management) postoperatively.48,72

    Notably, the lack of significant impact on treatment concordance between our study groups may be explained by the standard of care in the IPU, where set treatment plans during initial consultations are usually followed through (ie, treatment performed matches treatment selected) irrespective of decision aids more focused on the decision-making process itself than the outcome of the decision. Lastly, the decision aid in this study incorporates complex analytics that require careful and clear explanations to patients. While no formal training was provided to surgeons, future implementation of such tools should incorporate coaching for clinicians to put the data generated in human terms and effectively communicate insights as treatment options. Decision aids have incorporated communication aids to enhance the health care professional–patient interaction,41 and this tactic could be applied to our tool. Widespread uptake of such innovations will ultimately depend on surgeons attesting to data insights supporting their deductive reasoning and judgment during SDM, data flow automation (to minimize user burden), robust longitudinal PROM collection, coaching on SDM methods, and communication of outputs alongside legal and ethical considerations.59,64,65,73,74 Addressing these factors may accelerate the successful integration of AI-enabled decision aids using PROMs and drive uptake of these tools to unlock advanced insights as the machine learns from an accumulating set of clinical- and patient-focused data points to provide reliable, real-time estimations of outcomes.

    Limitations

    There are several limitations to this study. First, this work was performed in a specialized setting at a single institution delivering a comprehensive range of treatments. While this potentially limits generalizability, the study does demonstrate the feasibility of using this type of decision aid in any setting where PROMs are collected longitudinally. Future evaluations should account for nonoperative strategies delivered and assess outcomes in different settings, including traditional fee-for-service care. Second, because surgeons were not masked to the intervention, there is potential for contamination (ie, differential surgeon behavior expressed toward each group, such as motivational bias manifest in enhanced interactions with those in the intervention group). Such biases were challenging to accommodate in our study design. Third, we did not assess the effect of the decision aid on patient knowledge, because matching educational content was provided to both groups nor did we assess patient activation (defined as propensity to engage in adaptive health behaviors), choice awareness, aspects of deliberation (eg, decision conflict and decision regret), expectation management, or surgeon perceptions (eg, efficiency of the consultation). While research assistants were available to answer any questions patients had while they read the education material and used the decision aid, we did not formally assess health literacy, and some patients may not have been able to read or understand all content in the decision aid. Future iterations of the decision aid may also include audio or video options instead of having to read the material. Further work should explore these factors while minimizing questionnaire burden and account for health literacy, language barriers, and sociodemographic status.40 Fourth, the typical course of a formal OA in-clinic diagnosis poses a general limitation in limiting the timeframe over which the tool may be applied.

    Conclusions

    The findings of this study suggest that multifaceted decision aids integrating patient education, preference assessment, and AI-enabled analytics built with PROM data can provide a personalized, data-driven approach to SDM for patients with advanced knee OA considering TKR. Benefits to patients observed in this single-center study warrant further investigation across multiple sites routinely collecting PROMs, with careful consideration of institutional and health care professionals’ experiences of incorporating AI in practice. The patient-centered, data-driven approach to SDM in this study may mark a step-change in the application of patient decision aids in orthopedic practice.

    Back to top
    Article Information

    Accepted for Publication: November 22, 2020.

    Published: February 18, 2021. doi:10.1001/jamanetworkopen.2020.37107

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Jayakumar P et al. JAMA Network Open.

    Corresponding Author: Prakash Jayakumar, MBBS, DPhil, Dell Medical School at the University of Texas at Austin, 1701 Trinity St, Bldg B, Z0800, Austin, TX 78712 (prakash.jayakumar@austin.utexas.edu).

    Author Contributions: Dr Jayakumar had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Jayakumar, Moore, Furlough, Uhler, Andrawis, Koenig, Rathouz, Bozic.

    Acquisition, analysis, or interpretation of data: Jayakumar, Moore, Furlough, Aksan, Rathouz, Bozic.

    Drafting of the manuscript: Jayakumar, Moore, Furlough.

    Critical revision of the manuscript for important intellectual content: Jayakumar, Moore, Uhler, Andrawis, Koenig, Aksan, Rathouz, Bozic.

    Statistical analysis: Jayakumar, Moore, Aksan, Rathouz.

    Obtained funding: Moore, Andrawis, Bozic.

    Administrative, technical, or material support: Jayakumar, Moore, Furlough, Uhler, Andrawis.

    Supervision: Jayakumar, Moore, Koenig, Bozic.

    Conflict of Interest Disclosures: Dr Jayakumar reported receiving research support from Agency for Healthcare Research and Quality and the Kozmetsky Family Foundation and Commonwealth Fund, and consulting fees from Johnson and Johnson Value Creation. Dr Bozic reported receiving research support from the Agency for Healthcare Research and Quality, having stock options with Carrum Health, receiving consulting fees from Centers for Medicare & Medicaid Services and Embold Health, receiving publishing royalties with Wolters Kluwer, receiving personal fees from Cardinal Analytx, and serving in governance roles with the American Academy of Orthopaedic Surgeons.

    Funding/Support: This project was supported by Dell Medical School’s Texas Health Catalyst Program.

    Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: Contents are solely the responsibility of authors and do not necessarily represent official views of the University of Texas at Austin.

    Data Sharing Statement: See Supplement 3.

    References
    1.
    Deshpande  BR, Katz  JN, Solomon  DH,  et al.  Number of persons with symptomatic knee osteoarthritis in the US: impact of race and ethnicity, age, sex, and obesity.   Arthritis Care Res (Hoboken). 2016;68(12):1743-1750. doi:10.1002/acr.22897PubMedGoogle ScholarCrossref
    2.
    Wallace  IJ, Worthington  S, Felson  DT, Jurmain  RD, Wren  KT, Maijanen  H,  et al.  Knee osteoarthritis has doubled in prevalence since the mid-20th century.   Proc Natl Acad Sci USA. 2017;114(35):9332–9336. doi:10.1073/pnas.1703856114Google ScholarCrossref
    3.
    Hootman  JM, Helmick  CG, Barbour  KE, Theis  KA, Boring  MA.  Updated projected prevalence of self-reported doctor-diagnosed arthritis and arthritis-attributable activity limitation among US adults, 2015-2040.   Arthritis Rheumatol. 2016;68(7):1582-1587. doi:10.1002/art.39692PubMedGoogle ScholarCrossref
    4.
    Hootman  JM, Helmick  CG, Brady  TJ.  A public health approach to addressing arthritis in older adults: the most common cause of disability.   Am J Public Health. 2012;102(3):426-433. doi:10.2105/AJPH.2011.300423PubMedGoogle ScholarCrossref
    5.
    Wright  EA, Katz  JN, Cisternas  MG, Kessler  CL, Wagenseller  A, Losina  E.  Impact of knee osteoarthritis on health care resource utilization in a US population-based national sample.   Med Care. 2010;48(9):785-791. doi:10.1097/MLR.0b013e3181e419b1PubMedGoogle ScholarCrossref
    6.
    American Academy of Orthopaedic Surgeons. American Academy of Orthopaedic Surgeons OA Knee Recommendations. Accessed November 28, 2019. https://www.aaos.org/quality/quality-programs/lower-extremity-programs/osteoarthritis-of-the-knee/
    7.
    Elwyn  G, Frosch  D, Rollnick  S.  Dual equipoise shared decision making: definitions for decision and behaviour support interventions.   Implement Sci. 2009;4:75. doi:10.1186/1748-5908-4-75PubMedGoogle ScholarCrossref
    8.
    Bryant  D, Bednarski  E, Gafni  A.  Incorporating patient preferences into orthopaedic practice: should the orthopaedic encounter change?   Injury. 2006;37(4):328-334. doi:10.1016/j.injury.2006.01.030PubMedGoogle ScholarCrossref
    9.
    Barry  MJ, Edgman-Levitan  S.  Shared decision making—pinnacle of patient-centered care.   N Engl J Med. 2012;366(9):780-781. doi:10.1056/NEJMp1109283PubMedGoogle ScholarCrossref
    10.
    Rabi  DM, Kunneman  M, Montori  VM.  When guidelines recommend shared decision-making.   JAMA. 2020;323(14):1345-1346. doi:10.1001/jama.2020.1525PubMedGoogle ScholarCrossref
    11.
    Bozic  KJ, Chiu  V.  Emerging ideas: shared decision making in patients with osteoarthritis of the hip and knee.   Clin Orthop Relat Res. 2011;469(7):2081-2085. doi:10.1007/s11999-010-1740-7PubMedGoogle ScholarCrossref
    12.
    Christensen  DL, Dickens  JF, Freedman  B,  et al; MOTION Group.  Patient-reported outcomes in orthopaedics.   J Bone Joint Surg Am. 2018;100(5):436-442. doi:10.2106/JBJS.17.00608PubMedGoogle ScholarCrossref
    13.
    Bartlett  SJ, Ahmed  S.  Montreal Accord on patient-reported outcomes (PROs) use series–paper 1: introduction.   J Clin Epidemiol. 2017;89:114-118. doi:10.1016/j.jclinepi.2017.04.012PubMedGoogle ScholarCrossref
    14.
    Ayers  DC.  Implementation of patient-reported outcome measures in total knee arthroplasty.   J Am Acad Orthop Surg. 2017;25(suppl 1):S48-S50. doi:10.5435/JAAOS-D-16-00631PubMedGoogle ScholarCrossref
    15.
    Noonan  VK, Lyddiatt  A, Ware  P,  et al.  Montreal Accord on patient-reported outcomes (PROs) use series–paper 3: patient-reported outcomes can facilitate shared decision-making and guide self-management.   J Clin Epidemiol. 2017;89:125-135. doi:10.1016/j.jclinepi.2017.04.017PubMedGoogle ScholarCrossref
    16.
    Bingham  CO  III, Noonan  VK, Auger  C, Feldman  DE, Ahmed  S, Bartlett  SJ.  Montreal Accord on patient-reported outcomes (PROs) use series–paper 4: patient-reported outcomes can inform clinical decision making in chronic care.   J Clin Epidemiol. 2017;89:136-141. doi:10.1016/j.jclinepi.2017.04.014PubMedGoogle ScholarCrossref
    17.
    Lavallee  DC, Chenok  KE, Love  RM,  et al.  Incorporating patient-reported outcomes into health care to engage patients and enhance care.   Health Aff (Millwood). 2016;35(4):575-582. doi:10.1377/hlthaff.2015.1362PubMedGoogle ScholarCrossref
    18.
    Berliner  JL, Brodke  DJ, Chan  V, SooHoo  NF, Bozic  KJ.  Can preoperative patient-reported outcome measures be used to predict meaningful improvement in function after TKA?   Clin Orthop Relat Res. 2017;475(1):149-157. doi:10.1007/s11999-016-4770-yPubMedGoogle ScholarCrossref
    19.
    Moor  JH.  The Dartmouth College Artificial Intelligence Conference: the next fifty years.   AI Mag. 2006;27:87–91. doi:10.1609/aimag.v27i4.1911Google Scholar
    20.
    Wang  S, Summers  RM.  Machine learning and radiology.   Med Image Anal. 2012;16(5):933-951. doi:10.1016/j.media.2012.02.005PubMedGoogle ScholarCrossref
    21.
    Obermeyer  Z, Emanuel  EJ.  Predicting the future—big data, machine learning, and clinical medicine.   N Engl J Med. 2016;375(13):1216-1219. doi:10.1056/NEJMp1606181PubMedGoogle ScholarCrossref
    22.
    Bannuru  RR, Osani  MC, Vaysbrot  EE,  et al.  OARSI guidelines for the non-surgical management of knee, hip, and polyarticular osteoarthritis.   Osteoarthritis Cartilage. 2019;27(11):1578-1589. doi:10.1016/j.joca.2019.06.011PubMedGoogle ScholarCrossref
    23.
    Lee  VS, Kawamoto  K, Hess  R,  et al.  Implementation of a value-driven outcomes program to identify high variability in clinical costs and outcomes and association with reduced cost and improved quality.   JAMA. 2016;316(10):1061-1072. doi:10.1001/jama.2016.12226PubMedGoogle ScholarCrossref
    24.
    Sepucha  KR, Stacey  D, Clay  CF,  et al.  Decision quality instrument for treatment of hip and knee osteoarthritis: a psychometric evaluation.   BMC Musculoskelet Disord. 2011;12:149. doi:10.1186/1471-2474-12-149PubMedGoogle ScholarCrossref
    25.
    Elwyn  G, Barr  PJ, Grande  SW, Thompson  R, Walsh  T, Ozanne  EM.  Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters.   Patient Educ Couns. 2013;93(1):102-107. doi:10.1016/j.pec.2013.05.009PubMedGoogle ScholarCrossref
    26.
    Jayakumar  P, Teunis  T, Vranceanu  A-M, Lamb  S, Ring  D, Gwilym  S.  Relationship between magnitude of limitations and patient experience during recovery from upper-extremity fracture.   JB JS Open Access. 2019;4(3):e0002.1-7. doi:10.2106/JBJS.OA.19.00002PubMedGoogle Scholar
    27.
    Lyman  S, Lee  Y-Y, Franklin  PD, Li  W, Cross  MB, Padgett  DE.  Validation of the KOOS, JR: a short-form knee arthroplasty outcomes survey.   Clin Orthop Relat Res. 2016;474(6):1461-1471. doi:10.1007/s11999-016-4719-1PubMedGoogle ScholarCrossref
    28.
    Hays  RD, Bjorner  JB, Revicki  DA, Spritzer  KL, Cella  D.  Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items.   Qual Life Res. 2009;18(7):873-880. doi:10.1007/s11136-009-9496-9PubMedGoogle ScholarCrossref
    29.
    Skapinakis  P.  The 2-item generalized anxiety disorder scale had high sensitivity and specificity for detecting GAD in primary care.   Evid Based Med. 2007;12(5):149. doi:10.1136/ebm.12.5.149PubMedGoogle ScholarCrossref
    30.
    Spitzer  RL, Kroenke  K, Williams  JBW, Löwe  B.  A brief measure for assessing generalized anxiety disorder: the GAD-7.   Arch Intern Med. 2006;166(10):1092-1097. doi:10.1001/archinte.166.10.1092PubMedGoogle ScholarCrossref
    31.
    Kroenke  K, Spitzer  RL, Williams  JBW.  The Patient Health Questionnaire-2: validity of a two-item depression screener.   Med Care. 2003;41(11):1284-1292. doi:10.1097/01.MLR.0000093487.78664.3CPubMedGoogle ScholarCrossref
    32.
    Kroenke  K, Spitzer  RL, Williams  JB.  The PHQ-9: validity of a brief depression severity measure.   J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.xPubMedGoogle ScholarCrossref
    33.
    Harris  PA, Taylor  R, Thielke  R, Payne  J, Gonzalez  N, Conde  JG.  Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.   J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010PubMedGoogle ScholarCrossref
    34.
    Hochberg  Y.  A sharper Bonferroni procedure for multiple tests of significance.   Biometrika. 1988;75(4):800–2. doi:10.1093/biomet/75.4.800Google ScholarCrossref
    35.
    Kurtz  SM, Lau  E, Ong  K, Zhao  K, Kelly  M, Bozic  KJ.  Future young patient demand for primary and revision joint replacement: national projections from 2010 to 2030.   Clin Orthop Relat Res. 2009;467(10):2606-2612. doi:10.1007/s11999-009-0834-6PubMedGoogle ScholarCrossref
    36.
    Sloan  M, Premkumar  A, Sheth  NP.  Projected volume of primary total joint arthroplasty in the U.S., 2014 to 2030.   J Bone Joint Surg Am. 2018;100(17):1455-1460. doi:10.2106/JBJS.17.01617PubMedGoogle ScholarCrossref
    37.
    Williams  SN, Wolford  ML, Bercovitz  A.  Hospitalization for total knee replacement among inpatients aged 45 and over: United States, 2000-2010.   NCHS Data Brief. 2015;(210):1-8.PubMedGoogle Scholar
    38.
    Tew  M, Dalziel  K, Clarke  P, Smith  A, Choong  PF, Dowsey  M.  Patient-reported outcome measures (PROMs): can they be used to guide patient-centered care and optimize outcomes in total knee replacement?   Qual Life Res. 2020;29(12):3273-3283. doi:10.1007/s11136-020-02577-4Google ScholarCrossref
    39.
    Franklin  PD, Zheng  H, Bond  C, Lavallee  DC.  Translating clinical and patient-reported data to tailored shared decision reports with predictive analytics for knee and hip arthritis.   Qual Life Res. Online. June 2020:1-8. doi:10.1007/s11136-020-02557-8Google Scholar
    40.
    Mancuso  CA, Sculco  TP, Wickiewicz  TL,  et al.  Patients’ expectations of knee surgery.   J Bone Joint Surg Am. 2001;83(7):1005-1012. doi:10.2106/00004623-200107000-00005PubMedGoogle ScholarCrossref
    41.
    Bozic  KJ, Belkora  J, Chan  V,  et al.  Shared decision making in patients with osteoarthritis of the hip and knee: results of a randomized controlled trial.   J Bone Joint Surg Am. 2013;95(18):1633-1639. doi:10.2106/JBJS.M.00004PubMedGoogle ScholarCrossref
    42.
    Stacey  D, Taljaard  M, Dervin  G,  et al.  Impact of patient decision aids on appropriate and timely access to hip or knee arthroplasty for osteoarthritis: a randomized controlled trial.   Osteoarthritis Cartilage. 2016;24(1):99-107. doi:10.1016/j.joca.2015.07.024PubMedGoogle ScholarCrossref
    43.
    Sepucha  K, Atlas  SJ, Chang  Y,  et al.  Patient decision aids improve decision quality and patient experience and reduce surgical rates in routine orthopaedic care: a prospective cohort study.   J Bone Joint Surg Am. 2017;99(15):1253-1260. doi:10.2106/JBJS.16.01045PubMedGoogle ScholarCrossref
    44.
    Sepucha  K, Bedair  H, Yu  L,  et al.  Decision support strategies for hip and knee osteoarthritis: less is more: a randomized comparative effectiveness trial (DECIDE-OA Study).   J Bone Joint Surg Am. 2019;101(18):1645-1653. doi:10.2106/JBJS.19.00004PubMedGoogle ScholarCrossref
    45.
    Gooberman-Hill  R, Sansom  A, Sanders  CM,  et al.  Unstated factors in orthopaedic decision-making: a qualitative study.   BMC Musculoskelet Disord. 2010;11(1):213. doi:10.1186/1471-2474-11-213PubMedGoogle ScholarCrossref
    46.
    Youm  J, Chan  V, Belkora  J, Bozic  KJ.  Impact of socioeconomic factors on informed decision making and treatment choice in patients with hip and knee OA.   J Arthroplasty. 2015;30(2):171-175. doi:10.1016/j.arth.2014.09.006PubMedGoogle ScholarCrossref
    47.
    Shue  J, Karia  RJ, Cardone  D, Samuels  J, Shah  M, Slover  JD.  A randomized controlled trial of two distinct shared decision-making aids for hip and knee osteoarthritis in an ethnically diverse patient population.   Value Health. 2016;19(4):487-493. doi:10.1016/j.jval.2016.01.006PubMedGoogle ScholarCrossref
    48.
    Jain  D, Bendich  I, Nguyen  LL,  et al.  Do patient expectations influence patient-reported outcomes and satisfaction in total hip arthroplasty? a prospective, multicenter study.   J Arthroplasty. 2017;32(11):3322-3327. doi:10.1016/j.arth.2017.06.017PubMedGoogle ScholarCrossref
    49.
    Stevens  KN, Nadarajah  V, Jauregui  JJ,  et al.  Preoperative expectations of patients undergoing knee surgery.   J Knee Surg. 2019. doi:10.1055/s-0039-1698805PubMedGoogle Scholar
    50.
    du Long  J, Hageman  M, Vuijk  D, Rakic  A, Haverkamp  D.  Facing the decision about the treatment of hip or knee osteoarthritis: what are patients’ needs?   Knee Surg Sports Traumatol Arthrosc. 2016;24(5):1710-1716. doi:10.1007/s00167-016-3993-5PubMedGoogle ScholarCrossref
    51.
    Riddle  DL, Sando  T, Tarver  T,  et al.  Shared decision making applied to knee arthroplasty: a systematic review of randomized trials.   Arthritis Care Res (Hoboken). 2020. doi:10.1002/acr.24240PubMedGoogle Scholar
    52.
    Lützner  C, Postler  A, Beyer  F, Kirschner  S, Lützner  J.  Fulfillment of expectations influence patient satisfaction 5 years after total knee arthroplasty.   Knee Surg Sports Traumatol Arthrosc. 2019;27(7):2061-2070. doi:10.1007/s00167-018-5320-9PubMedGoogle ScholarCrossref
    53.
    Filardo  G, Roffi  A, Merli  G,  et al.  Patients control preferences and results in knee arthroplasty.   Knee Surg Sports Traumatol Arthrosc. 2017;25(2):552-558. doi:10.1007/s00167-016-4405-6PubMedGoogle ScholarCrossref
    54.
    Jayadev  C, Khan  T, Coulter  A, Beard  DJ, Price  AJ.  Patient decision aids in knee replacement surgery.   Knee. 2012;19(6):746-750. doi:10.1016/j.knee.2012.02.001PubMedGoogle ScholarCrossref
    55.
    Sansom  A, Donovan  J, Sanders  C,  et al.  Routes to total joint replacement surgery: patients’ and clinicians’ perceptions of need.   Arthritis Care Res (Hoboken). 2010;62(9):1252-1257. doi:10.1002/acr.20218PubMedGoogle ScholarCrossref
    56.
    Smith  S, Alvand  A, Locock  L,  et al.  Partial or total knee replacement? identifying patients’ information needs on knee replacement surgery: a qualitative study to inform a decision aid.   Qual Life Res. 2020;29(4):999-1011. doi:10.1007/s11136-019-02381-9PubMedGoogle ScholarCrossref
    57.
    Sepucha  K, Feibelmann  S, Chang  Y,  et al.  Factors associated with the quality of patients’ surgical decisions for treatment of hip and knee osteoarthritis.   J Am Coll Surg. 2013;217(4):694-701. doi:10.1016/j.jamcollsurg.2013.06.002PubMedGoogle ScholarCrossref
    58.
    Bansback  N, Trenaman  L, MacDonald  KV,  et al.  An individualized patient-reported outcome measure (PROM) based patient decision aid and surgeon report for patients considering total knee arthroplasty: protocol for a pragmatic randomized controlled trial.   BMC Musculoskelet Disord. 2019;20(1):89. doi:10.1186/s12891-019-2434-2PubMedGoogle ScholarCrossref
    59.
    Spatz  ES, Krumholz  HM, Moulton  BW.  Prime time for shared decision making.   JAMA. 2017;317(13):1309-1310. doi:10.1001/jama.2017.0616PubMedGoogle ScholarCrossref
    60.
    Utrillas-Compaired  A, De la Torre-Escuredo  BJ, Tebar-Martínez  AJ, Asúnsolo-Del Barco  Á.  Does preoperative psychologic distress influence pain, function, and quality of life after TKA?   Clin Orthop Relat Res. 2014;472(8):2457-2465. doi:10.1007/s11999-014-3570-5PubMedGoogle ScholarCrossref
    61.
    Huber  M, Kurz  C, Leidl  R.  Predicting patient-reported outcomes following hip and knee replacement surgery using supervised machine learning.   BMC Med Inform Decis Mak. 2019;19(1):3. doi:10.1186/s12911-018-0731-6Google ScholarCrossref
    62.
    Cronström  A, Dahlberg  LE, Nero  H, Hammarlund  CS.  “I was considering surgery because I believed that was how it was treated”: a qualitative study on willingness for joint surgery after completion of a digital management program for osteoarthritis.   Osteoarthritis Cartilage. 2019;27(7):1026-1032. doi:10.1016/j.joca.2019.04.004PubMedGoogle ScholarCrossref
    63.
    Cronström  A, Nero  H, Dahlberg  LE.  Factors associated with patients’ willingness to consider joint surgery after completion of a digital osteoarthritis treatment program: a prospective cohort study.   Arthritis Care Res (Hoboken). 2019;71(9):1194-1201. doi:10.1002/acr.23772PubMedGoogle ScholarCrossref
    64.
    Arterburn  D, Wellman  R, Westbrook  E,  et al.  Introducing decision aids at Group Health was linked to sharply lower hip and knee surgery rates and costs.   Health Aff (Millwood). 2012;31(9):2094-2104. doi:10.1377/hlthaff.2011.0686PubMedGoogle ScholarCrossref
    65.
    Trenaman  L, Stacey  D, Bryan  S,  et al.  Decision aids for patients considering total joint replacement: a cost-effectiveness analysis alongside a randomised controlled trial.   Osteoarthritis Cartilage. 2017;25(10):1615-1622. doi:10.1016/j.joca.2017.05.022PubMedGoogle ScholarCrossref
    66.
    Riddle  DL, Perera  RA.  Appropriateness and total knee arthroplasty: an examination of the American Academy of Orthopaedic Surgeons appropriateness rating system.   Osteoarthritis Cartilage. 2017;25(12):1994-1998. doi:10.1016/j.joca.2017.08.018PubMedGoogle ScholarCrossref
    67.
    Riddle  DL, Ghomrawi  H, Jiranek  WA, Dumenci  L, Perera  RA, Escobar  A.  Appropriateness criteria for total knee arthroplasty: additional comments and considerations.   J Bone Joint Surg Am. 2018;100(4):e22. doi:10.2106/JBJS.17.00405Google Scholar
    68.
    Riddle  DL, Jiranek  WA, Hayes  CW.  Use of a validated algorithm to judge the appropriateness of total knee arthroplasty in the United States: a multicenter longitudinal cohort study.   Arthritis Rheumatol. 2014;66(8):2134-2143. doi:10.1002/art.38685PubMedGoogle ScholarCrossref
    69.
    Bumpass  DB, Nunley  RM.  Assessing the value of a total joint replacement.   Curr Rev Musculoskelet Med. 2012;5(4):274-282. doi:10.1007/s12178-012-9139-6PubMedGoogle ScholarCrossref
    70.
    Katz  JN, Winter  AR, Hawker  G.  Measures of the appropriateness of elective orthopaedic joint and spine procedures.   J Bone Joint Surg Am. 2017;99(4):e15. doi:10.2106/JBJS.16.00473PubMedGoogle Scholar
    71.
    American Academy of Orthopaedic Surgeons. Appropriate use criteria. Accessed November 15,2020. http://www.orthoguidelines.org/go/auc/auc.cfm?auc_id=224986
    72.
    Ghomrawi  HMK, Lee  LY-Y, Nwachukwu  BU,  et al.  Preoperative expectations associated with postoperative dissatisfaction after total knee arthroplasty: a cohort study.   J Am Acad Orthop Surg. 2020;28(4):e145-e150. doi:10.5435/JAAOS-D-18-00785PubMedGoogle ScholarCrossref
    73.
    Parikh  RB, Obermeyer  Z, Navathe  AS.  Regulation of predictive analytics in medicine.   Science. 2019;363(6429):810-812. doi:10.1126/science.aaw0029PubMedGoogle ScholarCrossref
    74.
    Loftus  TJ, Filiberto  AC, Balch  J,  et al.  Intelligent, autonomous machines in surgery.   J Surg Res. 2020;253:92-99. doi:10.1016/j.jss.2020.03.046PubMedGoogle ScholarCrossref
    ×