[Skip to Navigation]
Sign In
Figure 1.  Participant Eligibility and Enrollment
Participant Eligibility and Enrollment

aAllowing for more than 1 eligible family member per patient.

bMore than 1 family member per patient surveyed; second family member excluded.

cConsecutive sample approached until enrollment goal was met.

dNot analyzed, administrative error.

Figure 2.  Impact of Events 6 (IES-6) and Hospital Anxiety and Depression Scale (HADS) Scores at 3 and 6 Months After Admission of Patient to the Intensive Care Unit (ICU)
Impact of Events 6 (IES-6) and Hospital Anxiety and Depression Scale (HADS) Scores at 3 and 6 Months After Admission of Patient to the Intensive Care Unit (ICU)

A, The ends of the boxes represent the IQRs, horizontal lines are the means, and the whiskers are the range. HADS-a indicates HADS anxiety; HADS-d, HADS depression. B, Each cutoff score indicates significant symptoms of posttraumatic stress, anxiety, or depression.

Table 1.  Patient and Participant Demographic Characteristicsa
Patient and Participant Demographic Characteristicsa
Table 2.  Fixed-Effect Estimates From Multivariable Linear Mixed Modelsa
Fixed-Effect Estimates From Multivariable Linear Mixed Modelsa
Table 3.  Qualitative Coding Patterns Across Subgroupsa
Qualitative Coding Patterns Across Subgroupsa
1.
Zhou  F, Yu  T, Du  R,  et al.  Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.   Lancet. 2020;395(10229):1054-1062. Erratum in: Lancet. 2020;395(10229):1038. doi:10.1016/S0140-6736(20)30566-3 PubMedGoogle ScholarCrossref
2.
Bhatraju  PK, Ghassemieh  BJ, Nichols  M,  et al.  Covid-19 in critically ill patients in the Seattle region—case series.   N Engl J Med. 2020;382(21):2012-2022. Published online March 30, 2020. doi:10.1056/NEJMoa2004500 PubMedGoogle ScholarCrossref
3.
Arentz  M, Yi  E, Klaff  L,  et al.  Characteristics and outcomes of 21 critically ill patients with COVID-19 in Washington state.   JAMA. 2020;323(16):1612-1614. doi:10.1001/jama.2020.4326 PubMedGoogle ScholarCrossref
4.
Wu  C, Chen  X, Cai  Y,  et al.  Risk factors associated with acute respiratory distress syndrome and death in patients with coronavirus disease 2019 pneumonia in Wuhan, China.   JAMA Intern Med. 2020;180(7):934-943. doi:10.1001/jamainternmed.2020.0994 PubMedGoogle ScholarCrossref
5.
Azoulay  E, Cariou  A, Bruneel  F,  et al.  Symptoms of anxiety, depression, and peritraumatic dissociation in critical care clinicians managing patients with COVID-19: a cross-sectional study.   Am J Respir Crit Care Med. 2020;202(10):1388-1398. doi:10.1164/rccm.202006-2568OC PubMedGoogle ScholarCrossref
6.
Davidson  JE, Jones  C, Bienvenu  OJ.  Family response to critical illness: postintensive care syndrome-family.   Crit Care Med. 2012;40(2):618-624. doi:10.1097/CCM.0b013e318236ebf9 PubMedGoogle ScholarCrossref
7.
Gries  CJ, Engelberg  RA, Kross  EK,  et al.  Predictors of symptoms of posttraumatic stress and depression in family members after patient death in the ICU.   Chest. 2010;137(2):280-287. Published online September 17, 2009. doi:10.1378/chest.09-1291 PubMedGoogle ScholarCrossref
8.
Kross  EK, Engelberg  RA, Gries  CJ, Nielsen  EL, Zatzick  D, Curtis  JR.  ICU care associated with symptoms of depression and posttraumatic stress disorder among family members of patients who die in the ICU.   Chest. 2011;139(4):795-801. doi:10.1378/chest.10-0652 PubMedGoogle ScholarCrossref
9.
Kross  EK, Gries  CJ, Curtis  JR.  Posttraumatic stress disorder following critical illness.   Crit Care Clin. 2008;24(4):875-887, ix-x. doi:10.1016/j.ccc.2008.06.002 PubMedGoogle ScholarCrossref
10.
Lautrette  A, Darmon  M, Megarbane  B,  et al.  A communication strategy and brochure for relatives of patients dying in the ICU.   N Engl J Med. 2007;356(5):469-478. doi:10.1056/NEJMoa063446 PubMedGoogle ScholarCrossref
11.
Curtis  JR, Treece  PD, Nielsen  EL,  et al.  Randomized trial of communication facilitators to reduce family distress and intensity of end-of-life care.   Am J Respir Crit Care Med. 2016;193(2):154-162. doi:10.1164/rccm.201505-0900OC PubMedGoogle ScholarCrossref
12.
Amass  TH, Villa  G, OMahony  S,  et al.  Family care rituals in the ICU to reduce symptoms of post-traumatic stress disorder in family members-a multicenter, multinational, before-and-after intervention trial.   Crit Care Med. 2020;48(2):176-184. doi:10.1097/CCM.0000000000004113 PubMedGoogle ScholarCrossref
13.
White  DB, Angus  DC, Shields  AM,  et al; PARTNER Investigators.  A randomized trial of a family-support intervention in intensive care units.   N Engl J Med. 2018;378(25):2365-2375. doi:10.1056/NEJMoa1802637 PubMedGoogle ScholarCrossref
14.
Davidson  JE, Aslakson  RA, Long  AC,  et al.  Guidelines for family-centered care in the neonatal, pediatric, and adult ICU.   Crit Care Med. 2017;45(1):103-128. doi:10.1097/CCM.0000000000002169 PubMedGoogle ScholarCrossref
15.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Lancet. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-X PubMedGoogle ScholarCrossref
16.
Harris  PA, Taylor  R, Thielke  R, Payne  J, Gonzalez  N, Conde  JG.  Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.   J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010 PubMedGoogle ScholarCrossref
17.
Thoresen  S, Tamps  K, Hussain  A, Heir  T, Johansen  VA, Bisson  JI.  Brief measure of posttraumatic stress reactions: impact of Event Scale-6.   Soc Psychiatry Psychiatr Epidemiol. 2010;45(3):405-412. Published online May 29, 2009. doi:10.1007/s00127-009-0073-x PubMedGoogle ScholarCrossref
18.
Hosey  MM, Leoutsakos  JS, Li  X,  et al.  Screening for posttraumatic stress disorder in ARDS survivors: validation of the Impact of Event Scale-6 (IES-6).   Crit Care. 2019;23(1):276. doi:10.1186/s13054-019-2553-z PubMedGoogle ScholarCrossref
19.
Pochard  F, Azoulay  E, Chevret  S,  et al; French FAMIREA Group.  Symptoms of anxiety and depression in family members of intensive care unit patients: ethical hypothesis regarding decision-making capacity.   Crit Care Med. 2001;29(10):1893-1897. doi:10.1097/00003246-200110000-00007 PubMedGoogle ScholarCrossref
20.
Zigmond  AS, Snaith  RP.  The hospital anxiety and depression scale.   Acta Psychiatr Scand. 1983;67(6):361-370. doi:10.1111/j.1600-0447.1983.tb09716.x PubMedGoogle ScholarCrossref
21.
Wall  RJ, Engelberg  RA, Downey  L, Heyland  DK, Curtis  JR.  Refinement, scoring, and validation of the Family Satisfaction in the Intensive Care Unit (FS-ICU) survey.   Crit Care Med. 2007;35(1):271-279. doi:10.1097/01.CCM.0000251122.15053.50 PubMedGoogle ScholarCrossref
22.
Creswell  JW, Creswell  JD.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications; 2017.
23.
Tong  A, Sainsbury  P, Craig  J.  Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.   Int J Qual Health Care. 2007;19(6):349-357. doi:10.1093/intqhc/mzm042 PubMedGoogle ScholarCrossref
24.
QRS International. Crosstab query, Nvivo 12. 2018. Accessed September 1, 2021. https://help-nv.qsrinternational.com/12/win/v12.1.90-d3ea61/Content/queries/crosstab-query.htm
25.
Glaser  BG.  The constant comparative method of qualitative analysis.   Soc Probl. 1965;12(4):436-445. doi:10.2307/798843 Google ScholarCrossref
26.
Heinze  G, Wallisch  C, Dunkler  D.  Variable selection—a review and recommendations for the practicing statistician.   Biom J. 2018;60(3):431-449. doi:10.1002/bimj.201700067 PubMedGoogle ScholarCrossref
27.
Cao  J, Zhang  S.  Multiple comparison procedures.   JAMA. 2014;312(5):543-544. doi:10.1001/jama.2014.9440 PubMedGoogle ScholarCrossref
28.
Benjamini  Y, Hochberg  Y.  Controlling the false discovery rate: a practical and powerful approach to multiple testing.   J R Stat Soc B. 1995;57(1):289-300. doi:10.1111/j.2517-6161.1995.tb02031.x Google ScholarCrossref
29.
Van Buuren  S, Groothuis-Oudshoorn  K.  Mice: multivariate imputation by chained equations in R.   J Stat Soft. 2011;45(3)1-67. doi:10.18637/jss.v045.i03 Google ScholarCrossref
30.
Van Buuren  S.  Flexible IMputation of Missing Data. 2nd ed. CRC Press; 2018. doi:10.1201/9780429492259
31.
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2022. Accessed May 14, 2021. https://www.R-project.org/
32.
Hsieh  HF, Shannon  SE.  Three approaches to qualitative content analysis.   Qual Health Res. 2005;15(9):1277-1288. doi:10.1177/1049732305276687 PubMedGoogle ScholarCrossref
33.
Hochendoner  SJ, Amass  TH, Curtis  JR,  et al.  Voices from the pandemic: a qualitative study of family experiences and suggestions regarding the care of critically ill patients.   Ann Am Thorac Soc. Published online August 26, 2021. doi:10.1513/AnnalsATS.202105-629OC PubMedGoogle ScholarCrossref
34.
Greenberg  JA, Basapur  S, Quinn  TV, Bulger  JL, Glover  CM, Shah  RC.  Psychological symptoms among surrogates of critically ill patients during and before the COVID-19 pandemic.   Chest. 2021;159(6):2318-2320. doi:10.1016/j.chest.2020.12.056 PubMedGoogle ScholarCrossref
35.
Nunez  ER, Villa  G, McFadden  R,  et al.  Differences in family involvement in the bedside care of patients in the ICU based on self-identified race.   Crit Care Explor. 2021;3(3):e0365. doi:10.1097/CCE.0000000000000365 PubMedGoogle ScholarCrossref
36.
Petrinec  AB, Daly  BJ.  Post-traumatic stress symptoms in post-ICU family members: review and methodological challenges.   West J Nurs Res. 2016;38(1):57-78. doi:10.1177/0193945914544176 PubMedGoogle ScholarCrossref
37.
Chen  C, Wittenberg  E, Sullivan  SS, Lorenz  RA, Chang  YP.  The experiences of family members of ventilated COVID-19 patients in the intensive care unit: a qualitative study.   Am J Hosp Palliat Care. 2021;38(7):869-876. doi:10.1177/10499091211006914 PubMedGoogle ScholarCrossref
1 Comment for this article
The effect of online communication of family members with their patients
Fatih Tufan, Assoc Prof | Istanbul Aydin University Florya Medical Park Hospital Department of Geriatrics

I congratulate the authors for this well designed study which provides very important information.

In family members of the patients, it seems that feelings of distrust and concern about the need to take clinicians’ information at face value without being present to see for themselves is an important factor underlying PTSD.

I wonder if providing online communication between the patients and their family members would be beneficial in this regard.

CONFLICT OF INTEREST: None Reported
Original Investigation
April 25, 2022

Stress-Related Disorders of Family Members of Patients Admitted to the Intensive Care Unit With COVID-19

Author Affiliations
  • 1Division of Pulmonary Sciences and Critical Care Medicine, Department of Medicine, University of Colorado School of Medicine, Aurora
  • 2Department of Veterans Affairs, Eastern Colorado Health Care System, Aurora
  • 3Departments of Medicine, Humanities and Public Health Sciences, Penn State College of Medicine, Hershey, Pennsylvania
  • 4Department of Anesthesiology, Columbia University, New York, New York
  • 5Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, New York
  • 6Division of Pulmonary, Critical Care, and Sleep Medicine, Cambia Palliative Care Center of Excellence, University of Washington, Seattle
  • 7Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Columbia University, New York, New York
  • 8Dana Farber Cancer Institute, Boston, Massachusetts
  • 9Center for Geriatric Surgery, Brigham and Women’s Hospital, Boston, Massachusetts
  • 10Division of Internal Medicine, Department of Medicine, University of Colorado School of Medicine, Aurora
  • 11Department of Medicine, Tulane University School of Medicine, New Orleans, Louisiana
  • 12Center for Surgery and Public Health, Brigham and Women’s Hospital, Boston, Massachusetts
  • 13South Shore Health, Weymouth, Massachusetts
  • 14Department of Pulmonary Critical Care, Tulane University School of Medicine, New Orleans, Louisiana
  • 15Department of Psychiatry, Tulane University School of Medicine, New Orleans, Louisiana
  • 16Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, Penn State College of Medicine, Hershey, Pennsylvania
  • 17University of Colorado School of Medicine, Aurora
  • 18Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora
  • 19Evergreen Health Care, Kirkland, Washington
  • 20Department of Vascular Surgery, Tufts University School of Medicine, Maine Medical Center, Portland
  • 21Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle
  • 22Columbia University Vagelos College of Physicians and Surgeons, New York, New York
  • 23Department of Medicine, University of Washington, Seattle
  • 24Division of Pulmonary and Critical Care Medicine, Department of Medicine, University of Vermont Larner College of Medicine, Burlington
JAMA Intern Med. 2022;182(6):624-633. doi:10.1001/jamainternmed.2022.1118
Key Points

Question  What are the psychological sequelae of having a family member with COVID-19 admitted to the intensive care unit (ICU)?

Findings  In this prospective, mixed-methods cohort study of 330 family members of patients admitted to the ICU with COVID-19, family members had significant symptoms of posttraumatic stress disorder (PTSD) 3 months after the patients’ admission to the ICU; higher PTSD symptoms scores were significantly associated with Hispanic ethnicity, female gender, and previous medication use for a psychiatric condition. Family members with higher scores more commonly described feelings of distrust and concern about the need to take clinicians’ information at face value without being present to see for themselves.

Meaning  Having a family member with COVID-19 in the ICU was associated with a high prevalence of symptoms of PTSD, and identified associations may guide future interventions.

Abstract

Importance  The psychological symptoms associated with having a family member admitted to the intensive care unit (ICU) during the COVID-19 pandemic are not well defined.

Objective  To examine the prevalence of symptoms of stress-related disorders, primarily posttraumatic stress disorder (PTSD), in family members of patients admitted to the ICU with COVID-19 approximately 90 days after admission.

Design, Setting, and Participants  This prospective, multisite, mixed-methods observational cohort study assessed 330 family members of patients admitted to the ICU (except in New York City, which had a random sample of 25% of all admitted patients per month) between February 1 and July 31, 2020, at 8 academic-affiliated and 4 community-based hospitals in 5 US states.

Exposure  Having a family member in the ICU with COVID-19.

Main Outcomes and Measures  Symptoms of PTSD at 3 months, as defined by a score of 10 or higher on the Impact of Events Scale 6 (IES-6).

Results  A total of 330 participants (mean [SD] age, 51.2 [15.1] years; 228 [69.1%] women; 150 [52.8%] White; 92 [29.8%] Hispanic) were surveyed at the 3-month time point. Most individuals were the patients’ child (129 [40.6%]) or spouse or partner (81 [25.5%]). The mean (SD) IES-6 score at 3 months was 11.9 (6.1), with 201 of 316 respondents (63.6%) having scores of 10 or higher, indicating significant symptoms of PTSD. Female participants had an adjusted mean IES-6 score of 2.6 points higher (95% CI, 1.4-3.8; P < .001) than male participants, whereas Hispanic participants scored a mean of 2.7 points higher compared with non-Hispanic participants (95% CI, 1.0-4.3; P = .002). Those with graduate school experience had an adjusted mean score of 3.3 points lower (95% CI, 1.5-5.1; P < .001) compared with those with up to a high school degree or equivalent. Qualitative analyses found no substantive differences in the emotional or communication-related experiences between those with high vs low PTSD scores, but those with higher scores exhibited more distrust of practitioners.

Conclusions and Relevance  In this cohort study, symptoms of PTSD among family members of ICU patients with COVID-19 were high. Hispanic ethnicity and female gender were associated with higher symptoms. Those with higher scores reported more distrust of practitioners.

Introduction

The COVID-19 pandemic has tested the limits of health care systems. Among those with COVID-19 respiratory failure, mortality rates were initially as high as 60% to 80%.1-4 To avoid viral transmission during the early pandemic months, many hospitals eliminated visitors and still limit visitation to varying degrees. These measures may have added significant stress for patients, family members, and clinicians.5

Symptoms of posttraumatic stress disorder (PTSD), among other stress-related symptoms, such as depression and anxiety, have been described in family members of patients admitted to intensive care units (ICUs), and the existence of any of these symptoms has been collectively named the post–intensive care syndrome family.6 The prevalence of symptoms of PTSD, depression, and anxiety in this population vary widely but were approximately 15% to 30% before the COVID-19 pandemic,6-9 depending on the condition. Studies have demonstrated that active engagement of families at the bedside reduces stress-related symptoms,10-13 and specifically a reduction in symptoms of PTSD has been associated with increased family member participation and control at the bedside.12 Therefore, bedside engagement in the ICU is recommend by the Society of Critical Care Medicine.14 Developing an understanding of the effect on patients’ family members may help identify mechanisms to mitigate the repercussions when family members are unable to be at the bedside because of the pandemic or for other reasons. This study investigated the potential consequences of visitation restrictions and pandemic stress for families of critically ill patients with COVID-19, examining factors associated with family member stress-related symptoms. We hypothesized that these family members would have high levels of symptoms of PTSD, anxiety, and depression.

Methods
Study Design

This prospective, multisite, explanatory, sequential, mixed-methods, observational cohort study was performed at 12 US hospitals in Colorado, Washington, Louisiana, New York, and Massachusetts (eTable 1 in the Supplement). All patients admitted to the ICU with increased oxygen requirements and a COVID-19 diagnosis from February 1 through July 31, 2020, were eligible. A consecutive sample of all patients were eligible, except in New York City, where a random-sequence generator selected 25% of eligible patients per study month because of the large volume of patients. Adult family members were eligible if complete contact information was available in the patient medical record and the family members were fluent in English or Spanish, except at sites without Spanish-speaking researchers (Colorado and Massachusetts), where eligibility was limited to English fluency. Institutional review boards at each site approved all procedures. Participants provided informed consent via telephone and received written consent materials, per local policy. A $10 stipend was provided to survey participants at the University of Washington. Interview participants at Penn State College of Medicine and the University of Washington received a $25 stipend. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

Data Collection

Eligible family members were called approximately 3 to 4 months after the patients’ ICU admission date. At least 3 recruitment calls were made to eligible family members. Figure 1 outlines eligibility and refusal per STROBE guidelines.15 At 10 of 12 sites (eTable 1 in the Supplement), participants were resurveyed at approximately 6 months, at least 2 months after first survey completion. Ten sites were purposively selected for qualitative interviewing based on demographic characteristics and geographic location for diverse representation (eTable 1 in the Supplement). Consecutive participants at those sites were asked about interest in a follow-up qualitative interview. Participants were enrolled until at least 10 interviews per site were conducted (Figure 1). Patient demographic characteristics were abstracted from the medical record. All data were stored within each institution’s REDCap (Research Electronic Data Capture) electronic data capture tool.16

Outcomes
Symptoms of PTSD

The Impact of Events Scale–revised (IES-r) is a widely used and validated questionnaire to screen for symptoms of PTSD in ICU survivors and their families,10,12 and the Impact of Events Scale 6 (IES-6) is a 6-item subset of the IES-r, used in this study to minimize question burden for participants. The IES-6 correlates well with the IES-r.17,18 Each item is scored from 0 to 4; total scores range from 0 to 24. A score of 10 or higher has been associated with significant symptoms of PTSD, although not diagnostic of clinical PTSD, and scores less than 10 may still indicate clinically important symptoms of posttraumatic stress.17

Symptoms of Anxiety and Depression

The Hospital Anxiety and Depression Scale (HADS) has been validated for family members of ICU patients and is a 14-item questionnaire with 7 anxiety-related questions (HADS-a) and 7 depression-related questions (HADS-d).19,20 Each item is scored from 0 to 3, with scores of 8 or higher on either HADS-a or HADS-d associated with significant symptoms of anxiety or depression, respectively, although not diagnostic of these conditions.12,19,20

Predictor Variables
Family and Patient Demographic Characteristics

After survey completion, family members were asked about characteristics known to impact family stress disorders,7,8 including age, gender, educational level, and medication use in the prior year for psychiatric conditions (Table 1).

Medical record abstraction was performed for patient characteristics that have been previously associated with family stress disorders,7,8 including Sequential Organ Failure Assessment scores at ICU admission, length of ICU stay, exposure to specific ICU therapies (eg, mechanical ventilation), hospital death, and preexisting comorbidities using the Charlson Comorbidity Index (Table 1). Thirty percent of medical records were abstracted by 2 separate researchers, with the plan to double abstract more records if less than 95% agreement was noted on more than 5% of the records; however, double abstracting was not needed.

Family Satisfaction With Care and Decision-making

The investigators selected 12 questions, a priori, from the Family Satisfaction in the ICU-27 (FS-ICU27) questionnaire that we believed would likely be affected by the pandemic. The FS-ICU27 is a validated survey21 for family members’ experience, divided into satisfaction with care and decision-making (eTable 2 in the Supplement).

Qualitative Data

Five trained interviewers (P.A., S.D., and others) used a semistructured guide (eAppendix 1 in the Supplement) to interview family members about 4 constructs: illness narrative, stress, communication, and satisfaction with care. The interview guide was designed using an explanatory, sequential, mixed-methods design22 in which participants who had IES-6 scores of 10 or higher were asked questions to explore their experiences in the context of their high stress, and those with IES-6 scores less than 10 were asked questions related to their experiences and how they coped with stress. Interviews were audio-recorded and transcribed verbatim.

eAppendix 2 in the Supplement provides analytic details relevant to codebook development, coding, and the Consolidated Criteria for Reporting Qualitative Studies.23 Briefly, an ontological philosophical assumption22 was applied to learn the nature of families’ experiences of having a critically ill loved one with COVID-19. Using a phenomenological approach, we conducted an inductive, thematic analysis with qualitative software (NVivo, version 12).24 The final codebook, developed by 5 qualitative researchers (T.A., L.J.V., S.D., P.D.W., and 1 other) (eAppendix 2 in the Supplement), was applied by 2 analysts (S.D. and 1 other) per transcript to code the data using the constant-comparison method.25 Interrater reliability was measured using the Cohen κ calculation; conflicts were adjudicated by a third analyst (L.J.V.). Coding patterns and frequencies were reviewed by 3 authors (T.A., L.J.V., and S.D.) to develop themes.

Statistical Analysis
Power Analysis

Studies conducted before the COVID-19 pandemic9,11-13 have found that significant symptoms of PTSD occur in approximately 30% of family members of ICU patients. We hypothesized that the prevalence of PTSD symptoms in this observational cohort would be 40%. We targeted 200 complete surveys to yield a 95% CI for the true proportion of individuals who experience PTSD within a margin of error of 6.8%. This sample size also yields the ability to compute baseline correlations within a margin of error of 0.14.

Descriptive Analysis

Continuous data were summarized using means (SDs) or medians (ranges), whereas categorical data were summarized using numbers (percentages). Before modeling, we used generalized variance inflation factors to explore the potential for variance inflation that might occur in models that use all the potential confounders listed in Table 2. Because all inflation factors were less than 3, a full model was determined to be feasible (see eAppendix 3 in the Supplement for additional statistical planning).

Main Analysis

Our main analysis consisted of 3 linear mixed models (LMMs) for each outcome: IES-6 total score, HADS-a score, and HADS-d score. The LMMs contained fixed effects for all Table 2 variables (including a fixed effect for follow-up time) and participant-level random intercepts, which accounted for dependence among scores at different follow-up times from the same respondent. Outcome-based variable selection was avoided to retain unbiased hypothesis test P values26; inferences were conducted using the fully specified model in all cases. In other words, all covariates were included as fixed effects in all models; thus, model coefficients represent covariate-adjusted differences in the expected outcomes. Continuous covariates were treated as linear effects in these models, although we investigated whether nonlinear functional forms were warranted by local polynomial regression of LMM residuals vs each continuous variable (eFigure 1 in the Supplement). Categorical covariates (race, ethnicity, educational level, relationship with patient, respiratory failure, and ICU admission month) were coded with the largest category as baseline or reference, and joint tests for each were performed using likelihood ratio goodness-of-fit tests. Results from main analyses are not adjusted for multiplicity.27 Secondary analyses investigating the associations between 12 questions on the FS-ICU survey and the IES-6 score were adjusted for multiplicity using the Benjamini-Hochberg false discovery rate control procedure.28

Missing Data Methods

Although LMMs accommodate participants who are missing outcome data at certain time points, they are not well suited to missing covariate data. Furthermore, participants who were missing covariate or outcome data were not missing completely at random (eTable 3 in the Supplement). Therefore, we used multiple imputation via chained equations, which fill in missing values stochastically using models built on other observed variables (3.4% of data cells were imputed). Results are then pooled after analyses are conducted on each imputed data set.29 This multiple imputation process reduces bias while improving power because all participants are included in the analysis.30 We implemented this process for our main analyses, whereas descriptive statistics were computed using only the complete-case data. All data were analyzed using R software, version 4.0.2.31

Mixed-Methods Analysis

Quantitative models identified variables of interest that were subsequently explored using mixed-methods crosstabs and code matrices.32 Two authors (S.D. and L.J.V.) explored coding patterns within each variable grouping using the constant comparison method25 to identify common concepts and themes. Code counts displayed in the analytic matrices that varied by 20% or more between groups were considered different.

Results
Quantitative Findings

Of the 955 eligible family members, we were able to contact 652, and 330 consented to participate at the 3-month time point, with participation of 34.6% (330 of 955) of eligible family members and 50.6% (330 of 652) of those contacted. Among the 330 participants, the mean (SD) age was 51.2 (15.1) years; 228 (69.1%) were women, 101 (30.6) were men, and 1 (0.3%) was nonbinary. Of 284 participants who reported race, 71 (25.0%) were Black, African American, or African; 24 (8.5%) were Asian American or Pacific Islander, Native American, or Native Indian; 150 (52.8%) were White; and 39 (13.7%) were of another race. Of 309 participants reporting ethnicity, 92 (29.8%) were Hispanic. The mean (SD) number of days from patient ICU admission to participant contact was 122.6 (21.8), and 136 patients (41.3%) died in the hospital. Complete demographic characteristics are given in Table 1.

Figure 2 reports the IES-6 and HADS scores. The mean (SD) IES-6 score at 3 months was 11.9 (6.1), with 201 of 316 respondents (63.6%) scoring 10 or higher, indicating significant symptoms of PTSD. At the 6-month follow-up call, the mean (SD) IES-6 score was 10.1 (5.6), with 75 of 155 respondents (48.4%) scoring 10 or higher. The mean (SD) HADS-a score at the 3-month call was 7.3 (4.5), and the mean (SD) HADS-d score was 5.4 (4.2), with 141 of 314 (44.9%) having a HADS-a score of 8 or higher and 97 of 313 (31.0%) having a HADS-d score of 8 or higher, indicating significant symptoms of anxiety or depression. At 6-month follow-up, the mean (SD) HADS-a score was 6.3 (4.2), and the mean (SD) HADS-d score was 5.0 (3.9), with 53 of 154 (34.4%) having a HADS-a score of 8 or higher and 39 of 155 (25.2%) having a HADS-d score of 8 or higher.

The pooled results of our primary models for IES-6 and HADS-a/d total scores are presented in Table 2. The mean adjusted IES-6 score for women was 2.6 points higher than for men (95% CI, 1.4-3.8; P < .001). Compared with those who had not used psychiatric medication, participants who used psychiatric medication in the year before the ICU admission had a mean adjusted IES-6 score 3.0 points higher (95% CI, 1.5-4.6; P < .001). Participants who identified as Hispanic had a mean adjusted IES-6 score of 2.7 points higher than that of non-Hispanic participants (95% CI, 1.0-4.3; P = .004). Participants whose family member was admitted in April 2020 had a mean adjusted IES-6 score of 1.2 points higher compared with those admitted from February to March 2020 (95% CI, 0.1-2.4; P = .04).

Higher levels of education were associated with lower mean IES-6 scores. Compared with participants with an education level of high school or equivalent and controlling for other covariates, participants with college degrees had a mean IES-6 score of 2.3 points lower (95% CI, −4.0 to −0.7; P = .005), and those with graduate school experience had an adjusted mean IES-6 score that was 3.3 points lower (95% CI, −5.1 to −1.5; P < .001). Similar results for HADS-d were observed. When adjusted for multiplicity, no association between any FS-ICU item and IES-6 score achieved significance below a false discovery rate threshold of 5% (eTable 4 in the Supplement).

A joint likelihood ratio test for effect modification by patient mortality was nonsignificant for both IES-6 and HADS-a, indicating that the effects of interest did not differ substantially for those whose family members died. The test was statistically significant for the outcome of HADS-d (likelihood ratio χ2 = 2.02, P = .004), suggesting that the effects of interest on depression were modified by whether the patient died. It appears that family members of patients who died experienced more pronounced effects of gender, comorbidity score, race, ethnicity, educational level, relationship with patient, and ICU admission month on their HADS-d score (eTable 5 in the Supplement).

When outcomes were stratified by site, IES-6 scores were different by site (Kruskal-Wallis H = 27.9, P < .001) (eFigure 2 in the Supplement). The association of Hispanic ethnicity within site was assessed at baseline with a series of Wilcoxon rank sum tests. Participants who identified as Hispanic at the New York City site and the Kirkland, Washington, sites had significantly higher IES-6 scores than those who identified as non-Hispanic (Wilcoxon rank sum statistic = 8.96, P = .003 for Hispanic patients; Wilcoxon rank sum statistic = 5.87, P = .02 for non-Hispanic patients). For the Denver, Colorado, Boston, Massachusetts, and Seattle, Washington, area sites, those who identified as Hispanic had higher IES-6 scores than those who identified as non-Hispanic, but these finding did not reach statistical significance.

Qualitative and Mixed-Methods Findings

Of the 102 participants enrolled for qualitative interviews, 74 completed interviews (Figure 1), with full qualitative results reported separately.33 Because quantitative analysis revealed a novel association between ethnicity and IES-6 scores, we developed qualitative matrices for these subgroups (Table 3), comparing themes and codes for those with high (≥10) vs low (<10) IES-6 scores. Although we did not identify themes that were unique to the subgroups, differences in coding patterns were identified. Family members with IES-6 scores of 10 or higher more commonly described feelings of distrust and concern about taking clinicians’ information at face value without being present to see for themselves. Forty-nine of the 74 family members with complete interviews (66.2%) had IES-6 scores of 10 or higher, and 25 (33.8%) had IES-6 scores below 10. Of those 49 with IES-6 scores of 10 or higher, 18 (36.7%) mentioned distrust, whereas 2 of the 25 (8.0%) with IES-6 scores less than 10 mentioned distrust; Hispanic participants more commonly reported distrust (Table 3).

Twenty-one family members reported above-and-beyond acts of compassion (eTable 5 in the Supplement), describing experiences of exceptional communication, developing more personal relationships, extra touches for patients (eg, allowing a patient a Coca-Cola for breakfast), or addressing isolation head-on (eg, hanging signs in windows so families could identify their loved one’s room). Such events were reported more commonly by participants with IES-6 scores below 10 (10 of 25 participants [40.0%]) or who were non-Hispanic (19 of 54 participants [35.2%]) than those with IES-6 scores of 10 or higher (11 of 49 participants [22.4%]) or who were Hispanic (2 of 17 participants [11.8%]). Notably, no substantial differences were found in coding patterns between groups with regard to experiences with communication, positive emotions, or negative emotions as it related to IES-6 score or ethnicity.

Discussion

In this prospective, multicenter, mixed-methods observational cohort study, having a family member admitted to the ICU with COVID-19 was associated with high levels of symptoms of PTSD, with 63.6% of participants having an IES-6 score of 10 or higher 3 to 4 months after ICU admission. Before the pandemic, symptoms of PTSD were estimated to occur in approximately 30% of family members of ICU patients.9,11-13 Our findings align with a smaller study34 that reported high levels of stress in family members of patients with COVID-19, and our study offers several new insights as to why prevalence of PTSD was high in this population.

Our explanatory, sequential, mixed-methods design provided the opportunity to qualitatively explore our findings, revealing that those with higher scores exhibited more distrust of practitioners. As the COVID-19 pandemic continues to challenge the ability of family members to build bedside relationships with clinicians, this loss of trust may translate to an increase in stress-related disorders. As such, establishing rapport with family members in creative and innovative ways may help to offset the physical distance. To our knowledge, no previous linkages between PTSD and distrust of health care practitioners have been reported; thus, this finding warrants further study.

Our study is the first, to our knowledge, to report an association of Hispanic ethnicity with higher IES-6 scores in the 3 to 6 months after ICU admission. Although there was site-to-site variation in IES-6 scores, within-site analysis suggested that the reported impact of ethnicity transcends site-level differences in demographic characteristics. Furthermore, qualitative examination found reports of above-and-beyond acts of compassion less commonly noted in those with Hispanic ethnicity. It is possible that receiving or perceiving fewer acts of compassion may help explain the association of increased IES scores or that bedside exclusion prevented families from engaging in culturally important bedside care rituals. Prior studies highlight that those identifying as Hispanic are more likely to use touch at the bedside and be involved in patient care35 and that bedside care rituals may help reduce psychological distress.12 Furthermore, when patients or families experience loss of control, they may be more prone to develop PTSD.7,36,37 Many of the acts of compassion restored, at least to some extent, aspects of control, and these acts may foster an increased trust with clinicians. Regardless, this finding warrants further exploration. Before the COVID-19 pandemic, additional factors associated with the development of stress-related symptoms included patients’ ICU length of stay and mortality, as well as family member female gender and use of a medication in the year before enrollment for emotions or mood6-9; these factors were also associated with increased symptoms in the current study.

These data, combined with recent qualitative analyses highlighting the centrality of visitation restriction and fractured communication with health care practitioners as primary drivers of family stress symptoms,33 support the hypothesis that visitation restriction plays a role in increasing stress-related disorders in family members who could not be present at the bedside of their critically ill family member. Our study provides the opportunity to quantify and understand the apparent risks associated with this exclusion. Future studies are needed to confirm this hypothesis and identify approaches to reduce the impact of visitation restriction. As the pandemic and visitation rules of hospitals continue to shift, our results should also alert the health care community to the diverse factors associated with significant psychological distress in family members of patients in the ICU. In addition, these data may inform us of the risks assumed by family members who cannot, for reasons beyond their control (eg, geography, work, or childcare), visit their loved one during an ICU stay.

Strengths and Limitations

Our study has several strengths. First, this was a multicenter, mixed-methods, national evaluation with a multisite, ethnically and racially diverse participant population, supporting generalizability. Second, by following participants sequentially, the data provide insight into how these stress-related disorders may evolve over time. Third, we used a rigorous, mixed-methods approach to contextualize the quantitative findings.

This study also has several limitations. First is the low enrollment rate of 34.6% of all eligible family members and 50.6% of those we contacted. Although the respondent population was demographically diverse, there are likely differences between those who chose to participate and those who did not, which introduces a risk of nonresponse bias. Because avoidance is a significant component of stress-related disorders, particularly PTSD, it is possible that nonresponse bias would lead to an underestimation of the prevalence of stress disorders. Second, given that questionnaires were completed 3 months after ICU stay, there is a risk for recall bias. Although recall bias is not relevant for assessing symptoms at the time of the telephone call, it could impact assessments of satisfaction or communication in the ICU. Third, because this is an observational study, no causal relationships may be assumed; unmeasured confounders may have affected findings. To minimize this possibility, factors previously known to be associated with the development of stress-related symptoms were adjusted for during analysis. Fourth, we are unable to separate the influence of the unprecedented social context of the pandemic and marked changes in visitation practices, and both phenomena may have introduced unmeasured confounders.

Conclusions

This cohort study describes the prevalence of and factors associated with psychological distress, particularly symptoms of PTSD, in family members of patients with COVID-19 admitted to ICUs across the US in the early months of the COVID-19 pandemic. Many family members reported significant symptoms of PTSD at 3 and 6 months, more than has been seen in prepandemic populations. The implications of these findings suggest that visitation restrictions may inadvertently generate a secondary public health crisis through an epidemic of stress-related disorders among family members of ICU patients. Furthermore, these data may have relevance beyond the COVID-19 pandemic because many family members are unable to visit their loved ones during an ICU stay because of other common barriers. Additional research is needed to explore opportunities to improve family members’ experiences when they cannot be present while their loved one is admitted to the ICU and to determine the degree to which these symptoms persist and for how long.

Back to top
Article Information

Accepted for Publication: March 1, 2022.

Published Online: April 25, 2022. doi:10.1001/jamainternmed.2022.1118

Corresponding Author: Timothy Amass, MD, ScM, Division of Pulmonary Sciences and Critical Care Medicine, University of Colorado School of Medicine, 1700 N Wheeling St, Room H3-309, Aurora, CO 80045 (timothy.amass@cuanschutz.edu).

Author Contributions: Dr Amass had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Amass, Van Scoy, Cruse, McGuirl, Moss, Rhoads, Stapleton, Curtis.

Acquisition, analysis, or interpretation of data: Amass, Van Scoy, Hua, Ambler, Armstrong, Baldwin, Bernacki, Burhani, Chiurco, Cooper, Csikesz, Engelberg, Fonseca, Halvorson, Hammer, Heywood, Duda, Huang, Jin, Johnson, Tabata-Kelly, Kerr, Lane, Lee, Likosky, McGuirl, Milinic, Nielsen, Peterson, Puckey, Rea, Rhoads, Sheu, Tong, Witt, Wykowski, Yu, Stapleton, Curtis.

Drafting of the manuscript: Amass, Van Scoy, Baldwin, Huang, Kerr, Moss, Nielsen, Rhoads.

Critical revision of the manuscript for important intellectual content: Amass, Van Scoy, Hua, Ambler, Armstrong, Baldwin, Bernacki, Burhani, Chiurco, Cooper, Cruse, Csikesz, Engelberg, Fonseca, Halvorson, Hammer, Heywood, Duda, Jin, Johnson, Tabata-Kelly, Lane, Lee, Likosky, McGuirl, Milinic, Moss, Peterson, Puckey, Rea, Rhoads, Sheu, Tong, Witt, Wykowski, Yu, Stapleton, Curtis.

Statistical analysis: Amass, Baldwin, Huang, Jin, Kerr, Peterson, Curtis.

Obtained funding: Van Scoy, Johnson, Lee, Puckey, Curtis.

Administrative, technical, or material support: Amass, Van Scoy, Hua, Armstrong, Bernacki, Chiurco, Cruse, Engelberg, Fonseca, Halvorson, Hammer, Heywood, Duda, Tabata-Kelly, Kerr, Lee, Likosky, McGuirl, Nielsen, Puckey, Rea, Rhoads, Sheu, Witt, Wykowski.

Supervision: Amass, Van Scoy, Hua, Bernacki, Moss, Nielsen, Curtis.

Conflict of Interest Disclosures: Dr Hua reported receiving grants from the National Institute on Aging and the American Federation for Aging Research during the conduct of the study and funding from the National Cancer Institute unrelated to submitted work. Dr Stapleton reported receiving grants from the National Institutes of Health outside the submitted work. Dr Curtis reported receiving grants from the National Institutes of Health, serving on the Cambia Health Foundation Advisory Board, and serving on the Francis Family Foundation Advisory Board outside the submitted work. No other disclosures were reported.

References
1.
Zhou  F, Yu  T, Du  R,  et al.  Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.   Lancet. 2020;395(10229):1054-1062. Erratum in: Lancet. 2020;395(10229):1038. doi:10.1016/S0140-6736(20)30566-3 PubMedGoogle ScholarCrossref
2.
Bhatraju  PK, Ghassemieh  BJ, Nichols  M,  et al.  Covid-19 in critically ill patients in the Seattle region—case series.   N Engl J Med. 2020;382(21):2012-2022. Published online March 30, 2020. doi:10.1056/NEJMoa2004500 PubMedGoogle ScholarCrossref
3.
Arentz  M, Yi  E, Klaff  L,  et al.  Characteristics and outcomes of 21 critically ill patients with COVID-19 in Washington state.   JAMA. 2020;323(16):1612-1614. doi:10.1001/jama.2020.4326 PubMedGoogle ScholarCrossref
4.
Wu  C, Chen  X, Cai  Y,  et al.  Risk factors associated with acute respiratory distress syndrome and death in patients with coronavirus disease 2019 pneumonia in Wuhan, China.   JAMA Intern Med. 2020;180(7):934-943. doi:10.1001/jamainternmed.2020.0994 PubMedGoogle ScholarCrossref
5.
Azoulay  E, Cariou  A, Bruneel  F,  et al.  Symptoms of anxiety, depression, and peritraumatic dissociation in critical care clinicians managing patients with COVID-19: a cross-sectional study.   Am J Respir Crit Care Med. 2020;202(10):1388-1398. doi:10.1164/rccm.202006-2568OC PubMedGoogle ScholarCrossref
6.
Davidson  JE, Jones  C, Bienvenu  OJ.  Family response to critical illness: postintensive care syndrome-family.   Crit Care Med. 2012;40(2):618-624. doi:10.1097/CCM.0b013e318236ebf9 PubMedGoogle ScholarCrossref
7.
Gries  CJ, Engelberg  RA, Kross  EK,  et al.  Predictors of symptoms of posttraumatic stress and depression in family members after patient death in the ICU.   Chest. 2010;137(2):280-287. Published online September 17, 2009. doi:10.1378/chest.09-1291 PubMedGoogle ScholarCrossref
8.
Kross  EK, Engelberg  RA, Gries  CJ, Nielsen  EL, Zatzick  D, Curtis  JR.  ICU care associated with symptoms of depression and posttraumatic stress disorder among family members of patients who die in the ICU.   Chest. 2011;139(4):795-801. doi:10.1378/chest.10-0652 PubMedGoogle ScholarCrossref
9.
Kross  EK, Gries  CJ, Curtis  JR.  Posttraumatic stress disorder following critical illness.   Crit Care Clin. 2008;24(4):875-887, ix-x. doi:10.1016/j.ccc.2008.06.002 PubMedGoogle ScholarCrossref
10.
Lautrette  A, Darmon  M, Megarbane  B,  et al.  A communication strategy and brochure for relatives of patients dying in the ICU.   N Engl J Med. 2007;356(5):469-478. doi:10.1056/NEJMoa063446 PubMedGoogle ScholarCrossref
11.
Curtis  JR, Treece  PD, Nielsen  EL,  et al.  Randomized trial of communication facilitators to reduce family distress and intensity of end-of-life care.   Am J Respir Crit Care Med. 2016;193(2):154-162. doi:10.1164/rccm.201505-0900OC PubMedGoogle ScholarCrossref
12.
Amass  TH, Villa  G, OMahony  S,  et al.  Family care rituals in the ICU to reduce symptoms of post-traumatic stress disorder in family members-a multicenter, multinational, before-and-after intervention trial.   Crit Care Med. 2020;48(2):176-184. doi:10.1097/CCM.0000000000004113 PubMedGoogle ScholarCrossref
13.
White  DB, Angus  DC, Shields  AM,  et al; PARTNER Investigators.  A randomized trial of a family-support intervention in intensive care units.   N Engl J Med. 2018;378(25):2365-2375. doi:10.1056/NEJMoa1802637 PubMedGoogle ScholarCrossref
14.
Davidson  JE, Aslakson  RA, Long  AC,  et al.  Guidelines for family-centered care in the neonatal, pediatric, and adult ICU.   Crit Care Med. 2017;45(1):103-128. doi:10.1097/CCM.0000000000002169 PubMedGoogle ScholarCrossref
15.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Lancet. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-X PubMedGoogle ScholarCrossref
16.
Harris  PA, Taylor  R, Thielke  R, Payne  J, Gonzalez  N, Conde  JG.  Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support.   J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010 PubMedGoogle ScholarCrossref
17.
Thoresen  S, Tamps  K, Hussain  A, Heir  T, Johansen  VA, Bisson  JI.  Brief measure of posttraumatic stress reactions: impact of Event Scale-6.   Soc Psychiatry Psychiatr Epidemiol. 2010;45(3):405-412. Published online May 29, 2009. doi:10.1007/s00127-009-0073-x PubMedGoogle ScholarCrossref
18.
Hosey  MM, Leoutsakos  JS, Li  X,  et al.  Screening for posttraumatic stress disorder in ARDS survivors: validation of the Impact of Event Scale-6 (IES-6).   Crit Care. 2019;23(1):276. doi:10.1186/s13054-019-2553-z PubMedGoogle ScholarCrossref
19.
Pochard  F, Azoulay  E, Chevret  S,  et al; French FAMIREA Group.  Symptoms of anxiety and depression in family members of intensive care unit patients: ethical hypothesis regarding decision-making capacity.   Crit Care Med. 2001;29(10):1893-1897. doi:10.1097/00003246-200110000-00007 PubMedGoogle ScholarCrossref
20.
Zigmond  AS, Snaith  RP.  The hospital anxiety and depression scale.   Acta Psychiatr Scand. 1983;67(6):361-370. doi:10.1111/j.1600-0447.1983.tb09716.x PubMedGoogle ScholarCrossref
21.
Wall  RJ, Engelberg  RA, Downey  L, Heyland  DK, Curtis  JR.  Refinement, scoring, and validation of the Family Satisfaction in the Intensive Care Unit (FS-ICU) survey.   Crit Care Med. 2007;35(1):271-279. doi:10.1097/01.CCM.0000251122.15053.50 PubMedGoogle ScholarCrossref
22.
Creswell  JW, Creswell  JD.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications; 2017.
23.
Tong  A, Sainsbury  P, Craig  J.  Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.   Int J Qual Health Care. 2007;19(6):349-357. doi:10.1093/intqhc/mzm042 PubMedGoogle ScholarCrossref
24.
QRS International. Crosstab query, Nvivo 12. 2018. Accessed September 1, 2021. https://help-nv.qsrinternational.com/12/win/v12.1.90-d3ea61/Content/queries/crosstab-query.htm
25.
Glaser  BG.  The constant comparative method of qualitative analysis.   Soc Probl. 1965;12(4):436-445. doi:10.2307/798843 Google ScholarCrossref
26.
Heinze  G, Wallisch  C, Dunkler  D.  Variable selection—a review and recommendations for the practicing statistician.   Biom J. 2018;60(3):431-449. doi:10.1002/bimj.201700067 PubMedGoogle ScholarCrossref
27.
Cao  J, Zhang  S.  Multiple comparison procedures.   JAMA. 2014;312(5):543-544. doi:10.1001/jama.2014.9440 PubMedGoogle ScholarCrossref
28.
Benjamini  Y, Hochberg  Y.  Controlling the false discovery rate: a practical and powerful approach to multiple testing.   J R Stat Soc B. 1995;57(1):289-300. doi:10.1111/j.2517-6161.1995.tb02031.x Google ScholarCrossref
29.
Van Buuren  S, Groothuis-Oudshoorn  K.  Mice: multivariate imputation by chained equations in R.   J Stat Soft. 2011;45(3)1-67. doi:10.18637/jss.v045.i03 Google ScholarCrossref
30.
Van Buuren  S.  Flexible IMputation of Missing Data. 2nd ed. CRC Press; 2018. doi:10.1201/9780429492259
31.
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2022. Accessed May 14, 2021. https://www.R-project.org/
32.
Hsieh  HF, Shannon  SE.  Three approaches to qualitative content analysis.   Qual Health Res. 2005;15(9):1277-1288. doi:10.1177/1049732305276687 PubMedGoogle ScholarCrossref
33.
Hochendoner  SJ, Amass  TH, Curtis  JR,  et al.  Voices from the pandemic: a qualitative study of family experiences and suggestions regarding the care of critically ill patients.   Ann Am Thorac Soc. Published online August 26, 2021. doi:10.1513/AnnalsATS.202105-629OC PubMedGoogle ScholarCrossref
34.
Greenberg  JA, Basapur  S, Quinn  TV, Bulger  JL, Glover  CM, Shah  RC.  Psychological symptoms among surrogates of critically ill patients during and before the COVID-19 pandemic.   Chest. 2021;159(6):2318-2320. doi:10.1016/j.chest.2020.12.056 PubMedGoogle ScholarCrossref
35.
Nunez  ER, Villa  G, McFadden  R,  et al.  Differences in family involvement in the bedside care of patients in the ICU based on self-identified race.   Crit Care Explor. 2021;3(3):e0365. doi:10.1097/CCE.0000000000000365 PubMedGoogle ScholarCrossref
36.
Petrinec  AB, Daly  BJ.  Post-traumatic stress symptoms in post-ICU family members: review and methodological challenges.   West J Nurs Res. 2016;38(1):57-78. doi:10.1177/0193945914544176 PubMedGoogle ScholarCrossref
37.
Chen  C, Wittenberg  E, Sullivan  SS, Lorenz  RA, Chang  YP.  The experiences of family members of ventilated COVID-19 patients in the intensive care unit: a qualitative study.   Am J Hosp Palliat Care. 2021;38(7):869-876. doi:10.1177/10499091211006914 PubMedGoogle ScholarCrossref
×