[Skip to Content]
[Skip to Content Landing]
Figure 1.
Life-years Gained by Study and Projection Year
Life-years Gained by Study and Projection Year

Shown are the estimated life-years gained by SWOG trial through 2015.

Figure 2.
Cumulative Life-years Gained Through 2015 by Study
Cumulative Life-years Gained Through 2015 by Study

Each color-coded area represents cumulative life-years for 1 of 23 studies.

1.
National Cancer Institute. Cancer trends progress report: person-years of life lost. http://www.progressreport.cancer.gov/end/life_lost. Accessed March 29, 2017.
2.
Nass SJ, Moses HL, Mendelsohn J, eds; Institute of Medicine Committee on Cancer Clinical Trials and the NCI Cooperative Group Program.  A National Cancer Clinical Trials System for the 21st Century: Reinvigorating the NCI Cooperative Group Program. Washington, DC: National Academies Press; 2010. PubMed
3.
SWOG Cancer Research. Fast facts. http://www.swog.org/Media/SWOG-Fast-Facts.pdf. Published 2014. Accessed March 29, 2017.
4.
National Cancer Institute. Cancer Moonshot Blue Ribbon Panel Report. http://www.cancer.gov/brp. Published 2016. Accessed March 29, 2017.
5.
National Cancer Institute. Surveillance, Epidemiology, and End Results  Program. SEER data, 1973-2013. Released April 2016, based on the November 2015 submission. http://www.seer.cancer.gov. Accessed June 14, 2016.
6.
Kaplan  EL, Meier  P.  Nonparametric estimation from incomplete observations.  J Am Statist Assoc. 1958;53(282):457-481.Google ScholarCrossref
7.
Lee  ET. Nonparametric methods of estimating survival functions. In: Lee  ET, Wang  JW, eds.  Statistical Methods for Survival Data Analysis. 3rd ed. Hoboken, NJ: John Wiley & Sons; 2003:73-76.Crossref
8.
National Center for Health Statistics, Centers for Disease Control and Prevention. Life tables. http://www.cdc.gov/nchs/products/life_tables.htm. Accessed June 10, 2016.
9.
R Foundation. R: The R Project for Statistical Computing. https://www.R-project.org/. Published 2016. Accessed June 13, 2016.
10.
Bates  DM, Chambers  JM. Nonlinear models. In: Chambers JM, Hastie TJ, eds. Statistical Models in S. Belmont, CA: Wadsworth & Brooks/Cole; 1992.
11.
Unger  JM, Barlow  WE, Martin  DP,  et al.  Comparison of survival outcomes among cancer patients treated in and out of clinical trials.  J Natl Cancer Inst. 2014;106(3):dju002.PubMedGoogle ScholarCrossref
12.
National Institutes of Health. Research Portfolio Online Reporting Tools (RePORT). https://projectreporter.nih.gov/reporter.cfm. Accessed December 2, 2016.
13.
U.S. Bureau of Labor Statistics. Databases, tables & calculators by subject: CPI inflation calculator. http://www.bls.gov/data/inflation_calculator.htm. Accessed December 2, 2016.
14.
Alexanian  R, Haut  A, Khan  AU,  et al.  Treatment for multiple myeloma: combination chemotherapy with different melphalan dose regimens.  JAMA. 1969;208(9):1680-1685.PubMedGoogle ScholarCrossref
15.
Gottlieb  JA, Rivkin  SE, Spigel  SC,  et al.  Proceedings: superiority of adriamycin over oral nitrosoureas in patients with advanced breast carcinoma: a Southwest Cancer Chemotherapy Study Group study.  Cancer. 1974;33(2):519-526.PubMedGoogle ScholarCrossref
16.
Glucksberg  H, Rivkin  SE, Rasmussen  S,  et al.  Combination chemotherapy (CMFVP) versus L-phenylalanine mustard (L-PAM) for operable breast cancer with positive axillary nodes: a Southwest Oncology Group study.  Cancer. 1982;50(3):423-434.PubMedGoogle ScholarCrossref
17.
Salmon  SE, Haut  A, Bonnet  JD,  et al.  Alternating combination chemotherapy and levamisole improves survival in multiple myeloma: a Southwest Oncology Group study.  J Clin Oncol. 1983;1(8):453-461.PubMedGoogle ScholarCrossref
18.
Samson  MK, Rivkin  SE, Jones  SE,  et al.  Dose-response and dose-survival advantage for high versus low-dose cisplatin combined with vinblastine and bleomycin in disseminated testicular cancer: a Southwest Oncology Group study.  Cancer. 1984;53(5):1029-1035.PubMedGoogle ScholarCrossref
19.
Crawford  ED, Eisenberger  MA, McLeod  DG,  et al.  A controlled trial of leuprolide with and without flutamide in prostatic carcinoma.  N Engl J Med. 1989;321(7):419-424.PubMedGoogle ScholarCrossref
20.
Alberts  DS, Liu  PY, Hannigan  EV,  et al.  Intraperitoneal cisplatin plus intravenous cyclophosphamide versus intravenous cisplatin plus intravenous cyclophosphamide for stage III ovarian cancer.  N Engl J Med. 1996;335(26):1950-1955.PubMedGoogle ScholarCrossref
21.
Moertel  CG, Fleming  TR, Macdonald  JS,  et al.  Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma.  N Engl J Med. 1990;322(6):352-358.PubMedGoogle ScholarCrossref
22.
Salmon  SE, Crowley  JJ, Grogan  TM, Finley  P, Pugh  RP, Barlogie  B.  Combination chemotherapy, glucocorticoids, and interferon alfa in the treatment of multiple myeloma: a Southwest Oncology Group study.  J Clin Oncol. 1994;12(11):2405-2414.PubMedGoogle ScholarCrossref
23.
Grossman  HB, Natale  RB, Tangen  CM,  et al.  Neoadjuvant chemotherapy plus cystectomy compared with cystectomy alone for locally advanced bladder cancer.  N Engl J Med. 2003;349(9):859-866.PubMedGoogle ScholarCrossref
24.
Miller  TP, Dahlberg  S, Cassady  JR,  et al.  Chemotherapy alone compared with chemotherapy plus radiotherapy for localized intermediate- and high-grade non-Hodgkin’s lymphoma.  N Engl J Med. 1998;339(1):21-26.PubMedGoogle ScholarCrossref
25.
Peters  WA  III, Liu  PY, Barrett  RJ  II,  et al.  Concurrent chemotherapy and pelvic radiation therapy compared with pelvic radiation therapy alone as adjuvant therapy after radical surgery in high-risk early-stage cancer of the cervix.  J Clin Oncol. 2000;18(8):1606-1613.PubMedGoogle ScholarCrossref
26.
Albain  KS, Barlow  WE, Ravdin  PM,  et al; Breast Cancer Intergroup of North America.  Adjuvant chemotherapy and timing of tamoxifen in postmenopausal patients with endocrine-responsive, node-positive breast cancer: a phase 3, open-label, randomised controlled trial.  Lancet. 2009;374(9707):2055-2063.PubMedGoogle ScholarCrossref
27.
Al-Sarraf  M, LeBlanc  M, Giri  PG,  et al.  Chemoradiotherapy versus radiotherapy in patients with advanced nasopharyngeal cancer: phase III randomized Intergroup study 0099.  J Clin Oncol. 1998;16(4):1310-1317.PubMedGoogle ScholarCrossref
28.
Hutchins  LF, Green  SJ, Ravdin  PM,  et al.  Randomized, controlled trial of cyclophosphamide, methotrexate, and fluorouracil versus cyclophosphamide, doxorubicin, and fluorouracil with and without tamoxifen for high-risk, node-negative breast cancer: treatment results of Intergroup Protocol INT-0102.  J Clin Oncol. 2005;23(33):8313-8321.PubMedGoogle ScholarCrossref
29.
Flanigan  RC, Salmon  SE, Blumenstein  BA,  et al.  Nephrectomy followed by interferon alfa-2b compared with interferon alfa-2b alone for metastatic renal-cell cancer.  N Engl J Med. 2001;345(23):1655-1659.PubMedGoogle ScholarCrossref
30.
Macdonald  JS, Smalley  SR, Benedetti  J,  et al.  Chemoradiotherapy after surgery compared with surgery alone for adenocarcinoma of the stomach or gastroesophageal junction.  N Engl J Med. 2001;345(10):725-730.PubMedGoogle ScholarCrossref
31.
List  AF, Kopecky  KJ, Willman  CL,  et al.  Benefit of cyclosporine modulation of drug resistance in patients with poor-risk acute myeloid leukemia: a Southwest Oncology Group study.  Blood. 2001;98(12):3212-3220.PubMedGoogle ScholarCrossref
32.
Berenson  JR, Crowley  JJ, Grogan  TM,  et al.  Maintenance therapy with alternate-day prednisone improves survival in multiple myeloma patients.  Blood. 2002;99(9):3163-3168.PubMedGoogle ScholarCrossref
33.
Wozniak  AJ, Crowley  JJ, Balcerzak  SP,  et al.  Randomized trial comparing cisplatin with cisplatin plus vinorelbine in the treatment of advanced non–small-cell lung cancer: a Southwest Oncology Group study.  J Clin Oncol. 1998;16(7):2459-2465.PubMedGoogle ScholarCrossref
34.
Petrylak  DP, Tangen  CM, Hussain  MH,  et al.  Docetaxel and estramustine compared with mitoxantrone and prednisone for advanced refractory prostate cancer.  N Engl J Med. 2004;351(15):1513-1520.PubMedGoogle ScholarCrossref
35.
Mehta  RS, Barlow  WE, Albain  KS,  et al.  Combination anastrozole and fulvestrant in metastatic breast cancer.  N Engl J Med. 2012;367(5):435-444.PubMedGoogle ScholarCrossref
36.
Durie  BG, Hoering  A, Abidi  MH,  et al.  Bortezomib with lenalidomide and dexamethasone versus lenalidomide and dexamethasone alone in patients with newly diagnosed myeloma without intent for immediate autologous stem-cell transplant (SWOG S0777): a randomised, open-label, phase 3 trial.  Lancet. 2017;389(10068):519-527.PubMedGoogle ScholarCrossref
37.
National Comprehensive Cancer Network. NCCN Drugs & Biologics Compendium (NCCN Compendium). https://www.nccn.org/professionals/drug_compendium/content/contents.asp. Accessed February 16, 2017.
38.
Saad  F, Hotte  SJ.  Guidelines for the management of castrate-resistant prostate cancer.  Can Urol Assoc J. 2010;4(6):380-384.PubMedGoogle ScholarCrossref
39.
Siegel  RL, Miller  KD, Jemal  A.  Cancer statistics, 2017.  CA Cancer J Clin. 2017;67(1):7-30. PubMedGoogle ScholarCrossref
40.
Horm  JW, Sondik  EJ.  Person-years of life lost due to cancer in the United States, 1970 and 1984.  Am J Public Health. 1989;79(11):1490-1493.PubMedGoogle ScholarCrossref
41.
American Cancer Society.  Cancer Facts & Figures 2016. Atlanta, GA: American Cancer Society; 2016.
42.
Alliance for Clinical Trials in Oncology. https://www.allianceforclinicaltrialsinoncology.org/main/. Accessed March 29, 2017.
43.
ECOG-ACRIN Cancer Research Group. http://ecog-acrin.org/. Accessed March 29, 2016.
44.
NRG Oncology. https://www.nrgoncology.org/. Accessed March 29, 2016.
45.
Children’s Oncology Group. https://www.childrensoncologygroup.org/. Accessed March 29, 2016.
46.
National Cancer Institute. NCI budget and appropriations. https://www.cancer.gov/about-nci/budget. Accessed March 29, 2016.
47.
National Cancer Institute. Cancer prevalence and cost of care projections. https://costprojections.cancer.gov/expenditures.html. Accessed March 29, 2016.
48.
Lippman  SM, Klein  EA, Goodman  PJ,  et al.  Effect of selenium and vitamin E on risk of prostate cancer and other cancers: the Selenium and Vitamin E Cancer Prevention Trial (SELECT).  JAMA. 2009;301(1):39-51.PubMedGoogle ScholarCrossref
49.
Omenn  GS, Goodman  GE, Thornquist  MD,  et al.  Effects of a combination of beta carotene and vitamin A on lung cancer and cardiovascular disease.  N Engl J Med. 1996;334(18):1150-1155.PubMedGoogle ScholarCrossref
50.
Rossouw  JE, Anderson  GL, Prentice  RL,  et al; Writing Group for the Women’s Health Initiative Investigators.  Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women’s Health Initiative randomized controlled trial.  JAMA. 2002;288(3):321-333.PubMedGoogle ScholarCrossref
51.
Warren  E.  Strengthening research through data sharing.  N Engl J Med. 2016;375(5):401-403.PubMedGoogle ScholarCrossref
52.
Omnibus Budget Reconciliation Act of 1993. http://www.gpo.gov/fdsys/pkg/BILLS-103hr2264enr/pdf/BILLS-103hr2264enr.pdf. Accessed February 10, 2017.
53.
Centers for Medicare & Medicaid Services. CMS Manual System: regulations and guidance transmittal. Subject: “Compendia and Authoritative Sources for Use in the Determination of a ‘Medically Accepted Indication’ of Drugs and Biologicals Used Off-Label in an AntiCancer Chemotherapeutic Regimen.” http://www.cms.gov/Regulations-and-Guidance/Guidance/Transmittals/downloads/r96bp.pdf. Published October 24, 2008. Accessed February 10, 2017.
54.
Unger  JM, Barlow  WE, Ramsey  SD, LeBlanc  M, Blanke  CD, Hershman  DL.  The scientific impact of positive and negative phase 3 cancer clinical trials.  JAMA Oncol. 2016;2(7):875-881.PubMedGoogle ScholarCrossref
55.
Stadtmauer  EA, O’Neill  A, Goldstein  LJ,  et al; Philadelphia Bone Marrow Transplant Group.  Conventional-dose chemotherapy compared with high-dose chemotherapy plus autologous hematopoietic stem-cell transplantation for metastatic breast cancer.  N Engl J Med. 2000;342(15):1069-1076.PubMedGoogle ScholarCrossref
56.
Farquhar  C, Marjoribanks  J, Lethaby  A, Azhar  M.  High-dose chemotherapy and autologous bone marrow or stem cell transplantation versus conventional chemotherapy for women with early poor prognosis breast cancer.  Cochrane Database Syst Rev. 2016;(5):CD003139.PubMedGoogle Scholar
57.
Rettig  RA, Jacobson  PD, Farquhar  CM, Wade  MA.  False Hope: Bone Marrow Transplantation for Breast Cancer. New York, NY: Oxford University Press; 2007.
58.
DiMasi  JA, Hansen  RW, Grabowski  HG.  The price of innovation: new estimates of drug development costs.  J Health Econ. 2003;22(2):151-185.PubMedGoogle ScholarCrossref
59.
DiMasi  JA, Grabowski  HG.  The cost of biopharmaceutical R&D: is biotech different?  Manage Decis Econ. 2007;28(4-5):469-479.Google ScholarCrossref
Original Investigation
October 2017

The Effect of Positive SWOG Treatment Trials on Survival of Patients With Cancer in the US Population

Author Affiliations
  • 1SWOG Statistical Center, Seattle, Washington
  • 2Fred Hutchinson Cancer Research Center, Seattle, Washington
  • 3SWOG Group Chair’s Office, Knight Cancer Institute, Oregon Health & Science University, Portland
JAMA Oncol. 2017;3(10):1345-1351. doi:10.1001/jamaoncol.2017.0762
Key Points

Question  How have the National Cancer Institute–sponsored network cooperative cancer research groups benefited patients with cancer in the general population?

Findings  Using data from 23 positive SWOG treatment trials, this study estimated that 3.34 million life-years were gained in the population of US patients with cancer through 2015, at a cost of $125 per life-year gained.

Meaning  The National Cancer Institute’s investment in its cancer cooperative group research program has provided exceptional value and benefit to the American public through its research programs generating positive cancer treatment trials.

Abstract

Importance  Recently, tremendous prominence has been given to the investigation of the effect of different research processes as part of the Cancer Moonshot. More than half a century ago, the National Cancer Institute (NCI) established a network of publicly funded cancer cooperative research groups to systematically evaluate new treatments for efficacy and safety.

Objective  To examine the extent to which positive NCI-sponsored cancer treatment trials have benefited patients with cancer in the US population.

Design, Setting, and Participants  This investigation used study data from SWOG, an NCI-sponsored network cooperative research group. All treatment trials during SWOG’s 60-year history (1956-2016) were identified for which the new, experimental therapy provided a statistically significant improvement in overall survival. It was assumed that the new, proven treatments from these trials established new standards for cancer care in the treatment community. Twenty-three positive SWOG treatment trials were identified from a variety of different disease settings.

Main Outcomes and Measures  This study estimated population life-years gained from the 23 treatment trials through 2015 by mapping the effect of the new treatments onto the US cancer population using an area under the Kaplan-Meier survival curve approach that combined trial-specific hazard function and hazard ratio results, along with Surveillance, Epidemiology, and End Results program and life table data. Calculations were age adjusted. The US dollar return on investment was estimated as the ratio of the total investment by the NCI in the treatment trial program divided by the estimate of life-years gained.

Results  In total, 12 361 patients were enrolled to the 23 positive trials from 1965 to 2012. The study estimated that 3.34 million (95% confidence limit, 2.39-4.15 million) life-years were gained from these 23 trials through 2015. Estimates were greater than 2 million life-years gained under most model simulations. The US dollar return on investment was $125 per life-year gained.

Conclusions and Relevance  SWOG treatment trials have had a substantial impact on population survival for patients with cancer over 60 years. The NCI’s investment in its cancer cooperative group research program has provided exceptional value and benefit to the American public through its research programs generating positive cancer treatment trials.

Trial Registration  clinicaltrials.gov Identifiers: NCT00004001, NCT00075764, and NCT00644228

Introduction

Cancer is the leading cause of life-years lost in the United States, exceeding all other diseases, including heart disease.1 Cancer clinical trials represent a vital step in evaluating the efficacy and safety of new therapeutic approaches for cancer. Treatments proven to work in clinical trials often become new standards of care for patients with cancer. More than half a century ago, the National Cancer Institute (NCI) established a network of publicly funded cancer cooperative research groups to systematically evaluate new treatments for efficacy and safety.2 Little attempt has been made to quantify the impact of this clinical research system in terms of population survival gains.

SWOG, one of the original cooperative research groups, reached its 60-year anniversary in 2016. SWOG is a member of the NCI’s National Clinical Trials Network and the NCI’s Community Oncology Research Program. SWOG was launched as a pediatric oncology group in 1956 under the name Southwest Cancer Chemotherapy Study Group. In 1958, the NCI directed the study group to extend its mandate to adult cancers. In 1973, the group was renamed the Southwest Oncology Group, later shortened to SWOG. The group now conducts cancer treatment trials for a variety of different cancers in adults. It has approximately 12 000 members from cancer clinics and centers at more than 650 institutions around the United States, with more than 200 000 patients previously enrolled to SWOG trials and approximately 80 trials now active.3

Recently, tremendous prominence has been given to the investigation of the impact of different research processes as part of the Cancer Moonshot.4 In this context and given the history and durability of the cooperative group structure, we examined the extent to which positive SWOG cancer treatment trials over the decades have benefited patients with cancer in the US population in terms of extending life.

Methods

We first established the denominator of all randomized phase 3 treatment trials conducted by SWOG during the course of its 60-year history. The estimation of impact was based solely on new treatments derived from trials that showed a positive overall survival benefit in favor of the experimental arm according to the prespecified protocol design. Overall survival was defined as the time from registration or the beginning of administration of chemotherapy until death due to any cause. Alive patients, including those lost to follow-up, were censored at the date of last contact if it occurred before 5 years. To better represent community practice, treatments with positive overall survival benefits that were too toxic for investigators to recommend as new treatment were excluded. The outcome of trials with respect to overall survival was identified through article publication or internal reports of studies or through analysis of patient-level data available in the SWOG database. Information on ethical review and informed consent of participants for each of the trials was included in their study reports. Institutional review board approval of the study was not required because this analysis used secondary data that were not identifiable.

Analytical Framework

We used life-years gained as the primary measure. We estimated life-years gained from positive SWOG treatment trials using a counterfactual modeling approach. We assumed that any new, trial-proven treatment would become the standard of care in the cancer treatment community beginning at the time of initial publication of the trial results. To establish a common impact across a panel of diverse cancer types, we assumed that the treatment effect from the new, trial-proven treatment endured for 5 years based on visual inspection of the survival functions by arm for the trials included in the analysis. Also, we assumed that the beneficial effect of the new treatment would endure for all future patients with the given cancer diagnosis.

Estimation of Life-years Gained

To estimate life-years gained, we mapped the treatment effect of new treatments onto the US cancer population using data from the Surveillance, Epidemiology, and End Results 9 registry areas.5 For each positive trial, we matched the major cancer type, stage, prior cancer, surgery, sex (where appropriate), age (ie, ≥18 years), and tumor characteristic eligibility criteria from the SWOG trial to corresponding data in the Surveillance, Epidemiology, and End Results program to estimate the number of patients in the United States to whom the new treatment would apply. Calculations were conducted separately by 5-year age intervals (because the number of life-years an individual may live varies by age) and were further calculated for each yearly cohort of patients with a given cancer type beginning the year of trial publication until the end of the estimation period. Given that the Surveillance, Epidemiology, and End Results 9 registry areas have consistently represented 9.5% of the US population over time, we inflated each age-specific and year-specific estimate of the number of patients with a given cancer by approximately 10.5 (1 divided by 0.095).

We used the area under the Kaplan-Meier survival curve (AUSC) to estimate life-years lived for an average individual.6,7 Within the first 5 years after diagnosis (the treatment benefit period), the hazard rate for those receiving the standard treatment was estimated from trial data (see the Derivation of Hazard Rates and Hazard Ratios subsection below). The hazard rate for patients with the new treatment was generated using the hazard ratio for overall survival from the trial. To estimate residual AUSC in the posttreatment benefit period (ie, after 5 years), we assumed (conservatively) that maximum survival extended only until the mean (age-specific) half-life, as indicated by life table data.8 The estimation of residual AUSC was based on an exponential linear regression model, with points placed annually past the end of the treatment effect until the median half-life. The convergence to approximately zero difference between the standard treatment and new treatment curves was parameterized by a point weight parameter at the final time point at the median expected life. Higher weight parameters generated more rapidly converging curves and consequently fewer residual life-years gained. We chose a weight parameter of 100, at which overall estimates of life-years gained were found to achieve asymptotic stability: that is, further increases in the weight parameter had little impact on the area between the curves. The model is fit by nonlinear least squares “nls” in R.9,10

The difference in the AUSCs between those with the standard treatment (STD) compared with those with the new, trial-proven treatment (EXP) represents the mean life-years gained (LYG) for an average single individual due to the new treatment. This relationship is shown by the following equation: LYG = AUSCEXP − AUSCSTD.

eFigure 1 in the Supplement shows representative patients from SWOG trial S9008, demonstrating how more life-years gained accrue to younger patients (1.58 years for a 25-year-old patient) than for older patients (0.85 year for a 75-year-old patient). In particular, because more patients will be alive at the end of the treatment period for the experimental arm, life-years on average continue to accrue after the treatment effect has ended until the median half-life is attained or until the end of the estimation period (ie, 2015).

We partitioned age into 14 intervals (20-25, 26-30, 31-35, 36-40, 41-45, 46-50, 51-55, 56-60, 61-65, 66-70, 71-75, 76-80, 81-85, or >85 years). Within each age interval, there are ai individuals, and life-years gained for the individuals within the age interval i will be given by the following equation: (AUSCEXP AUSCSTD)i×ai.

To obtain life-years gained within all 14 age intervals within a given year gives the following equation:

Image description not available.

Separate calculations were performed for each successive year starting at initial trial publication. If Z is the number of years from trial publication to the end of the estimation period, then the total life-years gained is represented by the following equation:

Image description not available.

Forward projections for 2020, 2025, and 2030 were also calculated but were based on positive trials identified through 2015 only.

Finally, we repeatedly conducted random sampling of the coefficient for the treatment effect from each trial, drawing from distributions based on the observed point estimate and its variation under a normal distribution. The upper and lower 2.5% quantiles from 500 iterations defined the 95% confidence limit (CL).

Derivation of Hazard Rates and Hazard Ratios

To estimate the hazard rate for the standard arm, we obtained yearly point estimates for overall survival from patient-level trial data (if available) or from trial publications. In all instances, these point estimates were adjusted downward to better reflect survival patterns anticipated for patients in the cancer treatment population given that trial-treated patients have been observed to have better survival than non–trial-treated patients, predominantly in the first year after treatment initiation (eMethods in the Supplement).11 We used the hazard ratio if it was explicitly reported in trial publications; otherwise, the hazard ratio was estimated from patient-level trial data (if available) or based on overall survival point estimates reported in the main trial publication.

Additional and Sensitivity Analyses

The treatment effect from a SWOG trial with respect to overall survival may not have translated fully into the cancer treatment community. Prior evidence suggests that trial participation explains at most 5% of the variation in survival outcomes.11 Accordingly, we allowed the effectiveness of the translation to vary from as low as 75% of the hazard ratio for overall survival identified in the trial up to 100%. We also allowed the effect of new treatments to vary from 3 to 7 years. Finally, we allowed the model weight parameter value to estimate residual AUSC to vary, with uninform probability, within the 5 category ranges of 1:10, 11:50, 50:100, 100:200, and 200:1000.

Support of the cooperative group system represents a sizable investment on the part of funding agencies. Therefore, we also estimated the US dollar return on investment from funding agencies as the ratio of the estimated total US dollars spent funding SWOG’s entire treatment trial program divided by the estimate of life-years gained during the estimation period. Federal direct funding to SWOG is publicly available by fiscal year from 1985 onward.12 Historical yearly funding amounts were inflated to constant 2015 US dollars.13 In the absence of historical data before 1985 and given that SWOG core grants typically operate on 5-year cycles, we assumed that yearly funding before 1985 was equivalent to the yearly mean during 5 years from 1985 to 1989, representing the earliest available funding data.

Results

We identified 205 randomized trials that were activated during the 60-year history of SWOG. Twelve of the trials are still ongoing, leaving 193 total trials available for analysis. The trials examined a broad range of cancers and comprised 83 157 patient enrollments from January 1962 through December 2014. Twenty-three trials (11.9%) resulted in statistically significant improvement in overall survival for patients receiving the new, experimental therapy (eFigure 2 in the Supplement). The positive trials included 12 361 patients enrolled from 1965 to 2012 (eTable in the Supplement).14-36 The primary experimental drugs (except estramustine) are listed in the National Comprehensive Cancer Network Drugs & Biologics Compendium.37,38

Life-years Gained

Figure 1 shows the estimated life-years gained by SWOG trial through 2015. Recently completed trials typically generated fewer life-years than older trials. Life-years gained through 2015 ranged from 313 for trial S0777 (which was completed in 2015) to 748 000 for trial S7436. In contrast, 102 000 life-years are projected to have been gained by trial S0777 and 1.26 million by trial S7436 by 2030.

In total, cumulative life-years gained from all 23 positive trials is estimated to be 3.34 million (95% CL, 2.39-4.15 million) through 2015. Figure 2 shows the pattern of cumulative life-years gained through 2015. Given the duration of time under examination, estimates of cumulative life-years gained were only modestly attenuated if new treatment uptake was assumed to occur after (rather than at) the trial publication date (3.19 million if 1 year later, 3.04 million if 2 years later, and 2.90 million if 3 years later). Based on these 23 trials alone, life-years gained were projected to be 4.32 million (95% CL, 3.29-5.45 million) through 2020, 5.38 million (95% CL, 3.98-6.73 million) through 2025, and 6.29 million (95% CL, 4.80-7.83 million) through 2030. Therefore, the estimate of life-years gained will almost double during the next 15 years.

Variation in Life-years Gained by Factors

We allowed the duration of the treatment effect to vary from 3 to 7 years by 1-year intervals, the effectiveness to vary from 75% to 100% of the efficacy hazard ratio, and the weight parameter to vary as specified in the Methods section. The results for life-years gained through 2015 are shown in eFigure 3 in the Supplement, indicating that life-years gained were greater than 2 million under most (95%) combinations of the duration of the treatment effect, effectiveness, and weight parameter assumptions.

Return on Investment

We estimated the US dollar return on investment for funding agencies through support of a cooperative cancer clinical trials group. Based on the estimation method we specified, the total federal investment in SWOG treatment trials during the course of its 60-year history was $418 million. This amount includes the cost for conducting all trials during the period, including negative trials. Therefore, the US dollar return on investment for funding agencies is the ratio of the total expenditures to life-years gained, or $125 per life-year gained through 2015. Estimates per life-year gained are $113 for 2020, $106 for 2025, and $104 for 2030.

Discussion

During the course of its 60-year history, SWOG treatment trials with positive results for overall survival are estimated to have generated 3.34 million life-years gained in the United States. The estimated US dollar return on investment was $125 per life-year gained. Therefore, SWOG treatment trials have had a sizable impact on population survival for patients with cancer, at a modest cost.

The national cancer mortality rate in the United States has decreased by 25% since 1991, in part due to advances in treatment.39 Our findings are consistent with this observation, with 84.2% of the estimated 3.34 million life-years gained occurring since the 1990s. To provide additional context, the life-year gains from positive SWOG trials have returned approximately 1% of the estimated 360 million life-years lost since 1969 due to cancer (eMethods in the Supplement).1,40 Alternatively, 3.34 million life-years gained would be sufficient to provide each of the approximately 600 000 individuals who died of cancer in the United States in 2016 with 5.6 more life-years.41 Our estimate may be conservative because the model did not account for life-years gained from treatments showing better progression-free, recurrence-free, or disease-free survival. Such end points are frequently used in trials as more timely measures for the effect of treatment. Some treatments showing benefits in these outcomes in trials would also have generated survival gains in the population. Moreover, there are 3 other major NCI-sponsored adult cancer cooperative groups in addition to SWOG (Alliance for Clinical Trials in Oncology,42 ECOG-ACRIN Cancer Research Group (formed by the merger of the Eastern Cooperative Oncology Group and the American College of Radiology Imaging Network),43 and NRG Oncology44) and one major pediatric cancer group (Children’s Oncology Group45). NRG Oncology was formed from the merger of the NSABP (National Surgical Adjuvant Breast and Bowel Project), RTOG (Radiation Therapy Oncology Group), and GOG (Gynecologic Oncology Group). These groups also have long histories of successful conduct of trials leading to sizable gains in population survival.42-45

The total budget of the NCI was $5.2 billion in 2016, of which $151 million was designated for the NCI’s National Clinical Trials Network.46 Only a portion of that funding is further designated toward the conduct of phase 3 cancer treatment trials, even as national expenditures for cancer care in the United States were about $140 billion.47 Our evidence indicates that the cost of this investment per life-year gained is low ($125), an estimate that will continue to decrease over time given that population life-years gained from prior positive trials are growing at a faster rate than costs for trial conduct.

Investments in cooperative group clinical trials have benefited patients in ways that were not included in our model. Clinical trial discoveries have led to reductions in treatment toxicity and cancer morbidity, especially in the new era of targeted treatments. Large prevention trials conducted by the cooperative groups have identified interventions to prevent the development of new cancers and have shown that commonly used drugs and supplements are ineffective and even harmful.48-50 Decades of NCI-sponsored clinical trial design, monitoring, and analysis have generated important insights into the science of trial conduct, which has benefited the broader clinical trial community. In an era of increased data sharing, the clinical trial databases of the cancer cooperative groups promise more important insights that could serve as the foundations and hypotheses for future studies.51

Limitations

Our model for calculating life-years gained is limited due to its simplified representation of the complex manner in which new, trial-proven treatments translate to the cancer population. The model also required assumptions that do not fully represent real-world settings. We assumed that each of the trials changed practice, an assumption reinforced by the observation that almost all of the trial-proven treatments were included in common medical compendia, enabling payment coverage by Medicare.52,53 However, it is likely that not all of the trials changed practice alone: in some instances, trials from other research groups may also have provided positive supporting evidence for the new treatments. The model did not reflect the value of negative trials.54 As one example, many thousands of lives have likely been saved based on randomized clinical trials showing that autologous bone marrow transplant for breast cancer (a treatment that had achieved wide dissemination in the cancer treatment community, despite lack of conclusive evidence in trials) in fact did not work.55-57 Finally, the total research costs (as opposed to the phase 3 trial costs alone) per life-year gained for any new drug are inevitably higher because the conduct of a phase 3 comparative trial represents the end of the life cycle of treatment development, which also includes drug discovery, development, and early-stage trials. However, even accounting for these additional development costs, the investment US dollar per life-year gained is likely still modest because phase 3 trial costs comprise a substantial portion of the total drug development costs.58,59

Conclusions

These findings quantify the impact of one of the NCI’s large cooperative cancer clinical trial research groups on people touched by cancer. They show that the combined efforts of patient volunteers, clinical researchers, and cancer scientists have had a measurable impact on extending life for patients with cancer. Moreover, the amount of the investment required to produce these gains has been modest. Therefore, the NCI’s investment in its cancer cooperative group research program has provided exceptional value and benefit to the American public through its research programs generating positive cancer treatment trials.

Back to top
Article Information

Accepted for Publication: February 23, 2017.

Corresponding Author: Joseph M. Unger, PhD, MS, SWOG Statistical Center, Fred Hutchinson Cancer Research Center, 1100 Fairview Ave N, Room M3-C102, PO Box 19024, Seattle, WA 98109 (junger@fredhutch.org).

Published Online: June 5, 2017. doi:10.1001/jamaoncol.2017.0762

Author Contributions: Dr Unger had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: Unger, Blanke.

Drafting of the manuscript: Unger, Blanke.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Unger, LeBlanc.

Obtained funding: Blanke.

Administrative, technical, or material support: LeBlanc, Blanke.

Study supervision: Blanke.

Conflict of Interest Disclosures: None reported.

Funding/Support: Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under the following grant award numbers: 5U10CA180888-03 (Dr Blanke), 5U10CA180819-03 (Dr LeBlanc), and 5UG1CA189974 (Dr Blanke).

Role of the Funder/Sponsor: The funding organization had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Meeting Presentation: This paper was presented at the annual meeting of the American Society of Clinical Oncology; June 5, 2017; Chicago, Illinois.

Disclaimer: The content of this publication is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

References
1.
National Cancer Institute. Cancer trends progress report: person-years of life lost. http://www.progressreport.cancer.gov/end/life_lost. Accessed March 29, 2017.
2.
Nass SJ, Moses HL, Mendelsohn J, eds; Institute of Medicine Committee on Cancer Clinical Trials and the NCI Cooperative Group Program.  A National Cancer Clinical Trials System for the 21st Century: Reinvigorating the NCI Cooperative Group Program. Washington, DC: National Academies Press; 2010. PubMed
3.
SWOG Cancer Research. Fast facts. http://www.swog.org/Media/SWOG-Fast-Facts.pdf. Published 2014. Accessed March 29, 2017.
4.
National Cancer Institute. Cancer Moonshot Blue Ribbon Panel Report. http://www.cancer.gov/brp. Published 2016. Accessed March 29, 2017.
5.
National Cancer Institute. Surveillance, Epidemiology, and End Results  Program. SEER data, 1973-2013. Released April 2016, based on the November 2015 submission. http://www.seer.cancer.gov. Accessed June 14, 2016.
6.
Kaplan  EL, Meier  P.  Nonparametric estimation from incomplete observations.  J Am Statist Assoc. 1958;53(282):457-481.Google ScholarCrossref
7.
Lee  ET. Nonparametric methods of estimating survival functions. In: Lee  ET, Wang  JW, eds.  Statistical Methods for Survival Data Analysis. 3rd ed. Hoboken, NJ: John Wiley & Sons; 2003:73-76.Crossref
8.
National Center for Health Statistics, Centers for Disease Control and Prevention. Life tables. http://www.cdc.gov/nchs/products/life_tables.htm. Accessed June 10, 2016.
9.
R Foundation. R: The R Project for Statistical Computing. https://www.R-project.org/. Published 2016. Accessed June 13, 2016.
10.
Bates  DM, Chambers  JM. Nonlinear models. In: Chambers JM, Hastie TJ, eds. Statistical Models in S. Belmont, CA: Wadsworth & Brooks/Cole; 1992.
11.
Unger  JM, Barlow  WE, Martin  DP,  et al.  Comparison of survival outcomes among cancer patients treated in and out of clinical trials.  J Natl Cancer Inst. 2014;106(3):dju002.PubMedGoogle ScholarCrossref
12.
National Institutes of Health. Research Portfolio Online Reporting Tools (RePORT). https://projectreporter.nih.gov/reporter.cfm. Accessed December 2, 2016.
13.
U.S. Bureau of Labor Statistics. Databases, tables & calculators by subject: CPI inflation calculator. http://www.bls.gov/data/inflation_calculator.htm. Accessed December 2, 2016.
14.
Alexanian  R, Haut  A, Khan  AU,  et al.  Treatment for multiple myeloma: combination chemotherapy with different melphalan dose regimens.  JAMA. 1969;208(9):1680-1685.PubMedGoogle ScholarCrossref
15.
Gottlieb  JA, Rivkin  SE, Spigel  SC,  et al.  Proceedings: superiority of adriamycin over oral nitrosoureas in patients with advanced breast carcinoma: a Southwest Cancer Chemotherapy Study Group study.  Cancer. 1974;33(2):519-526.PubMedGoogle ScholarCrossref
16.
Glucksberg  H, Rivkin  SE, Rasmussen  S,  et al.  Combination chemotherapy (CMFVP) versus L-phenylalanine mustard (L-PAM) for operable breast cancer with positive axillary nodes: a Southwest Oncology Group study.  Cancer. 1982;50(3):423-434.PubMedGoogle ScholarCrossref
17.
Salmon  SE, Haut  A, Bonnet  JD,  et al.  Alternating combination chemotherapy and levamisole improves survival in multiple myeloma: a Southwest Oncology Group study.  J Clin Oncol. 1983;1(8):453-461.PubMedGoogle ScholarCrossref
18.
Samson  MK, Rivkin  SE, Jones  SE,  et al.  Dose-response and dose-survival advantage for high versus low-dose cisplatin combined with vinblastine and bleomycin in disseminated testicular cancer: a Southwest Oncology Group study.  Cancer. 1984;53(5):1029-1035.PubMedGoogle ScholarCrossref
19.
Crawford  ED, Eisenberger  MA, McLeod  DG,  et al.  A controlled trial of leuprolide with and without flutamide in prostatic carcinoma.  N Engl J Med. 1989;321(7):419-424.PubMedGoogle ScholarCrossref
20.
Alberts  DS, Liu  PY, Hannigan  EV,  et al.  Intraperitoneal cisplatin plus intravenous cyclophosphamide versus intravenous cisplatin plus intravenous cyclophosphamide for stage III ovarian cancer.  N Engl J Med. 1996;335(26):1950-1955.PubMedGoogle ScholarCrossref
21.
Moertel  CG, Fleming  TR, Macdonald  JS,  et al.  Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma.  N Engl J Med. 1990;322(6):352-358.PubMedGoogle ScholarCrossref
22.
Salmon  SE, Crowley  JJ, Grogan  TM, Finley  P, Pugh  RP, Barlogie  B.  Combination chemotherapy, glucocorticoids, and interferon alfa in the treatment of multiple myeloma: a Southwest Oncology Group study.  J Clin Oncol. 1994;12(11):2405-2414.PubMedGoogle ScholarCrossref
23.
Grossman  HB, Natale  RB, Tangen  CM,  et al.  Neoadjuvant chemotherapy plus cystectomy compared with cystectomy alone for locally advanced bladder cancer.  N Engl J Med. 2003;349(9):859-866.PubMedGoogle ScholarCrossref
24.
Miller  TP, Dahlberg  S, Cassady  JR,  et al.  Chemotherapy alone compared with chemotherapy plus radiotherapy for localized intermediate- and high-grade non-Hodgkin’s lymphoma.  N Engl J Med. 1998;339(1):21-26.PubMedGoogle ScholarCrossref
25.
Peters  WA  III, Liu  PY, Barrett  RJ  II,  et al.  Concurrent chemotherapy and pelvic radiation therapy compared with pelvic radiation therapy alone as adjuvant therapy after radical surgery in high-risk early-stage cancer of the cervix.  J Clin Oncol. 2000;18(8):1606-1613.PubMedGoogle ScholarCrossref
26.
Albain  KS, Barlow  WE, Ravdin  PM,  et al; Breast Cancer Intergroup of North America.  Adjuvant chemotherapy and timing of tamoxifen in postmenopausal patients with endocrine-responsive, node-positive breast cancer: a phase 3, open-label, randomised controlled trial.  Lancet. 2009;374(9707):2055-2063.PubMedGoogle ScholarCrossref
27.
Al-Sarraf  M, LeBlanc  M, Giri  PG,  et al.  Chemoradiotherapy versus radiotherapy in patients with advanced nasopharyngeal cancer: phase III randomized Intergroup study 0099.  J Clin Oncol. 1998;16(4):1310-1317.PubMedGoogle ScholarCrossref
28.
Hutchins  LF, Green  SJ, Ravdin  PM,  et al.  Randomized, controlled trial of cyclophosphamide, methotrexate, and fluorouracil versus cyclophosphamide, doxorubicin, and fluorouracil with and without tamoxifen for high-risk, node-negative breast cancer: treatment results of Intergroup Protocol INT-0102.  J Clin Oncol. 2005;23(33):8313-8321.PubMedGoogle ScholarCrossref
29.
Flanigan  RC, Salmon  SE, Blumenstein  BA,  et al.  Nephrectomy followed by interferon alfa-2b compared with interferon alfa-2b alone for metastatic renal-cell cancer.  N Engl J Med. 2001;345(23):1655-1659.PubMedGoogle ScholarCrossref
30.
Macdonald  JS, Smalley  SR, Benedetti  J,  et al.  Chemoradiotherapy after surgery compared with surgery alone for adenocarcinoma of the stomach or gastroesophageal junction.  N Engl J Med. 2001;345(10):725-730.PubMedGoogle ScholarCrossref
31.
List  AF, Kopecky  KJ, Willman  CL,  et al.  Benefit of cyclosporine modulation of drug resistance in patients with poor-risk acute myeloid leukemia: a Southwest Oncology Group study.  Blood. 2001;98(12):3212-3220.PubMedGoogle ScholarCrossref
32.
Berenson  JR, Crowley  JJ, Grogan  TM,  et al.  Maintenance therapy with alternate-day prednisone improves survival in multiple myeloma patients.  Blood. 2002;99(9):3163-3168.PubMedGoogle ScholarCrossref
33.
Wozniak  AJ, Crowley  JJ, Balcerzak  SP,  et al.  Randomized trial comparing cisplatin with cisplatin plus vinorelbine in the treatment of advanced non–small-cell lung cancer: a Southwest Oncology Group study.  J Clin Oncol. 1998;16(7):2459-2465.PubMedGoogle ScholarCrossref
34.
Petrylak  DP, Tangen  CM, Hussain  MH,  et al.  Docetaxel and estramustine compared with mitoxantrone and prednisone for advanced refractory prostate cancer.  N Engl J Med. 2004;351(15):1513-1520.PubMedGoogle ScholarCrossref
35.
Mehta  RS, Barlow  WE, Albain  KS,  et al.  Combination anastrozole and fulvestrant in metastatic breast cancer.  N Engl J Med. 2012;367(5):435-444.PubMedGoogle ScholarCrossref
36.
Durie  BG, Hoering  A, Abidi  MH,  et al.  Bortezomib with lenalidomide and dexamethasone versus lenalidomide and dexamethasone alone in patients with newly diagnosed myeloma without intent for immediate autologous stem-cell transplant (SWOG S0777): a randomised, open-label, phase 3 trial.  Lancet. 2017;389(10068):519-527.PubMedGoogle ScholarCrossref
37.
National Comprehensive Cancer Network. NCCN Drugs & Biologics Compendium (NCCN Compendium). https://www.nccn.org/professionals/drug_compendium/content/contents.asp. Accessed February 16, 2017.
38.
Saad  F, Hotte  SJ.  Guidelines for the management of castrate-resistant prostate cancer.  Can Urol Assoc J. 2010;4(6):380-384.PubMedGoogle ScholarCrossref
39.
Siegel  RL, Miller  KD, Jemal  A.  Cancer statistics, 2017.  CA Cancer J Clin. 2017;67(1):7-30. PubMedGoogle ScholarCrossref
40.
Horm  JW, Sondik  EJ.  Person-years of life lost due to cancer in the United States, 1970 and 1984.  Am J Public Health. 1989;79(11):1490-1493.PubMedGoogle ScholarCrossref
41.
American Cancer Society.  Cancer Facts & Figures 2016. Atlanta, GA: American Cancer Society; 2016.
42.
Alliance for Clinical Trials in Oncology. https://www.allianceforclinicaltrialsinoncology.org/main/. Accessed March 29, 2017.
43.
ECOG-ACRIN Cancer Research Group. http://ecog-acrin.org/. Accessed March 29, 2016.
44.
NRG Oncology. https://www.nrgoncology.org/. Accessed March 29, 2016.
45.
Children’s Oncology Group. https://www.childrensoncologygroup.org/. Accessed March 29, 2016.
46.
National Cancer Institute. NCI budget and appropriations. https://www.cancer.gov/about-nci/budget. Accessed March 29, 2016.
47.
National Cancer Institute. Cancer prevalence and cost of care projections. https://costprojections.cancer.gov/expenditures.html. Accessed March 29, 2016.
48.
Lippman  SM, Klein  EA, Goodman  PJ,  et al.  Effect of selenium and vitamin E on risk of prostate cancer and other cancers: the Selenium and Vitamin E Cancer Prevention Trial (SELECT).  JAMA. 2009;301(1):39-51.PubMedGoogle ScholarCrossref
49.
Omenn  GS, Goodman  GE, Thornquist  MD,  et al.  Effects of a combination of beta carotene and vitamin A on lung cancer and cardiovascular disease.  N Engl J Med. 1996;334(18):1150-1155.PubMedGoogle ScholarCrossref
50.
Rossouw  JE, Anderson  GL, Prentice  RL,  et al; Writing Group for the Women’s Health Initiative Investigators.  Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women’s Health Initiative randomized controlled trial.  JAMA. 2002;288(3):321-333.PubMedGoogle ScholarCrossref
51.
Warren  E.  Strengthening research through data sharing.  N Engl J Med. 2016;375(5):401-403.PubMedGoogle ScholarCrossref
52.
Omnibus Budget Reconciliation Act of 1993. http://www.gpo.gov/fdsys/pkg/BILLS-103hr2264enr/pdf/BILLS-103hr2264enr.pdf. Accessed February 10, 2017.
53.
Centers for Medicare & Medicaid Services. CMS Manual System: regulations and guidance transmittal. Subject: “Compendia and Authoritative Sources for Use in the Determination of a ‘Medically Accepted Indication’ of Drugs and Biologicals Used Off-Label in an AntiCancer Chemotherapeutic Regimen.” http://www.cms.gov/Regulations-and-Guidance/Guidance/Transmittals/downloads/r96bp.pdf. Published October 24, 2008. Accessed February 10, 2017.
54.
Unger  JM, Barlow  WE, Ramsey  SD, LeBlanc  M, Blanke  CD, Hershman  DL.  The scientific impact of positive and negative phase 3 cancer clinical trials.  JAMA Oncol. 2016;2(7):875-881.PubMedGoogle ScholarCrossref
55.
Stadtmauer  EA, O’Neill  A, Goldstein  LJ,  et al; Philadelphia Bone Marrow Transplant Group.  Conventional-dose chemotherapy compared with high-dose chemotherapy plus autologous hematopoietic stem-cell transplantation for metastatic breast cancer.  N Engl J Med. 2000;342(15):1069-1076.PubMedGoogle ScholarCrossref
56.
Farquhar  C, Marjoribanks  J, Lethaby  A, Azhar  M.  High-dose chemotherapy and autologous bone marrow or stem cell transplantation versus conventional chemotherapy for women with early poor prognosis breast cancer.  Cochrane Database Syst Rev. 2016;(5):CD003139.PubMedGoogle Scholar
57.
Rettig  RA, Jacobson  PD, Farquhar  CM, Wade  MA.  False Hope: Bone Marrow Transplantation for Breast Cancer. New York, NY: Oxford University Press; 2007.
58.
DiMasi  JA, Hansen  RW, Grabowski  HG.  The price of innovation: new estimates of drug development costs.  J Health Econ. 2003;22(2):151-185.PubMedGoogle ScholarCrossref
59.
DiMasi  JA, Grabowski  HG.  The cost of biopharmaceutical R&D: is biotech different?  Manage Decis Econ. 2007;28(4-5):469-479.Google ScholarCrossref
×