Patient-Important Outcomes in Registered Diabetes Trials

Context Concerns about the safety and efficacy of diabetes interventions persist, in part because randomized clinical trials (RCTs) have not measured their effect on patient-important outcomes, ie, death and quality of life (morbidity, pain, function).

Objective To systematically determine the extent to which ongoing and future RCTs in diabetes will ascertain patient-important outcomes.

Data Sources On November 10, 2007, we searched primary RCT registries ClinicalTrials.gov (http://www.clinicaltrials.gov), International Standard Randomized Controlled Trial Number Register (http://isrctn.org), and Australian New Zealand Clinical Trials Registry (http://www.anzctr.org.au).

Study Selection We identified phase 2 through 4 RCTs enrolling patients with diabetes. Of 2019 RCTs, 1054 proved eligible. We randomly sampled 50% of the eligible RCTs (527 of 1054) and selected 436 registered since registration became mandatory (2004).

Data Extraction Pairs of reviewers working independently collected study characteristics and determined the outcomes measured and their type (physiological outcomes, surrogate outcomes thought to reflect an increased risk for patient-important outcomes, and patient-important outcomes).

Results Of the 436 registered RCTs included in this analysis, 24 (6%) had not started enrollment, 109 (25%) were actively enrolling, and 303 (69%) had completed enrollment. Primary outcomes were patient-important outcomes in only 78 of 436 RCTs (18%; 95% confidence interval [CI], 14%-22%), physiological and laboratory outcomes in 69 of 436 (16%; 95% CI, 13%-20%), and surrogate outcomes in 268 of 436 (61%; 95% CI, 57%-66%). Patient-important outcomes were reported as primary or secondary outcomes in 201 of 436 (46%; 95% CI, 41%-51%). In multivariate analysis, large trials (odds ratio [OR], 1.10; 95% CI, 1.02-1.19 for every additional 100 patients) and trials of longer duration (OR, 1.03; 95% CI, 1.01-1.06 for every additional 30 days) were more likely while parallel design RCTs (OR, 0.15; 95% CI, 0.05-0.44) and type 2 diabetes trials (OR, 0.23; 95% CI, 0.09-0.61) were less likely to assess patient-important outcomes as a primary outcome.

Conclusion In this sample of registered ongoing RCTs in diabetes, only 18% included patient-important outcomes as primary outcomes.

Trials measuring biochemical and surrogate markers may help researchers understand how and to what extent interventions could affect health. The value of these interventions remains unclear until trials test their effect on outcomes that are important to patients and clinicians. We have previously reported¹ that only 21% of diabetes randomized clinical trials (RCTs) published in high-impact medical journals as of 2003 reported on patient-important outcomes, that is, death and quality of life (morbidity, pain, function)^2,3; in contrast, 60% measured physiological or laboratory outcomes.¹ Diabetologists have engaged in limited debate regarding the paucity of trials measuring patient-important outcomes.^4-6

In the wake of policy debates concerning rosiglitazone, those involved have called for trials of antidiabetes agents that measure cardiovascular end points important to patients.⁷ Nevertheless, authors of a recent consensus statement on how to choose antidiabetic agents to treat patients with type 2 diabetes based their recommendations on trials that measure the effect of these agents on physiological or surrogate markers of poor validity.⁸ In both cases, the authors looked forward to ongoing trials' assessing patient-important outcomes.

Are ongoing diabetes trials likely to be more informative to patients and clinicians? We used a novel approach of querying large public clinical trial registries to systematically determine the extent to which ongoing and future registered RCTs plan to measure patient-important outcomes in patients with diabetes.

Eligible registered RCTs exclusively studied patients with any form of diabetes receiving at least 2 interventions, 1 of which served as control. When trials were registered more than once, we retained the initial registration. We included registered RCTs regardless of their language, size, or their primary objectives. We did not exclude trials that were completed or closed for enrollment.

On November 10, 2007, we searched for phase 2 to 4 RCTs enrolling patients with diabetes registered in 3 large public-access trial registries: ClinicalTrials.gov (http://www.clinicaltrials.gov), International Standard Randomized Controlled Trial Number Register (http://isrctn.org), and Australian New Zealand Clinical Trials Registry (http://www.anzctr.org.au/). We chose to use these 3 registries because they meet the proposed WHO criteria for a primary register.⁹

Teams of 2 reviewers working independently screened all potentially eligible studies identified in the initial search. When a disagreement was noted, the first step in resolving it was by consensus (ie, the 2 reviewers discussed among themselves to reach a decision). If they did not agree, then we used arbitration (ie, a third reviewer who is an experienced diabetologist and methodologist made the decision). In this process, we often had to Google the trial to find if it had an official Web site that offered more details than what is in registries. The majority of the disagreements were resolved by consensus among the 3 reviewers. There were only 4 instances requiring arbitration. Eligible studies included RCTs enrolling patients with diabetes. They had to report outcomes of interest in the registries.

Because we planned to test up to 10 predictors of trials that sought to measure patient-important outcomes in multivariate analyses, we estimated that we needed at least 100 trials that measured patient-important outcomes and 100 trials that did not to be able to test these models with a reasonably low risk of overfitting. We extracted data from 30 eligible clinical trials as a pilot and noted that about 25% of these chose patient-important outcomes as primary outcome measures. This meant that we needed 400 trials. We chose to focus on trials registered after registration became mandatory in 2004.

Working in duplicate, we used a standardized form to extract data regarding study characteristics that included registration details, trial funding, trial phase, region and number of centers, planned sample size, enrollment status, whether trials were completed or were active, design, blinding, patient characteristics (age, diabetes type), and type of intervention (drug vs placebo, active intervention, usual care, or no intervention; nutrition; education, counseling or physiological interventions, or other interventions).

Two reviewers working independently and in a blinded fashion classified trial outcomes into 3 categories.

Patient-Important Outcomes: death and quality of life (major morbid events such as stroke, myocardial infarction, amputation, loss of vision, and end-stage renal disease; minor morbid events such as hypoglycemic events, delayed wound healing, infection, and visual disturbances; and pain and functional status).

Surrogate Outcomes: intermediate end points that may indicate disease progression and increased risk for patient-important outcomes (eg, glycated hemoglobin, cholesterol levels, worsening renal function, and need for retinal photocoagulation).

Physiological and Laboratory Outcomes: assessed response to physiological or laboratory maneuvers without direct tangible effects on patients (eg, insulin, C-peptide levels).

We used the κ statistic to measure chance-adjusted interobserver agreement about the nature of outcomes.

We used descriptive statistics to characterize the reporting of outcomes in registered diabetes trials. Univariate analyses explored associations between trial characteristics and likelihood of reporting patient-important outcomes. To test predictors, we constructed a multivariate model (using logistic regression) with patient-important outcomes (yes/no) as the dependent variable and the following planned independent predictors: study sponsor (for profit or mixed vs not for profit), trial design (parallel vs crossover), trial phase (3 or 4 vs 2), number of patients, type of diabetes (type 2 diabetes vs others). The rationale for choosing these predictors is (1) measuring patient-important outcomes, such as death or stroke, is less likely to occur in studies with crossover design; (2) phase 2 trials are probably less likely to report patient-important outcomes because they focus on dosing, safety, and feasibility of interventions; (3) smaller trials may be less likely to report patient-important outcomes because they are not powered for these outcomes; (4) trials enrolling patients with type 2 diabetes, who are at higher risk due to comorbidities, may be more likely to assess patient-important outcomes; and (5) trials with shorter follow-up duration are expected to report less patient-important outcomes because many complications of diabetes require longer time to occur.

Odds ratios (ORs) and their associated 95% confidence intervals (CI) characterize the strength of association between predictors and outcome. This is similar to setting a significance level at 5% and conducting 2-sided hypothesis testing. When this 95% CI does not include 1 then the respective P value would be <.05 and the association statistically significant. We entered data in duplicate and conducted analyses using StatsDirect 2.5.4 (StatsDirect Ltd, Cheshire, England). Considering that trials are usually powered to detect predetermined significant differences in their primary outcomes, we considered primary outcomes in our main analysis and conducted sensitivity analysis using both primary and secondary outcomes.

The Figure describes the flow of articles. After rigorously screening 2019 potentially eligible RCTs in duplicate, 1054 proved eligible. We randomly sampled 50% of the eligible RCTs (527 of 1054) and selected those registered since registration became mandatory in 2004 (436 trials; trial numbers are available in the online eTable).

The majority of registered trials were phase 3 or 4, parallel design RCTs conducted in Europe or North America involving adults with type 2 diabetes, testing a drug and registered in ClinicalTrials.gov. Of these, 6% (24 of 436) had not started enrollment, 25% (109 of 436) were actively enrolling, and 69% (303 of 436) had finished enrollment. Sixty-four (21%) of the 303 trials that finished enrollment were designated as active and not completed. A MEDLINE search for publication status was conducted on a 10% random sample (46 trials) and revealed that 2 (4%) of the 46 included trials had been published.

Primary outcomes were patient-important outcomes in only 78 of 436 RCTs (18%; 95% CI, 14%-22%), physiological and laboratory outcomes in 69 of 436 (16%; 95% CI, 13%-20%), and surrogate outcomes in 268 of 436 (61%; 95% CI, 57%-66%). Included trials and the classification of primary outcomes are described in Table 1. Patient-important outcomes were reported as primary or secondary outcomes in 201 of 436 (46%; 95% CI, 41%-51%). Included trials and the classification of primary and secondary outcomes are described in Table 2. Because of poor reporting, we could not classify the outcomes in 21 RCTs. Of the 201 RCTs reporting patient-important outcomes, 27 assessed mortality, 38 major morbidity, 87 minor morbidity, 33 quality of life, 25 pain, and 15 functional status, as primary or secondary end points. Chance-adjusted interreviewer agreement for outcome determination ranged from κ = 0.64 to 0.88 between teams of reviewers.

The results of univariate and multivariate logistic regression analyses are summarized in Table 3 depicting 2 models; the first is based on primary outcomes and the second is based on both primary and secondary outcomes.

In multivariate analysis, independent predictors of patient-important outcomes were larger trial size, longer trial duration, and crossover-design RCTs. Trials of patients with type 2 diabetes were significantly less likely to report patient-important outcomes. Study sponsor and trial phase were not associated with consideration of patient-important outcomes. The exclusion of phase 2 trials from analyses changed the proportion of trials reporting patient-important outcomes as a primary outcome from 18% to 19%.

In multivariate analysis, the independent predictors of patient-important outcomes were trial length and trial phase (phase 3 or 4 vs phase 2). Trials of patients with type 2 diabetes were significantly less likely to report patient-important outcomes. Trial size, study sponsor, and trial design (parallel vs crossover) were not associated with consideration of patient-important outcomes. The exclusion of phase 2 trials from analysis changed the proportion of trials reporting patient-important outcomes as their primary or secondary outcome from 46% to 52%.

Only a minority of registered randomized trials in diabetes measure outcomes of importance to patients as primary end points. Trial size, duration, crossover design, and patient population (less patient-important outcomes in trials of type 2 diabetes) proved independent predictors of trials assessing patient-important outcomes, reflecting the need for larger and longer trials to assess the impact of interventions on outcome.

The inferences from this review are limited by inherent limitations of trial registries. Some trials did not report all the variables assessed in this review. Sample size of trials, if reported, did not contain information about how it was determined and which of the outcomes drove the sample-size calculations. Furthermore, registries did not indicate whether a trial was a pilot study, in which case surrogate end points may be justified. There was significant variability in the reporting of details of trial design as well as outcomes within and between the 3 trial registries. In addition, the registries have limited advanced search capabilities compared with MEDLINE or other electronic databases of published literature. Another key area that was lacking is the ability to allow Web site links in critical fields. Researcher-controlled Web sites may be changed at any time, therefore potentially affecting the integrity of the record allowing for information deficits or unacknowledged changes after the study starts enrolling. These factors may have had an effect on the completeness of assessment of trial description and on the reproducibility of our outcomes classification.

There are few trials reporting physiological and laboratory outcomes represented in our findings compared with our previous report based on published diabetes RCTs¹: while 119/199 (60%) of published RCTs in our previous report assessed physiological and laboratory measures, we noted that only 57 of 436 (13%) registered trials in the present study measured physiological and laboratory outcomes. This suggests that RCTs' assessing such outcomes are not getting registered, that registration bias leads to overregistration of trials measuring patient-important outcomes, or that publication bias differentially affects trials that measure patient-important outcomes. This concern is reinforced because many major journals have endorsed a policy of mandatory RCT registration prior to publication,¹⁰ the policy statement indicated the registries we used as meeting all their requirements, and trialists tend to submit trials measuring patient-important outcomes to higher-impact journals (ie, 62% of published diabetes RCTs appeared in the top general medical journals). Thus, our results may represent an overestimate of the extent to which future RCTs plan to measure patient-important outcomes and thus provide an excessively optimistic forecast.

The validity of the findings is strengthened by our reasonably reproducible judgments about the type of outcomes and our parsimonious analyses. We used trial registries as a novel source of data for studying and forecasting clinical trial methodology.

The use of registries, while promising, has several challenges. The search engines are not user friendly and do not serve the needs of heterogeneous users. As others have reported, we have also found duplicate trial registration.¹¹ Finally, we found inconsistency in the extent of use of the registries' fields and in the detail reported (in some cases linking the registration to study or sponsor Web sites in which the information promised was not available). Notwithstanding these concerns, clinical trial registries represent a potentially comprehensive resource for forecasting evidence in a given field thanks to journal editors' policy initiatives requiring prospective trial registration,¹² and the World Health Organization's efforts to standardize and integrate trial registries.¹³

There have been 2 previous reports assessing published RCTs related to diabetes and reporting of patient-important outcomes.^1,14 McIver and Dineen evaluated 79 RCTs published in 1994-1995 and indexed in MEDLINE¹⁴ and found that 9% measured patient-important outcomes. Montori et al reported that 42 of 199 (21%) RCTs published in 2003 in top biomedical journals reported patient-important outcomes as their primary outcomes, whereas surrogate outcomes were reported in 38 of 199 (19%) and the rest of the trials measured various laboratory and physiological outcomes.¹ Although it is difficult to directly compare, the use of surrogate end points seems to have increased from 19% of trials in 2003 to 61% of trials noted in this investigation.

One potential reason for the apparent increase in trials measuring surrogate end points is the preference of researchers and funding agencies to obtain results faster, with fewer patients and at lower costs. A major downside of such trials is that the results cannot be used with confidence in patient care because they do not provide information about benefits that patients would consider important, given the paucity of validation of surrogate end points in diabetes and elsewhere.¹⁵

However, large and long trials are also using surrogate outcomes of doubtful validity, an opportunity lost to ascertain the effect of antihyperglycemic agents on more informative patient-important outcomes. Surrogate markers such as glycated hemoglobin are commonly used in diabetes trials but are problematic as guides for patient care.¹⁶ For instance, the validity of hemoglobin A_1c as a surrogate for cardiovascular diabetes complications in patients with type 2 diabetes is highly questionable, as the results of the UK Prospective Diabetes Study (UKPDS)¹⁷ and Rosiglitazone Evaluatedfor Cardiac Outcomes and Regulation of Glycaemia in Diabetes (RECORD)¹⁸ trials as well as the emerging data about the glitazars and the thiazolidinediones suggest.^19,20 Large trials measuring patient-important outcomes can determine the extent to which current and candidate surrogate outcomes accurately and fully capture the effect of antihyperglycemic agents on patient-important outcomes. Up to now, RCT results suggest the answer is “not very well.”

One approach to increasing patient-important outcomes in RCTs as primary end points is to conduct very large randomized trials that will provide definitive evidence (eg, a typical RCT looking for a 20% relative risk reduction in a key dichotomous patient-important outcome will require about 10 000 patients if the event rate is approximately 5%). Large trials are expensive and challenging to organize, conduct, and monitor. They require simple protocols and preclude collection of extremely detailed data regarding patients, process of care, and outcomes. It is more feasible in the era of meta-analyses for trialists to carefully measure patient-important outcomes, even as secondary outcomes, and to transparently report their results making their reports subject to pooling.²¹ Thus, the importance of our findings is that (1) without pooling, the individual diabetes trials will largely fail in providing information about the effect of interventions on patient-important outcomes; and (2) trials that are planning to measure patient-important outcomes as secondary end points need to overcome the temptation to selectively report outcomes with statistically significant results^22,23 as well as to report these findings transparently and carefully. Journals also may need to publish the less-than-interesting results to enable meta-analyses of these results to produce precise enough estimates that can guide practice. This approach is not without controversy. For example, experts have expressed concerns about the validity of pooling cardiovascular events across trials of rosiglitazone,^20,24 since trials reported very few events (with some reporting no events), and these trials did not have coronary events as primary end points.^25,26 The limited number of events across inconsistent trials makes pooled estimates imprecise despite pooling. Whether the field could be advanced more effectively with fewer and larger trials measuring patient-important outcomes deserves greater debate.

The extent to which our findings apply to other chronic conditions is uncertain, but we have noted similar reliance on surrogates in conditions other than diabetes. Examples include the Effect of Ezetimibe Plus Simvastatin Versus Simvastatin Alone on Atherosclerosis in the Carotid Artery (ENHANCE) trial which assessed the effect of ezetimibe and simvastatin on plaque progression,²⁷ not on cardiovascular events, and trials of testosterone in men assessed the effect on bone markers²⁸ and lipid fractions,²⁹ not on fractures or cardiovascular events.

We believe the time has come for a broad consensus on a standard set of important outcomes for patients in diabetes trials, similar to the Outcome Measures in Rheumatology (OMERACT) initiative.³⁰ The OMERACT approach allows for the uniform measurement of outcomes in RCTs with emphasis on outcomes that experts—and ultimately patients—thought would better capture the experience of rheumatological conditions.

Only 18% of ongoing RCTs in diabetes will measure outcomes of importance to patients as primary end points.

Corresponding Author: Victor M. Montori, MD, MSc, Knowledge and Encounter Research Unit, Mayo Clinic College of Medicine, 200 First St SW, Rochester, MN 55905 (montori.victor@mayo.edu).

Author Contributions: Dr Montori had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs Murad and Ghandi contributed equally to this article.

Study concept and design: Gandhi, Murad, Isley, Montori.

Acquisition of data: Gandhi, Murad, Fujiyoshi, Mullan, Flynn, Elamin, Swiglo.

Analysis and interpretation of data: Gandhi, Murad, Fujiyoshi, Flynn, Elamin, Swiglo, Isley, Guyatt, Montori.

Drafting of the manuscript: Gandhi, Murad, Fujiyoshi, Flynn, Isley, Guyatt, Montori.

Critical revision of the manuscript for important intellectual content: Gandhi, Murad, Fujiyoshi, Mullan, Flynn, Elamin, Swiglo, Isley.

Statistical analysis: Gandhi, Murad, Flynn, Montori.

Administrative, technical, or material support: Gandhi, Murad, Flynn, Elamin, Montori.

Study supervision: Montori.

Funding/Support: There were no funding sources for this study.