Association of Journal Quality Indicators With Methodological Quality of Clinical Research Articles

Context The ability to identify scientific journals that publish high-quality research would help clinicians, scientists, and health-policy analysts to select the most up-to-date medical literature to review.

Methods To assess whether journal characteristics of (1) peer-review status, (2) citation rate, (3) impact factor, (4) circulation, (5) manuscript acceptance rate, (6) MEDLINE indexing, and (7) Brandon/Hill Library List indexing are predictors of methodological quality of research articles, we conducted a cross-sectional study of 243 original research articles involving human subjects published in general internal medical journals.

Results The mean (SD) quality score of the 243 articles was 1.37 (0.22). All journals reported a peer-review process and were indexed on MEDLINE. In models that controlled for article type (randomized controlled trial [RCT] or non-RCT), journal citation rate was the most statistically significant predictor (0.051 increase per doubling; 95% confidence interval [CI], 0.037-0.065; P<.001). In separate analyses by article type, acceptance rate was the strongest predictor for RCT quality (−0.113 per doubling; 95% CI, –0.148 to –0.078; P<.001), while journal citation rate was the most predictive factor for non-RCT quality (0.051 per doubling; 95% CI, 0.044-0.059; P<.001).

Conclusions High citation rates, impact factors, and circulation rates, and low manuscript acceptance rates and indexing on Brandon/Hill Library List appear to be predictive of higher methodological quality scores for journal articles.

It is difficult for clinicians, scientists, and health policy analysts to keep up with the more than 2 million new research articles published each year in medical and scientific journals.¹ Furthermore, many published reports are of poor-to-average methodological quality,^2-6 and most scientific articles are never cited.^7,8

One approach to facilitating identification of sound medical evidence is to identify high-quality journals that are likely to publish high-quality research. Peer-review and bibliometric methods (such as journal citation rates, impact factors, circulation, manuscript acceptance rates, and indexing on MEDLINE or Brandon/Hill Library List) may be useful in evaluating the quality of a journal.^9-14 However, these methods are controversial due to potential biases in citation, impact factor, and inherent limitations of the sources of information used to calculate them.^8,12,15-28 Currently, none of these bibliometric parameters have been validated as predictors of journal quality.

We determined whether journal characteristics of peer-review status, citation rate, impact factor, circulation, manuscript acceptance rate, and indexing on MEDLINE or the Brandon/Hill Library List are associated with the methodological quality of original research articles they publish. Studies have also suggested that the source of research funding is associated with article quality.^3,4,6,29-31 Therefore, we also estimated the effect of funding source on article quality score.

Using a computer-generated list of random numbers, we randomly selected 30 journals from 107 categorized as general internal medical journals by the Institute for Scientific Information.³² We excluded journals that were not in English or were unavailable through the University of California library system. Original research articles published in the journals were identified by searching MEDLINE and HealthSTAR from January 1, 1999, through December 31, 1999, using exact journal title, human subjects only, and publication type (journal article). We excluded reviews, historical articles, meta-analyses, case reports or case series, clinical conferences, comments, and consensus development conferences because they would require a different instrument for quality assessment. For each journal, we initially randomly sampled 3 randomized controlled trial (RCT) articles (or all, if <3 were published) and 3 other (non-RCT) articles. We scored these as described below and examined the article-to-article variability within each article type (RCT or non-RCT) within each journal. We then randomly sampled up to 6 additional articles of a type from the journals with the greatest variability, for a total of 97 RCT and 146 non-RCT articles. This additional sampling from the most variable journals improved the amount of statistical information provided per article scored.

We collected data on the following 7 journal characteristics for each of the journals: (1) peer-review status, defined as manuscript review by a journal's editor(s) and outside experts, was verified by examining the journal's published peer-review policy or contacting the journal's editorial office; (2) citation rate, defined as the average number of times current articles in a specific journal were cited during the year they were published, was obtained from the Institute for Scientific Information³²; (3) impact factor, defined as the total number of citations during a given year that a journal receives to articles from the 2 previous years, divided by the total number of "source" articles published in the journal during that same 2-year period, was obtained from the Institute for Scientific Information³²; (4) circulation, defined as the number of subscriptions for a journal publication, was obtained from Ulrich's International Periodicals Directory³³ or the journal's editorial office or publisher; (5) manuscript acceptance rate, defined as the percentage acceptance of original research articles in a given year, was obtained from the journal's editorial office or publisher; (6) whether each journal was indexed in MEDLINE in 1999; (7) and whether each journal was indexed on the Brandon/Hill Library List in 1999.³⁴ (The Brandon/Hill Library List is a selected list of books and journals that are recommended for the small medical library.³⁴)

Two reviewers independently assessed the quality of each article using an instrument previously tested for validity and reliability.⁵ Our quality assessment instrument includes 22 items designed to measure the methodological quality of articles (defined as the minimization of systematic bias and the consistency of conclusions with results) with a wide range of study designs, regardless of article topic. We selected this instrument rather than an instrument that only assesses the quality of RCTs because our objective was to assess the validity of quality characteristics of journals based on the quality of RCTs and non-RCTs that the journal publishes. We chose the instrument because it compared favorably in terms of validity and reliability with instruments that assess the quality of RCTs only,³⁵ and it performed similarly to other well-accepted instruments for scoring the quality of trials included in meta-analyses.³⁶ We did not choose the component approach for assessing quality of research articles, since empirical evidence for the approach applies primarily to RCTs.^37,38

Reviewers were trained to use the instrument and were given detailed written instructions. Because previous studies suggest that masking of the articles to the reviewers does not influence quality scores,^39,40 reviewers were not blinded to the identity of the articles or the hypothesis of the study, but they did not have access to the data on the journal quality characteristics.

Scores can range continuously from 0 (lowest quality) to 2 (highest quality).⁵ The average of the scores of the 2 reviewers was used for the analyses unless the reviewers' scores differed by more than 1 SD. In this case, the article was discussed by both reviewers until consensus was achieved, and the consensus score was used in the analyses. Ten percent of methodological quality scores required adjudication. The interrater reliability of overall scores, which was measured by intraclass correlation,⁴¹ was fair (r = 0.45).

For each article, funding sources were categorized as government, private nonprofit, industry, government plus private nonprofit, industry plus any other source, unable to be determined, or none disclosed.

Quality scores were modeled in terms of journal characteristics and article characteristics, with a random journal effect included to account for within-journal correlation (Mixed Procedure, Version 8.2, SAS Institute Inc, Cary, NC). Because predictor variables had skewed distributions, we used log transformations to prevent large values from being too influential. Because some journals had much more variability in their quality scores than others, models allowed for different variances for each journal.

Logistic regression with a random journal effect was used to model article type and journal characteristics as predictors of whether funding source was disclosed.

The mean (SD) methodological quality score of all articles was 1.37 (0.22) (range, 0.62-1.88) on a scale of 0 to 2. All 30 journals had non-RCT articles and 28 had RCT articles. Non-RCT articles had an estimated average quality score of 1.31, and RCT articles were estimated to average 0.13 points higher (95% confidence interval [CI], 0.10-0.17; P<.001).

All journals reported a peer-review process and were indexed on MEDLINE, so these variables could not be analyzed. There were significant associations between quality scores and higher citation rates (P<.001; citation rate range, 0.03-6.06), higher impact factors (P<.001; impact factor range, 0.22-28.66), higher circulation (P = .001; circulation range, 1080-3.7 million), lower manuscript acceptance rates (P<.001; manuscript acceptance rate range, 7.5%-72.0%), and indexing on Brandon/Hill Library List (P<.001; 33.3% indexed and 66.7% not indexed) (Table 1). Residuals and fitted random journal effects from these models appeared to be approximately normally distributed, and there were no extreme outliers.

In models that controlled for article type (RCT or non-RCT), citation rate was the predictor with the smallest P value (0.051 increase per doubling, 95% CI, 0.037-0.065; P<.001). No other predictors substantially improved this model.

In separate analyses by article type, acceptance rate was the strongest predictor for RCT quality (−0.113 per doubling; 95% CI, −0.148 to −0.078; P<.001), while journal citation rate was the most predictive factor for non-RCT quality (0.051 per doubling; 95% CI, 0.044-0.059; P<.001). This means that the estimated effect of acceptance rate on the quality of RCTs is that for every doubling of acceptance rate, article quality score decreases by 0.11 point. The estimated effect of citation rate on the quality of non-RCTs is that for every doubling in journal citation rate, article quality score increases by 0.05 point.

For the entire sample, only 66% (160/243) of articles reported a source of study funding. These studies reported funding sources solely by government (34%; 54/160), private nonprofit (19%; 30/160), and industry (14%; 23/160). We were unable to determine the type of funding for 4 studies (2%). Thirty-one percent (49/160) reported multiple sources of funding, with 53% (26/49) being funded by government and private nonprofit followed by 37% (18/49) by government and industry, 6% (3/49) by government, industry, and private nonprofit, and 4% (2/49) by industry and private nonprofit. The authors of RCTs (77%; 75/97) were more likely to disclose a funding source than authors of non-RCTs (58%; 85/146) (P = .05 from random-effects logistic regression).

After controlling for article type (RCT or non-RCT), articles disclosing funding sources were estimated to score 0.022 higher than those without disclosure (95% CI, −0.029 to 0.074; P = .39). All of the journal characteristics had statistically significant associations with disclosure. Significance ranged from P<.001 for impact factor and citation rate to P = .006 for circulation.

Table 2 shows the estimated effects of different funding sources on article quality. For RCTs, private nonprofit funding was associated with the highest scores, while industry-funded studies received the lowest scores, particularly those with a mix of industry and other funding. For non-RCTs, the pattern was qualitatively similar, but effects were smaller and did not reach statistical significance.

Articles of higher methodological quality are published in journals whose articles are cited more frequently (higher citation rates and impact factors), read more widely (higher circulation, indexed on the Brandon/Hill Library List), and scrutinized more carefully by editors and outside peer-reviewers (lower manuscript acceptance rates). These 5 journal characteristics may be valid predictors of journal quality when evaluating journals within the same category, such as general internal medicine. Journal citation and manuscript acceptance rates were the best predictors of the quality of research articles published in the journals.

One limitation of our study is that we used a scale, rather than a component method, to assess the quality of research articles.^36,38 However, there is limited empirical data to support the selection of components to measure the quality of non-RCTs that are published by the journals in our study. The quality scores for the articles in this study are slightly higher than previous studies using the same instrument,^3,5,6 which may be the result of improvements in study design and/or the quality of reporting over the last few years. The interrater reliability score in this study is somewhat lower than previous reports using the same instrument,^3,5,6 possibly because of the difficulty in assessing the quality of non-RCT articles, which are more variable in their design.³⁵

Our findings that about one third of research articles did not disclose any funding source and that those with disclosed funding sources were of higher quality than those without, suggests that journal editors should continue to encourage disclosure of sources of study funding, affiliations, and other potential financial conflicts of interest.^42,43 Our finding that studies with industry support have lower quality scores than government-funded studies, as well as other studies showing associations between industry-funded sources and outcomes, suggest that funding should be considered when assessing the usefulness of an article.^3,4,6,29-31