From the Center for Bioethics, University of Pennsylvania, Philadelphia (Dr Cho); Division of General Internal Medicine and Program for Health Care Research, Department of Veterans Affairs Medical Center, Cleveland, Ohio, and University Hospitals of Cleveland, Case Western Reserve University (Dr Justice); Division of Biostatistics, Department of Biostatistics and Epidemiology and Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine, Philadelphia (Dr Berlin); Department of Emergency Medicine, University of Missouri Kansas City School of Medicine (Dr Waeckerle); Division of Emergency Medicine, University of California, San Francisco School of Medicine (Dr Callaham); Institute for Health Policy Studies, University of California, San Francisco (Dr Rennie). Dr Winker is Senior Editor and Dr Rennie is Deputy Editor (West), JAMA.
Context.— In a previous study, we found that masking success was higher at a journal
that masked reviewers to author identity. We hypothesized that masking policy
or other factors could be associated with masking success.
Objectives.— To evaluate differences in success of masking reviewers to author identity
at 7 biomedical journals and to identify factors associated with these differences.
Design.— Written questionnaire.
Participants.— Reviewers at 3 journals with a long-standing policy of masking author
identity (Annals of Emergency Medicine, Epidemiology,
and Journal of the American Geriatrics Society) and
4 journals without a policy of masking author identity (Annals of Internal Medicine, JAMA , Obstetrics &
Gynecology , and Ophthalmology).
Main Outcome Measures.— Masking success (percentage of reviewers successfully masked) and reviewer
characteristics associated with masking.
Results.— There was no significant difference in masking success between journals
with a policy of masking (60%) and those without (58%) (P=.92). We found no association between masking success and a policy
of masking when adjusted for the reviewer characteristics of age, sex, years
of reviewing experience, number of articles published, number of articles
reviewed, percentage of time spent in research, editorial experience, or academic
rank (odds ratio [OR], 1.3; 95% confidence interval [CI], 0.64-2.8; P=.43). In multivariable analysis of reviewer characteristics,
reviewers spending a greater percentage of time in research, the only significant
predictor of masking success, were less likely to be successfully masked (OR,
1.01; 95% CI, 1.00-1.02) (P=.04).
Conclusions.— Masking success appears unrelated to a journal policy of masking, but
is associated with reviewers' research experience and could be affected by
other characteristics. Using reviewers with less research and reviewing experience
might increase masking success, but the effect on review quality is unknown.
MASKING peer reviewers to author identity has been suggested as a means
to improve quality and fairness of reviews. In a previous study,1
we found that masking reviewers to author identity in the peer review process
was not uniformly successful at different journals, and that incomplete masking
success limited our ability to assess the effect of masking on review quality.
In particular, we noted that the Annals of Emergency Medicine had significantly higher rates of masking success than the other journals.
Because the Annals of Emergency Medicine was the
only journal in our study that had a long-standing policy of masking, we hypothesized
that their policy might be responsible for the high rate, either because reviewers
were accustomed to not knowing author identity or because the way in which
the manuscript was prepared differed from other journals. Differences in characteristics
of specialties might also be responsible.
In this study, we explored 2 questions: (1) is masking success associated
with having a long-standing policy of masking? and (2) is masking success
associated with reviewer characteristics?
In this study we included 7 journals, 3 with a long-standing policy
of masking author identity (Annals of Emergency Medicine, Epidemiology, and Journal
of the American Geriatrics Society) and 4 journals that did not mask
author identity (Annals of Internal Medicine, JAMA , Obstetrics & Gynecology, and Ophthalmology).
Between August and December 1996, each journal enrolled manuscripts
that met the following inclusion criteria: (1) the manuscript was sent for
external peer review, (2) the manuscript reported original research, including
meta-analyses but excluding case reports or letters, and (3) the authors did
not object to having their manuscripts enrolled. Each journal masked eligible
manuscripts by removing author and institutional identity from the title page,
running headers or footers, and acknowledgments of manuscripts. Self-references
in the text were not removed. Each journal then sent the manuscript to at
least 2 reviewers along with a 1-page questionnaire. The reviewers were asked
to return the manuscript and review to the journal and to return the questionnaire
to the study coordinator in a separate, preaddressed envelope.
Each journal enrolled at least 20 manuscripts. Approximately one quarter
of these were randomized to the usual reviewing process and were not analyzed
in this study if the usual process was not to mask reviewers. Data from Annals of Internal Medicine, JAMA , Obstetrics & Gynecology, and Ophthalmology
and data from 22 of 42 total manuscripts at Annals of Emergency
Medicine were collected in a previous study,1
in which 2 reviewers per manuscript were enrolled, one of whom was masked
to author identity. For this study, we used data only from the masked reviewer.
For Epidemiology and Journal of
the American Geriatrics Society and additional manuscripts from Annals of Emergency Medicine, between 1 and 4 reviewers
per manuscript were included in the study, all of whom were masked to author
Reviewer Questionnaire. Using the reviewer questionnaire, we collected data on masking success
and reviewer familiarity with the author from all 7 journals and collected
data on reviewer characteristics from 6 journals (not Journal
of the American Geriatrics Society). (Copies of the questionnaire will
be provided to interested readers on request.)
We determined masking success by asking reviewers the question, "Do
you think you can identify any of the principal authors of this manuscript?"
If they answered yes, we instructed reviewers to list any authors they thought
they could identify. If any reviewer correctly identified at least 1 author,
we considered the reviewer unmasked to author identity. Thus, we measured
masking success based on the percentage of reviewers who were able to guess
author identity. For reviewers who thought they knew at least 1 of the authors,
we asked whether they were familiar with the work described in the manuscript
before receiving it to review and whether the reviewer was familiar with the
Reviewers were asked to provide information on age, sex, academic rank,
reviewing experience, editorial experience, number of original research articles
published in the last 5 years (including meta-analyses), number of manuscripts
reviewed in the last 3 months, and percentage of time spent in research, clinical,
and administrative activities.
Analysis. We used χ2 tests for analysis of categorical variables,
and t tests for analysis of continuous variables.
We used unconditional logistic regression models with single variables to
estimate the unadjusted odds ratio (OR) for the outcome variable—masking
success—associated with each of the potential predictor variables. Subsequently,
we fit a multivariable logistic regression model including all the potential
predictors to estimate the ORs for each variable, adjusted for all other variables
in the model. Non–normally distributed continuous variables were also
analyzed after log transformation, but transformation did not substantially
change the results or conclusions. Results of untransformed data are reported
here. For all tests of significance, we used 2-tailed α=.05.
The overall response rate of reviewers was 87% (287/328). No eligible
reviewers declined to participate in the study. Overall, 60% of reviewers
were successfully masked (95% confidence interval [CI], 54%-65%). The response
rates and masking success rates by journal are shown in Table 1.
Masking success was significantly higher at Annals
of Emergency Medicine (83% [95% CI, 74%-92%]; N=78) than at all other
journals combined (N=209) (P<.001). There was
no significant difference in masking success between journals with a policy
of masking and those without (P=.92). There was also
no significant association between masking success and a policy of masking
when adjusted for the reviewer characteristics of age, sex, years of reviewing
experience, number of articles published, number of articles reviewed, percentage
of time spent in research, editorial experience, or academic rank (full professor
vs all other ranks) (OR=1.3; 95% CI, 0.64-2.8; P=.43).
Reviewer Characteristics and Factors Associated With Masking Success. We performed univariate analyses of reviewer characteristics that we
hypothesized might be related to masking success. For reviewers from all journals
combined (except Journal of the American Geriatrics Society, which did not collect data on reviewer characteristics), masking
success was not associated with age, academic rank, or having editorial experience
(P>.05). However, compared with reviewers who guessed
author identity correctly, successfully masked reviewers had fewer years of
reviewing experience, published fewer research articles, and spent less time
in research (Table 2, unadjusted
In a multivariable logistic regression model including reviewer age,
sex, years of reviewing experience, number of articles published, number of
articles reviewed, percentage of time spent in research, editorial experience,
academic rank (full professor vs all other ranks), and journal (Annals of Emergency Medicine vs all other journals), we investigated
the factors associated with correctly identifying an author. The unadjusted
OR for masking success for reviewers at Annals of Emergency
Medicine, compared with reviewers for other journals, was 4.8 (95%
CI, 2.5-9.3; P <.001). However, after adjusting
for the other 8 variables, the OR was 3.3 (95% CI, 1.4-7.2; P=.009), indicating a reduced, but still substantial, difference in
masking success between Annals of Emergency Medicine
and other journals. Thus, reviewer characteristics did not explain most of
the difference in ability to identify authors between reviewers at Annals of Emergency Medicine and reviewers at other journals.
Because factors such as age, number of publications, and reviewing experience
are likely to be correlated, we performed regression analysis to examine which,
if any, of the reviewer characteristics (other than the journal for which
they were reviewing or the journal policy of masking) was an independent predictor
of masking success. Of the reviewer characteristics we analyzed, percentage
of time spent in research was the only significant independent predictor (P=.04; Table 2,
Because some manuscripts were reviewed by more than 1 reviewer, we repeated
the logistic regression analyses, adjusted for clustering by manuscripts.
The results were essentially the same as the unadjusted results (data not
We confirmed the results of our previous study1
that the rate of masking success varies widely across journals, but found
that a long-standing policy of masking did not increase masking success. Previous
have found rates of masking success similar to those in this study. In our
study, masking success was not associated with having a long-standing policy
of masking, per se. In multivariable analysis, we found that masking success
was associated with 1 reviewer characteristic. Reviewers spending a greater
proportion of time in research were more likely to identify authors, suggesting
that using reviewers with less research experience will increase masking success.
Because research experience might be a desirable trait in reviewers for many
manuscripts, however, excluding reviewers with this trait could be detrimental
to the quality of reviewing. Thus, journals that mask author identity may
have to balance their desire for improving the success of masking with the
need for high-quality reviewers.
Characteristics of reviewers and other factors we examined did not completely
explain differences in masking success. One possible factor might have been
in differences in masking procedures among journals. Although the minimum
masking procedure followed a standard protocol, Annals of
Internal Medicine and Annals of Emergency Medicine used additional procedures that may have affected masking success. Annals of Internal Medicine removed names and journal identification
(but not titles or other reference information) from self-references in the
text and reference section. However, the masking success rate at Annals of Internal Medicine was very close to the overall average.
At Annals of Emergency Medicine, the instructions
to authors stated that the journal's policy was to mask author identity and
requested that authors not include identifiers in running heads, but running
heads would have been removed as part of normal study procedures. Although
all 3 of the journals in our study with a policy of masking have had these
policies for years, different journal "cultures" might have arisen that affected
masking success (eg, authors at some journals might be less likely to self-reference).
Characteristics such as the age and size of medical specialties could
influence the type of research performed and the probability that researchers
will be able to identify others' work. Contrary to what one might expect,
however, JAMA and Annals of Internal Medicine,
journals representing large and broad fields, had lower masking success rates
than Annals of Emergency Medicine and Obstetrics & Gynecology. Our analyses suggest that to determine
whether masking might be more successful in one field than another specialty
characteristics other than size of the field would need to be explored.
Our results suggest that reviewers' research experience is associated
with masking success, but that other unidentified factors are also involved.
Indeed, results from a previous study1 suggested
that author renown may in part affect masking success; this factor may be
related to reviewer experience. In this study, we did not collect data on
this author renown for all journals, and were thus unable to further investigate
its effects. The study was also limited by the relatively small number of
manuscripts enrolled at journals without a policy of masking. Additional research
on a larger number of journals from a variety of medical specialties would
be needed to determine what other characteristics of reviewers (eg, specialty
or their relationships to authors), authors, journals, or medical specialties
are associated with masking success.
Cho MK, Justice AC, Winker MA, Berlin JA, Waeckerle JF, Callaham ML, Rennie D, and the PEER Investigators . Masking Author Identity in Peer ReviewWhat Factors Influence Masking Success?. JAMA. 1998;280(3):243-245. doi:10.1001/jama.280.3.243