Dr Lundberg is Editor and Ms Fritz is Administrative Assistant, JAMA . Dr Paul is with HCI, Inc, Princeton, NJ.
Context.— Journal editors are responsible to many publics, and their choices of
articles to publish are a frequent source of dispute.
Objective.— To assess the extent of agreement between topics identified by experts
and by JAMA readers as most important for publication.
Design and Participants.— Modified Delphi process of polling of JAMA Editorial Board members
and senior staff (ie, experts) in 1996, and masked direct mail survey of a
stratified sample of JAMA readers in late 1996 and early 1997.
Main Outcome Measures.— Agreement between experts and readers on the topics most important for JAMA
to deal with in 1997.
Results.— Of 55 experts polled, the 40 respondents (73% response rate) proposed
178 topics. Editing to combine similar topics left 73. The same 55 persons
were asked to stratify all 73 alphabetically arranged topics on a scale of
1 to 5 (85% [47/55] response rate). They were then given the results of this
ballot and asked to vote again (76% [42/55] response rate). Of the 55 experts,
40 attending the annual editorial board meeting were given all results; 39
attendees voted on the final topics. In response to the mail survey, a single
pass of the same 73 topics yielded a response rate of 41.6% (208 returns).
Nonresponders were roughly equivalent to responders demographically. Readers
agreed with the experts on only 3 of the top 10 subjects: managed care, cancer,
Conclusion.— Expert opinion and the opinion of readers as to what JAMA should emphasize
SCIENTISTS, physicians, and editors of their journals often confront
issues of great importance to their profession and the people it serves. Many
of these problems can be studied by using scientific methods, posing questions,
creating hypotheses, and testing such by scientific study. But when the question
is one for which no apparent scientific study can be devised to answer, and
when decisions are necessary for daily work, the people involved often seek
opinion to guide them. Opinion can be from 1 individual, a few individuals,
or a group.1
Editors of medical and science journals must decide what articles to
publish. They usually simply respond to what is sent to them by authors, using
peer review and editorial judgment, and publishing the best of what they receive.
We at JAMA decided many years ago that, although this reactive approach
has worked well for many medical journals, we wanted to be proactive as well.
Since 1990 we at JAMA and the 10 American Medical Association (AMA) Archives (specialty) Journals have used a multipass, modified
Delphi voting approach to ascertain the subjects of greatest importance for JAMA
to deal with in the next year.2,3
In addition, we have consulted the editors of the Archives Journals to determine if there is 1 subject near the top of each year's
list that all 11 journals should publish about concurrently with theme issues
during that year. The purposes of these theme issues and the resulting annual
publication topics have been reported elsewhere.4
The persons voting in the annual Delphi process were senior editorial
staff of JAMA and JAMA Editorial Board members, which include the chief
editors of the 10 AMA Archives Journals. All are
considered experts in their fields and represent a wide range of areas of
interest. This entire process is seen by us as a great success and has spawned
the notion of global theme issues, which have now occurred twice and are highly
regarded by editors and readers worldwide.5,6
In 1996 we decided that we should test the ability of these expert opinions
to predict what ordinary readers might wish to read about, by assessing the
level of congruence of subject interest among these groups. Thus, we performed
our usual criterion standard design survey using the Delphi process involving
55 recognized experts who were asked to propose the topics most important
for JAMA to deal with in 1997. All voters were informed of the process,
consented to participate, and were provided the results.
A total of 40 (73%) of the 55 experts surveyed responded with 178 ideas
for topics. Editing for crossovers and groupings left 73 topics. The same
55 persons were asked to stratify all 73 topics, which had been alphabetically
arranged, on a scale of 1 to 5. The response rate was 85% (47/55). The same
55 persons were given the results of this Delphi pass and asked to vote again.
The response rate was 76% (42/55). After all results had been provided to
40 attendees (of the original 55) at the regular 1996 JAMA Editorial Board
meeting, the topics were discussed and debated. Thirty-nine voted on the final
ballot—a response rate of 98% of those present.
To assess the topic interest of JAMA readers, in late 1996 and early
1997 a direct mail survey from an independent (not AMA) company was sent to
a stratified sample of 500 practicing physicians (not basically academic)
who are regular JAMA readers. A random draw from the AMA Masterfile of confirmed JAMA
readers was taken to compile a list for survey. All who participated were
informed of the process in the cover letter and by virtue of responding consented.
They were not called in advance. They were not provided results.
A response rate of 41.6% (208 readers) was realized for the single-pass
ballot of the same 73 topics. Nonresponders were roughly equivalent in available
parameters to the responders (age, reading habits, specialties, sex, type
of practice, number of patients seen per week). We compared and contrasted
the results across all 73 topics.
The final top 10 topics determined by the experts were (in order) managed
care, death and dying, genetics, quality of care, violence, aging, caring
for the uninsured and underinsured, outcomes research, human immunodeficiency
virus/acquired immunodeficiency virus, and cancer. Readers agreed with the
experts on only 3 of the top 10 subjects: managed care, cancer, and aging
The data are substantial in amount and detail and are reported here
only as the full comparison of the 2 total groups for the 73 topics, since
this is considered a pilot project.
It is clear that according to the methods described here, there is substantial
variation between expert opinion and the opinion of readers as to what JAMA
should emphasize. We recognize weaknessess in this pilot study. For example,
the methods for the 2 groups—experts and readers—were different;
one was a single-pass mail survey, while the other was a 4-pass, sequential
modified Delphi. The sample sizes were small: 55 in the expert group and 500
in the reader group. Response rates were less than ideal, from 73% to 98%
in the expert group but only 46% in the reader group. But the differences
in results were startling and we believe important. For example, the experts
ranked alternative medicine 68th of 73 topics JAMA should publish about,
while the readers ranked it seventh of 73. Similarly, the experts ranked tobacco
17th of 73, while the readers ranked tobacco 55th of 73. We were so impressed
that we have altered our manuscript acceptance process to become even more
Among the potential explanations for the marked differences is the viewpoint:
"Where one stands often depends upon where one sits." Perhaps the experts
told us what they thought physicians needed to know, but also the experts
tended to know what research was being performed and in what subjects the
journal was likely to get good papers. The readers, however, mostly practicing
physicians, perceived what they needed to know, and may not know, in order
to care best for the patients walking into their offices that day. They also
want journals to be useful for their practices.
Editors of primary-source, peer-reviewed medical journals have trust
relationships with many publics. Key among these are readers, authors, and
owners. Having hard data on which to make publishing choices can be useful,
since the principal interests of these diverse publics frequently come into
It is the editor's job to keep all of these publics happy at all times.
That, of course, is impossible. We should strive to serve the perceived needs
of all these publics but recognize that we may not be able to serve all their
wants as well. We have embarked on substantial new research aimed at balancing
these inputs in a more positive way.
Lundberg GD, Paul MC, Fritz H. A Comparison of the Opinions of Experts and Readers as to What Topics
a General Medical Journal (JAMA ) Should Address. JAMA. 1998;280(3):288-290. doi:10.1001/jama.280.3.288