The IRB data for academic year 1998 were unavailable.
Basic science indicates basic science projects; prospective, prospective clinical studies; and retrospective, retrospective clinical studies. Error bars represent the SE of the mean.
Chang CWD, Mills JC. Effects of a Reward System on Resident Research Productivity. JAMA Otolaryngol Head Neck Surg. 2013;139(12):1285-1290. doi:10.1001/jamaoto.2013.5303
Copyright 2013 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.
With the changing academic medical environment, nontraditional methods may need to be considered to foster research and scholarly activity during the otolaryngology residency experience.
To evaluate the results of a reward system on resident research activity.
Design, Setting, and Participants
Retrospective review of publications and approved institutional review board (IRB) projects between July 1, 1997, and June 30, 2011, among otolaryngology residents at a single otolaryngology residency program at the University of Missouri.
A resident reward system, which awards points for research efforts for each progressive step along the research path from project design to acceptance for publication, was implemented on July 1, 2004. Residents can convert points to a monetary amount to be used for academic enrichment.
Main Outcomes and Measures
Publication rate and IRB approval rate were compared before and after institution of the reward system. Study design types that were published and study design types that received IRB approval were evaluated as well. We hypothesized that the reward system would increase research quantity and quality.
The mean publication output per resident per year increased from 0.13 (95% CI, 0.03-0.23) before commencement of the reward system to 0.43 (95% CI, 0.26-0.60) after implementation of the reward system (P = .004). Significantly more case reports were published compared with other study design types. The number of approved IRB projects before institution of the reward system was 0.47 (95% CI, 0.18-0.75) per resident per year. After instituting the reward system, this increased significantly to 1.29 (95% CI, 0.96-1.63) per resident per year (P = .007). Significant increases in IRB-approved case reports and retrospective clinical studies were noted.
Conclusions and Relevance
After implementation of a point-based reward system, resident research activity increased. The data suggest that this system may encourage resident research, although further refinement may be required to promote higher-quality research endeavors.
Research is viewed as an important aspect of otolaryngology training. The Accreditation Council for Graduate Medical Education approved changes in the otolaryngology residency program requirements effective July 1, 2013, that provide more detailed recommendations for resident scholarly activity.1 Under the new requirements, at least 3 months of structured research experience should be provided for residents (program requirement IV.B.2.a). Furthermore, the research should result in a manuscript suitable for publication in a peer-reviewed journal (program requirement IV.B.2.a.2). Although the word should rather than must is used in each of these requirements, the new wording of the requirements makes more implicit the level of research expected of residents.
A recent coeditorial put forth by the chief editors of the major American otolaryngology journals further affirms the importance of resident research as part of the education process.2 Residents who participate in poster andoral presentations at meetings, as well as prepare manuscripts for potential presentation, are stimulated to appraise the literature, develop projects, analyze data, and contextualize results. Support of this process requires not only intellectual input but also mentorship, financial input, and appropriate encouragement and recognition.
We sought to increase self-directed research productivity among residents by implementing a positively reinforcing reward system. The objective of this study was to evaluate the results of this reward system on resident research activity. We hypothesized that the reward system would increase research quantity and quality.
This study received an exemption by the University of Missouri Health Sciences institutional review board (IRB). A reward system was implemented on July 1, 2004, that rewards residents for each progressive step along the research path from project design to acceptance for publication (Table). For example, a retrospective review would accumulate 100 points after IRB approval, another 200 points after initial manuscript submission, and 400 and 500 points once accepted for publication to a journal with a low and high impact factor, respectively. In addition, point distribution is designed to provide greater reward for higher-level research projects. Furthermore, points are awarded for poster and oral presentation of the research depending on the venue, with more points given for presentation at national or international meetings. Residents accrue points in their individual accounts, with yearly net earnings capped at 2500 points per academic year (AY). Each point is worth $1, which may be used for allowable educational expenses such as books, academic travel, educational software, medical equipment (operative loupes, etc), and professional organization membership dues. Residents are each gifted a start-up balance of 500 points at the beginning of their first year of training, with another 200 points given at the beginning of their second year.
Publications for analysis were discovered using the PubMed database. Those between July 1, 1997, and June 30, 2012, were limited by searching for University of Missouri AND otolaryngology in all fields. The list was then reviewed to include only publications with residents listed as an author if the research was conducted during that person’s residency. Study design, publication date, and impact factor of the journal in which the article was printed were recorded. Studies were categorized as reviews, basic science projects, case reports (<3 patients), prospective clinical studies, and retrospective clinical studies.
A list of IRB project approvals was obtained from the University of Missouri IRB from January 1, 1999, to June 30, 2011. The projects before 1999 were unavailable because of a limitation of the IRB’s electronic database. At the University of Missouri, case reports are required to be submitted to the IRB, although most of them will receive an exemption. All approved IRB projects were counted regardless of whether the application was initiated by a faculty member or a resident. Internal department records of points earned per resident per year were reviewed as well.
Each publication and approved IRB project was grouped by year as defined by the academic calendar. The AY is considered the year in which its final day occurs. For example, AY 2011 spans from July 1, 2010, to June 30, 2011. From AY 1998 to AY 2007, the department size was 10 residents. From AY 2008 to AY 2011, the department size was 11 residents. Therefore, to fairly compare academic output, the number of publications or IRB approvals per year was divided by the total number of residents during that year to calculate a mean output per resident per year. The mean output of IRB approvals per resident per year and the mean output of publications per resident per year were compared before and after the reward system era using independent samples 2-tailed t test.
Forty-one unique publications were identified. The mean publication output was 0.13 (95% CI, 0.03-0.23) publications per resident per year for articles submitted before commencement of the reward system. This increased significantly to 0.43 (95% CI, 0.26-0.60) publications per resident per year after implementation of the reward system (P = .004). The number of approved IRB projects before institution of the reward system was 0.47 (95% CI, 0.18-0.75) per resident per year. After instituting the reward system, this increased significantly to 1.29 (95% CI, 0.96-1.63) per resident per year (P = .007). The raw data are shown in Figure 1. No significant difference was observed in the impact factors of the publications in which the articles were printed.
A change in the distribution of study types was found after institution of the reward system. Although IRB approvals for case reports, retrospective clinical studies, and prospective clinical studies increased, only the differences in the first 2 categories were statistically significant (P = .04 and P = .05, respectively). In the analysis of acceptance for publication, the number of case reports published significantly increased (P = .004) after the reward system was started. Other publication types demonstrated no statistically significant change. Figure 2 shows the distribution of study types before and after institution of the reward system.
The number of points earned per resident per year was reviewed between July 1, 2004, and June 30, 2011. The findings revealed a median of 850 points and a mean (SD) of 1224 (1020) points.
Incentive compensation is an accepted practice familiar to most everyone accustomed to the US market system. Industrial businesses throughout the United States have some sort of performance-based incentives that guide compensation strategies for workers. These incentives are seen as strong drivers to motivate employees toward achieving desired performance measures. For example, an employer can reward an employee for his or her efficiency and quality by paying by piecework for acceptable products. Management can be rewarded by tying compensation to the financial health of the company by issuing stock options.
Likewise, the use of incentives is not a foreign concept in the business of clinical medicine. Patient care is incentivized in a piecework fashion by reimbursing productivity in a fee-for-service model. In the future, third-party payment systems may change the reimbursement structure by creating monetary incentives to meet benchmark quality, efficiency, and cost goals. Incentive plans sponsored by Medicare and Medicaid are already under way to promote adoption and meaningful use of electronic health record systems.
The concept of applying research incentives to spur productivity may seem distasteful to traditional academicians. Research is often idealized as the pursuit of knowledge free from external influences and biases that may pollute the intellectual process. In this narrative, the academician’s reward is through appealing to his or her intrinsic values such as the reception of accolades, recognition among peers, or simply the satisfaction of fulfilling a noble cause (improving patient care, discovering new treatments, and advancing medical knowledge). While these remain at the heart of all research, it is impossible to ignore the external economic influences that abound in the medical research arena. The influence of research funding extends beyond merely supporting the scientist; it is an enormous economic driver that provides a large source of revenue that powers academic institutions. Unsurprisingly, intrinsic rewards can be (and often are) leveraged to increase extrinsic rewards such as salary, further funding, and academic standing.
Such rewards may not translate easily to residents. More than 50% of otolaryngology residents surveyed from 2002 to 2011 intended to pursue private practice careers.3 For these residents, the rewards of pursuing research in residency may not be readily apparent and may not tangibly align with intended career objectives. In addition, regardless of future career aspirations, all training physicians face competing and conflicting interests that pull attention away from research. Trainees have to master and practice a widening body of knowledge that comprises clinical otolaryngology. Care environments are increasingly more complex and systems more complicated. Time is eroded by potential work hour restrictions, increasing volume of patient care, and necessary educational time, as well as family and social needs. Simultaneously, public funding that supports postgraduate medical training has not kept up with the increased work and educational demands and is even being closely scrutinized for reduction. In sum, these forces may create a disincentive for research.
The influences that limit resident incentive for research are similar to those faced by academic medical faculty as well. Typically, academic medical departments are charged with the tripartite mission of teaching, research, and clinical care. Teaching and non–grant-funded research have traditionally been supported by clinical revenue, as well as Medicare dollars allocated to graduate medical education. Reduced reimbursements and threatened graduate medical education funding pressure departments to offset these reductions through increased clinical productivity. If valuation of faculty endeavors is heavily weighted toward revenue generation, academic physicians practicing in today’s health care environment face an opportunity cost related to participating in research activities, namely, a loss of compensation associated with the time not spent on clinical activities. Competition for grant funding is fierce, often requiring successful researchers to invest significant effort to specialize in a physician-scientist career pathway if they aspire to be competitive candidates. Various cultural, economic, and family and social obstacles influence the decision to pursue such a rigorous career pathway in addition to the years already spent in medical school and residency.4- 7
In an effort to uphold the tripartite mission, some academic leaders have espoused new compensation strategies.8 If departments want to value and encourage participation in research and teaching, then incentives must be aligned to reward achievement in these areas. However, before implementing such a performance-based incentive plan, adequate performance measures need to be established. For clinical medicine, activity is measured in work relative value units. For academic activity, academic relative value units have been proposed that quantify effort spent on nonreimbursed activities such as teaching, committee membership, and peer-reviewed publications. Assigning academic relative value units to scholarly activity may seem arbitrary and can be contentious to stakeholders. However, such a system provides a quantifiable metric to measure productivity, from which incentive programs can be developed.9- 11 Implementation of academic performance–based incentive plans has been linked to increases in revenue, faculty salaries, morale and satisfaction, and publications and National Institutes of Health grant funding.8,12
The resident reward system outlined herein borrows the same strategy guiding faculty performance–based compensation but applies it to residents. Typically, academic otolaryngology departments have discretionary funds that are spent to provide residents with a rich and rewarding residency experience beyond what is absolutely required by the Accreditation Council for Graduate Medical Education. Otolaryngology residency programs spend anywhere from $1000 to $100 000 on resident-oriented discretionary expenditures per year, according to a survey sent out by one of us (C.W.D.C.) to otolaryngology residency programs throughout the United States (unpublished data, 2012). These funds are often used for travel expenses, the purchase of books, meeting and course registration fees, surgical equipment (eg, operative loupes), as well as research costs. Discretionary funds can be distributed in various ways. One method is to distribute a fixed amount to each resident, akin to receiving a salary guarantee. In our case, we have chosen to use the reward system algorithm to guide the distribution of discretionary resident funds by tying disbursement of monies to scholarly activity measures.
Like the academic relative value unit model, implementation of the resident reward system requires defining measurable metrics and assigning value to efforts performed. For our model, it was thought that the combination of IRB project data and publication data provided a more comprehensive assessment of research activity than simply using the publication metric alone. If the number of publications was the exclusive measure for research activity, only completed, polished projects would be recognized, ignoring research efforts invested in setting up proposals that did not progress to publication. Even if their projects are never printed in the pages of a journal, residents nonetheless educationally benefit from the experience. While publication may be a mark of good research, works may go unpublished because of myriad factors, including limitations of time, funding, research mentorship and quality, and meaningful data outcomes (independent of a good research question and appropriate study method) and editorial biases.13,14
Residents at the University of Missouri are allotted 3 months of dedicated research, which is divided into a 2-month block during the postgraduate year 3 and a 1-month block during the postgraduate year 4. This allocation did not change during the studied period. Successful completion of a project is anticipated of each resident, although the short time frame can be challenging with complicated and rigorous projects. In addition, scholarly works are expected to occur throughout residency, not just during the allocated research time. As a result, we thought that continuous academic activity would be better encouraged by rewarding not only for completed projects but also for the initiation and development of new investigations.
One could argue that it should not be necessary to reward something that otherwise is an expectation of residency. However, perhaps the socioeconomic environment that influences residents’ choices is a reflection of the same socioeconomic environment that shapes the greater academic medical environment. As faculty members grapple with fulfilling nonclinical academic duties, so do residents. If departments want to promote the value of research, it may be helpful to translate this into something tangible to all participants, including residents.
Implementation of any system or process has the risk of creating unintended consequences. While the reward system stimulated increased publications, it did not stimulate increased publication of higher-level research. Two possible reasons should be considered to account for this. First, publications are a lagging metric compared with IRB approvals. Indeed, increased proposals were noted for higher-level research projects, including retrospective and prospective clinical studies, after the implementation of the reward system. However, more time may be needed to see these projects reach fruition.
Second, the incentive valuation may require adjustment to rectify imbalances. Point values assigned to research tasks may need to be revised to shift incentive. It is debatable if monetary rewards are always the proper incentive to produce the best results. Large monetary rewards have been associated in certain circumstances with decreased performance, whether due to performance anxiety or by stifling creative and innovative exploration in favor of the tried and true.15 The latter rationale may partially account for the increased number of case reports seen in this study. Although the reward system more richly incentivizes higher-quality research efforts, this may not completely offset the increased amount of effort required. While the reward amounts could be modified to provide stimulus away from case reports, it is uncertain whether this alone would result in the output of higher-quality publications.
Some behavioral psychology evidence suggests that extrinsic rewards (money) have a negative influence on intrinsic motivation. Known as the undermining effect, a decrease in task engagement can occur after the removal of an extrinsic reward compared with controls.16 Influencing motivation through emotion and social rewards may better tap into intrinsic motives. The application of extrinsic rewards should not occur to the detriment of intrinsic ones.
Potential issues should be kept in mind when analyzing the results of this study. In social sciences, determining causation is difficult because the social milieu cannot always be practically observed in a strictly controlled fashion. It is almost impossible to account for the varied dynamic factors of a real social environment. Therefore, the reward system cannot be completely credited with the increase in scholarly output with absolute certainty. However, it seems to be circumstantially related. Other influencing factors may be present. The department underwent a leadership change (both the chair and program director positions) in 2004 and 2009, which may have resulted in increased emphasis on scholarly activity. The study period also saw a turnover of various PhD faculty members, which could have affected the basic science research output. We can approximate causation as best we can with correlative data, recognizing the possibility of post hoc ergo propter hoc fallacy.
Future investigations may want to measure the influence of incentives on other residency objectives. The reward system outlined above is part of a larger incentive program in our department that enables residents to earn points for achievements within each of the 6 Accreditation Council for Graduate Medical Education core competencies. Suitable metrics measuring achievement in the core competencies would need to be found. As our study was designed, it was difficult to find metrics consistently applied during the studied time span to measure the subjective qualities found in the other core competencies.
In conclusion, a monetary reward system may provide a way to encourage academic productivity, serving as a mechanism for positive reinforcement. Based on this retrospective analysis, the reward system presented herein may effectively encourage resident scholarly activity but may also have the unintended effect of skewing publication output toward case reports. Additional reward systems based on other than monetary incentives need to be considered to fully encourage the potential of each resident.
Corresponding Author: C. W. David Chang, MD, Department of Otolaryngology–Head and Neck Surgery, University of Missouri, One Hospital Drive, Room MA 314, Columbia, MO 65212 (email@example.com).
Submitted for Publication: March 17, 2013; final revision received August 6, 2013; accepted August 21, 2013.
Published Online: October 24, 2013. doi:10.1001/jamaoto.2013.5303.
Author Contributions: Dr Chang had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: All authors.
Acquisition of data: All authors.
Analysis and interpretation of data: All authors.
Drafting of the manuscript: All authors.
Critical revision of the manuscript for important intellectual content: Chang.
Statistical analysis: Chang.
Administrative, technical, and material support: Chang.
Study supervision: Chang.
Conflict of Interest Disclosures: None reported.
Previous Presentation: This study was presented at the American Academy of Otolaryngology–Head and Neck Surgery Foundation 2012 Annual Meeting and Otolaryngology Expo; September 10, 2012; Washington, DC.