Association Between Financial Incentives and Participant Deception About Study Eligibility

Key Points español 中文 (chinese)

Question  Is payment associated with participant deception about research eligibility and, if so, is higher payment associated with more deception?

Findings  In this randomized survey experiment of a nationally representative sample of 2275 US adults, offers of payment to participate in an online survey were associated with substantial deception by participants about their eligibility compared with the control condition, with estimated proportions of ineligible individuals who engaged in deception ranging from 10.5% to 22.8%. Larger payments were not associated with increased rates of deception.

Meaning  Payment may be associated with deception about eligibility for study participation, but higher payment may not lead to higher rates of deception.

Importance  Offers of payment for research participation are ubiquitous but may lead prospective participants to deceive about eligibility, jeopardizing study integrity and participant protection. To date, neither the rate of payment-induced deception nor the influence of payment amount has been systematically studied in a nationally representative randomized survey experiment.

Objectives  To estimate payment-associated deception about eligibility for an online survey and to assess whether there is an association between payment amount and deception frequency.

Design, Setting, and Participants  Randomized, 7-group survey experiment. Data were collected in March 2018 and analyzed from March to August 2018. The setting was a nationally representative online survey among US adults drawn from the GfK KnowledgePanel.

Interventions  Varying payment amounts for participation ($5, $10, or $20 cash equivalent) and direction of eligibility criterion (having received or not having received an influenza vaccination in the past 6 months).

Main Outcomes and Measures  Proportion of respondents reporting recent influenza vaccination.

Results  In total, 2275 individuals participated in the survey, a 59.4% (2275 of 3829) response rate; 51.8% (1108) were female, and 21.1% of respondents (399) were aged 18 to 29 years, 24.9% (532) were aged 30 to 44 years, 26.0% (601) were aged 45 to 59 years, and 28.0% (738) were 60 years or older. For participants offered a $5 incentive, the reported frequency of recent influenza vaccination was 16.6% higher (95% CI, 9.1%-24.1%) among those told that eligibility (and thus payment) required recent vaccination than among those told that eligibility required no recent vaccination. The corresponding differences were 21.0% (95% CI, 13.5%-28.5%) among those offered $10 and 15.4% (95% CI, 7.8%-23.0%) among those offered $20. Estimated proportions of ineligible individuals who responded deceptively regarding eligibility ranged from 10.5% to 22.8% across study groups. There was no evidence that higher payment was associated with higher frequency of deception.

Conclusions and Relevance  In a nationally representative randomized survey experiment to evaluate whether and to what extent payment is associated with participants misleading investigators about their research eligibility, this study found evidence of significant deception. However, no association was observed between payment amount and frequency of deception. Further research is needed to extend these findings to clinical research. These data suggest that, when possible, investigators should rely on objective tests of eligibility rather than self-report.

Offering payment for research participation is often both practically necessary and ethically fraught. Relying on participant altruism or the prospect of direct medical benefit may be insufficient to achieve recruitment goals, necessitating offers of reimbursement, compensation, and incentives to encourage enrollment and retention.¹ Concern about undue influence stemming from such payment can be minimized by the requirement that institutional review boards (IRBs) approve research only when its balance of risks and benefits is objectively reasonable.² However, even if ethical concerns can be managed, an important pragmatic concern persists: anecdotal evidence and common sense suggest that payment may encourage participants to mislead investigators about their eligibility or other aspects of participation by concealing or fabricating information to secure payment. Intentional misrepresentation of this nature, otherwise known as deception, may have serious consequences for participant safety and the validity of research results.³

Systematic, prospective data regarding whether and to what extent participants will deceive when offered payment in a real study are lacking.^4-9 Therefore, we designed a randomized survey experiment in a nationally representative sample of US adults participating in an online panel, with the objectives of quantifying payment-associated deception about eligibility and evaluating whether variable payment amounts are associated with variable rates of deception.

The University of Pennsylvania IRB approved the study as involving no greater than minimal risk to participants. To avoid influencing participant behavior, the IRB approved an alteration of informed consent to omit a statement disclosing the study’s true objective of evaluating payment-associated deception. Instead, at the end of the survey, each participant was told that “results will be used to help us learn how offering payment affected people’s responses to survey questions. This will help better understand how payment should be used in future research studies.”

Primary data collection for this project was conducted in collaboration with GfK Knowledge Networks, an online survey company. GfK is a member of the American Association for Public Opinion Research (AAPOR) Transparency Initiative and follows all of the association’s reporting guidelines. GfK uses probability-based recruitment procedures to populate its nationally representative survey panel, known as the GfK KnowledgePanel (GfK Custom Research North America), and standing KnowledgePanel members are randomly selected to participate in ongoing studies. Selected members receive an email invitation with a unique password permitting a single use; GfK provides internet access and connected devices to members who lack these tools. The GfK KnowledgePanel allows the generation of fully representative samples to produce statistically valid inferences for study populations.¹⁰

For this study, GfK invited a random sample to complete a 1-minute online questionnaire about attitudes toward vaccines. The email invitation sent to prospective participants stated the amount of payment offered but not the eligibility criteria, which were disclosed only on clicking the survey link.

Participants were randomly assigned into 7 groups, and the survey was held open from March 2 to 13, 2018, until there were 325 respondents in each group. Control group participants (group 1) were told that they were eligible if they had received any vaccine in their lifetime, a condition intended to achieve near-universal eligibility and avoid any incentive to provide either affirmative or negative answers to a subsequent question specifically about influenza vaccination status (the primary outcome measure). In the 6 experimental groups, participants were exposed to the following 2 interventions: (1) direction of the self-reported eligibility criterion (ie, whether having received an influenza vaccination in the past 6 months made them eligible or ineligible) and (2) amount of incentive. Unlike group 1, these participants were told that they were eligible if they had (A groups) or had not (B groups) received an influenza vaccine in the past 6 months, a period that covered the 2017-2018 influenza season. Eligible participants in groups 2A and 2B were offered a $5 cash equivalent for survey completion, groups 3A and 3B were offered $10, and groups 4A and 4B were offered $20.

Although group 1 participants’ eligibility was not tied to their influenza vaccination status, they were also offered $5 to complete the survey. This was necessary to ensure that the only difference between group 1 participants and groups 2A and 2B participants was whether payment depended on their answer to the question about recent influenza vaccination status; this allowed for isolation of the association between payment and rate of deception regarding self-reported eligibility. Inclusion of groups receiving higher levels of payment (groups 3A and 3B and groups 4A and 4B) allowed testing for dose response according to incentive amount.

In the survey, group 1 participants were first asked if they had ever received a vaccination in their lifetime (question 0 [Q0]); all groups were asked whether they had received an influenza vaccine in the past 6 months (Q1). Because eligibility in group 1 was not dependent on recent influenza vaccination status, group 1 participants had no reason to answer the corresponding question, Q1, deceptively. Accordingly, we used the proportion of group 1 respondents reporting recent influenza vaccination as an estimate of actual baseline prevalence of recent influenza vaccination among the study population (ie, as the control). Inclusion of the A and B groups at each payment amount offered a further internal control; absent deception, reported influenza vaccination rates should not differ between groups 2A and 2B, between groups 3A and 3B, and between groups 4A and 4B.

After Q1, all participants who self-reported eligibility for the survey were presented with 3 statements selected from the Parent Attitudes About Childhood Vaccines (PACV) survey tool.¹¹ They were asked to state their level of agreement on a 5-point scale (1 is strongly agree, 2 is agree, 3 is not sure, 4 is disagree, and 5 is strongly disagree) with the following statements: “Children get more shots than are good for them” (Q2), “I believe that many of the illnesses that shots prevent are severe” (Q3), and “It is better for a child to develop immunity by getting sick than to get a shot” (Q4). These statements were included to provide content to the survey so that it would not end immediately after Q1, as well as to test an exploratory hypothesis that deception about eligibility may be associated with biased responses to other survey questions. More specifically, if deception was detected in the A and B groups, we sought to test whether reported vaccine attitudes in these groups would differ from those expressed by similar participants in group 1. The objective was not to assess vaccine attitudes in the study population but rather to evaluate whether participants who were deceptive about their eligibility might also adjust their responses to subsequent survey questions, perhaps based on speculation as to how a truly eligible participant might respond.

GfK provided participants’ demographic data along with survey results. KnowledgePanel members self-report demographics in a multiple-choice format.

Data were analyzed from March to August 2018. We performed χ² tests within pairs of groups (group 2A v 2B, group 3A v 3B, group 4A v 4B) to compare the proportions of participants who reported influenza vaccination in the past 6 months at each of the 3 payment levels ($5, $10, or $20). Sample sizes (325 respondents per group) were calculated to achieve greater than 90% power to detect a true difference of at least 13 percentage points in reported influenza vaccination rates within each A and B pair (2-sided test, α = .05). We also tested for an association between payment amount and reported influenza vaccination in the A and B groups separately using binary logistic regression.

To estimate the percentage of truly ineligible participants in each experimental group who answered the eligibility question (Q1) deceptively, we assumed that, in the absence of incentives, the distribution of responses in each experimental group would mirror that in the control group. For respondents incentivized to report that they had received an influenza vaccine (A groups), we calculated the percentage of respondents answering deceptively as follows: (% reporting vaccination in the experimental group − % reporting vaccination in the control group) / % reporting no vaccination in the control group. Conversely, for respondents incentivized to report that they had not received an influenza vaccine (B groups), we calculated the percentage of respondents answering deceptively as follows: (% reporting no vaccination in the experimental group − % reporting no vaccination in the control group) / % reporting vaccination in the control group. We assumed that all participants who self-reported ineligibility to participate in the survey were truthful because that response led to no payment. Therefore, they were excluded from these calculations.

To evaluate responses to PACV statements, we developed a single index score for each eligible participant by averaging the responses to the 3 questions after reverse scoring Q3.¹² Individual index scores were then averaged by group to create a single mean score for those in group 1 reporting recent influenza vaccination, those in group 1 reporting no recent influenza vaccination, those in the A groups reporting recent influenza vaccination (with incentive to do so), and those in the B groups reporting no recent influenza vaccination (with incentive to do so). The mean scores could range from 1 (reflecting maximal vaccine hesitancy) to 5 (reflecting minimal vaccine hesitancy). The mean scores were compared across groups using paired t tests.

For this study, design weights were computed to reflect the selection probabilities of assignees. Subsequently, final analysis weights were computed by balancing design weights with respect to geodemographic distributions of the target population for which benchmarks were obtained from the March Supplement 2017 US Census Bureau Current Population Survey.¹³ Respondents were weighted to represent the US adult population 18 years or older. We accounted for final design weights in all analyses. All P values are 2-sided, and statistical significance is defined as P < .05. Statistical analyses were conducted with a software program (SAS, version 9.4; SAS Institute Inc).

Among 3829 individuals invited to participate in the survey, 2275 responded, for a response rate of 59.4%. Of the respondents, 51.8% (1108) were female; 21.1% (399) were aged 18 to 29 years, 24.9% (532) were aged 30 to 44 years, 26.0% (601) were aged 45 to 59 years, and 28.0% (738) were 60 years or older. Characteristics of participants, including age, sex, educational level, race/ethnicity, and household income, are listed in Table 1. The study was not designed or powered to assess results by demographic category; demographics are provided solely to demonstrate sample diversity.

In the control group, 52.2% (weighted 95% CI, 46.3%-58.1%) of respondents (173 respondents unweighted) who reported ever having received any vaccine reported having received an influenza vaccine in the past 6 months (“recent vaccination”). Absent deception, this rate should have remained constant across all study groups. However, it increased to 63.1% (95% CI, 57.4%-68.7%) in group 2A, 62.8% (95% CI, 57.3%-68.4%) in group 3A, and 62.1% (95% CI, 56.5%-67.8%) in group 4A, all of which were incentivized to self-report recent influenza vaccination (Table 2). Conversely, it decreased to 46.5% (95% CI, 40.5%-52.4%) in group 2B, 41.8% (95% CI, 36.0%-47.5%) in group 3B, and 46.7% (95% CI, 40.9%-52.5%) in group 4B, all of which were incentivized to self-report no recent influenza vaccination (P < .001 for the 7-way comparison). This reflects differences of 16.6% (95% CI, 9.1%-24.1%) in reported recent vaccination between groups 2A and 2B, 21.0% (95% CI, 13.5%-28.5%) between groups 3A and 3B, and 15.4% (95% CI, 7.8%-23.0%) between groups 4A and 4B (P < .001 for all 2-way comparisons).

Based on the estimated true proportion of recent influenza vaccination in the study population (52.2%), estimated proportions of participants who were actually not eligible to participate but who answered deceptively about their eligibility ranged from 10.5% to 22.8% in the 6 experimental groups (Table 2). There was no evidence of a dose-response relationship with increasing payment amounts either among participants incentivized to report or not report recent influenza vaccination. In both the A and B groups, self-reported influenza vaccination rates remained almost flat across increasing incentives to deceive, as noted earlier (Figure).

The mean score on PACV questions among group 1 (control) participants reporting recent influenza vaccination was 3.87 (95% CI, 3.75-4.00) of 5; among group 1 participants reporting no recent influenza vaccination, it was 3.58 (95% CI, 3.44-3.72). Eligible participants in the A groups (reporting recent influenza vaccination) had a combined mean score of 3.81 (95% CI, 3.74-3.87), and eligible participants in the B groups (reporting no recent influenza vaccination) had a combined mean score of 3.43 (95% CI, 3.36-3.50). We observed no significant difference in the mean scores between group 1 respondents reporting recent influenza vaccination and those reporting eligibility in the A groups (3.87 vs 3.81, P = .35) or between group 1 respondents reporting no recent influenza vaccination and those reporting eligibility in the B groups (3.58 vs 3.43, P = .06) (Table 3).

In this randomized survey experiment conducted in a nationally representative sample of US adults, we observed evidence of substantial payment-associated deception regarding self-reported eligibility even at the lowest amount of payment offered. However, we did not observe evidence that higher payment was associated with more frequent deception. These findings provide novel insight regarding the potential association between offers of payment and deception by participants.

Our study was motivated by gaps in the empirical literature on payment-induced deception. Prior studies have raised concern that payment may lead research participants to mislead investigators, but the studies are limited to identifying cases in the published literature, retrospective self-report, hypothetical projections of potential behavior, and nonrepresentative populations. For example, Lee et al³ identified 103 instances of deception, including concealment, fabrication, drug holidays, and collusion, in 90 studies selected from a literature review of articles that included the terms deception, deceit, and subversive subjects, among others. Most of these instances of deception were incidentally detected, and the authors of the review article acknowledge that “few studies have investigated deceit in research participants, and fewer studies have examined deceit as a primary objective.”^3(p154) Our study did both.

Walker et al⁴ reported evidence of “rule-breaking” with implications for study validity and participant safety, much of which was motivated by payment, based on 5 case studies of repeat participants in phase 1 clinical trials drawn from a larger set of interviews with healthy volunteers. Relying on the same set of interviews with healthy volunteers (more than two-thirds of whom had participated in ≥5 clinical trials), McManus and Fisher⁵ found that payment can influence reporting of adverse events, but participants may be willing “to forgo their full compensation if they believe not reporting their symptoms jeopardizes their own safety or the validity of the research.”^5(p82) In a survey of individuals who had participated in at least 2 studies in the past year (of any type but most frequently “medication trials”), Devine et al⁶ found high rates of self-reported lifetime deceptive behavior related to eligibility for clinical trials, with significant correlation between deception and monetary reward.

Other studies have evaluated the hypothetical influence of payment on deception. Bentley and Thacker⁷ asked US pharmacy students to rate willingness to participate in hypothetical clinical studies and the likelihood they would engage in various disclosures to investigators. They found that higher levels of payment may influence behaviors regarding concealing information about restricted activities, but payment did not have a significant association with the likelihood of reporting adverse events. Slomka et al⁸ interviewed “economically disadvantaged African-American crack cocaine smokers”^8(p1403) participating in HIV prevention studies to solicit their general views about payment for research participation and identified some concern that high payments might cause people to provide false information to investigators. Finally, outside of the clinical trial context, Chandler and Paolacci⁹ carried out a series of online experiments using Amazon’s Mechanical Turk platform and determined that participants often misrepresent relevant characteristics to meet the eligibility criteria.

Our study builds on these findings by analyzing deception in a broader population sample offered payment to participate. The randomized nature of the experiment allowed us to assess the association of payment, payment amount, and direction of the incentive with aggregate deception about self-reported eligibility to participate. Although this study was carried out in the context of survey research, the data offer a rigorous and internally valid signal about the potential influence of payment on deception by research participants. Our main finding that there was significant payment-associated deception is broadly relevant to the work of all scientists who carry out data collection with human participants, including clinical trialists using survey methods to evaluate participant-reported outcomes or otherwise supplement clinical data, as well as those conducting trials of interventions relying on self-reported adherence, because these data may be influenced by participant desire to continue receiving study payment.^14,15 Our results offer a compelling reason to carry out similar research in the clinical trial context, where payment amounts and other benefits—such as access to potentially desirable investigational medicines—are often more substantial than typically associated with survey research. They also suggest that trialists should use objective rather than self-reported eligibility (and other) criteria whenever possible and that deception regarding eligibility may be a particular concern for pragmatic trials enrolling participants on the basis of self-reported behaviors and conditions.

The absence of higher rates of deception at higher payment amounts in our study may reflect that some proportion of individuals will not deceive even when the financial reward of doing so is great or that some individuals have “tipping points” higher than the amounts we offered. Addressing these questions about dose response is important because it may be that attempts to avoid deception by keeping offers of payment low will be ineffective (based on our finding that deception may occur even at low payment amounts) and that high payment offers may have the benefit of speeding recruitment without posing greater risk to study integrity (based on our finding that deception may not increase at higher payment amounts). In either case, as noted in the previous paragraph, adopting objective eligibility criteria should be a goal whenever possible.

A potential result of deception about eligibility is that the validity of other information reported by participants may also be affected. We hypothesized that individuals who deceived about eligibility might seek to answer subsequent questions regarding vaccine attitudes in a biased manner according to their estimate of how someone who was truly eligible might answer. However, it is unclear whether participants engaged in such role playing. Based on the absence of statistical significance on comparative mean vaccine attitude scores, we found no evidence that deception about eligibility by participants who were incentivized to report having either received or not received a recent influenza vaccine influenced responses to other survey questions.

The primary limitation of our study is that it was conducted in the context of a survey relying on self-reported eligibility; therefore, it cannot directly answer questions about the possibility and influence of payment-induced deception in the clinical trial context. In addition, there was no comparison of responders (59.4%) with nonresponders, which may limit generalizability of these findings. In contrast to some clinical trials that may entail high levels of burden and risk, participants in this brief online survey were asked to undertake little burden and no risk for proportionately high payment. They could also engage in deception without fear of being caught or of incurring physical risk by evading eligibility criteria intended to protect them. It is unclear whether clinical trial participants will be more inclined to deceive as a result of typically higher offers of payment in that setting or less inclined given greater potential consequences for themselves or investment in the outcome of the research.

Our study design cannot rule out the possibility that reported rates of recent influenza vaccination varied according to the eligibility criteria not as a result of payment-associated deception but rather as a result of participant desire to report eligibility in a way that would assist researchers (a social desirability or “helpfulness” bias). However, there is no evidence to suggest that individuals are eager to participate in research for which they are not eligible in the absence of incentive payments.

Another limitation is that we tested only 3 payment levels, which could obscure signals that may have become evident by evaluating more payment amounts at tighter increments. In addition, our assessment of whether payment was associated with different information provided by research participants after deception about eligibility was limited to an exploratory measure based on a small number of additional questions regarding vaccine attitudes not validated for that particular purpose.

One pressing research need is to replicate these findings among participants in clinical trials that have some self-reported criteria for initial and continuous eligibility. Further research is also needed to understand whether a dose-response relationship between payment amount and deception rates might be seen in other contexts or at different amounts, as well as to systematically assess the downstream influence of the inclusion of individuals in research who were deceptive about their eligibility. As further data are developed, research evaluating the effectiveness of approaches to minimize payment-induced deception will also be important.

Ethical concerns about undue influence have dominated discussions about payment for research participation, but investigators and IRBs also should be aware of the potential for payment to interfere with a study’s scientific integrity and participant protections. Our study found that as many as 10.5% to 22.8% of ineligible participants were motivated to deceive about their eligibility to enroll in an online survey to secure payment, although higher rates of payment were not associated with higher rates of deception. Because offering payment can be an important and ethical tool for recruiting, retaining, and fairly compensating participants,¹ we caution against responding to these findings by eliminating or reducing offers of payment. Instead, it is necessary to develop, test, and implement comprehensive alternative approaches to minimize the likelihood of participant deception.³

Accepted for Publication: November 29, 2018.

Published: January 25, 2019. doi:10.1001/jamanetworkopen.2018.7355

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Fernandez Lynch H et al. JAMA Network Open.

Corresponding Author: Holly Fernandez Lynch, JD, MBE, Department of Medical Ethics and Health Policy, Perelman School of Medicine, University of Pennsylvania, 423 Guardian Dr, Philadelphia, PA 19104 (lynchhf@pennmedicine.upenn.edu).

Author Contributions: Professor Fernandez Lynch and Dr Xie had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Fernandez Lynch, Joffe, Thirumurthy, Largent.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: All authors.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Xie.

Obtained funding: Fernandez Lynch, Thirumurthy, Largent.

Supervision: Fernandez Lynch.

Conflict of Interest Disclosures: Professor Fernandez Lynch reported receiving personal fees from the US Department of Health and Human Services for committee work related to payment of research participants and other research ethics and regulatory issues. No other disclosures were reported.

Funding/Support: This project was supported in part by grant UL1TR001878 from the National Center for Advancing Translational Sciences. It was also supported in part by the Institute for Translational Medicine and Therapeutics (ITMAT) and the Department of Medical Ethics and Health Policy, both at the University of Pennsylvania Perelman School of Medicine.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Center for Advancing Translational Sciences or the National Institutes of Health.

Additional Information: GfK was contracted to execute this study. Wendy Mansfield, PhD, senior vice president at GfK, collaborated to determine how best to conduct the survey through GfK’s KnowledgePanel, and Ying Wang, MS, senior research manager at GfK, assisted with survey programming, data collection, and raw results. Neither was compensated outside their salaries provided by GfK under the contract.