To help increase survey response rates and ensure the final sample is representative of the target population, researchers often offer respondents incentives. These incentives take different forms; for example, prepaid incentives are given to respondents before they complete the survey, and post-paid incentives are provided to respondents only after they complete it.
Gallup and other leading survey researchers, some of whom are listed at the bottom of this article, have thoroughly investigated the use of prepaid and post-paid incentives in self-administered mail surveys, finding that relatively small prepaid incentives can significantly improve response rates in these surveys. But researchers know less about how effective post-paid incentives are, particularly in relation to web-based surveys, which have become more common with the proliferation of the internet.
In November 2017, Gallup partnered with a large U.S. university to test the effectiveness of prepaid and post-paid incentives in a web-based survey of its alumni. The survey included questions about employee engagement, well-being and specific experiences in college.
The Experiment
Gallup randomly assigned all sample members to one of three groups. More than 10,000 sample members were assigned to group 1, which received no prepaid or post-paid incentive. One thousand sample members were assigned to group 2, which was promised a $5 gift card upon survey completion. Another 1,000 sample members were assigned to group 3, which received a $5 gift card in the initial survey invitation.
The Results: Prepaid and Post-Paid Incentives Equally Effective
The prepaid and post-paid incentives both improved response rates. The final response rate among group 1, which didn’t receive an incentive, was 13%. Group 2, in which respondents received a post-paid incentive, generated a 20% response rate, and the final response rate among those provided a prepaid incentive was 19%. The difference between groups 2 and 3 is not statistically significant.
| Final response rates | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| % | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Treatment group | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1: No incentive | 13 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 2: Post-paid incentive | 20 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 3: Prepaid incentive | 19 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Total response rate | 14 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| GALLUP, NOVEMBER 2017 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Demographic Distributions Similar Across Groups
Despite differences in the response rates across the three groups, the final, unweighted, sample demographic characteristics of each group were similar. In examining respondent characteristics, including age, race/ethnicity and education level, there were little to no statistically significant differences among the groups. The similarity in respondent characteristics suggests that the incentives did not disproportionately encourage specific subgroups of the population to participate.
Conclusion
Gallup’s experiment shows that prepaid and post-paid incentives are equally effective in improving response rates for web-based surveys. However, post-paid incentives are more cost-efficient than prepaid incentives are. For example, in a study of 1,000 respondents with a response rate of 20%, a prepaid incentive of $5 would cost researchers $5,000. In contrast, a post-paid incentive achieving a 20% response rate would cost only $1,000. In studies of larger populations, the cost differences between these strategies could be even more significant, further justifying the use of a post-paid incentive given the equal effectiveness of both incentive types.
The final decision about which incentive type to use depends on the data collection method and the target population. For example, in the case of a self-administered mail survey, a small cash incentive that can be directly provided to the sample member without condition may be particularly effective and operationally efficient. Providing a post-paid incentive to these respondents may be operationally inefficient because it would require multiple mailings. However, in web-based surveys, the operational efficiency and total effect of a post-paid incentive may justify using a post-paid incentive rather than a prepaid one.
Although the findings from this experiment are instructive, more research is needed about the type and amount of incentives required to maximize response rates. This is particularly important because of the challenges facing survey researchers as response rates decline across all data collection modes.
See the sources below for more information about the use of prepaid incentives:
- Singer, E., and Ye, C. (2013). The use and effects of incentives in surveys. The ANNALS of the American Academy of Political and Social Science, 645(1): 112-141.
- Mercer, A., Caporaso, A., Cantor, D., & Townsend, R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79(1), 105-129. Retrieved from https://www.bls.gov/cex/workshop/2016/symposium/monetary-incentives-and-response-rates.pdf
- Jones, J., Saad, L., Newport, F., and Marken, S. (2015). A mail survey experiment using Gallup’s annual crime survey. [Presentation]. Paper presented at the 70th Annual Conference of the American Association for Public Opinion Research. Hollywood, Florida.
by Stephanie Marken and Zac Auter
