Innovative design and analysis approaches for addressing increasing rates of attrition

All network activities in this second thematic area will focus on: 1) innovative design approaches for maintaining or increasing response rates in longitudinal studies of aging, and 2) state-of-the-art analytic approaches to adjusting survey estimates for attrition that may be non-ignorable in nature, which could lead to biased population estimates of trajectories in health outcomes that are of substantive interest. From a design perspective, methodological innovations in this area will focus on mixed-mode study designs (e.g., optimizing models for assigning participants to different data collection modes and contact protocols); methods to minimize respondent burden (e.g., questionnaire design features such as modular survey design; online schedulers); incentive strategies (e.g., cost-efficient approaches such as two-phase sampling; panel management; targeted incentives); respondent contact protocols (e.g., messaging; periodicity of contact attempts and types); and more efficient screening techniques for aging populations that are designed to improve screener response rates (e.g., use of commercial data to over-sample households with age-eligible individuals). From an analytic perspective, methodological innovations in this area will focus on appropriate adjustment of baseline and time-varying sampling weights to account for differential rates of attrition along key population subgroups; weighting and imputation methods for adjusting estimates of trajectories to allow for both ignorable and non-ignorable attrition; and estimation techniques for non-probability samples of small minority subgroups followed over time.

Bibliography

Biemer, P.P., Harris, K.M., Burke, B.J., Liao, D. & Halpern, C.T. (2022) Transitioning a panel survey from in-person to predominantly web data collection: Results and lessons learned. Journal of the Royal Statistical Society: Series A (Statistics in Society), 185, 798– 821. https://doi.org/10.1111/rssa.12750

Summary: In recent years, costs associated with in-person interviews have increased, while the data quality advantages typically associated with this data collection method have lessened. Therefore, many longitudinal surveys have begun to transition from in-person to web data collection, which has prompted the proposal of a multi-sample, multi-phase responsive design in an effort to minimize data quality risks associated with the transition from in-person to web-based data collection.

Boye, K., Mood, C., Tahiln, M.  (2022). Impact of incentives and response mode on non-response rates and non-response bias in a Swedish general population survey [PowerPoint slides]. Sweedish Institute for Social Research, Stockholm University Sigtuna 26-27 Sept Boye Mood Tahlin_participants.pdf

Summary: Utilizing more than one response mode and providing respondents with an economic incentive to participate in a survey are shown to help improve response rates while not increasing non-response bias.

Goodman, A., Brown, M., Silverwood, R.J., Sakshaug, J.W., Calderwood, L., Williams, J., et al. (2022) The impact of using the Web in a mixed-mode follow up of a longitudinal birth cohort study: Evidence from the National Child Development Study. J R Stat Soc Series A, 822– 850. Available from: https://doi.org/10.1111/rssa.12786

Summary: A mixed-mode design utilizing both telephone and web participation was investigated in the National Child Development Study in 2013. Compared to using telephone-only data collection methods, the offer of a web survey increased overall participation rates by 5.0 percentage points.

Lynn, Peter (2022). Non-response on Longitudinal Surveys [PowerPoint slides]. Economic and Social Research Council, University of Essex. Peter Lynn.pdf

Summary: Nonresponse in longitudinal studies can be impacted by many factors, such as increased burden on respondents and a failure to locate participants due to longer study duration. Various survey adjustments can be implemented to help combat nonresponse in longitudinal studies, including utilizing survey weights and auxiliary data.

Maslovskaya, O., Struminskaya, B. and Durrant, G. (2022), The future of online data collection in social surveys: Challenges, developments and applications. J R Stat Soc Series A, 185: 768-772. https://doi.org/10.1111/rssa.12895

Summary: Online data collection has become increasingly popular as widespread use of technology has increased, especially during the COVID-19 pandemic, as face-to-face data collection was not an option. Online data collection has some unique challenges, such as coverage issues, data quality, and measurement problems. 

McGonagle, K.A., Sastry, N., and Freedman, V.A. (2022). The Effects of a Targeted “Early Bird” Incentive Strategy on Response Rates, Fieldwork Effort, and Costs in a National Panel Study. Journal of Survey Statistics & Methodology, 00, 1-22.

Summary: Adaptive survey design is a powerful tool that can be used to combat declining response rates in household surveys. One way to implement adaptive survey design is to utilize incentives. The use of early-bird incentive (EBI) has been found to increase response rates as well as lower study costs in a sample of high-effort respondents from the 2019 wave of the US Panel Study of Income Dynamics. 

Sastry, N. & McGonagle, K.A.(2022) Switching from telephone to web-first mixed-mode data collection: Results from the Transition into Adulthood Supplement to the US Panel Study of Income Dynamics. J R Stat Soc Series A, 933– 954. Available from: https://doi.org/10.1111/rssa.12840

Summary: In an effort to reduce costs and increase respondent cooperation, web interviews were conducted in the 2019 wave of the Transition into Adulthood Supplement (TAS) to the US Panel Study of Income Dynamics (PSID). Compared to using telephone-only interviews, it was found that there were higher response rates, interviews were conducted faster, and data quality was higher and with lower associated costs when using the web-first approach.

Schwarz, H., Revilla, M. & Struminskaya, B.(2022) . J R Stat Soc Series A, 981– 1003. Available from: https://doi.org/10.1111/rssa.12857

Summary: As more surveys are conducted, respondents are likely to have previous survey experience, which can make the use of incentives important for continued participation. Quality of survey responses does not seem to be associated with previous web survey experience, or participating only to receive an incentive. 

Watson, N., & Cernat, A. (2022, January 11). Simulating the Consequences of Adaptive Survey Design in Two Household Panel Studies. Retrieved December 20, 2022, from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smab050/6503715?redirectedFrom=fulltext

Summary: Adaptive survey design techniques have typically been used in effort to help improve response rates and lower overall survey costs. Much of the existing research surrounding these techniques have focused largely on cross-sectional surveys, whereas this study seeks to explore the potential efficacy of this approach within longitudinal surveys. Implementing this approach appeared to reduce the number of follow-up calls to participants, but reduced the overall response rate across time.