Evaluation of Survey Data Quality Based on Interviewers’ Assessments: An Example from Taiwan’s Election and Democratization Study

Abstract

Researchers usually examine the quality of survey data by several conventional measures of reliability and validity. However, those measures are mainly designed to examine the quality of an individual measurement, rather than the quality of a data set as a whole. There is a relative lack of methods for evaluation of the overall data quality. This paper attempts to fill this gap. We propose using interviewers’ assessments as one of criteria for evaluating the overall data quality. Interviewers are the ones who literally conduct and thus directly observe interviews. Taiwan’s Election and Democratization Studies (TEDS) have required interviewers to assess how trustworthy the responses of each of their interviewees are, and to provide several descriptions about the process and environment of the interviews. We use this information to evaluate the data quality of TEDS surveys and compare it with the results from the conventional test-retest method. The findings are that the interviewer assessment is a fair indicator of the overall reliability of attitudinal questions but not a good indicator when factual questions are examined. Regarding the evaluation of data validity, more data is required to see whether or not interviewers’ assessment is informative in terms of data quality.

Keywords

• data quality
• interviewers’ assessment
• reliability
• validity
• Taiwan’s Election and Democratization Study