Chapter 10: Introduction to Survey Methodology and Design
Prepared for the Interdisciplinary and Global Studies Division,
Worcester Polytechnic Institute
By James K. Doyle
Assistant Professor
Department of Social Sciences and Policy Studies
Response Rates
Selecting an unbiased sample of sufficient size is an important component of any scientific survey, but is not enough to ensure that the people who answer are representative of the larger population from which they were drawn. It is also important to obtain a high response rate. The response rate of a survey is simply the number of completed, usable surveys obtained divided by the number of people who were asked to complete a survey. If this fraction is too low, there is a strong possibility of "nonresponse error," that is, that estimates are biased because those who didn't respond to the survey have different characteristics or opinions than those who did respond. To illustrate the potential biasing effect of a low response rate, suppose that 500 surveys are sent to randomly selected WPI students asking which of two alternate food service plans they prefer. Suppose further that 150 students (30%) respond, and that 98 of them choose Plan A and 52 choose Plan B. Thus a clear majority of 65% are in favor of Plan A. However, this simple analysis assumes that nonrespondents have the same opinions as respondents. What if, for example, the 350 nonrespondents in fact have just a slight preference, say, 55% vs. 45%, for Plan B? The true percentage preferring Plan A in this case would in fact be just 0.3 x 65% + 0.7 x 45% = 51%, which is not significantly different from 50%. What if, instead, the 350 nonrespondents had a strong preference, say 70% vs. 30%, for Plan B? In this case the true percentage preferring Plan A would be only 0.3 x 65% + 0.7 X 30% = 40.5. Under this scenario, the results obtained from the 150 respondents completely misrepresent the preferences of the student body! Of course, a low response rate to a survey does not guarantee that results will be biased. In the hypothetical food service survey, for example, it is entirely possible that nonrespondents had the same preferences (65% in favor of Plan A) as those who did respond. The problem is there is simply no way of knowing for sure what nonrespondents are like or how they are thinking. The only sure way of reducing this uncertainty is to obtain a high response rate in the first place. For example, suppose that the food service survey which reported that 65% of students preferred Plan A had achieved a response rate of 70% rather than 30%. Under this scenario, even if nonrespondents strongly disliked Plan A (say, 70% preferred Plan B instead), Plan A, as suggested by the survey, would still be the preference of the student body: 0.7 x 65% + 0.3 x 30% = 54.5%. Obtaining a high response rate to a survey requires a substantial investment of time and effort. A single mailing or phone call to each potential respondent is completely inadequate and is likely to result in a response rate of 20% or lower. This happens not because most people aren't interested or don't want to help, but simply because they are busy and have many competing demands on their time. Thus, some mail surveys never get opened because they are confused with junk mail and get thrown out; those that are opened often end up under a large stack of correspondence that people never find the time to get around to. And, telephone interviewers, if they don't get a busy signal or an answering machine, are highly likely to get a person who refuses to talk because they have company, or are eating dinner, or are on their way out the door, or have to put the kids to bed. There are two things a researcher can do to increase the chances that people will take the time to respond: (1) Design a questionnaire (and accompanying material) or telephone introduction that makes it immediately clear to people that responding to the survey is both important and easily accomplished (see Section VII). (2) Design an implementation plan that includes multiple mailings or phone calls, if necessary, directed at each potential respondent (see Section IX). If these established techniques are employed, response rates of 60-70% or higher can be achieved. However, in some cases, in spite of a researcher's best efforts, the response rate turns out to be less than 60%. In such cases the researcher should investigate and attempt to identify, insofar as possible, how similar or different nonrespondents are from respondents on relevant variables. Sometimes this can be accomplished with available information, for example, in most cases the gender and geographical location of nonrespondents is known, and occasionally sample lists contain other variables that can be compared. It is also possible to randomly select and telephone a small sample of nonrespondents and launch an all-out effort to get them to answer at least a couple of brief questions that will allow them to be compared with respondents. Even people who refused to participate initially will often agree to an interview that takes just a minute or two. For further information on response rates see Chapter 2 of Salant and Dillman (1994) and Chapter 2 of Groves (1989).
Maintained by webmaster@wpi.eduLast modified: November 14, 2006 11:41:31