Academic Technology Center
Teaching with Technology Collaboratory

Survey Design Best Practices

Teaching Goal:

To design and develop surveys that produce accurate, reliable, objective, and valid data for use in your instruction and research.

Here at WPI, a number of technology-oriented programs are available to assist you with the design and deployment of surveys:

Benefits of Addressing - Research and Theoretical Base

Surveys that are designed well...

Designing a Survey

  1. Selecting your sample (who, how, how many)
  2. Designing good questions (question types, wording, response/measure types)
  3. Developing the survey (layout, format, question order)

1. Selecting the Sample

The "survey population" is the larger population from whence the actual "sample" is selected. All samples are subject to sampling error, though this error can be minimized by accounting for two factors:

  1. Larger samples produce smaller sampling errors than smaller samples
  2. Homogenous samples produce smaller sampling errors than heterogeneous samples
Probability Sampling Designs

This type of sampling is purposefully random, guaranteeing that each person in the survey population has an equal chance of being selected (or not) for inclusion in the sample

Definition
Simple Random Sampling (SRS)

Each person in the survey population is assigned a random number, which is then randomly selected for inclusion in the sample.

This is rarely used in practice because it is inefficient and laborious if manually selected and coded.

Systematic Sampling

The survey population is systematically numbered 1-X, and each Nth person is selected. For example, each 10th person is selected for inclusion in the sample.

This method differs little from SRS, though it is more accurate in many instances than SRS. One danger to consider is periodicity: if the survey population is coded cyclically (e.g. - by apartment number), the sample may be biased.

A second danger to be aware of is implicit stratification: if the persons in the survey population are arranged in a certain pattern (e.g. - alphabetically by last name), the sample may be biased.

Stratified Sampling

Instead of drawing the sample from the entire survey population, the survey population is broken down into categories that are then sampled appropriately (though not necessarily equally). For example, a community might be broken down by Race, Income Level, or Age before being sampled.

Stratified sampling produces a more representative sample from the survey population, markedly reducing sampling error.

Multistage Cluster Sampling

Cluster sampling is best used when the survey population is too large to list individually for sampling (e.g. Ė all homeowners in the United States). It involves listing possible data sources (or clusters) and either sampling or stratifying those clusters until a representative, usable sample is defined.

For example, you might list all states in the United States and then sample or stratify the states into one cluster; then you might list all counties in each selected state and sample or stratify the counties into another cluster; and so forth, until you arrive at a reasonable and representative sample.

Probability Proportionate to Size (PPS)

In some instances of Multistage Cluster Sampling, very large populations stand a chance of being overrepresented, and very small populations stand a similar chance of being underrepresented.

In such cases, combining several small populations into one larger cluster is common practice, as is pre-sampling larger populations to account for their overrepresentation.

Non-probability Sampling Designs

When to Use

Purposive Sampling

Purposive sampling involves the selection of a sample based on the researcherís knowledge of the larger community or sampling frame. Though this provides very well-articulated results, they often fall victim to biases of the researcher. As such, great care must be taken when selecting the sample and an objective arbiter judging the selections is often recommended.

Quota Sampling

For quota sampling, the researcher must have considerable demographic data about the sample population. The sample population is broken down into demographically-delineated groups which are then appropriately weighted according to their portion of the total population, resulting in a reasonable representation of the sample population.

Available Subjects Sampling Surveys of students enrolled in particular courses only provides localized knowledge about select student populations. The data should not be used to describe students as a whole, as most universities employ uniquely selective methods for admitting and enrolling students.

2. Design Good Questions

Best Practices

Example

Closed-ended questions

Closed-ended questions force respondents to select an answer from a list of options provided by the researcher. This makes for data uniformity and easy coding after the surveys are collected.

Two things to remember when using closed-ended questions: first, the list of responses should be exhaustive so as to include any possible answer from a respondent; second, the options should be mutually exclusive so that respondents are not forced to choose between two similar responses.

Open-ended questions

Open-ended questions provide space for respondents to write-in their own responses to questions. This gives the respondent greater flexibility to answer question as they see fit, though coding the data is usually more time-consuming for the researcher.

Though open-ended questions usually provide richer data, they can also result in answers that are irrelevant to the studyís focus. In addition, open-ended responses usually produce less useful data if the survey is self-administered.

Likert scales

Questions about attitudes are often asked using likert scales which force the respondent to place themselves on a continuum of responses that typically include "strongly agree," "agree," "neutral," "disagree," and "strongly disagree". Other variations of these categories can easily be incorporated.

It is often assumed that a likert scale should include a "neutral" or "donít know" response on the scale, but if the purposes of the study are to force a respondent to take a stand, it is entirely justifiable to not include such a middle-of-the-road response.

Avoid negative items

When asking questions, avoid the use of negatives as they often confuse respondents into answering a question in a manner they do not intend to.

For example, when asked to agree or disagree with the statement, "The United States should not reduce its funding of bilingual education programs," many respondents will either read over the word not, or will be confused by the double negative in the question and their response.

Be conscientious of the Halo Effect

When questions are written, they often reference a person, group, or belief that inherently biases the question and subsequent response in relation to the respondentís feelings about the person, group, or belief. In some cases, even a particular word can bias a question. The following represent questions biased by the Halo Effect:

  • "Donít you agree with the President that...?"
  • "Do you agree with the recent Supreme Court decision that...?"
  • "Do you disagree that welfare is a problem?"
Contingency questions should be clearly marked

Researchers often use contingency questions to ask follow-up questions to respondents who answered a question a particular way (e.g. - if you answered "Yes" to this question, go to question 8).

Contingency questions are a good way to get more targeted information from a select group of respondents, but the answer path must be clear to the respondents. Visually drawing out the contingency question using an indented box is the most effective means of doing this.

Avoid double-barreled questions

Researchers often ask respondents for a single answer to a combination of questions. This forces the respondent to agree or disagree with all the questions, instead of giving them the opportunity to independently assess each question. The inclusion of the word ďandĒ is often a red flag for a double-barreled question. The following is an example of a double-barreled question:

"Do you agree that the United States should abandon its military programs and spend the money on educational programs?"

Use adequate wording

Ask questions in complete sentence to avoid any possible misunderstandings.

For example, instead of asking "Income?" you might ask "In the last tax year, what was your total personal income, not including that of your spouse/partner?"

Ensure consistent meaning to all respondents

Be sure to minimize any ambiguity in the question so that all respondents read the question the same way.

For instance, asking the question, "How many times in the past year have you talked with a local school board member?" could be read several ways. Do phone or email discussions count? Does attendance at a school board meeting count?

The "Donít Know" option

Providing "Donít Know" as a response gives respondents an easy way out from thinking about and answering a difficult question. However, if the nature of the question could reasonably elicit a "Donít Know" response, it should be provided as a response alternative.

3. Developing the Survey

Best Practices

Rationale

Include demographic questions at the end of the survey

The most important questions of the survey should be asked first when the respondents are most enthusiastic about participating in the survey. Demographic questions take little thought, are often indicative of a dull form, and are best included at the end of a survey when respondents are most tired.

Maximize your use of "white space"

There is a ill-founded fear that respondents react negatively to longer surveys. It is rather the case that short surveys, in which the questions are packed into a few pages, are often more imposing, confusing, and demoralizing than longer surveys in which only a few questions are asked on each page.

This is especially important when using open-ended questions in which longer responses are both desired and expected.

Use boxes for multiple choice questions

Boxes that respondents are expected to fill in completely are the most reliable method for gathering responses to multiple choice questions because there is little ambiguity as to which answer was chosen.

Open spaces for checkmarks are generally considered the least reliable method of collecting multiple choice data, as respondents tend to make large checkmarks that make it difficult to determine which answer they chose.

Question order is important

For example, asking a question about a personís religion will bias their answers to subsequent questions.

Controlling for this is difficult, though acknowledging that there is a question order bias is a helpful first step when designing the survey and analyzing the resultant data.

Provide instructions

This seems like a fundamental concept, but many surveys lack adequate instructions, leading respondents to answer questions incorrectly or without the desired depth.

Instructions are especially important when the question type changes - respondents should be appraised of any new expectations of the researcher.

Field test your survey before implementing Time and money permitting, surveys should be field tested to account for any bias, problematic wording, or confusing structure. This ensures more reliable and valid data and limits the financial loss if significant problems are overlooked during the initial design.

The preceding is a simplified explanation of the complex science of survey design. For a more extensive description that provides more practical examples and best practices, please see the below references.

References:

Babbie, E. Survey research methods (2nd ed.). Belmont, CA: Wadsworth Publishing Co.

Fowler, F.J. (2001). Survey research methods (3rd ed.). Newbury Park, CA: SAGE Publications.

Maintained by itweb@wpi.edu
Last modified: Jul 02, 2007, 13:52 EDT
[WPI] [ATC] [Home] [Back] [Top]