Interdisciplinary & Global Studies Division
Global Perspective Program

Chapter 10: Introduction to Survey Methodology and Design

Prepared for the Interdisciplinary and Global Studies Division,
Worcester Polytechnic Institute
By James K. Doyle
Assistant Professor
Department of Social Sciences and Policy Studies

Conclusion

This document has provided an introduction to the basic principles of scientific survey design and outlined the steps that all beginning survey researchers should take, including:

  1. Determining if a survey study is the best way to answer your research questions.
  2. Obtaining a random or representative sample of sufficient size.
  3. Making an informed choice of survey method.
  4. Creating a questionnaire that is valid, reliable, and unbiased.
  5. Designing a questionnaire and implementation plan that achieve a high response rate.
  6. Developing procedures that ensure that people are treated ethically.

1. Conducting a scientifically defensible statistical analysis of the survey data.

Decades of research on survey design and methodology have demonstrated the importance of these basic steps. When they are followed, survey data can be a critically important source of information that supports decision making and policy formation. When they are ignored, survey data can be inaccurate and biased, and the decisions and policies based upon them ineffective or even harmful. Many students are surprised when they find out how complicated survey design and methodology is and how much time and effort it takes to produce a high-quality survey project. However, there is no substitute for the level of effort and attention to detail that is required to obtain high quality: social science research methods of all types are inherently complicated because people and societies are complicated. In the case of survey research, at least, the basic information is now available to undergraduate students in this document and other easily accessible and highly readable books, so high quality surveys should be achievable by the majority of IQP projects. Of course, in some cases quality must be sacrificed to some degree due to constraints on time and budget. This is understandable and such surveys are still worth doing: as long as the limitations of the project are understood and acknowledged, most surveys will yield useful information.

Finally, it should be emphasized that this document is intended to serve as an introduction to survey design and methodology and is therefore incomplete in several important ways. For example, only the simplest sampling techniques are described, many important issues related to question wording have been omitted, detailed examples of questionnaires, cover letters, and other documents have not been included since they are readily available from other sources, and data analysis is presented in a particularly abbreviated fashion. Given these limitations, it is imperative that students consult additional sources on survey design and methodology and data analysis when planning their projects. The recent books by Salant and Dillman (1994) and Rosenthal and Rosnow (1996) are probably the best places to start, but even these books are overly simplistic and incomplete. Most survey projects will also need to consult the wider literature on survey design and methodology and data analysis for more detailed information related to the specific concerns of their project.

Answers to True/False Questions

The answer to all of the questions is false:

  1. Once a population reaches a certain size, the size of the sample necessary to estimate opinions within a few percentage points is fairly constant. A carefully selected sample of about 1200 people is sufficient to determine the opinions of the entire U. S. population with a sampling error of plus-or-minus 3%.
  2. Since telephone directories don't include people who have unlisted numbers, people without phones, and people whose listings have been changed or added since the last publication, they cannot be relied upon to provide a random sample. Better choices for choosing a sample of telephone numbers are random digit dialing and add-a-digit dialing. See Chapter 5 of Salant and Dillman (1994).
  3. Actually, it is essential for survey questions to be grouped categorically to minimize the burden on the respondent and to demonstrate that careful thought went into the design of the questionnaire. In addition, in some cases an alternate ordering of identical questions can change responses by as much as 30%, so question ordering requires careful consideration.
  4. Although posting a survey on the web can give you access to a huge audience, this audience is not representative of the general population. Even if you wish to generalize only to Web users, this strategy does not allow you to determine the response rate to your survey, a step which is absolutely necessary to be able to interpret survey results. See Section IV.
  5. It is not acceptable practice to keep adding names to your sample until you get a sufficient number of respondents, since this results in a very low response rate. The correct approach is to adopt methods and procedures that will ensure a large percentage of your original sample will respond.

References Cited

Agresti, A. (1997). Statistical Methods for the Social Sciences, 3rd. ed. Upper Saddle River, NJ: Prentice-Hall.
American Psychological Association (1982). Ethical Principles in the Conduct of Research with Human Participants. Washington, D.C.: American Psychological Association.
Clayton, P., Applebee, A., and Pascoe, C. (1996). Email surveys: Old problems with a new delivery medium. LASIE: Information Bulletin of the Library Automated Systems Information Exchange, 27(2), 30-39.
Dillman, D. A. (1978). Mail and Telephone Surveys: The Total Design Method. New York: Wiley.
Emerson, R. B. (1983). Contemporary Field Research. Boston: Little, Brown.
Fowler, F. J., Jr. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA: Sage Publications.
Frankfort-Nachmias, C., and Nachmias, D. (1996). Research Methods in the Social Sciences, 5th ed. London: Edward Arnold.
Groves, R. M. (1989). Survey Errors and Survey Costs. New York: Wiley.
Henry, G. T. (1990). Practical Sampling. Newbury Park, CA: Sage.
Judd, C. M., Smith, E. R., and Kidder, L. H. (1991). Research Methods in Social Relations, 6th ed. Fort Worth, TX: Holt, Rinehart, and Winston.
Labaw, P. J., (1980). Advanced Questionnaire Design. Cambridge, MA: Abt Books.
Martin, P., and Bateson, P. (1986). Measuring Behavior: An Introductory Guide. Cambridge: Cambridge Univ. Press.
McCall, R. B. (1998). Fundamental Statistics for the Behavioral Sciences, 7th ed. Pacific Grove, CA: Brooks/Cole.
McGrath, R. E. (1997). Understanding Statistics: A Research Perspective. New York: Longman.
McKenna, R. J. (1995). The Undergraduate Researcher's Handbook: Creative Experimentation in Social Psychology. Boston: Allyn and Bacon.
Morgan, D. L. (1988). Focus Groups as Qualitative Research. Newbury Park, CA: Sage Publications.
Rosnow, R. L., and Rosenthal, R. (1996). Beginning Behavioral Research: A Conceptual Primer, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall.
Salant, P. A., and Dillman, D. A. (1994). How to Conduct Your Own Survey. New York: Wiley.
Shavelson, R. J. (1996). Statistical Reasoning for the Behavioral Sciences, 3rd ed. Boston, MA: Allyn and Bacon.
Survey Research Center (1983). General Interviewing Techniques: A Self-
Instructional Workbook for Telephone and Personal Interviewer Training. University of Michigan, Ann Arbor: Survey Research Center.
Weber, R. P. (1985). Basic Content Analysis. Beverly Hills, CA: Sage.
Yin, R. K. (1989). Case Study Research: Design and Methods. Newbury Park, CA: Sage Publications.

Maintained by webmaster@wpi.edu
Last modified: November 14, 2006 11:57:20