Loading presentation...

Present Remotely

Send the link below via email or IM


Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.


Designing Quality Surveys - ACUI

In this presentation, learn about principles of good survey design, common pitfalls of survey questions, and strategies for designing quality survey surveys so you can improve your surveys and obtain better assessment data.

Kaitlyn Moran Schmitt

on 20 February 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Designing Quality Surveys - ACUI

Designing Quality Surveys Kaitlyn Moran, North Carolina State University ACUI Region 5 Conference 2012 Purpose Collection Method So You're Going to Do a Survey... Other Methods: Portfolios Free Writes Validity Threats Getting Started Question Types Writing Good Questions How the Questions Shape the Answers Common Pitfalls Helpful
Strategies Additional Reading Advice Validity: The extent to which "one can draw meaningful and useful inferences from scores on the instruments." Inadequate procedure Unrepresentative sample Poor questions Schuh, M. L. & Upcraft, J. H. (1996). Assessment in Student Affairs: A Guide for Practitioners. San Francisco: Jossey-Bass.
Schwarz, N. (1999). Self-reports: How questions shape the answers. American Psychologist, 54, 93-105.
Sudman, S. & Bradburn, N. M. (1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.
Umbach, P. (2005). Getting back to the basics of survey research. New Directions for Institutional Research, 127, 91-100.
Upcraft, J. H. & Schuh, M. L. (2001) Assessment Practice in Student Affairs: An Applications Manual. San Francisco: Jossey-Bass. Observation Mining Existing Data Source: Crewswell, J. W. (2003). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, (2nd ed.) (p. 153-178). Thousand Oaks, CA: Sage. Format Open-Ended Multiple Choice Ranking Single Answer Content Knowledge Attitudes,
Opinion & Value Demographics Assumptions Maxim of Relation Maxim of Quantity Maxim of Manner Maxim of Quality Relevant to the conversation Clear, not ambiguous or wordy As informative as required, no more or less Nothing false or without evidence Wording Question Order Adjacent Questions (to Determine Context) "Educational contribution" viewed more positively when preceded by question about students receiving financial support than question about tuition Question Placement (to Determine Attitudes) Consistency: Responses may vary based on other questions in an effort for respondents to be consistent Limiting: Questions that restrict a concept may impact attitudes about the concept later in the survey Social Desirability Sensitive or Threatening Questions Clear and Concise Free from bias Predictions How likely are you to patron the food court in the student union? How many times this year did you eat at the food court in the student union? What genre of music would you like to see featured for the spring concert? Behavior vs. What was the genre of music of the last concert you attended? Specificity, Applicability, and Assumptions Assuming knowledge Assuming characteristics Assuming exclusivity To how many recognized student organizations do you belong? How many hours did you have completed when you transferred? Are you the first person in your family to go to college? Ranges and Scales Response Alternatives Inclusiveness Normalizing How much time do you spend studying on a typical weekday?
1-2 hours
3-6 hours
6-10 hours
11+ hours 2-3 hours How many hours do you spend watching TV each day?
Up to 0.5 hour
0.5 hour to 1 hour
1 hour to 1.5 hours
1.5 hours to 2 hours
2 hours to 2.5 hours
More than 2.5 hours How many hours do you spend watching TV each day?
Up to 2.5 hours
2.5 to 3 hours
3 hour to 3.5 hours
3.5 hours to 4 hours
4 hours to 4.5 hours
More than 4.5 hours Source: Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, Vol. 54, No. 2, p. 93-105. Study from Schwarz, N., Hippler, H. J., Deutsch, B. and Strack, F. (1985). Response Categories: Effects on Behavioral Reports and Comparative Judgments. Public Opinion Quarterly, Vol. 49, p. 391. Compare Scales... 16.2 % watched more than 2.5 hours daily 37.5% watched more than 2.5 hours daily Clarity How often do you visit the student union?
Very Often
All of the time How often do you visit the student union?
Less than Once a Month
Once a Month
2-3 Times a Month
Once a Week
2-3 Times a Week
Daily Compare Scales... Double-Barreled Questions Behavioral Questions Aided Recall Demographic Questions "Other" and "None of the Above" "Questions in which two opinions are joined together, so that respondents must answer two questions at once when their opinions about the two may diverge." Source: Sudman, S., & Bradburn, N. M. (1982). Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass. Example Do you support raising student fees and building a new student center? One-And-A-Half Barreled Questions Questions that start out asking about one opinion object, but introduce another opinion object mid-way through the question Example Which one of the following statements is closest to your opinion on increasing the student fee for the programming board?
I strongly support increasing the programming board's fee.
I support increasing the programming board's fee, but not for the full amount requested.
I would like to see the programming board start charging for tickets before increasing their fee.
I oppose the increase to the programming board's fee. Do I have to feel the same
way about both? That's a
different issue! Boring! Ask them last unless they are used as a filter Inclusiveness Remember the importance of Gender Race Age Class Year Religion Provides another option in case those you've listed do not apply Example Example Example What additional perks, if any, do your graduate assistant(s) receive?
Meal Plan
Health Insurance
Parking Permit
Professional Development Funds
Tuition Waiver
Free Tickets to Events
Other (please describe): ____________________ Focus Groups Behavior The best format and content for your question will depend on what you want to know Human memory is fallible, so
can help to gather more accurate data Memory cues included in the question Asking for behavior within a specific time period Providing examples in the question Asking about multiple specific behaviors to obtain information about a general behavior Example Methods How many times did you visit the student union in the last two weeks? How many student organizations do you belong to -- for example, clubs, fraternities, sororities, religious groups, political groups? How do you spend your free time? Please mark which activities you've done in the past month.
Attended a movie
Dined at a restaurant
Went to a theater or concert
Read for pleasure
Went hunting or fishing
Went to a college sports event
Participated in a club or student organization event
Played pool or darts
Went on a walk or hike
None of the Above Any question that respondents can think has a "right" or a "wrong" answer The "right" answer may be over-reported Examples of Topics: Voting
Use of Alcoholic Beverages
Sexual Activity
Drug Use
Library Card Ownership Examples Examples Advice: Ask more closed questions and limit the number of open questions Closed questions are more likely to be answered by participants and are easier to analyze Advice: Bury sensitive questions in the middle of the survey Use open questions and longer questions Examples Avoid mutually non-exclusive response categories in single-answer questions What is your major? Which describes your residence?
Commute from permanent home Do you work on campus?
What type of on-campus position do you hold?
Work study
Student leadership (e.g., RA, Orientation Leader)
Research Assistant Advice: Use behavioral questions to get more accurate responses Advice: Sexual Orientation What has prevented you from getting involved?
I live too far away
Family responsibilities
I don't feel welcome
Lack of interest
Conflicts with my schedule What student elections have you voted in?
Student Government Association
Student organization officer election (Not SGA)
Other Avoid asking for predictions Past behavior is a stronger indicator of future behavior (i.e., lie) Tendency toward the middle Tendency toward agreement Tendency toward social norms Used to help understand what the researcher is looking for Example: "really irritated" "clarify the intended meaning" Source: Schwaz, N. (1999) Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93-105. Frequency scales
Reference periods
Rating scales Example: not at all successful to
extremely successful In the past month? In the past year? 0 to 10
-5 to 5 Asking convoluted questions can confuse your participants Example: "Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?" Source: Radwin, D. (2009, Oct. 5). High response rates don't ensure survey accuracy. The Chronicle of Higher Education. Including the Middle Advice: Use the literature to determine the average Phrase questions with both the positive and negative statements Be wary of asking threatening questions In a 1944 study, Americans enlisting in the German army before 1941 was viewed more favorably when it was preceded by a question about enlisting in the British army than when it was followed by the same question Reuse good-quality questions Remember the literature Evaluate your survey Stay Sharp: Test Your Skills Keep it brief Conduct Focus Groups Consult Experts Pilot Your Survey E.g., "Neither Agree Nor Disagree" Secret: Doesn't really matter! Doesn't change the results significantly of Survey Respondents The first questions should be interesting so respondents are inspired to complete the survey in response choices
Full transcript