Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Understanding Assessment Principles through Survey Design - NASPA

In this presentation, learn about basic assessment concepts, principles of good survey design, common pitfalls of survey questions, and strategies for designing quality survey so you can reduce measurement error and obtain better assessment data.
by

Kaitlyn Moran Schmitt

on 25 July 2016

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Understanding Assessment Principles through Survey Design - NASPA

Understanding Assessment Principles
Through Survey Design

Kaitlyn Moran, College of William & Mary
NASPA Investing in Our Future Webinar
Purpose
Collection Method
So You're Going to Do a Survey...
Other Methods:
Portfolios
Free Writes
Getting Started
Question Types
Writing Good Questions
How the Questions Shape the Answers
Common Pitfalls
Helpful
Strategies

Additional Reading
Advice
Groves, R. M. et al. (2004). Survey Methodology. Hoboken: Wiley-Interscience.
Schuh, M. L. & Upcraft, J. H. (1996). Assessment in Student Affairs: A Guide for Practitioners. San Francisco: Jossey-Bass.
Schwarz, N. (1999). Self-reports: How questions shape the answers. American Psychologist, 54, 93-105.
Sudman, S. & Bradburn, N. M. (1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.
Umbach, P. (2005). Getting back to the basics of survey research. New Directions for Institutional Research, 127, 91-100.
Upcraft, J. H. & Schuh, M. L. (2001) Assessment Practice in Student Affairs: An Applications Manual. San Francisco: Jossey-Bass.
Observation
Mining Existing Data
Format
Open-Ended
Multiple Choice
Ranking
Single Answer
Content
Knowledge
Attitudes,
Opinion & Value
Demographics
Assumptions
Maxim of Relation
Maxim of Quantity
Maxim of Manner
Maxim of Quality
Relevant to the conversation
Clear, not ambiguous or wordy
As informative as required, no more or less
Nothing false or without evidence
Wording
Question Order
Adjacent Questions (to Determine Context)
"Educational contribution" viewed more positively when preceded by question about students receiving financial support than question about tuition
Question Placement (to Determine Attitudes)
Consistency: Responses may vary based on other questions in an effort for respondents to be consistent
Limiting: Questions that restrict a concept may impact attitudes about the concept later in the survey
Social Desirability
Sensitive or Threatening Questions
Clear and Concise
Free from bias
Predictions
How likely are you to patron the food court in the student union?
How many times this year did you eat at the food court in the student union?
What genre of music would you like to see featured for the spring concert?
Behavior
vs.
What was the genre of music of the last concert you attended?
Specificity, Applicability, and Assumptions
Assuming knowledge
Assuming characteristics
Assuming exclusivity
To how many recognized student organizations do you belong?
How many hours did you have completed when you transferred?
Are you the first person in your family to go to college?
Ranges and Scales
Response Alternatives
Inclusiveness
Normalizing
How much time do you spend studying on a typical weekday?
None
1-2 hours
3-6 hours
6-10 hours
11+ hours
2-3 hours
How many hours do you spend watching TV each day?
Up to 0.5 hour
0.5 hour to 1 hour
1 hour to 1.5 hours
1.5 hours to 2 hours
2 hours to 2.5 hours
More than 2.5 hours
How many hours do you spend watching TV each day?
Up to 2.5 hours
2.5 to 3 hours
3 hour to 3.5 hours
3.5 hours to 4 hours
4 hours to 4.5 hours
More than 4.5 hours
Source: Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, Vol. 54, No. 2, p. 93-105. Study from Schwarz, N., Hippler, H. J., Deutsch, B. and Strack, F. (1985). Response Categories: Effects on Behavioral Reports and Comparative Judgments. Public Opinion Quarterly, Vol. 49, p. 391.
Compare Scales...
16.2 % watched more than 2.5 hours daily
37.5% watched more than 2.5 hours daily
Clarity
How often do you visit the student union?
Never
Rarely
Sometimes
Often
Very Often
All of the time
How often do you visit the student union?
Less than Once a Month
Once a Month
2-3 Times a Month
Once a Week
2-3 Times a Week
Daily
Compare Scales...
Double-Barreled Questions
Behavioral Questions
Aided Recall
Demographic Questions
"Other" and "None of the Above"
"Questions in which two opinions are joined together, so that respondents must answer two questions at once when their opinions about the two may diverge."
Source: Sudman, S., & Bradburn, N. M. (1982). Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass.
Example
Do you support raising student fees and building a new student center?
One-And-A-Half Barreled Questions
Questions that start out asking about one opinion object, but introduce another opinion object mid-way through the question
Example
Which one of the following statements is closest to your opinion on the usefulness of the Undergraduate Tutorial Center?
I believe the UTC is very useful to all students.
I believe the UTC is useful to some students and not to others.
I believe the UTC is useful, but I think the UTC should be funded by user fees and not student fees.
I believe the UTC is not useful at all.
Do I have to feel the same
way about both?
That's a
different issue!
Boring!
Ask them last
unless they are used as a filter
Inclusiveness
Remember the importance of
Gender
Race
Age
Class Year
Religion
Provides another option in case those you've listed do not apply
Example
Example
Example
What additional perks, if any, do your graduate assistant(s) receive?
Meal Plan
Health Insurance
Housing
Parking Permit
Professional Development Funds
Tuition Waiver
Free Tickets to Events
Other (please describe): ____________________
Focus Groups
Behavior
The best format and content for your question will depend on what you want to know
Human memory is fallible, so
can help to gather more accurate data
Memory cues included in the question
Asking for behavior within a specific time period
Providing examples in the question
Asking about multiple specific behaviors to obtain information about a general behavior
Example Methods
How many times did you visit the student union in the last two weeks?
How many student organizations do you belong to -- for example, clubs, fraternities, sororities, religious groups, political groups?
How do you spend your free time? Please mark which activities you've done in the past month.
Attended a movie
Dined at a restaurant
Went to a theater or concert
Read for pleasure
Went hunting or fishing
Went to a college sports event
Participated in a club or student organization event
Played pool or darts
Went on a walk or hike
None of the Above
Any question that respondents can think has a "right" or a "wrong" answer
The "right" answer may be over-reported
Examples of Topics:
Voting
Exercising
Use of Alcoholic Beverages
Sexual Activity
Drug Use
Library Card Ownership
Examples
Examples
Advice:
Ask more closed questions and limit the number of open questions
Closed questions are more likely to be answered by participants and are easier to analyze
Advice:
Bury sensitive questions in the middle of the survey
Use open questions and longer questions
Examples
Avoid mutually non-exclusive response categories in single-answer questions
What is your major?
Which describes your residence?
On-campus
Off-campus
Commute from permanent home
Do you work on campus?
What type of on-campus position do you hold?
Work study
Student leadership (e.g., RA, Orientation Leader)
Research Assistant
Advice:
Use behavioral questions to get more accurate responses
Advice:
Sexual Orientation
What has prevented you from getting involved?
I live too far away
Family responsibilities
Cost
I don't feel welcome
Lack of interest
Conflicts with my schedule
What student elections have you voted in?
None
Student Government Association
Student organization officer election (Not SGA)
RHA
Other
Avoid asking for predictions
Past behavior is a stronger indicator of future behavior
(i.e., lie)
Tendency toward the middle
Tendency toward agreement
Tendency toward social norms
Used to help understand what the researcher is looking for
Example:
"really irritated"
"clarify the intended meaning"
Source: Schwaz, N. (1999) Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93-105.
Frequency scales
Reference periods
Rating scales
Example:
not at all successful to
extremely successful
In the past month? In the past year?
0 to 10
vs.
-5 to 5
Asking convoluted questions can confuse your participants
Example:
"Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?"
Source: Radwin, D. (2009, Oct. 5). High response rates don't ensure survey accuracy. The Chronicle of Higher Education.
Including the Middle
Advice:
Use the literature to determine the average
Phrase questions with both the positive and negative statements
Be wary of asking threatening questions
In a 1944 study, Americans enlisting in the German army before 1941 was viewed more favorably when it was preceded by a question about enlisting in the British army than when it was followed by the same question
Reuse good-quality questions
Remember the literature
Evaluate your survey
Stay Sharp: Test Your Skills
Keep it brief
Conduct Focus Groups
Consult Experts
Pilot Your Survey
E.g., "Neither Agree Nor Disagree"
Secret: Doesn't really matter!
Doesn't change the results significantly
of Survey Respondents
The first questions should be interesting so respondents are inspired to complete the survey
in response choices
When to Do a Survey
vs.
quantitative
To get
responses
To answer
"What?"
or
"How many?"
To understand
opinions
To obtain responses from a
number of people
surface-level
large
qualitative
To get
responses
To answer
"Why?"
or
"How?"
To understand
opinions
To obtain responses from a
number of people
in-depth
small
When You Want:
Not:
Inadequate procedure
Unrepresentative sample
Sampling error
Coverage error
Poor response
Measurement error
Nonresponse error
Validity
The extent to which "one can draw meaningful and useful inferences from scores on the instruments."
Poor questions
Source: Crewswell, J. W. (2003). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, (2nd ed.) (p. 153-178). Thousand Oaks, CA: Sage.
Threats to Validity:
#SAInvest
Sampling
Definitions
Population
Sample
The group of people about which you want to know something
E.g., all students at my college/university
The group of people from your population whom you choose to assess
E.g., a group of 50 students from my college/university
Techniques
Probability

Non-Probability
Simple random
Stratified
Cluster/Multi-stage
Purposive
Convenience
Sampling Error
Assessing a part of your population whose characteristics differ significantly from the characteristics of the population as a whole
E.g., a sample of graduating students for a population of all students
Measurement Error
The result of inaccurately measuring what you're trying to measure
Bad Questions In = Bad Responses Out
Talk to your Assessment Office or Director
Length
How important is it for you to have athletic teams on your campus?
Extremely Important
Very Important
Important
Somewhat important
A little important
Neither important nor unimportant
A little unimportant
Somewhat unimportant
Unimportant
Very unimportant
Not at all important
Example
Demographic questions can decrease your response rates if respondents think you might be able to identify them
Necessary
Only ask them if they are
And you plan to use the responses
Writing Good
Surveys

Provide a description of your
Keep it
Start with - but - questions, then ask more difficult questions
Ask questions in an that makes (e.g., group similar questions together)
Use
Be careful with how you use
Advice for Good Surveys
Distribution
Nonresponse Error
The result of the people who respond to the assessment being significantly different from the people who do not respond
Ways to Increase Response Rate
Utilize principles of good survey design!
Survey the right people
Contact your participants multiple times
Be considerate
Explain your purpose
Offer incentives
Thank your participants
Kaitlyn Moran
Coordinator, Programming
College of William & Mary
kaitlyn@wm.edu
Past Surveys:
Survey of NC State students to determine their attitudes about the Campus Cinema, especially usage and support of an increase of student fees)
Survey of W&M students to determine their satisfaction with and interest in events sponsored by the campus programming board
Additional Assessments:
Benchmarking assessment of late-night programming (Boston College)
Assessment of student learning of time management skills through participation in the campus programming board (NC State)
In process: Needs assessment of resources for student organization leaders (William & Mary)
Student Leadership Development
Distribution
purpose
short
easy
order
sense
skip logic!
forced responses
interesting!
E.g., only students involved in Greek Life respond
Advice
Incentives are more effective when everyone is given a small token for participation than when only a few winners receive a large prize
Full transcript