Loading presentation...

Present Remotely

Send the link below via email or IM


Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.


Questionnaire Design

No description

Nina Sabarre

on 12 October 2015

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Questionnaire Design

Questionnaire Design
in the International Context

Learning Objectives
After this session, you will be able to...

Describe attitude questions, knowledge questions, behavior questions, and sensitive questions

Use strategies to write these types of questions with cultural considerations in mind
to construct questions that allow for the collection of reliable and valid data that is representative of the population of interest.
Attitude Questions
The International Context
It's complicated!

You now have to anticipate how different cultural frameworks will impact the respondent's understanding of a question.
Did you vote in the last election?
What was your biggest motivation for voting in the last election?
Answers to attitude questions are fluid and can change often.

Beliefs and perceptions are complex ideas.
Q2a. Would you say security in your local area is good, fair or poor? Is that ‘very good/poor'?

Q2b. Would you say security on local roads is good, fair or poor? Is that 'very good/poor'?
Question order can have profound effect on responses to your attitude question.
Election Survey A and Election Survey B were used to project the number of likely voters in the Afghanistan presidential election (held April 2014).

But the percentage of respondents who were likely to vote was off by 15%.... WHY?
Knowledge Questions
Behavior Questions
Intent: to accurately record past, present or future behaviors
Sensitive Questions
Any survey question dealing with a sensitive or potentially threatening topic, where special measures should be taken to increase respondent ease and encourage truthful responses.

Understand & clearly define the attitude object.

Revisit your research question.
Avoid double barreled questions.
Example of a double barreled question:
The likelihood of voting question in Election Survey A followed questions about past voting behavior, election media consumption, and expectations for candidates.

The likelihood of voting question in Election Survey B followed a series of questions about security and safety.
Example of Question Order
Example of fluid attitudes
Example of breaking down attitude objects
Research question: How do Egyptians perceive legitimacy of the newly formed government?
Helpful Hints for Question Order
Q-11. Do you think the Iraqi Government provides services and aid to all Iraqis equally or do they favor one group or actor over the others?

1. Yes, provides services and aid to all Iraqis equally
2. No, favors one group over others
98. Refused (vol.)
99. Don’t Know (vol.)

Q-11a. Do you think the Iraqi Government provides services and aid to all Iraqis equally?

1. Yes
2. No
98. Refused (vol.)
99. Don’t Know (vol.)

Q-11b. Do you think the Iraqi Government favors one group or actor over the others?

1. Yes
2. No
98. Refused (vol.)
99. Don’t Know (vol.)

Revised Question Wording:
Drill Down Battery Questions
Q30. Do you believe women in your country have fewer rights than men, the same or equal rights, or more rights than men when it comes to:

Consider how other questions frame respondents' thinking
Open-ended questions before closed-ended questions
Start with warm-up questions
Consider respondent fatigue

Stacey Frank
Nina Sabarre
Measure someone's views or feelings on a topic
Cannot be fact checked
You must understand the behavioral norms in the country where you’re working
Example of the Impact of Behavioral Norms
a. Exercise or play sports
b. Walk or ride a bicycle
c. Do strenuous household chores like cleaning or gardening

Recall is related primarily to time elapsed and saliency
Time Elapsed:

It's easier to remember something that happened last week than something that happened last year
Negative Interplay of Timeframe and Salience
Long time elapse, low salience:
How much did you, yourself spend in total on potato chips in January 2011?

Short time elapse, high salience:
Did you, yourself get married last week?

Short time elapse, low salience:
How much money, if any, did you spend at restaurants yesterday?

A Positive Interplay of Timeframe and Salience
Be aware of how different perceptions of time will impact your survey questions
What Year is it in Iran?
2014: Gregorian calendar year
1393: Persian calendar year
1435: Islamic calendar year
Tools to Mitigate Telescoping:
Landmark Events
People remember events more easily than arbitrary time frames.

Tools to Mitigate Telescoping:
Aided Recall
Aided recall provides the respondent with memory cues or external sources of information to reduce recall errors.
Aided Recall: External Data
Available Records
Tax survey
Education survey

Media survey
Health survey
Example of the Use of Landmark Events
Q57. Were you personally or someone in your household injured prior to 2001?
1. Yes
2. No

Q58. Were you personally or someone in your household injured after 2001?
1. Yes
2. No

Understand the true ramifications of asking a sensitive question.
Example of Sensitive Question: Iraq

Q6.    Who provides security in your neighborhood? (Open-ended with pre-codes, DO NOT READ LIST)

a. ___________________

IMPORTANT: If respondent says “The Islamic State of Iraq and Al-Sham, ISIS, Islamic State and Da’esh” Skip to Demographics and END interview 

There are several possible approaches for asking sensitive questions.
Understand the potential impact of social desirability bias.
The tendency to answer questions in a way that will be viewed favorably by others.

over-reporting 'good' behavior
under-reporting 'bad' behavior

Tips: Socially Undesirable Questions
Deliberately load the question:
“Everybody does it…”
Assume the behavior, ask about frequency

Ask “have you ever” before asking “do you currently”

Tips: Socially Desirable Questions
Use the casual approach “Do you happen to…”
Ask “do you currently” rather than “do you usually”

The Casual Approach
Q-36. Thinking about voting, if you happen to know, can you tell me the location where people living in this voting precinct will go to cast their ballot on election day, or is this something you don’t know?
1. Yes, knows the location
2. No, does not know location

The Everybody Approach
The Some/Other Approach:
Q40. Some people think that sending girls to school is doing what the Qur’an and Hadith teach. Other people think that sending girls to school goes against the teachings of the Qur’an and Hadith. Which is closer to your view?

Ask Simple Questions!
If it is difficult to capture the whole picture in one question, you may need a series of drill down questions.
Ask Simple Questions!
Measure effectiveness of an educational process

Measure knowledge in order to effectively design and implement information programs or advertising campaigns

Measure intelligence, which may be required to help explain behavior and attitudinal variables
Before asking attitude questions about issues or people, ask knowledge questions to screen out respondents who lack sufficient information
Know the education
level of your sample.
Consider the use of pictures and other nonverbal procedures for determining knowledge
Knowledge Questions as screening variables
Q-27. Biometric fingerprint scanners are scheduled to be used in the 2015 presidential election in Yemen. Are you aware of Yemen’s planned new biometric voting registry?

1. Yes
2. No
98. Refused (vol.)
99. Don’t Know (vol.)
Q-28. (If ‘1’ in Q27) Do you support or oppose the creation of the new voting system, or do you neither support nor oppose it?

1. Support it
2. Neither support nor oppose it
3. Oppose it
96. Not Asked
98. Refused (vol.)
99. Don’t Know (vol.)
Are you aware of Yemen's planned new biometric voting registry?
I'm now going to read a list of Yemeni political leaders. Do you have a very favorable, somewhat favorable, somewhat unfavorable, or very unfavorable opinion of the following leaders,
or do you not know much about them

President Abd Rabuh Mansur Hadi
Sanaa Mayor Abd al-Qadir Hilal
Major General Ali Muhsin al-Ahmar
Huthi movement leader Abd al-Malik al-Huthi
Former President Ali Abdullah Salih
Businessman Hamid al-Ahmar
Former Republican Guard Commander Ahmad Ali Salih
Defense Minister Mohammed Naser Ahmed

6% DK
59% DK
27% DK
18% DK

5% DK
25% DK
27% DK
53% DK

Write knowledge questions with the research question in mind
Please look at this list of candidates with their photos and party symbols. After I read out the list of 11 candidates for president, please tell me if the election were today, for whom would you vote?
Purpose of Knowledge Question
On what date will the 2014 elections be held? (OPEN-ENDED)
1. On April 5
2. Other date (within one month of April 5th, between March and April)
3. Other date (any other date outside of March and April)
4. Don't know exact date
98. Refused (vol.)
99. Don’t Know (vol.)
Understand external conditions that influence attitudes
What is Social Desirability Bias?
Asking Questions: The Definitive Guide to Questionnaire Design - For Market Research, Political Polls, and Social and Health Questionnaires.
Bradburn, Norman N., Seymour Sudman, and Brian Wansick.

John Wiley and Sons Inc., 2004.

Cognitive Interviewing: A Tool for Improving Questionnaire Design.
Willis, Gordon B. Sage Publications Inc., 2005.

Cross-Cultural Survey Methods.
Janet A. Harkness, et. al. John Wiley and Sons Inc., 2003.

Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
, Third Edition. Don A. Dillman, et. al. John Wiley and Sons Inc., 2008.

"Telescoping of Landmark Events: Implications for Survey Research." Gaskell, George D., Daniel B. Wright, and Colm A. O'Muircheartaigh. Public Opinion Quarterly Volume 64: 77-89.

Q36a. Do you have a very favorable, somewhat favorable, somewhat unfavorable, or very unfavorable view of the government?

Q36b. How do you feel about the government's ability to govern fairly – very confident, somewhat confident, not so confident or not confident at all?

Q36c. How do you feel about most decisions made by the government – do you support them strongly, support them somewhat, oppose them somewhat or oppose them strongly?
Q15. In a typical week, please tell me if you engage in each of the following activities AT LEAST 2 DAYS WEEK for at least 10 minutes each day?
The effect of a respondent remembering that an event occurred but forgetting the exact date

Long time elapse, high salience:
Where were you when the terrorist attacks in New York and Washington DC on September 11, 2001 happened?
1. Sending girls to school is doing what the Quran and Hadith teach
2. Sending girls to formal school goes against the teachings of the Qur’an and Hadith.
Need to be carefully written to encourage truthful responses and avoid unintended distortions

the more salient the event, the easier it will be for a respondent to recall details related to that event

Economic or social costs and benefits
Continuing consequences
Remember that all of these factors will be culture-specific!
Use when concerned with under-reporting, not over-reporting.
Q6. As you know, many people in this area grow poppy along with other crops like wheat. Do you yourself happen to grow poppy?

1. Yes, grow poppy.
2. No, do not grow poppy.
Don't use biasing language
Ask about one concept at a time
Construct exhaustive and mutually exclusive response options
Specify the response task
Check your assumptions
Know the characteristics of your respondent population
Know the strengths and weaknesses of the different question types
Test your questionnaires!
Cognitive interviews can help uncover hidden misinterpretations and misunderstandings
'Sensitive' subjects will differ from culture to culture
Example of 'Sensitive' Question: Haiti

Q12. Have you ever given birth to a boy or girl who was born alive but later died?

PROBE: Any baby who cried or showed signs of life but did not survive?

Factors that Impact Sensitivity
Legal restrictions on free speech
Religious or ethnic taboos
Different concepts of public and private information
In all cases...
avoid reliability checks within the same questionnaire!
Aided Recall: Descriptive Paragraph
Q53. On average, how long does it take for one round trip to obtain water from this source?
This includes travel time to the source, wait time at the source, and return time. For bottled water, this refers to the amount of time it takes to travel to the shop, buy the water, and return.
Full transcript