close

Вход

Забыли?

вход по аккаунту

код для вставкиСкачать
Applied Opinion Research Training
Workshop
Day 2
Designing Quantitative
Instruments
Mary McIntosh
Quantitative Instruments
Designing a Questionnaire – Keep in Mind:

Don’t forget the big picture
Follow your objectives and hypotheses to determine the content of
your questions




The quality of your research depends on the quality of your
questionnaire
Ask others to review and proofread your survey
Always pretest, do a trial run
Utilize other sources to assist you in designing your
questionnaire
Previous studies, expert input, etc.

Make instructions as clear as possible
Quantitative Instruments
Designing a Questionnaire – Step by Step:






Screening respondents
Wording questions
Response options
Order of questions
Length of questionnaire
How the questionnaire will be administered
Quantitative Instruments
Screening Respondents: Sample Parameters
 First,
you want to decide if you will need to specify
certain criteria that your respondents need to satisfy
• Age (e.g., over 18, under 65)
• Profession (e.g., government official, private sector, etc.)
• Status in household (e.g., mother)
 These
questions should come first
Quantitative Instruments
Wording Questions
 Second,
start writing your questions, keeping in
mind:



Questions should be kept short
Use wording appropriate to your respondents’ level of
education
Make all definitions, qualifiers, and assumptions explicit
(e.g., Do you think the Bank’s program was effective?
Which program?)
Quantitative Instruments
Wording Questions





Be careful when using abbreviations, acronyms, and
jargon
Be careful with word choice
Highlight words that require extra emphasis
Make sure your questions are technically accurate
Make sure questions are relevant and applicable to
your respondents
Quantitative Instruments
Wording Questions
 The time
span for recall in a question should reflect
the saliency of the topic
 Also be
careful about asking questions about future
intentions; keep to the short-term future
 Time spans, past or future, should be
possible (e.g., don’t use “lately”)
as specific as
Quantitative Instruments
Wording Questions

Do not use leading questions that suggest a particular answer
or the researcher’s viewpoint


Do not use loaded questions that bias people towards
particular answers


(e.g., Why does working in the public sector make people lazy
and careless?)
(e.g., Do you think it is important to have a strong police force
in this time of crisis?)
Be wary of double-barreled questions. Rephrase them into
two questions whenever possible

(e.g., Should the government reduce its financial help and its
technical assistance to other countries?)
Quantitative Instruments
Wording Questions



Avoid double negatives (e.g., not prohibit), they can be
confusing. If absolutely necessary, emphasize the “not”
by underlining it
Make sure your questions are valid. The question must
ask for information which the respondent is capable of
providing an informed response
Make the question as specific as possible
(e.g., instead of asking “Is privatization a good idea?” ask “Is
privatization of utilities a good idea?”)
Quantitative Instruments
Wording Questions

Explicitly state alternatives


Use face-saving phrasing



(e.g., instead of asking “To balance the budget, would you be in
favor of a higher income tax?” ask “To balance the budget,
would you be in favor of a higher income tax, a higher sales tax,
or reduced government services?”)
(e.g., instead of “Did you vote?” ask “Were you able to vote?”)
Be sure the wording of your question matches your
response options
Consider “Don’t Know” options
Quantitative Instruments
Response Options
Common response options are:
– Likert/Rating scale
– Dichotomous choice/Trade-off
– Multiple choice
– Open-ended
– Ranking
Quantitative Instruments
Likert/Rating Scale
Shouldn’t use more than 2-3 scales in each survey effort
Numbered Scales
 Asks for responses on a numbered scale with descriptions for what
the numbers represent (even and odd numbered scales)

 e.g., Please indicate how favorable your impression of the World Bank
is on a scale of 1-10, 1 being very unfavorable, 10 being very favorable



Numbered scales work well in written surveys
Numbered scales work well with educated population
Numbered scales must be appropriate for culture (e.g., 10 point
scale vs. 7 or 8 point scale)
Quantitative Instruments
Likert/Rating Scale
Word Scales

Word scales are similar but meet different respondent needs.
 e.g., Please indicate how much you agree with this statement, using a
scale of “strongly disagree”, “somewhat disagree”, “neither agree nor
disagree”, “somewhat agree”, “strongly agree”
 What is your opinion of the Bank’s work in the area of infrastructure
development in your country. Is it “very good,” “somewhat good,”
“somewhat bad,” or “very bad.”



Word scales are usually 3, 4, or 5 point scales
Very common, easily used in written form
or verbally
Word scales sometimes preferred because the opinion is clearly
defined for respondent
Quantitative Instruments
Likert/Rating Scale
Provides a great deal of information, quickly and
easily
 Can be used to compute summary statistics (e.g.,
frequencies, means) and significance tests
 Best used to measure opinions, attitudes, extent of
agreement/intention, etc. Not as good for measuring
facts
 Consider a “Don’t Know” or “Refuse” response

Quantitative Instruments
Likert/Rating Scale




Common formats are 4-points, 5-points, 7-points, and 10points (will vary based on how much knowledge respondents
have; how nuanced their opinions may be. When in doubt,
keep it simple (4 point scale.)
The greater the number of points, the greater variation you
will be able to measure in respondents’ responses
An odd number of points allows respondents to give a middle
or “neutral” response. Even numbered scale forces
respondents to commit.
Odd point middle point can have a number of meanings:
don’t know; neutral; neither positive nor negative
Quantitative Instruments
Likert/Rating Scale
Respondents need to be comfortable with the scales you
utilize
 If possible, use a balanced scale

• e.g., 1-not at all effective, 5-very effective
• If it is not possible to use a balanced scale, make sure your endpoints
are clear antonyms

In your number descriptions, 1 should describe the smallest,
most negative response and 4/5/7/10 should describe the
largest, most positive response
Quantitative Instruments
Dichotomous Choice/Trade-Off

Asks for an either/or response
•
•

e.g., Were you able to vote in the last election? Yes/No
e.g., Which of the following two statements best represents your
views on taxation in our country?
1) I believe our tax dollars are not used wisely by our
government.
2) I believe our tax dollars are used appropriately by our
government.
Flexible and easy to administer in written or verbal format
Quantitative Instruments
Dichotomous Choice/Trade-Off
Provides limited information - Does not tell you the
extent of opinion
 Yields dichotomous data
 Can be used to compare response percentages and
to examine how different groups responded
 Best used to measure facts. Not good for
measuring opinions, attitudes

Quantitative Instruments
Multiple Choice
 Gives respondent a variety of responses to choose from
e.g., Which organization do you trust the most? (Oxfam,
Save The Children, Catholic Charities)
e.g., What do you consider the most critical development
challenge facing your country? (Poverty, Education, Corruption,
etc.)
e.g., Which of the following describes your position?
(Government official, Private sector, NGO, etc.)
Quantitative Instruments
Multiple Choice
 Very
common, flexible and easily used in written
form; verbal ease depends on the number of choices
 Provides a great deal of information, quickly and
easily
 Yields categorical or ordered data
 Can be used to compare response percentages and
to examine how different groups responded
Quantitative Instruments
Multiple Choice

Under some circumstances, you may want the respondent to
be able to choose more than one answer
• e.g., In your opinion, which are the most important Bank programs in
your country? (choose up to 3)

Response options must be mutually exclusive
• e.g., How much of your country’s GDP do you think is spent on
foreign assistance:
< 1%
1-4%
5-8%
9+%
Quantitative Instruments
Multiple Choice

Response options need to be exhaustive, but not
overwhelming
• Include an “Other” choice in which respondents can
write in a response that you did not include

When applicable, order choices from smallest to
largest, most negative to positive
• “Other” and “Don’t Know” options should be last
Quantitative Instruments
Open-Ended
 Allows
respondent to provide any response that they want
• e.g., Why do you think economic reforms have not been implemented
by the government?
• e.g., How would you improve the effectiveness of Bank programs?
 Common, however,
must be transcribed and, if not in English,
translated
 Limited to skill of interviewer and knowledge of respondent
Quantitative Instruments
Open-Ended
 Provides a great deal of information
 Yields
data unique to each respondent;
once it is coded, response percentages
can be computed and compared
• Risk: coding is subjective, time consuming,
and expensive
 Best used for exploratory
research
• Use when many answers are possible and multiple-choice options
are too extensive or unknown
Quantitative Instruments
Ranking
 Asks respondent to rank a variety of responses
according to some criterion
• e.g., Please rank the following in order of priority from 110, 1 being top priority: Education, Communicable
Disease, Weak Institutions, The Economy, Corruption,
etc.
 Less common, best used in written form,
difficult to administer verbally
Quantitative Instruments
Ranking
Can be difficult for respondents to complete
 Provides limited information; assumes that
respondents feel differently about each item
 Data is difficult to analyze and interpret
 Best used to measure limited attitudes and opinions

Quantitative Instruments
Order of Questions
If necessary, screening questions should be first to
eliminate ineligible respondents early
 Then, using the funnel approach, proceed from
broad, general questions to more specific questions
 Opening questions should be easy and
non-threatening to keep respondents’ interest
 If possible, make the opening questions interesting
to peak respondents’ interest

Quantitative Instruments
Order of Questions





Consider the advantages/disadvantages of starting a
questionnaire with an open-ended question
Avoid questions that get monotonous
Consider the advantages/disadvantages of completing all
questions on an issue before moving to a new issue
Order issues in a logical fashion to facilitate the flow of the
questionnaire
Obtain historical information in chronological order, either
forward or backward
Quantitative Instruments
Order of Questions
Use transition statements to facilitate shifts from
one issue to another or one series of questions to
another
 Eliminate order bias due to the sequencing of
questions as much as possible

• e.g., do not ask about economic growth as a major
development challenge before you ask respondents
what they consider to be the major development
challenges for their country
Quantitative Instruments
Order of Questions
 Do not ask sensitive
or difficult questions until a rapport has
been established with the respondent
 Any demographic/classification questions should be asked at
the end of the questionnaire, unless required for screening
or cultural tradition
• Demographic/classification questions should be ordered from the
least to the most sensitive items
 Avoid complex
or frustrating skip patterns
(if self-administered or face-to-face)
Quantitative Instruments
Length of Questionnaire

Don’t burden respondents with a long questionnaire
• The shorter the questionnaire, the higher the quality of the data

Only ask questions that meet your specific research
objectives
• For each question, ask is this question necessary?

Pre-test your questionnaire, even if it is just among your
colleagues, to determine how long it will take respondents to
complete it
• Language expansion factor
• Cultural traditions of long introduction
Quantitative Instruments
Administering the Questionnaire
 Confidentiality
must be ensured
 Consider not using a political polling firm
 Keep in mind how you will be administering the
questionnaire (e.g., mail-in, phone, in person, within
groups, electronic) when deciding on the wording
and length of the questionnaire
Quantitative Instruments
Administering the Questionnaire
 Mail-in Questionnaires:
• Best for longer questionnaires that may include sensitive issues for
which respondents will want to maintain confidentiality
• Should include postage-paid return envelopes to encourage
participation
 Phone
Surveys:
• Best for moderate length questionnaires, around twenty minutes
• Should be done at appropriate times with interviewers fluent in the
respondents’ language
Quantitative Instruments
Administering the Questionnaire
 In-person Surveys:



Best for longer questionnaires
Should be done at appropriate times with local
interviewers who are fluent in the respondents’ language
Skill of the interviewer will influence the quality of the data
Hands-on-Work:
Designing a Quantitative Questionnaire
Designing Qualitative Instruments
Sharon Felzer
Qualitative Instruments
Designing a Guideline – Keep in Mind:
 This is a questionnaire
for discussion
 Don’t forget the big picture
• Follow your objectives and hypotheses to determine the content of your
questions
 The quality
of your research depends on the quality of your
guideline and how well the guideline is administered
 Ask others to review your guideline
 Utilize other sources to assist you in designing your guideline
• Previous studies, expert input, etc.
Qualitative Instruments
Designing a Guideline – Step by Step:
 Introduction
 Wording questions
 Order of questions
 Length of guideline
 Administration
of guideline
Qualitative Instruments
Introduction
 The point of the introduction period is to make the
respondent(s) feel more comfortable in the situation



The researcher should introduce themselves and the
research
Discuss anonymity and confidentiality
Introduce respondent(s)
Qualitative Instruments
Wording Questions
 Writing questions for a guideline
is much like writing
a questionnaire with all open-ended questions;
therefore, many of the recommendations are the
same, keeping in mind:
• You want a logical, systematic conversation
• Replicate as much as possible by other interviewers
Qualitative Instruments
Wording Questions
Most importantly:
• Your questions must encourage discussion:
– Why?
– How?
– What do you think about…?
– In what ways…?
– What do you know about…?
» e.g., What is your perspective on privatizing the
airlines in our country?
Qualitative Instruments
Wording Questions
Provide probes for the researcher to use if the initial
question does not elicit sufficient discussion. For instance:
– Question: What do you think are the major
development challenges facing your country?
– Probe: Why are these more pressing than other
challenges?
– Probe: Describe how the government is meeting
these challenges (probe: resources, partnerships,
etc.?)
Qualitative Instruments
Wording Questions




Occasionally, a close-ended question may be
necessary, just be sure to include follow-ups
 e.g., Do you believe families should have to pay for
their children’s education? Why/why not?
Although you want to encourage discussion, make sure
your questions are sufficiently focused that respondents
do not go too far astray
You should set approximate time allotments for each
section to be sure the researcher covers all the issues
For certain questions, you may want to take notes on a
board for the respondents to see; indicate this in the
guideline
Qualitative Instruments
Order of Questions
As with quantitative questionnaires, use the funnel
approach, proceeding from broad, general
questions to more specific questions
 Opening questions should be easy and nonthreatening

Qualitative Instruments
Length of Guideline

Keep in mind, the objective of this research is to encourage
discussion. You should not expect to get through a lengthy
guideline
• The shorter the guideline, the more you will be able to encourage a
thorough discussion

Only ask questions that meet your specific research
objectives. For each question, ask:
• Is this question necessary?
• Will this question further the relevant discussion?
• Will it meet my objectives?
Qualitative Instruments
Administering the Guideline




A skilled moderator/interviewer is essential to gathering
quality data
Confidentiality must be ensured
Consider not using political polling firms
Keep in mind how you will be administering the
guideline (e.g., focus group, in-depth interview) when
deciding on the wording and length of the guideline
Qualitative Instruments
Administering the Guideline
Focus Group:
Professional moderators
 Best for encouraging discussion of a range of
respondents’ opinions
 Guidelines should be short to moderate in length.
To keep a lively discussion, focus groups should
not go too long (about 90 minutes, depending
on the culture)
 Be careful not to include sensitive issues about which
respondents will want to maintain confidentiality and
may be reluctant to discuss

Qualitative Instruments
Administering the Guideline
Focus Group:
Moderators should be instructed on what to do if the
discussion goes too far astray
 Moderators must also maintain objectivity, so that
respondents do not feel that they should respond in a
certain manner
 Moderators must be fluent in the respondents’ language

Qualitative Instruments
Administering the Guideline
In-depth Interview:
Professional interviewer
 Best for a longer guideline, a guideline that does not
lend itself to group discussion, or when a group format
would not be appropriate with the desired respondents
(e.g., elite government officials)
 Confidentiality must be ensured
 Should be done at appropriate times with interviewers
fluent in the respondent’s language

Qualitative Instruments
Administering the Guideline
In-depth Interview:


Be careful asking about sensitive issues as responses
may be vague or respondent may be uncomfortable
responding, impacting the entire interview
As with focus groups, interviewers should be instructed
on what to do if the discussion goes too far astray and
how to maintain objectivity
Hands-on-Work
Designing a Qualitative Guideline
1/--страниц
Пожаловаться на содержимое документа