THE ART OF FORMULATING AND ASKING SURVEY QUESTIONS
Principal Sources of Error in Survey Results
• Sampling error
The sample of interviewees is not representative of the universe from which it
is drawn; it is skewed in various ways.
• Respondent error
Response is not appropriate/relevant to the question, due to
misunderstanding (= poorly worded question?)
Response is deliberately misleading (= lying)
Faulty recall (retrospective questions)
Social desirability response set: Response is determined by what
interviewee believes is socially acceptable behavior or attitude
Acquiescence response set: Respondent agrees with whatever
statement is presented to him/her, for fear of offending interviewer
• Interviewer error
Question not asked (skipped), question was asked of inappropriate
respondent, response categories were read (directing respondent), response
was inaccurately recorded, etc.
• Data processing error
“GIGO”: garbage in, garbage out. Errors in transferring data from
questionnaire into data base.
From research objective to survey question: An example
• Objective: Document and explain change in rate of settlement in U.S.
(vs. circular migration).
• Explanatory hypotheses:
Such changes may be a function of:
-- Continued tightening of border controls, increasing coyote fees,
indebtedness, and physical risk
-- Family reunification migration |
(fewer relatives to maintain in hometown)
-- Maturation of social networks
-- Less seasonal work in U.S., more employment in “permanent”
year-round jobs
-- “Anchoring” investments in U.S. (home ownership, small businesses)
-- Life events (marriage, children reaching school age, etc.)
-- Changing gender roles in migrant families (greater female
labor force participation, etc.)
-- Decisions to naturalize (gain U.S. citizenship), for “defensive”
purposes, employment eligibility, etc.
• Empirical tests/indicators?
• Survey questions?
Types of Survey Questions
Filter questions:
1. ¿Es Ud. de Tunkás?
(si NO es de Tunkás, agradecerle a la persona y
busca otra persona)
2. ¿Cuántos años tiene Usted?
(si tiene menos de 15 años o más de 65 años,
pedir hablar con otra persona)
The answers to these questions are not marked on the cuestionario.
Standardized Question, Closed Response
Non-standardized Questions, Closed Response
Viaje
Lugar de Destino (Ciudad, Estado)
1er
Códigos: Documentos
1= Cuidadano
2 = Residente Legal
3 = Contratado (H2A/H2B)
4 = Contratado (Bracero)
5 = Turista
6 = Tarjeta cruce local
7 = Visa de estudiante
8 = Indocumentado
8888 = N/A
9999 = NS/NQR
Año en
que llegó
Tiempo
que duró
este viaje
en EEUU
(meses)
DOC
(usar
códigos)
Standardized Question, Open Response
(avoid as much as possible)
(Si no usó coyote:) ¿Cómo logró cruzar la frontera, sin
usar coyote?
(Anotar respuesta completa)
___________________________________________
_________________________
___________________________________________
_________________________
___________________________________________
_________________________
___________________________________________
_________________________
Open Question, Open Response
• Ya no es encuesta.
• Questions soliciting answers that will involve a lot of
complexity, nuance, and detail should be in a semistructured interview schedule. Qualitative interviews
are also generally better for getting at emotion,
meaning, and narrative.
• E.g.: ¿Qué es los que más/menos le gusta de los
EE.UU.?
Different Types of Closed Responses
Questions with a numeric response:
A poorly constructed question:
A better question with a simple numerical response:
¿Cuántas veces a la semana fuma Ud.? ________
(n.b. answer may be zero).
Questions with a dichotomous yes/no
response:
Questions with scaled responses:
“Thermometer” scaled response
(MOSTRAR
TARJETA ) Le voy a preguntar su opinión acerca de
algunas personas e instituciones. En una escala de 0 a 10,
donde cero significa que su opinión es muy mala y el 10 que su
opinión es muy buena, ¿cuál es su opinión acerca de….?
(LEER Y ROTAR)
a. Vicente Fox
b. Carlos Salinas de Gortari
c. El Papa Benedicto Dieciséis
d. George Bush, presidente de Estados Unidos
e. Fidel Castro, presidente de Cuba
f. Felipe Calderón
g. Santiago Creel
h. Andrés Manuel López Obrador
Dichotomous vs Scaled Responses:
55. Actualmente, si uno no tiene papeles, ¿es muy peligroso cruzar la
frontera?
Si
1
No
2
N/A
8888
NS/NQR
9999
Better:
55. Actualmente, ¿qué tan peligroso es cruzar la frontera, sin papeles?
Muy peligroso
Algo peligroso
Poco peligroso
Nada peligroso
N/A
NS/NQR
1
2
3
4
8888
9999
Questions with unscaled
categorical responses:
Comparative opinion question
(paired alternatives):
91. Algunas personas dicen que los jovenes nacidos
en Tunkas pueden progresar en la vida sin salir del
pueblo. Otras personas dicen que para superarse, los
jovenes nacidos en Tunkas tienen que salir.
¿Qué diria Ud.?
puede superarse sin salir 1
tiene que salir
2
N/A
NS/NQR
8888
9999
Questions asking respondent for a
rank ordering:
OTHER ISSUES IN SURVEY DESIGN
Define complex concepts precisely
¿Cuántos viajes a hecho a los Estados Unidos, para trabajar o vivir?
Definición de ‘un viaje’:
Un viaje es una estancia en los EE.UU. que dura por lo menos 1 mes y cuyo
motivo principal es trabajar o vivir.
No incluye vacaciones ni turismo.
Si el entrevistado regresa a Tunkás por menos de 1 mes y luego regresa a
los EE.UU., consideramos que es el mismo viaje a los EE.UU.
“Proxy reporting” vs. “Self-reporting”
24. ¿En esta última temporada en los EEUU, ¿se quedó más tiempo de lo que tenía
planeado?
24ª. ¿Por qué?
25. ¿Conoce a alguien que se quedó en los EEUU en vez de regresar a Tunkás?
Better:
25. ¿Tiene algún familiar que se quedó en los EEUU en vez de regresar a Tunkás,
durante los últimos 12 meses?
Always specify the time frame for reporting
21. En los últimos 5 años, ¿dónde ha pasado más tiempo,
en Tunkás o en los EEUU?
San Miguel Tlacotepec
Otra localidad en México
EEUU
N/A
NS/NQR
1
2
3
8888
9999
22. En los últimos 5 años, ¿cuántas veces regresó Ud. a Tunkás?
Nº de veces:
N/A
NS/NQR
_________
8888
9999
Avoid questions that don’t elicit sufficiently detailed responses
55. Actualmente, si uno no tiene papeles, ¿es muy peligroso cruzar la
frontera?
Si
1
No
2
N/A
8888
NS/NQR
9999
Better:
55. Actualmente, ¿qué tan peligroso es cruzar la frontera, sin papeles?
Muy peligroso
Algo peligroso
Poco peligroso
Nada peligroso
N/A
NS/NQR
1
2
3
4
8888
9999
Softening the question/reducing sensitivity
Ahora quiero hacerle algunas preguntas sobre su último intento
o éxito en cruzar la frontera.
30. ¿Lo hizo con papeles, o tuvo que entrar sin papeles?
Con papeles legales
Con papeles prestados
Con papeles falsos o chuecos
Sin papeles
N/A
1
2
3
4
8888
NS/NQR
9999
Sensitivity of items
Organization of questionnaire/sequence of items
0
Time elapsed in interview
The art
(and a little of the
science) of
field interviewing
Interview each other to gain confidence
Get your
bearings
Build rapport with the entire community
Do geological explorations to gain access
Key issues/challenges
Introducing yourself
How to explain what you are doing to the potential interviewee.
“How will I – or my community – benefit?”
(Para que sirve todo esto?)
•
•
•
•
•
•
•
Hints:
Have an honest “sales pitch.”
We are writing the “history” of the town and its migration.
Our UCSD group was here three years ago.
“Confident assertiveness” – Fowler
We’re not from the US or Mexican government.
Information is anonymous.
Participation is voluntary.
What are the benefits for Tunkás?
• Don’t promise anything we can’t deliver.
• Community leaders are interested in high-quality data about
where migrants go, how much they remit, etc…
• Our research findings can be used to educate U.S. public and
“funcionarios” who make U.S. immigration policy – para
promover la reforma migratoria.
• Passing out free books from Tunkás II study. Smaller
expressions of gratitude. Map and satellite image of town,
photos, etc…
• Benefits tend to be at the community-level for the town &
binational society rather than for specific individuals.
Managing social distance issues:
You are an “elite” college student: perceived as upperclass, highly educated, culturally different. How to
make yourself less threatening?
Hints:
• Allow the interviewee to ask you questions about yourself,
to reduce social distance and establish trust.
• Trade on your student status. You are there to learn.
• Emphasize our team’s previous experience in Tunkas and
southern California; common social ties.
• We are an “equipo binacional.”
• Draw on the gender/ ethnic/ national diversity of the team
to manage difficult interview situations.
How to manage time during the interview
Need to keep to a schedule, to complete interview in a
reasonable amount of time, while maintaining the flexibility
needed to build rapport and stimulate informative exchanges
of information with interviewee.
Hints:
-- allow interviewee to tell you (some) of his own stories;
-- use pauses, allowing enough time for a response (don’t
interrupt silence);
-- internalize the questionnaire (memorize it!) so you don’t
waste time looking for the next relevant question)
Read the question the first time EXACTLY as it is written in the
cuestionario. This is especially important for attitudinal
questions. If the respondent doesn’t understand the question,
you may explain it in your own words.
Probing: How deep to probe, to get a response that’s more
detailed, more complete, more reliable, more relevant to the
question?
Caution: Danger is that, with extensive probing, interviewee
may “invent” an opinion that in reality he doesn’t have, to be
courteous or to avoid the shame of not having an opinion.
Detecting and correcting errors of
inconsistency in responses
Hints:
• “Active listening” is important.
• Pay very close attention to responses. Think about
the response content.
• Repeat the respondent’s own words to signal your
understanding of him, and to set up follow-up
question to clarify a contradiction or inconsistency.
Reducing respondent error
(“reporting error”)
• Interviewee doesn’t understand question, gives inappropriate
response.
• Interviewee gives incomplete answers (not telling the whole
truth)
• Interviewee gives deliberately misleading answers (untruthful
responses)
• Interviewee has recall problems (poorly recalled facts, e.g., in
response to retrospective questions about migration &
employment history)
Hint: Prompt respondent with questions about major life
events like birth dates of kids, marriage, to help establish a
timeline for other events
Other sources of respondent error
• Social desirability response set (bias): Interviewee tells you
what he thinks YOU want him to say, or what is most socially
acceptable in his community, in the U.S.
• Absurdly high values: For example, 40-year-old man tells you
that he has had 10 wives or has been to the U.S. 100+ times,
to impress you, or to signal he’s not taking interview seriously.
• Non-response: Interviewee refuses to answer certain
questions at all (due to fear, shame, desire to conceal income,
etc.)
Hint: Try to convince respondent of the non-threatening
nature of the study. Try to get answers for all questions, but if
the respondent refuses, move on to other questions.
Still more challenges
• How to record responses (note-taking) without
interrupting flow of the interview?
• How to assure confidentiality?
• How to respond to interviewee requests, for money,
transportation, introductions, information about “El
Norte,” etc.?
• What “body language” should you use to build
rapport and elicit more useful responses?
Descargar

Slide 1