Quality strategies in crossnational surveys
The case of the European Social
Survey
Ineke Stoop
Quality and comparability
Harmonisation strategies and cross-national surveys
Optimal comparability
European Social Survey
Nonresponse
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Harmonisation strategies
(Körner and Meyer, 2005)
Harmonised
concept
Harmonised
concept
Harmonised
concept
Measurement
procedure
Measurement
procedure
Measurement
procedure
National survey/
Specific concept
National survey/
Specific concept
National survey/
Specific concept
Input harmonisation
Ex-ante output
harmonisation
Ex-post output
harmonisation
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Type of harmonisation
Ex post output harmonisation
• Use existing sources
Ex ante output harmonisation
• Set up new survey (or develop new instrument)
Output harmonisation
• Best national quality
• Or, national survey tradition
• But, what about optimal comparability
Input harmonisation
• Design new survey
• Optimal comparability
• Or, are identical methods and instruments really equivalent in different countries?
• But, what about optimal quality
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Strategies in comparative survey
Quality
Optimal national
approach
• Comparability?
• Poor performance
accepted
Some restrictions and
standards
Consistency
Identical approach
• Lowest standards
• Possible?
• Pull down level
• Highest standards
• Possible?
• Pull up level
High standards and
optimal national approach
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Mixed approach
Focus on those key aspects that may hamper
comparability
•
•
•
•
Data collection mode
Sampling approach
Response rates
Question formulation
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
European Social Survey
Attitudes, values and beliefs
Bi-annual: 2002, 2004, 2006, 2008
Fixed core, rotating modules
Effective size 1500 or more
Central co-ordination
National implementation
• 30 National Coordinators responsible for fieldwork
Face-to-face
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Aims European Social Survey
To monitor and interpret
public attitudes and values
within Europe and to
investigate how they
interact with Europe’s
changing institutions
• Provide data on beliefs,
attitudes and values for
scientific and policy
making purposes
• Measure attitude change
in a changing Europe
To advance and consolidate
improved methods of crossnational survey
measurement in Europe
and beyond
• Collect data according to
highest standards
• Generate methodological
research
• Develop and disseminate
new best practices
• Develop and improve
social survey research
infrastructure in Europe
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
EVERYTHING AVAILABLE
EVERYTHING DOCUMENTED
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Nonresponse: consistency
Strict probability sampling
No substitution
Fieldwork monitoring
Guidelines response
enhancement
• Incentives
• Brochure
Guidelines interviewer
training
Target noncontact rate: 3%
• Minimum number of calls
• Timing of calls (evening,
weekend)
Target response rate: 70%
• Refusal conversion
suggested
Contact form and call
records
Standard response rate
calculation
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Nonresponse: divergence
Population
• Residential population
• Minority languages
Sampling frame
• Addresses, households
• Individuals
At home pattern
• Female employment
• When does the evening start?
Fieldwork organisation
• Field director’s meeting
Interviewers
• Experience
• F2F
• Random sampling
• Training and briefing
• Remuneration
• Response, hours, staff
Mode of contacting
Use of advance letters
Incentives
Contact forms
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Response rates ESS (deviation 70%)
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Why do response rates differ?
Survey climate
• Used to surveys?
Popularity topic
• Europe?
Country size, urbanicity and contactability
Survey modes
• Predominantly telephone?
Fieldwork efforts
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
UP TILL NOW
NO UNIVOCAL PREDICTOR
OF
RESPONSE RATES
ACROSS COUNTRIES
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
% noncontact and refusal (R1, R2, R3)
% refusal
60
CH
50
60
60
50
50
CH
40
FR
ES
GB
DE
PT
NL
BE
NODK
20 PL FISE
SI
HU
40
NL
GB
DE
30
30
30
NL
NOBE ES
DK
FI
SE
20
PLPT
NOHUDEGB
BE
SE
FI
ES
PT
20
PLSI
SI
HU
10
10
10
0
0
0
0
4
8
CH
FR
DK
40
FR
12
16
0
4
8
12
16
0
4
8
12
16
% noncontact
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Age, gender, education (Vehovar & Zupanic)
Age: underrepresented
•
•
•
35+ (Austria)
55+ (Belgium and Luxemburg)
15-34 (Spain, Ireland, Netherlands, UK)
Education: underrepresented
•
•
Middle (Austria, France, Iceland, Luxembourg, the Netherlands, Norway,
Sweden and the United Kingdom)
Less (Switzerland, Czech Republic, Germany, Estonia, Hungary, Norway,
Slovenia, Slovakia and Ukraine)
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Results multivariate analysis: reluctance
(Beullens, Billiet and Loosveldt)
Switzerland
• Members of larger families
Germany
• Female, aged, city dweller
• Internet, unemployment history
• Less political participation
Estonia
• Female, not in village
• Employed
Slovakia
• Older, average education, church
life
• Never job, less safe, comfortable
family income
Netherlands
• Female, average education, more
TV, internet
• Paid job, healthy
• Immigrants threat, trust political
institutions
• Civil obedience, political
participation
• Socially isolated, dissatisfied
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Nonresponse bias in cross-national studies
How to measure bias?
• Different type of auxiliary information available
• Sample frame/register information?
• Reluctant respondents?
• Neighbourhood information?
Different size of bias?
Different type of bias?
Different bias over time?
ESS-experiments in several countries
• Mixed results
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Is output harmonisation better?
Maybe, but in that case do you know about
•
•
•
•
•
Response rates (standard calculation)
Intensity and effect of field efforts
Nonresponse composition
Underrepresented groups
Nonresponse bias
Black box:
• Differences between countries due to methodology or
substance?
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Are comparative surveys possible?
If not, national surveys are not possible either
• Subgroups
• Socio-demographic
• Survey interest
• Topic relevance
• Regions
• National languages
• Interviewers
There is so much we don’t know, and now we know
so much more
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Thank you for your attention
www.europeansocialsurvey.org
European Conference on Quality in Official Statistics 8-11 July 2008
Descargar

PowerPoint-presentatie