Metadata and Quality Indicators
Reuse for Quality reporting
Work Session on Statistical Metadata (METIS)
Giovanna Brancato, Marina Signore
Istat
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Outline
1. Istat information System on Survey Documentation
(SIDI/SIQual)
2. Istat Quality Reporting for external users
3. Metadata and Quality documentation reuse for Istat
quality reporting
4. Metadata and quality reporting reuse for ESMS
Standards
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Quality documentation and dissemination at
Istat: SIDI and SIQual
Quality report
navigation system
(intranet/internet)
Process report
Quality pilots
Quality indicators
Programming
Management Systems
Geneva, 10-12 March 2010
SIQual
S
I
Quality metadata
D
I
Process metadata
Statistical
Yearbook
Work Session on
Statistical Metadata
(Metis)
SIDI metadata
Key concept
Statistical production process:
producing statistical information
process
aimed
at
 primary survey: data collection from reporting units
(households,
businesses,
institutions)
including
administrative sources surveys
225 documented primary surveys
156 active, 53 ceased, 16 suspended
 secondary study: processing of statistical data
already available from previous primary surveys or
secondary studies
101 documented secondary studies
81 active, 14 ceased, 6 suspended
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIQual
SIDI navigation environment
Intranet: metadata and standard quality indicators
functionalities for queries on organisational units
(direction/service/unit)
Internet: metadata SIQual
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIDI Standard Quality Indicators
Completeness status
Group
Coverage and Unit nonresponse
Coding
Editing and Imputation
Revision policy *
Timeliness and punctuality *
Comparability *
Coherence btw provisional and final *
Coherence with other sources
% of active processes
with indicators
67. 20
23.10
28. 60
7.50
92.24
60.99
17.78
12.30
Resources and Costs
* Including Primary Surveys and Secondary Studies
Geneva, 10-12 March 2010
65.98
Work Session on
Statistical Metadata
(Metis)
Istat quality reporting for external users
Tools
 Short methodological notes (SMNs)
 Process reports (PR)
 Quality reports (QR)
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Istat short methodological note (SMN)
Use
Compact reports included in the Statistical Yearbook
Structure
General conceptual metadata: observed issues, analysis unit
Methodological process: reporting unit, periodicity;
regulation, sampling design, data collection modes
EU
Quality metadata: synthesis of the main activities carried out
to prevent, monitor and evaluate nonsampling errors (unit
nonresponse and measurement error), validation techniques
Dissemination of provisional and final results: Timeliness,
geographical and sector level of aggregation
Languages
Italian (English)
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIDI metadata reuse in the SMN
Example
SIDI documentation on observed issues:
Attendance at museums and art exhibitions
 Chronic diseases
 Cinema attendance
 Friendship relations
Games and other leisure activities
Health conditions
Housing conditions
…
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIDI metadata reuse in the SMN
Example
SIDI documentation on the activities to prevent unit non
response:
- Survey presentation letter signed by Istat President
- Guarantees on statistical confidentiality
- Interviewer identification badge
- Description of survey objectives by interviewers
- Telephone contacts to make an appointment for the interview
- Establishing a toll free line or telephone number for further
explanations
Compacted description in the short methodological note:
“Activities for preventing unit nonresponse”
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIDI quality indicators reuse in SMN
Examples
SIDI documentation on Timeliness:
single value
average over a period
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Istat process report (PR)
Use
Standard process documentation to be included in
publications on survey results as methodological annex.
General structure
- Introduction
- Objectives
- Survey design
- Data capturing
- Data pre-treatment
- Processing
- Documentation
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIDI metadata reuse in PR
The report is pre-filled in with the information drawn from SIDI
An editor window allows to adjust and modify the
text to turn it in a fluent language
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Istat quality report (QR)
Use
It is a short report, including a subset of quality indicators, to
be disseminated together with press releases on Istat web site
Structure
It is organised into the following sections:
- Relevance
- Accuracy
- Timeliness and punctuality
- Clarity and accessibility
- Coherence and comparability
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Indicators included in the quality report (QR)
Qualtiy indicators
- Sampling error (confidence interval or coefficient of
variation)
- Weighted/Unweighted Response rate [respondent units /
(eligible units and unresolved units)]
- Net imputation Rate [imputations from blank to nonblank
values/imputable values]
- Modification Rate [changes from nonblank to nonblank
values/imputable values]
- Mean Revision (size and direction of the revisions)
- Absolute mean revision (average size of revisions)
- Timeliness (of provisional/final results, of data sources)
- Comparability (length of the series)
- Coherence between provisional and final estimates (difference
or relative difference btw provisional and final estimates)
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIDI metadata and quality indicators reuse in
QR
• Implementation phase
• Development of the product quality report integrated to SIDI
management environment
• Limited reuse of metadata
• Reuse of standard quality indicators following the short
methodological approach
• Need of developing new quality indicators and revising some
existing ones
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
ESMS requirements
Definition
Statistical metadata to be used by Member States for
documenting statistical data to Eurostat
 metadata
 quality indicators (ongoing work)
 harmonisation within Eurostat, OECD, IMF
Uses
 Comparisons and evaluations
 Metadata and quality information for the users
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Nature of ESMS metadata
Organization: Institutional mandate, confidentiality legislation,
release policy, quality management
Data set and variables: classification system, sector coverage,
statistical concepts and definitions, statistical unit and
population, reference area, time coverage, base period, unit
of measure, reference period
Dissemination frequency and formats
Eurostat quality components: relevance, accuracy, …
Statistical processing
Standard quality indicators
[Simulation: compilation of ESMS template with metadata
and quality indicators for a survey
~ 60%]
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Issues on ESMS
Type of data and frequency of data collection
Organisational data: univocally defined throughout the institution
Metadata: stable over time
Quality indicators: survey edition
Standard quality indicators
Harmonisation: among different Member States, among different
EU regulations  Eurostat Handbook/Standards for Quality
Reports (2009)
Priority setting: identification of a minimum set
Use: additional indicators upon request for expert users
Aggregation: survey edition, annual averages (for monthly
surveys)
Dissemination: summary measures, ranges of variations,
graphical representation
Geneva, 10-12 March 2010
Metadata and quality indicators reuse for quality
reporting
Thank you
for your attention !
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
SIDI main features
Metadata documentation standards
Thesaura: lists of standard items to be used to document
process activities and quality control actions
Content: Topics of the survey, analysis units,
questionnaire
Process: Reporting unit (sources of the secondary study),
survey design, data collection, data transformation, data
processing
Quality: Activities carried out to prevent, monitor and
evaluate survey errors
Metadata qualitative descriptions: free notes supporting
standard metadata items
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Istat process report (PR)
Use
Standard process documentation to be included in publications
on survey results as methodological annex.
General structure
- Introduction (name, period, description)
- Objectives (information needs and periodicity, issues)
- Survey design (population, sampling, reporting unit and frame,
questionnaire design and testing, pilot survey)
- Data capturing (data collection mode, field operations,
sensitive questions, unit nonresponse monitoring, mandatory
participation, response error)
- Data pre-treatment (coding, manual revision, data entry,
editing and imputation)
- Processing (estimation, validation, use of generalised
software)
- Documentation (list of relevant documents)
Geneva, 10-12 March 2010
Work Session on
Statistical Metadata
(Metis)
Istat quality report (QR)
Use
It is a short report, including a subset of quality indicators, to be
disseminated together with press releases on Istat web site
Structure
It is organised into the following sections:
- Relevance (regulations and use of the data)
- Accuracy (methodological context, sampling error,
nonsampling error, revision policy)
- Timeliness and punctuality (production of preliminary
results, timeliness)
- Clarity and accessibility (links to data and
documentation)
- Coherence and comparability (national or international
regulations and manuals, over time data comparability)
Geneva, 10-12 March 2010
Descargar

Diapositiva 1