Quality Standards in
The Basis for Valid and Reliable Data
for Educational Decision Making
Ina V.S. Mullis, Michael O. Martin, & Pierre Foy
Russian Education in the Mirror of the International Comparative Studies
June 19, 2013
Our Mission: Provide Internationally
Comparable Data of High Quality for
Improving Education
• Data about student achievement
– Reading, Mathematics, and science
• Data about the contexts for teaching and
– Key factors influencing achievement
– Relevant for educators and policy makers
“Internationally Comparable Data
of High Quality”
• Requires 100% attention to doing high quality
• With quality assurance steps along the way
• Classic attributes of high quality achievement
– Reliability
– Validity
– International Comparability
Instruments measure consistently what they
are intended to measure
• Instruments are the same
• Environment for using the instruments is the same
• Persons respond to the instruments in the same
• Instruments are scored in the same way
Ensure that comparisons are made based on “real”
achievement and not affected by extraneous factors
Inferences drawn from results can be
supported by evidence
• Requires unified agreement
– about how the construct has been conceptualized
and articulated… e.g., is this mathematics?
– on how it has been operationalized… e.g., do these
items measure mathematics?
In other words, does a student with a high score
on the mathematics test actually know a lot of
What About International
• Our Curricula are different!
• Our languages are different!
• Our school systems are organized differently!
– Different age of entry
– Duration of compulsory schooling
– Percentage of students attending school
– Stages of schooling (primary, elementary, etc.)
– Different promotion and retention policies
Validity in an International Context
• Need to ensure that data are internationally
• Inferences made about achievement differences
between countries can be substantiated
Accomplished by setting high quality standards
in all TIMSS and PIRLS procedures for developing
and administering the achievement assessments
Ensuring Reliability and Validity of the
TIMSS and PIRLS Achievement Data
• Assessment Framework
• Test Development
• Field Test
• Translation Verification
• Target Population
• Sampling
Ensuring Reliability and Validity of the
TIMSS and PIRLS Achievement Data
• Data Collection
• Constructed Response Scoring
• Database Construction
• Achievement Scaling
• Reporting Achievement Data
Assessment Frameworks
Dealing with different curricula
• Define the constructs in detail
– TIMSS: content and cognitive domains
– PIRLS: Purposes and processes
Assessment Frameworks
Developed through widespread collaboration
with participating countries
• Literature reviews, current perspectives
• Surveys to align assessments with countries’
• Iterative reviews by National Research
– Within country and in plenary
• Iterative reviews by expert panels – SMIRC, RDG
Assessment Frameworks
Updated with each assessment cycle
• Incorporate fresh perspectives
• Accommodate new countries
• Evolve over time
Test Development
In accordance with Assessment Framework
• Assess topics/content in Framework
• Ambitious frameworks require many items for
adequate measurement
– Each domain requires sufficient representation
• Trend measurement requires many items
– Items released and replaces with each cycle
TIMSS and PIRLS have lots of items!
Test Development
• Developed in proportion to emphases agreed in
• According to decisions about item format
– 50% multiple choice; 50% constructed response
• With scoring guides for constructed-response
• According to careful plan for measuring trends
– Approximately 60% trend, 40% new
Field Test
Essential for confirming appropriateness
and comparability of items – different
And to verify the proper implementation of
all procedures
• Twice as many items as needed
• Translation by each country
• Scoring guides for constructed-response items
and training
Field Test
• TIMSS & PIRLS ISC develops manuals
describing standardized procedures
• IEA DPC checks and processes data
• TIMSS & PIRLS ISC conducts item analyses
– Difficulty
– Discrimination
– Scoring reliability
Finalizing Item Selection
• Task force and TIMSS & PIRLS ISC make initial
recommendation about items to retain
• Field test data and initial recommendation
reviewed by expert committees – SMIRC, RDG
• Field test data and expert committee
recommendation about item selection reviewed
by the NRCs from participating countries
• Assessment items adopted by NRCs
Reliability and Validity in Data
Collection, Analysis, and Reporting
• Are the target populations comparable?
• Was sampling conducted properly?
• Are translations comparable?
• Were the tests administered appropriately?
• Was scoring done correctly?
• Are the data comparable?
• Are the achievement results comparable?
Comparable Target Populations?
School systems organized differently
Amount of instruction
Years of schooling
• PIRLS: 4 years of schooling, counting from 1st
year of primary 4th grade
• TIMSS: 4 & 8 years of schooling
4th & 8th grades
• Based on ISCED definitions
Comparable Target Populations?
Why grade and not age as the basis?
– Better for improving education!
• Education is organized by grade, so gradebased data easier to use for implementing
• Amount of instruction, not maturation, the
primary determinant of achievement
– Students learn through instruction, not simply by
growing older
Comparable Target Populations?
• Has country chosen correct grade?
• Are all eligible students included in definition?
– Generally yes, for most countries
– If less than 100%, annotated in International
• Are exclusions kept to a minimum?
– Generally yes, for most countries
– If more than 5%, annotated in International Reports
Sampling Conducted Correctly?
TIMSS and PIRLS Requirements
• Random sample design
– Developed and authorized by Statistics Canada
• Accurate school sampling frame
– School sampling done by Statistics Canada
• Proper classroom sampling
– Use of WinW3S software mandatory
Sampling Conducted Correctly?
TIMSS and PIRLS goals for sampling
• Participation rates for schools, classes, and
– 100%!!!
• Sampling precision goals
– Percentages ± 5%
– Means ± 0.1 S.D.
• Usually 150 schools and one or two classes per
school (Approx. 4,000 students)
Sampling Conducted Correctly?
• Procedures acceptable and fully documented?
– Reviewed by Statistics Canada and Sampling Referee
• Acceptable participation rates?
– At least 85% schools, 95% classes, 85% students
– Generally yes, for most countries
– Others annotated in International Reports, or below
a line
Population coverage and participation rates
published in International and Technical Reports
Translations Comparable?
• Has country correctly translated all assessment
– IEA provides guidelines and instructions
– IEA Secretariat verifies each translation
– Issues referred to National Research Coordinator for
• Do the test booklets conform to international
– TIMSS & PIRLS ISC verifies final layout before printing
• Countries check final printed booklets
Tests Administered Correctly?
Data collection a National responsibility
• TIMSS & PIRLS ISC develops manuals
describing standardized procedures
– School Coordinator Manual
– Test Administrator Manual
Tests Administered Correctly?
How do we verify that data collection procedures
have been followed?
• IEA Secretariat and TIMSS & PIRLS ISC conduct
program of international quality control monitoring
– IEA Secretariat recruits Quality Control Monitor (QCM)
in each country
– TIMSS & PIRLS ISC conducts training sessions for QCMs
– The QCM visits a sample of 15 schools at each grade;
records observations and interviews school coordinator
and test administrator
Tests Administered Correctly?
• TIMSS & PIRLS ISC analyzes and reports the
results in the Technical Report
– Generally, QCM reports are very positive
– Data collected according to procedures specified in
manuals, with very few exceptions
• Country also conducts quality control
observations at 15 schools
• NRCs complete online Survey Activities Report
Constructed-response Item
Scoring Done Correctly?
• About 50% of TIMSS and PIRLS items are in
constructed response format
• Each constructed-response item has its own
tailored scoring guide
• Scoring training materials prepared for each
constructed-response item
– Scoring guide
– Anchor or exemplar papers
– Practice papers
Constructed-response Item
Scoring Done Correctly?
• Scoring training conducted separately for
Southern Hemisphere and Northern
Hemisphere countries
• Training materials updated based on field test
– Scoring guides refined
– Enhanced sets of exemplar responses and practice
Constructed-response Item
Scoring Done Correctly?
How do we know the scoring was done well?
• Monitor reliability through double scoring
– Within-country for the current assessment
• 200 responses per item
– Within-country across trend assessments
• 200 responses per item scanned from previous
assessment and delivered electronically for recoding with
current assessment
– Across countries for the current assessment
• 200 responses per item from English-speaking countries
delivered electronically
Constructed-response Item
Scoring Done Correctly?
What happens if an item is not scored reliably?
• Vast majority of items have high scoring
• Items with less than 70% agreement for
within-country or trend reliability are removed
from scaling
– Extremely rare
• Scoring reliability results for all countries
documented in technical reports
Are the Data Comparable?
• IEA DPC provides data entry software and
variable codebooks to standardize data
– Software provides data checking and validation tools
applied by the countries
• IEA DPC provides extensive training seminars
• IEA DPC checks each country’s data files for
internal consistency and accuracy
• IEA DPC interacts with countries to resolve
data issues
Are the Data Comparable?
• IEA DPC creates database and sends to TIMSS
& PIRLS ISC and Statistics Canada for analysis
and reporting
• Statistics Canada computes sampling weights
based on data and sampling documentation
– Compares estimated population size using weights
against estimate from sampling frame
– Interacts with countries to resolve issues
• Statistics Canada creates final sampling
weights, including adjustments for nonresponse, for analysis and reporting
Are the Data Comparable?
Initial review of item statistics, before scaling
• TIMSS & PIRLS reviews achievement item
statistics – every item for every country
• Investigates items for poor discrimination or
unreliable scoring – sometimes caused by a
translation or printing error
• Rare, but such “faulty” items are not included
in scaling achievement results for that country
Are the Data Comparable?
Review of item-by-country interactions
• For each item, examine each country’s
performance on the item in light of its overall
– Outliers may be due to translation error, printing
error, etc.
• For trend, compare item-by-country interaction
patterns for current and previous assessments
– If different, may delete that item for that country for
Are the Scaled Achievement
Results Comparable?
Use IRT scaling to summarize achievement
data by modeling item difficulty and
discrimination – one scale for all countries
• Scaling procedure fits a model to each item
– The better the fit, the more accurate the results
• Check fitted model against observed data for
each item
– Typically, any issues were discovered during initial
Are the Scaled Achievement
Results Comparable?
For trend items,
• Data from current assessment and previous
assessment scaled together
• Item fit plotted separately to ensure that the
item is a good fit for both sets of assessment
Are the Scaled Achievement
Results Comparable?
Now that we have item parameters – difficulty and
discrimination – we can place students on the scale,
i.e., produce student achievement scores (plausible
• Done separately for each country
• Done separately for each achievement scale
– Reading, mathematics, science
– TIMSS content and cognitive domains and PIRLS
purposes and processes
• Each achievement distribution for each country
checked separately
Are the Scaled Achievement
Results Comparable?
Scaling generally is very successful
• For most TIMSS and PIRLS countries,
achievement score distributions are very
satisfactory and provide an excellent basis for
analysis and reporting
• Plots provide a good quality control check
Are Achievement Results in the TIMSS
and PIRLS International Reports
• All reported statistics accompanied by standard
• Tests of statistical significance performed for
many differences
– Between countries, across countries
• Annotations for countries not fully meeting
sampling standards
• Achievement results presented in context
Why Do We Go to All This Trouble?
• Provide evidence of the comparative validity of
the TIMSS and PIRLS achievement data
• The data can be trusted for important decision
making based on comparisons among countries
• TIMSS and PIRLS can form the basis for
evidence-based policy making
Thank You!
Ina V.S. Mullis, Michael O. Martin, & Pierre Foy
Russian Education in the Mirror of the International Comparative Studies
June 19, 2013

Slide 0