Mapping the Dutch Foreign Language
State Examinations onto the Common
European Framework of Reference
Report of a Cito research project commissioned by
the Dutch Ministry of Education, Culture and Science
José Noijons & Henk Kuijper
© Stichting CITO Instituut voor Toetsontwikkeling 1
Project Goals
• To establish links between the existing
examinations in French, German and English and
the CEFR, following the steps as outlined in the
Manual published by the Council of Europe.
• To study the possibilities of developing more
comprehensive CEFR-related examinations in
the foreign languages.
© Stichting CITO Instituut voor Toetsontwikkeling 2
Four Project Phases
Phase 1: Familiarisation
Phase 2: Specification
Phase 3: Standardisation
Phase 4: Empirical validation
© Stichting CITO Instituut voor Toetsontwikkeling 3
Phase 2: Specification
Specification of texts
•
•
•
Related to the reading scales for
communicative language activities (CEFR
chapter 4)
Related to text type, source, topic, domain
(Dutch Grid)
Related to the scales for communicative
language competence (CEFR chapter 5 &
Dutch Grid)
© Stichting CITO Instituut voor Toetsontwikkeling 4
Phase 2: Specification
Specification of items
• Question types
• Task dimensions 1 (recognising,
inferences, evaluation)
• Task dimensions 2 (explicit vs. implicit)
• Task dimensions 3 (content of
operations)
© Stichting CITO Instituut voor Toetsontwikkeling 5
Specification texts: reading scales
communicative activities German
200
173
150
100
50
21
5
3
0
reading
correspondence
reading for orientation
reading for information
and argument
reading instructions
© Stichting CITO Instituut voor Toetsontwikkeling 6
Specification texts: text type, source,
topic, domain
• Great variation in text source and
communicative themes (topics)
• Increase of proportion of expository and
argumentative texts from lower to higher
educational levels
• Domain: mainly personal
© Stichting CITO Instituut voor Toetsontwikkeling 7
Specification texts: communicative
language competences
•
•
•
•
Level of abstraction
Grammatical complexity
Vocabulary
Text length
Dutch grid
© Stichting CITO Instituut voor Toetsontwikkeling 8
Specification texts: communicative language
competences, level of abstraction
4
3,5
3
2,5
English
2
German
1,5
1
0,5
0
vmbo-bb
vmbo-kb
vmbo-gl/tl
havo
vwo
Only concrete
1
Mostly concrete
2
Fairly extensive abstract
3
Mainly abstract
4
© Stichting CITO Instituut voor Toetsontwikkeling 9
Specification texts: communicative
language competences, grammar
4
3,5
3
2,5
English
2
German
1,5
1
0,5
0
vmbo-bb
vmbo-kb
vmbo-gl/tl
havo
vwo
Only simple structures
1
Mainly simple structures
2
Limited range of complex structures
3
Wide range of complex structures
4
© Stichting CITO Instituut voor Toetsontwikkeling 10
Specification items
From lower to higher educational level:
• more inferences and evaluation, less
recognising
• Relatively more implicit information
• Greater variation of operations
© Stichting CITO Instituut voor Toetsontwikkeling 11
Specification: general conclusions
• Increase of linguistic and cognitive
complexity of texts
• Increase of variation of operations
demanded in the items
© Stichting CITO Instituut voor Toetsontwikkeling 12
Claims through Specification and
Standardisation
If a claim of a link to the CEFR is based on
specification only, we do not know what
score a candidate needs to claim that the
candidate’s ability is at the CEFR-level the
test claims to be at.
Claims can be further substantiated through
standardisation of judgements: itemdifficulties are judged in relation to CEFR
levels.
© Stichting CITO Instituut voor Toetsontwikkeling 13
Standardisation
Standard-setting
Basket procedure: judges are to assign
items (texts + tasks) to CEFR-levels.
What minimum CEFR level does one need
to master this item?
© Stichting CITO Instituut voor Toetsontwikkeling 14
Results for Exams in English
Required minimum CEFR level for Exams in English
© Stichting CITO Instituut voor Toetsontwikkeling 15
Results for Exams in French
Required minimum CEFR level for Exams in French
© Stichting CITO Instituut voor Toetsontwikkeling 16
Results for Exams in German
Required minimum CEFR level for Exams in German
© Stichting CITO Instituut voor Toetsontwikkeling 17
Scores & Standards: English
Distribution of scores & cut score vwo-exam English
VWO English
Score
distribution
Score
distribution
© Stichting CITO Instituut voor Toetsontwikkeling 18
Scores & Standards: French
Distribution of scores & cut score vwo-exam French
VWO VWO
French
French
Score distribution
© Stichting CITO Instituut voor Toetsontwikkeling 19
Scores & Standards: German
Distribution of scores & cut score vwo-exam German
VWO German
English
Score
distribution
Score
distribution
© Stichting CITO Instituut voor Toetsontwikkeling 20
Conclusions standard-setting
• In the Netherlands a candidate can “pass” an
exam without attaining the relevant CEFR level
for that exam.
• Only scores at, or higher than the CEFR cut
score indicate that the candidate is at or above
the CEFR level the exam has been estimated to
be at.
• Empirical validation may help to show that a
CEFR level (a score) attained on one exam is
equivalent to a CEFR level on another exam.
© Stichting CITO Instituut voor Toetsontwikkeling 21
General Conclusions
• A test that is linked or validated through
specification only, cannot provide sufficient
information on how candidates need to perform
on the test to claim they have reached relevant
attainment targets.
• It is necessary through standard setting to
compute minimum scores that are needed for
candidates to claim they have reached relevant
attainment targets.
• External validation is needed to verify claims of
links to the CEFR.
© Stichting CITO Instituut voor Toetsontwikkeling 22
Descargar

AZERBAIJAN & PISA