Implementing a New Multilingual
Selection Model Across 27
European Countries
Gilles Guillard, Markus Nussbaum
The European Union
Commission
Economic &
Social Committee
Council
Committee of
the Regions
Parliament
Ombudsman
Court of Justice
Data Protection
Supervisor
Court of Auditors
The EU Institutions
Posts in 2012
European Parliament
6655
Council of the European Union
3153
European Commission
Court of Justice
25478
1952
European Court of Auditors
887
European Economic and Social Committee
724
Committee of the Regions
531
European Ombudsman
66
European Data Protection Supervisor
43
European External Action Service
Total
1670
41159
European
Court
of Auditors
2%
Court of
Justice
5%
European
External
Action Service
European
4%
Parliament
Others
16%
3%
European
Commission
62%
Council of
the
European
Union
8%
Entry Streams
Administrators (AD)
e.g. policy officers, lawyers, auditors, scientific
officers, translators, interpreters, communication
& press officers …
Qualification: University degree (bachelor)
Assistants (AST)
e.g. secretaries, HR assistants, financial
assistants, conference technicians …
Qualification: Secondary education (at least) and relevant
professional experience
Selection in Cycles
The AD Cycle 2012
Competition launched in March 2012
45000+ candidates for 296 posts
5 fields: Audit, EU Public Administration,
Communication, Law, External Relations
Computer Based Tests
Verbal reasoning
20 items from a database, 35 min, 23 languages
Numerical reasoning
10 Items from a database, 20 min, 23 languages
Abstract reasoning
10 Items in fixed test forms, 10 min
Situational Judgement
20 Items in fixed test forms, 30 min, EN/DE/FR
Verbal Reasoning
Numerical Reasoning
Abstract Reasoning
Situational Judgement
X
X
Results
EU Public Admin
Audit
Communication
Law
External Rel
Sum
AD5
validated application
AD7
AD5
AD7
AD5
AD7
AD5
AD7
AD5
AD7
12287
7892
2675
1389
3467
2023
4636
2130
6519
2338
45356
booked CBT
9557
6076
2182
1005
2525
1430
3615
1592
5016
1706
34704
participated CBT
8323
5336
1942
893
2132
1263
3113
1330
4412
1470
30214
reached CBT pass-marks
5502
3767
1427
603
1253
723
2076
830
2935
948
20064
190
102
108
34
77
51
144
43
64
21
834
succeeded CBT
Score Distribution - VR
Count
AD177: VR
3750
3500
3250
3000
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
participated
admitted
reserve list
0
1 2
3 4
5 6
7 8
9 10 11 12 13 14 15 16 17 18 19 20
Test score
Score Distribution - NR
AD177: NR
6000
5500
5000
4500
Count
4000
3500
participated
admitted
reserve list
3000
2500
2000
1500
1000
500
0
0
1
2
3
4
5
6
Test score
7
8
9
10
Score Distribution - AR
Count
AD177: AR
7500
7000
6500
6000
5500
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
participated
admitted
reserve list
0
1
2
3
4
5
6
Test score
7
8
9
10
Situational Judgement
AD-2011: SJ-PO
AD-2011: SJ-QR
8000
8000
8000
7000
7000
7000
6000
6000
5000
4000
6000
5000
Count
Count
9000
5000
4000
3000
3000
2000
2000
1000
1000
2000
1000
1
2
3
4
5
6
7
8
9
0
0
10 11 12 13 14 15 16
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16
0
1
2
3
4
Score
Score
AD-2011: SJ-WO
8000
7000
7000
6000
6000
5000
5000
4000
4000
3000
3000
2000
2000
1000
1000
0
0
0
1
2
3
4
5
6
7
8
Score
9
10 11 12 13 14 15 16
0
5
6
7
8
Score
AD-2011: SJ-RE
8000
Count
0
4000
3000
0
0
Count
Count
AD-2011: SJ-AP
9000
1
2
3
4
5
6
7
8
Score
9
10 11 12 13 14 15 16
9
10 11 12 13 14 15 16
Intercorrelations
Verbal Reasoning
Numerical Reasoning
Abstract Reasoning
SJT Analysis & Problem
Solving
SJT Planning & Organising
SJT Quality & Results
SJT Resilience
SJT Working with others
-
.41
.29
.16
.23
.15
.17
.18
Numerical Reasoning
.41
-
.32
.15
.20
.14
.15
.15
Abstract Reasoning
.29
.32
-
.14
.17
.12
.14
.14
SJT Analysis & Problem Solving
.16
.15
.14
-
.40
.33
.31
.35
SJT Planning & Organising
.23
.20
.17
.40
-
.32
.35
.35
SJT Quality & Results
.15
.14
.12
.33
.32
-
.35
.33
SJT Resilience
.17
.15
.14
.31
.35
.35
-
.32
SJT Working with others
.18
.15
.14
.35
.35
.33
.32
-
Verbal Reasoning
Item Analysis
How EPSO monitors the
behaviour of items
Item Analysis
Based on the dicotomous /
polytomous Rasch model
Conducted after every major
competition
Calibrate items and monitor
their behaviour
Georg Rasch (1901-1980)
Making psychometrics as vivid as possible
Example:
Cumulative Analysis VR+NR
Test
Responses
Candidates
Items
VR
2 295 360
114 768
5 071
NR
1 147 680
114 768
3 490
Total
3 443 040
Oct 2012
8 561
Item Difficulty Adjustment
Across Languages (IDAAL)
 How EPSO ensures fair opportunities
for all candidates although testing
in 23 different languages
Aligned Averages Algorithm (AAA)
5
5
-4
4
4
-3
3
3
-2
2
2
-1
1
1
0
0
0
+1
-1
-1
+2
-2
-2
+3
-3
-3
+4
EN
-4
A
A
EN
-4
Aligned Averages Algorithm (AAA)
citem = common item in en and lang
n
m

citem ( lang )
citem ( lang )  1
n


citem ( en )
citem ( en )  1
m
  lang
Minimum Meansquare Method (MMM)
5
5
4
4
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
C
A
B
EN
-4
A
B
C
EN
-4
Minimum Meansquare Method (MMM)


n
m

    item , lang   lang 
item  1 lang  1


2
m

item , lang
lang  1
m
  lang



 min!



Person Parameter Contrast Measure
(PPCM)
5
4
3
2
1
0
Cdts A
-1
-2
-3
Cdts EN
-4
Person Parameter Contrast Measure
(PPCM)
n
m

cdt ( lang )
cdt ( lang )  1
n


cdt ( en )
cdt ( en )  1
m
  lang
Item Difficulty Adjustment
Across Languages (IDAAL)
... is a combination of these three
approaches:
Aligned Averages Algorithm (AAA)
Minimum Meansquare Method (MMM)
Person Parameter Contrast Measure
(PPCM)
How to assemble test forms
 EPSO Rasch Item Combination
Algorithm (ERICA)
Framework
Candidates have to be treated equally:


Same number of items
Same difficulty of test form
➾ Adaptive testing is not possible
EPSO has developed an algorithm involving
two steps:
1.
2.
Set up a "master test form"
Create test forms corresponding to master test
form
Target (quality criteria)

Actual test forms correspond to master
test form

Limited overlap between test forms

No over-exposed items

Further improving gender balance
Defining the
master test form

Master test form difficulty: 5
easy

Master test form difficulty: 8

Master test form difficulty: 2
difficult
Weighting
Items are approximately standard normal
distributed
Most items are around the average (0)
available
Weighting was implemented to better take
account of item availabilities
Sample Test Form
Target
Item
Actual
-1.00
EN1626VEN
-1.001
-0.56
EN1555V
-0.550
-0.22
EN1170V
-0.227
0.06
EN1063V
0.062
0.30
EN2361V
0.295
0.51
EN1003VEN
0.517
0.71
EN1686V
0.703
0.89
EN2089V
0.891
1.06
EN2041VEN
1.064
1.22
EN2133V
1.211
1.38
EN2191V
1.394
1.54
EN2261V
1.532
1.69
EN2260VEN
1.684
1.84
EN2180VEN
1.811
2.00
EN2196VEN
2.032
2.16
EN2264V
2.198
2.33
EN2142V
2.333
2.51
EN1578VEN
2.451
2.73
EN1292VEN
2.809
3.00
EN2371V
2.941
Sum SqDev:
0.9
Thank you very much
for your attention!
Any questions?
Descargar

Slide 1