Language proficiency evaluation
(testing)
Selection and development
of language tests in the aviation context
Dr Jeremy Mell
Head of Language Studies, ENAC, Toulouse, France
Member PRICE SG, ICAO
[email protected]
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
1
Options for assessing language
proficiency for aviation
• informal observations
– « line checks », « inspections »
– classroom assessments
• formal language tests
– direct/semi-direct/indirect
– simulated language use
• most direct
• integrative: performance samples are matched to rating scales
– paper and pencil OR screen and mouse tasks
• most indirect
• discrete items/skills: favour numerical scores
• but « juggling with only one ball »….
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
2
Language tests
• Definition
•
•
•
•
structured events or procedures
to elicit performances as samples of test-taker ’s language skills
in a standardised way
enabling reliable inferences to be made concerning his/her level
of competence
• and possibility of reproducing those skills at that level of
competence consistently over time
– adapted from Carrol (1968) and Douglas (2000)
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
3
Language testing
• A well-developed domain of intellectual activity:
– academic
• body of research, cross-disciplinary links, international
associations,...
• codes of ethics, codes of practice
• institutions of excellence, « chapels », « gurus », feuds
– commercial
• international testing service providers
• general, academic, business,…
• Attempts to link different rating scales/test results
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
4
Types of language tests
• entry
• placement
• diagnostic
• progress/achievement
• proficiency
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
5
Entry/placement tests
• Purposes
– ab initio training:
• recruitment
• form level groups
– recurrent training:
• benchmarking a
population
• form level groups
• Characteristics
• items chosen to cover a
broad range of levels of
general language
• formats reflect previous
education
• scores tend to form a
« bell curve »
• stakes
– high (recruitment)
– mid (benchmarking)
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
6
Diagnostic tests
• Purposes
– identify specific
areas of
skill/knowledge for
improvement in
subsequent training
– individualisation of
training programmes
• Characteristics
• each item chosen to
represent a single
significant area of
knowledge/skill
• overall score less
important than analysis
of right/wrong
responses - focus on
errors
• stakes: low
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
7
Progress/achievement tests
• Purposes
– to measure
effectiveness of a
phase of learning
– to allow access by
learner to next phase
of learning
• Characteristics
– items chosen to closely
reflect content and
methodology of preceding
training phase
– scores will often be
interpreted with regard to
average or pre-established
norm
– may refer to intermediate
levels (3,5; 4,5;…)
– stakes: low to medium
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
8
Proficiency tests
• Purposes
– to establish the
competence of
candidate to exercise
language skills in
operational
conditions
• Characteristics
• items chosen to
resemble real-world
tasks
• overall scores are
holistic: YES/NO
• stakes: very high
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
9
Testing needs of aviation
• Ab initio populations
– entry
• selection
• local education vs
aviation training policies
– progress/achievement
• intermediate training
objectives
• continuation/curtailment
– initial proficiency
• Qualified populations
– placement/diagnostic
• benchmarking
populations
• individual training needs
• acceptance by takers
– (progress/achievement)
• degree and speed
– proficiency renewal
• recurrent
• licensing
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
10
Language test quality
• qualities of tests in general
– validity
– face validity
– reliability
– washback effect
– practicality
• conflicting qualities
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
11
Test design specifications (ICAO)
• “As of 5 March 2008, aeroplane and helicopter pilots, air traffic
controllers and aeronautical station operators shall demonstrate
the ability to speak and understand the language used for
radiotelephony communications to the level specified in the
language proficiency requirements in the Appendix.” (Annex 1,
1.2.9.4)
– formal evaluation
• “An individual must demonstrate proficiency at level 4 in all
categories in order to receive a level 4 score.” (Doc 9835,
Manual on the Implementation of ICAO LPRs, 2.8.4)
– profile evaluation
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
12
Test design teams
• Tasks
– develop test
specifications
– construct test
– evaluate test
– ensure ongoing test
maintenance
• Qualifications
– operational expertise
– language test
development
expertise
– linguistic expertise
– item writers
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
13
Test administration teams
• Tasks
– ensure advance
access to test
samples
– schedule test
– manage pre-test
preparations
– manage test
• Qualifications
– knowledge of
administration
guidelines
– interlocutor expertise
• operational
• linguistic
– rater expertise (min 2
raters)
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
14
Rater/interlocutor qualifications
• 3 basic qualifications
– language level
– aviation background + familiarity with ICAO LPRs
– principles of language proficiency and language testing
• suitable candidates
– aviation personnel
– language trainers
• L1 background
– native speakers(NS)/non-native speakers (NNS)
– risk of familiarity with a given form of NNS spoken English
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
15
Institutional context
•
need for independent international oversight to ensure:
– validity of testing procedures used
– comparability of testing procedures and outcomes
•
testing outcomes are linked with personnel licensing
– testing may not be in the sole hands of the ELT community
– partnerships with CAA, service provider, airline,...
•
system cannot tolerate a high failure rate
– waste of expensive training investment
– staffing levels must ensure continuity of service
•
prior knowledge of test procedures for positive “washback” on training
– trainee motivation and attitude, reassurance of candidates
– development of appropriate training systems
– BUT unsuitability of “cramming”, test item banks
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
16
A common test for ATC and pilots?
• pilots and controllers are partners in R/T communication
• but they are set apart by:
– different interactive roles
– complementary passive/active competencies
– opportunity to use other job-related language uses to extend
speech sample:
• controller: telephone co-ordinations, report to supervisor, ...
• pilot: pre-flight, intra-cockpit, cabin announcements, ground staff, ….
• a possible solution?
– common core test
– job-specific components
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
17
Need for testing services
•
time/space constraints
•
– dispersed locations (recurrent)
– irregular schedules and limited
availability of test-takers
(recurrent)
– availability of technologies
– test/re-test
•
test components
– single event, integrative?
– different item designs to test
separate skills?
standardisation of results
– extensive trialling
– comparable conditions of test
administration
– examiner training/auditing
• interlocutors
• raters
•
test security
– secure storage and transfer of
test materials
– multiple parallel versions
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
18
Appropriate technologies
•
voice-only
– telephone, 2-way radio, language laboratories, training simulators
•
input delivery
– analogue players, computer screen (multimedia)
•
performance storage and access
– analogue recorders, computer sound files (rapid access)
•
speech recognition
• standardise perception of intelligibility, save rating costs
• BUT needs to filter all possible speech variations
• no real interaction
•
« interactions » need to be human-human
– initiate and maintain exchanges
– deal with misunderstandings
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
19
Standardisation
• operationalisation of the rating scale
– speech samples
– « can do » statements
• rater training
– initial
– refresher
• rating protocols
– paired rating
– multiple rating
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
20
Assessment of level 6
• NOT a safety issue, but
raises concerns about:
• resources ( no re-test)
• NS/NNS distinctions
• Speakers may be:
• mother-tongue
• non-mother tongue
– intranational use
– non-intranational use
• Candidacy on basis of
biographical criteria
• citizenship, educational
background, residency, ...
• Protocols for assessment
– admissible evidence
– assessor qualification
– appropriate documentation
• Subsequent checking and
oversight must address:
– adherence to standard
ICAO phraseologies
– ICAO LPRs for
• intelligibility
• appropriateness
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
21
Informal evaluations
• Within the initial or continuing training process,
language trainers can identify 3 categories of
trainee:
- well-below level 4
- approximately at level 4 (confirm by formal evaluation)
- comfortably above above level 4
• …thus enabling informed decisions to be made on
• possible access to professional functions
• further training required
• benchmarking of populations
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
22
Formal tests for ICAO level 4
• Placement (entry)
– must relate to ALL 6 skills in
ICAO scale
– must measure distances above
and below level 4
– may have diagnostic capability
– tasks/topics/input materials
focus on general or job-related
skills
– development requires some
statistical validation
• Proficiency (exit)
– must relate to ALL 6 skills in
ICAO scale
– must characterise test-takers
as BELOW or AT levels 4, 5 or
6
– tasks/topics/input material must
reflect radiotelephony language
competencies (including
standardized phraseology)
– development requires strong
statistical validation (high
stakes)
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
23
Detailed specifications
(proficiency)
•
•
•
•
•
•
•
•
•
provide a representative range of intelligible international accents as input for
comprehension;
provide a professionally relevant format for candidates to display
comprehension;
elicit an adequate continuous speech sample to test fluency/pronunciation;
provide a voice-only setting for “diadic” (2-person) interactions;
provide examples of routine and unexpected events in a work-related context;
allow the candidate to use basic grammatical structures creatively;
allow the candidate to demonstrate ability to paraphrase;
allow the candidate to change between rehearsed/formulaic speech and
spontaneous interaction;
simulate unexpected events to create opportunities for misunderstanding.
– EUROCONTROL/ENAC preliminary feasibility study 2004
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
24
Some existing specific tests
• Controllers
– ab initio
•
•
•
entry: EPT/FEAST(Eurocontrol)
progress: APRO (ENAC, F)
proficiency: PELA (Eurocontrol)
– recurrent
•
•
placement: TNP (DGAC, F)
proficiency: ELPAC (Eurocontrol)
–
• Other tests
–
–
–
–
–
–
–
TEA (Mayflower, UK)
TELLCAP (Russia)
TOEFA (Peru),
TELPA (IAES, Korea)
TELAP (CA Flight University,China)
RELTA (RMIT, Australia)
….
under development for 2007
• Pilots
– ab initio
•
proficiency: FCL 1.200 (DGAC, F)
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
25
TNP (DGAC, France)
•
•
developed by national language and subject-matter experts (controllers
and ATC management) as a placement/diagnostic tool
3 papers cover professional language content:
• a written multiple choice test of language knowledge
• a tape-mediated written test of listening comprehension (RT messages)
• an individual oral interview using paper-mediated graphic and text input
•
•
expresses results in terms of the ICAO level chart
used since 1999 with qualified air traffic controllers:
• benchmarking to establish national training needs
• formulating periodic individualised language development and maintenance
programmes in order to meet national qualification renewal requirements
•
NOT a proficiency test
• lacks robust validation
ICAO Regional Workshop on Language Proficiency Requirements Implementation;
Paris, France, 6-7 September 2006
26
Descargar

Aucun titre de diapositive