Assessment as a washback tool:
is it beneficial or harmful?
Nick Saville
Director, Research and Validation
University of Cambridge
ESOL Examinations
October 2008
Outline
1. Introduction
2. Washback and impact: some definitions
3. Impact and the law of unintended
consequences
4. Researching impact: the Cambridge ESOL
approach
5. Conclusion
2
Nick Saville, Cambridge ESOL
1. Introduction
•
Examinations
• access to opportunity
• fairness
•
Exert a powerful influence on educational
processes
•
•
possibly negative?
concern for social consequences, ethicality,
accountability
3
Nick Saville, Cambridge ESOL
Questions to be addressed:
•
How can educators and assessment providers
foster beneficial effects from their tests and
examinations?
•
How can harmful consequences be avoided or
their impact mitigated?
4
Nick Saville, Cambridge ESOL
“Impact by design”
•
Integral part of a framework for developing and
validating examination systems
•
A concept akin to social impact assessment
5
Nick Saville, Cambridge ESOL
2. Washback and impact: definitions
•
Washback (or backwash) has been broadly
defined in the assessment literature as the
effect of testing on teaching and learning
•
One aspect of the broader phenomenon known
as impact – as we shall see later
6
Nick Saville, Cambridge ESOL
Washback
• Alderson and Wall, 1993:
• 15 washback hypotheses
• Based on who or what might be affected:
•
•
•
•
•
•
•
•
Teaching
Learning
Content
Rate of learning
Sequence of teaching/learning
Degree/depth of curriculum coverage
Attitudes of teachers/learners
Etc.
7
Nick Saville, Cambridge ESOL
Washback
•
A continuum - stretching from harmful at one end,
through neutral to beneficial at the other end
Negative
-
Neutral
Positive
+
8
Nick Saville, Cambridge ESOL
Washback
•
Negative?
• Restriction of content – narrowing of curriculum
• Too much time practising for the test
•
Positive?
• Transparent objectives and outcomes
• Increased motivation of learners
• Increased accountability of teachers (?)
•
BUT – cause and effect explanations are rarely
adequate …..
9
Nick Saville, Cambridge ESOL
Washback Models
In the language testing literature:
• Bailey (1996)
• Watanabe (2004)
• Cheng (2004, 2005)
• Green (2007)
10
Nick Saville, Cambridge ESOL
Bailey’s Model (1996) - based on Hughes,1993
3 Ps:
Participants
• students
• teachers
Processes
Products
• learning
• teaching
• materials
• curricula
11
Nick Saville, Cambridge ESOL
Watanabe’s Model (2004)
•
Five dimensions:
•
•
•
•
•
•
12
Specificity
Intensity
Length
Intentionality
Value
Factors influencing the process of washback are
related to:
•
•
•
the test itself
status
stakeholders
Nick Saville, Cambridge ESOL
W a sh b a ck d ir e ctio n
Green’s Model
Te st d e sig n
ch a ra cte ristics
Fo ca l
Co n stru ct
item for m at
c ontent
c om plex ity
etc .
O ve rla p
See: Studies in Language Testing, 25,
2007: IELTS Washback in Context
P o te n ti a l fo r
P o te n ti a l fo r
p o s i ti v e b a c k w a s h n e g a ti v e b a c k w a s h
W a sh b a ck va r ia b ility
Washback will be most intense –
Oth e r s ta k e h o l d e rs
P a rticip a n t ch a ra cte ristics a n d va lu e s


K n o w l e d g e / u n d e rs ta n d i n g o f te s t d e m a n d s
R e s o u rc e s to m e e t te s t d e m a n d s

A c c e p ta n c e o f te s t d e m a n d s
C o u rs e p ro v i d e rs
M a te ri a l s w ri te rs
P u b l i s h e rs
Te a c h e rs
L e a rn e rs
Im por ta nt
P e rce p tio n o f
te st d ifficu lty
E asy
B a ckwa sh to
p a rticip a n t
N o ba c k wa s h
C ha lle nging
U nim
13 por ta nt
on teaching and learning
behaviours – where participants
see the test as challenging and
W a sh b a ck in te n sity
P e rce p tio n o f
te st im p o rta n ce
have the most powerful effects
U na c hie v a ble
the results as important (high
stakes)
SEE BLUE ARROW
Inte ns e ba c k wa s h
Nick Saville, Cambridge ESOL
Impact
Impact concerns the effects and consequences a test can
have beyond the classroom and immediate learning
context:
•
On individual career or life chances
•
In educational systems and in society
14
Nick Saville, Cambridge ESOL
Impact
•
Impact deals with wider influences and includes the
“macro contexts” - tests and examinations in society
•
Washback is an aspect of impact related to the “micro
contexts” of the classroom and the school
15
Nick Saville, Cambridge ESOL
Impact
• Investigating impact is integral to validation
• an essential component in establishing the usefulness of an
assessment system - fitness for specific purposes and
contexts of use
•
Consistent with Messick views of validity (1989, 1996)
• Consequential aspects of validity
• fairness and ethics
16
Nick Saville, Cambridge ESOL
Impact
•
Educational systems – complex phenomena
•
Stakeholder constituencies
• many different stakeholders in educational processes
• complex network of relationships
17
Nick Saville, Cambridge ESOL
Stakeholders in the testing constituency
Learners
Teachers
Test writers/examiners
Receiving institutions
School owners
Future employers
Government agencies
Professional bodies
Test centre administrators
Materials writers
Publishers
etc
Test constructs
Test format
Test conditions
Test
scores
Test
assessment
criteria
Testing System
Inputs to test design
Learners
Parents/carers
Teachers
Receiving
institutions
Employers
School owners
Examiners
Government
agencies
Professional
bodies
Academic
researchers
Test writers/
Examiners
etc
Contexts of test use - consequences
18
Nick Saville, Cambridge ESOL
Stakeholders in the testing constituency
Learners
Teachers
Test writers/examiners
Receiving institutions
School owners
Future employers
Government agencies
Professional bodies
Test centre administrators
Materials writers
Publishers
etc
Test constructs
Test format
Test conditions
Test
scores
Test
assessment
criteria
Testing System
Inputs to test design
Learners
Parents/carers
Teachers
Receiving
institutions
Employers
School owners
Examiners
Government
agencies
Professional
bodies
Academic
researchers
Test writers/
Examiners
etc
Contexts of test use - consequences
19
Nick Saville, Cambridge ESOL
Stakeholders in the testing constituency
Learners
Teachers
Test writers/examiners
Receiving institutions
School owners
Future employers
Government agencies
Professional bodies
Test centre administrators
Materials writers
Publishers
etc
Test constructs
Test format
Test conditions
Test
scores
Test
assessment
criteria
Testing System
Inputs to test design
Learners
Parents/carers
Teachers
Receiving
institutions
Employers
School owners
Examiners
Government
agencies
Professional
bodies
Academic
researchers
Test writers/
Examiners
etc
Contexts of test use - consequences
20
Nick Saville, Cambridge ESOL
Impact
•
Multiple voices
• views, beliefs and attitudes
•
Important for examination providers
• to collect feedback from the stakeholder
• to take stakeholder perspectives into account
•
Dynamic relationship between micro and macro contexts
21
Nick Saville, Cambridge ESOL
3. The “law” of unintended consequences
•
“Any purposeful action will produce some unintended
consequences” or side-effects
•
“Goodhart’s Law” (or “Campbell’s Law” in the USA)
• a variant of the “law” of unintended consequences
22
Nick Saville, Cambridge ESOL
“Goodhart’s Law”
•
“All performance indicators lose their meaning when
adopted as policy targets”
•
Examples:
• England - school achievement targets - school league tables
• USA – No Child Left Behind (NCLB)
•
The clearer you are about what you want, the more
likely you are to get it – but the less likely it is to mean
what you wanted it to!
(Dylan Wiliam, 2008)
23
Nick Saville, Cambridge ESOL
Perverse incentives?
•
Assessment policy can create a tension between
•
educational objectives at the micro level (teaching
and learning in schools) and
•
a requirement for accountability at the macro level
24
Nick Saville, Cambridge ESOL
What role can examination providers play?
How can examination providers ensure that :
•
•
examination systems are “fit for purpose”?
•
mitigating action is are carried out if/when negative
effects and consequences occur?
research is carried out to “find out what is going on”
in contexts of use?
25
Nick Saville, Cambridge ESOL
4. Researching impact
•
Cambridge ESOL
• an international examinations board
•
Maxims of Test Impact
• An attempt to integrate an action-oriented approach
to investigating impact into working practices
• Milanovic and Saville, 1996
26
Nick Saville, Cambridge ESOL
Maxims for achieving/monitoring impact
Maxim 1 PLAN
Use a rational and explicit approach to test development
Maxim 2 SUPPORT
Support stakeholders in the testing process
Maxim 3 COMMUNICATE
Provide comprehensive, useful and transparent information
Maxim 4 MONITOR and EVALUATE
Collect all relevant data and analyse as required.
Milanovic and Saville, 1996 - Considering the impact of the Cambridge examinations
27
Nick Saville, Cambridge ESOL
Maxim 1 Plan
Use a rational and explicit approach to test development
•
•
a process model – cyclical and iterative
creates the necessary conditions for appropriate tests
to be developed and for on-going validation to take
place
•
begins with the purpose - including anticipating
• how the test should (or might) be used
• how relevant and useful it is likely to be
- social consequences and value implications
28
•
potential (unplanned) side-effects
Nick Saville, Cambridge ESOL
Maxim 2 Support
Support stakeholders in the testing process
•
Involvement of stakeholders - during test design and
development
• consultation on specifications/syllabus design
• participation in pilot tests
•
Professional support programmes
• for institutions and individual teachers/students etc.
•
Training of suitable personnel to work on all aspects of
the examination cycle
• item writers, examiners, etc.
29
Nick Saville, Cambridge ESOL
Stakeholders in the testing constituency
Learners
Teachers
Test writers/examiners
Receiving institutions
School owners
Future employers
Government agencies
Professional bodies
Test centre administrators
Materials writers
Publishers
etc
Test constructs
Test format
Test conditions
Test
scores
Test
assessment
criteria
Testing System
Inputs to test design
Learners
Parents/carers
Teachers
Receiving
institutions
Employers
School owners
Examiners
Government
agencies
Professional
bodies
Academic
researchers
Test writers/
Examiners
etc
Contexts of test use - consequences
30
Nick Saville, Cambridge ESOL
Maxim 3 Communicate
Provide comprehensive, useful and transparent information
•
Explain issues related to assessment adapted for the
different stakeholders
• the nature of the language constructs being assessed
• the meaning of language test results
• etc.
•
A major challenge for all test providers!
31
Nick Saville, Cambridge ESOL
Maxim 4 Monitor and Evaluate
Collect all relevant data and analyse as required
•
•
For example, feedback from the candidates and their teachers
•
from local contexts where the test is used (i.e. at the micro level)
A long-term endeavour as it involves:
•
•
the development of suitable instruments for the collection of adequate
data
appropriate research methodologies
e.g. mixed method designs, case studies etc.
•
Evaluate the test’s usefulness routinely
•
determine the need for changes and periodic revisions
32
Nick Saville, Cambridge ESOL
Outcomes of impact studies
•
The World-wide survey of the impact of IELTS
•
The Italian Progetto Lingue 2000 (PL2000
project)
•
See Hawkey, (2006)
•
Studies in Language Testing, 24
The theory and practice of impact studies: Messages from
studies of the IELTS test and Progetto Lingue 2000
33
Nick Saville, Cambridge ESOL
Progetto Lingue 2000
•
The policy was
•
“.... to introduce innovation into the teaching and learning of other
languages by putting greater emphasis on the development of
communicative competence in all grades of the school system”
•
It included:
•
the adoption of the Council of Europe’s Common European Framework
of Reference (CEFR) as the basis for learning objectives and standards
•
certification of proficiency – the testing
•
(by a certificating body recognised internationally)
34
Nick Saville, Cambridge ESOL
Students
Italy - PL2000 Impact Project
Parents
Students
Parents
Materials
Teachers
Teachers
Teachertrainers
Methodology
Curriculum
developers
Teachertrainers
Learning goals,
curriculum,
syllabus
Teacher Support
Curriculum
developers
Testers
Publishers
Testers
Publishers
Testing
Receiving
institutions
Employers
Receiving
institutions
Main interdependent language programme
stakeholders and dimensions
Employers
35
Nick Saville, Cambridge ESOL
5. Conclusion
•
Impact research now reflects the growing importance of
evidence-based approaches to educational planning
and evaluation
•
By adopting an “impact by design” approach - through
careful test development and validation strategies, and
by using the findings from impact research to guide
future actions - more effective assessment policies
and practices can be developed to meet the needs of
contemporary education
36
Nick Saville, Cambridge ESOL
Thank you
37
Nick Saville, Cambridge ESOL
38
Nick Saville, Cambridge ESOL
Descargar

Slide 1