DIAGNOSTIC TESTING – A TALE OF TWO INITIATIVES
Using diagnostic tests to assist in
(i) widening access and (ii) post hoc curriculum design
NAN YELD
The Alternative Admissions
Project
The Exit-level Testing
Project
To identify talented students
whose school results would not
qualify them for entry
To provide useful information
for curriculum design
To place these students into
appropriate programmes of study
To explore the effectiveness
(for entry level curriculum
design) of diagnostic
information on graduating
students
To deliver meaningful diagnostic
information for curriculum design
To assess the suitability of
using existing entrance
selection/diagnostic tests in
this attempt
Selected TIMSS 1999 performance indicators
Benchmark
Description of benchmark
% students reaching benchmark
Highest
performing
group (e.g.
Singapore,
Korea, Japan)
Lowest
performing
group (South
Africa,
Philippines, and
Morocco)1
Top 10%
Students can organise information,
make generalisations, and explain
solution strategies in non-routine
problem solving situations.
≥ 33 %
‹1%
Upper
quarter
Students can apply their understanding
and knowledge in a wide variety of
relatively complex situations.
± 66 %
1%
Median
Students can apply basic Mathematical
knowledge in straightforward
situations.
± 90 %
‹ 10 %
Lower
quarter
Students can do basic computations
with whole numbers.
± 99 %
31 %
1%
69 %
Percentage of students scoring below benchmark levels
Note 1: South Africa scored the lowest of this group.
South African School-Leaving Cohort
2001 (estimate)
failed
19%
rest of age
cohort
51%
certificate
22%
endorsement
8%
Percentage candidates in
aggregate categories
School-leaving examination data 2001 - black
candidates obtaining exemptions
80
70
60
50
40
30
20
23,430
10
0
7,260
2,310
40-50
50-60
≥ 60
Examination results in aggregate categories
Some factors impacting on test development
Issues to be considered
Consequences for test development
Affordability and feasibility Pen and paper
‘mark-ability’ format issues
Impact on system (e.g.
importance of not
undermining the Sen.
Certificate)
Tests not in school subject areas
Performance on SC remains
important
Impact on individuals
(importance of not creating
another barrier to access)
Writing of tests voluntary
Candidates assessed on EITHER
SC results OR project test results
The tests need to do
something different (avoid
‘predicting the past’)
Develop and implement innovative
ways of ‘getting at’ underlying
ability - scaffolding
The tests must not
The tests need to be able to
perpetuate the ‘revolving
inform curriculum development –
door’ tendency seen in many must have some diagnostic power.
access projects
In a higher education context,
students are required to:
• Make meaning from what they read;
• Understand and interpret conceptual and metaphorical
language;
• Identify and track academic argument;
• Follow discourse structure in text;
• Make inferences about and extrapolate on what they
read;
• Demonstrate familiarity with and understanding of the
conventions of visual literacies, such as reading and
interpreting graphs, flow-charts and diagrams; and
• Cope with basic numeracy demands.
Getting to grips with these requirements is seriously
impeded by approaches to texts and epistemic practice
such as:
• A tendency towards verbatim reproduction or plagiarism
in essays
• A tendency to describe rather than analyse, and to offer
tautologies in place of justification
• A tendency to focus on examples (tokens) rather than on
principles (types), and the relation between them
• A tendency to write from a highly subjective viewpoint
without depersonalising
• A tendency to be prescriptive or normative when asked
to be analytic
(Slonimsky and Shalem 2005)
Assessing the effectiveness of the scaffolding approach
Did the approach provide a greater range of scores (spread), so
that capable students could be more clearly differentiated from
weaker students?
Did the approach increase the predictive validity of the test (that
is, did the test correctly distinguish between weaker and stronger
students)?
Did the approach improve (raise) the level of stronger students’
scores?
ELPT and ALPT Scores
600
No. of Candidates
500
400
300
ELPT
ALPT
200
100
0
10
20
30
40
50
60
Test Results (% )
70
80
90
100
Task Preparation Study
Question 6.1: In note form (i.e. not full sentences), list the
main points the author makes about the Japanese educational
system.
For this exercise, do not include comparisons or
references to the American system.
Question 6.2: Drawing on the text of the whole article, but using
your own words, summarise the points the author makes about the
Japanese system of education. Note: your summary should not
include comparisons or references to the American system.
Points made in notes and
carried over to summary
Top 30% of
candidates
(n=114)
Bottom 30% of
candidates
(n=114)
.62
- 0.35
Semester-specific hazard rates for the Admissions
Project group (511) and the control group (563)
0.18
0.16
hazard rate
0.14
0.12
Admissions Group
0.1
Control group
0.08
0.06
0.04
0.02
0
6
12
18
24
semesters
30
36
42
% items at challenge levels
'Challenge levels' of papers
70
60
50
40
30
20
10
0
Level 1
Level 2
Level 3
Item challenge levels
1992 JMB
2001 1st National Paper
2003 2nd National Paper
ASSUMPTIONS UNDERLYING EXIT-LEVEL TESTING PROJECT
1 By and large, and with many notable exceptions,
academics …. do not take seriously/ are not well
informed about / are not very interested in ….
information about the learning needs of incoming
students.
2 Academics are more likely to take information about the
learning needs of their students seriously if these needs
are demonstrably still present in the students they are
about to graduate.
3 In order to demonstrate these needs, a valid and
credible (and feasible) instrument is needed.
1. By and large, academics …. do not take seriously/ are not well informed
about / are not very interested in …. information about the learning
needs of incoming students. Why might this be so?
• They believe it would be lowering standards to ‘pander to’ needs?
• They believe it’s the responsibility of the schooling sector to prepare
students, not theirs?
• They assume that massification will entail higher failure rates, it’s “only
natural”?
• They are really only interested in students who will be majoring in their
disciplines, and so it’s the survivors from the first-year shakedown that
they are focused on?
• Because doing so requires a shift from an inward looking, introjected
orientation, to an outward looking, ‘projective’ orientation (Barnett
2000), and a willingness to negotiate curricula across departmental
boundaries?
• The outsider status of those providing the information?
DESCRIPTIVE STATISTICS
Entering
Group
N = 95
Mean
Graduating
group
N = 67
36.9
45.1
1.3
1.2
Median
35.4
46.1
Mode
39.9
39.3
Standard Deviation
13.0
9.5
Minimum
11.8
27.0
Maximum
71.9
65.2
Range
60.1
38.2
Standard Error
Quintile boundaries: Entering and graduating
'disadvantaged schools' groups
60
Percentage
50
40
Entering
30
Graduating
20
10
0
1 (top)
2
3
4
Quintile lower boundaries
5
(bottom)
Skill cluster descriptions (examples)
Skill Assessed
Explanation of Skill Area
Extrapolation,
application and
inferencing
Students’ capacities to draw conclusions and apply
insights, either on the basis of what is stated in texts
or is implied by these texts.
Understanding the
communicative
function of
sentences
Students’ abilities to ‘see’ how parts of sentences /
discourse define other parts; or are examples of ideas;
or are supports for arguments; or attempts to persuade
Separating the
essential from the
non-essential
Students’ capacities to ‘see’ main ideas and supporting
detail; statements and examples; facts and opinions;
propositions and their arguments; being able to classify,
categorise and ‘label’
Understanding
basic numerical
concepts
Students’ abilities to make numerical estimations;
comparisons; calculate percentages and fractions; make
chronological references and sequence events /
processes; do basic computations
Cliff, A., Hanslo, M. and Visser, A-J. (2003). Faculty of Engineering and the Built Environment: AARP tests and
first-year students’ academic performance for the 2002/3 cycles. Unpublished report: University of Cape Town,
December 2003.
st
to
ta
l
%
ua
l
se
nt
i
N
um al
O
wn
er
ic
Vo
O
al
ic
wn
e
To
Vo
ta
ic
e
l
Co
O
nt
wn
Vo ent
O
ic
wn
e
O
Vo
rg
ic
e
O
Vo
wn
O
ca
wn
Vo
b
ic
Vo
e
ic
La
e
ng
M
ec
ha
ni
cs
Es
Vi
s
Vo
ca
M
b
et
ap
ho
Ex
r
tr
ap
ol
at
Se
e
nt
en
ce
Re
s
la
ti
on
s
Ge
nr
e
Te
Performance (%) on skills clusters
Skills clusters: graduating group 'disadvantaged
schools'
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
Skills clusters
REFERENCES
• Barnett, R. (2000). Supercomplexity and the curriculum.
Studies in Higher Education, 25, 255-2665.
• Cliff, A., Hanslo, M. and Visser, A-J. (2003). Faculty of
Engineering and the Built Environment: AARP tests and
first-year students’ academic performance for the 2002/3
cycles. Unpublished report: University of Cape Town,
December 2003.
• Slonimsky, L. and Shalem, Y. (2004). Pedagogic
responsiveness for academic depth. In Griesel, H.
(Ed.)(2004). Curriculum responsiveness – case studies
in higher education. South African Universities ViceChancellors’ Association: Pretoria, 81 - 101.
Descargar

www.brookes.ac.uk