DIAGNOSTIC TESTING – A TALE OF TWO INITIATIVES Using diagnostic tests to assist in (i) widening access and (ii) post hoc curriculum design NAN YELD The Alternative Admissions Project The Exit-level Testing Project To identify talented students whose school results would not qualify them for entry To provide useful information for curriculum design To place these students into appropriate programmes of study To explore the effectiveness (for entry level curriculum design) of diagnostic information on graduating students To deliver meaningful diagnostic information for curriculum design To assess the suitability of using existing entrance selection/diagnostic tests in this attempt Selected TIMSS 1999 performance indicators Benchmark Description of benchmark % students reaching benchmark Highest performing group (e.g. Singapore, Korea, Japan) Lowest performing group (South Africa, Philippines, and Morocco)1 Top 10% Students can organise information, make generalisations, and explain solution strategies in non-routine problem solving situations. ≥ 33 % ‹1% Upper quarter Students can apply their understanding and knowledge in a wide variety of relatively complex situations. ± 66 % 1% Median Students can apply basic Mathematical knowledge in straightforward situations. ± 90 % ‹ 10 % Lower quarter Students can do basic computations with whole numbers. ± 99 % 31 % 1% 69 % Percentage of students scoring below benchmark levels Note 1: South Africa scored the lowest of this group. South African School-Leaving Cohort 2001 (estimate) failed 19% rest of age cohort 51% certificate 22% endorsement 8% Percentage candidates in aggregate categories School-leaving examination data 2001 - black candidates obtaining exemptions 80 70 60 50 40 30 20 23,430 10 0 7,260 2,310 40-50 50-60 ≥ 60 Examination results in aggregate categories Some factors impacting on test development Issues to be considered Consequences for test development Affordability and feasibility Pen and paper ‘mark-ability’ format issues Impact on system (e.g. importance of not undermining the Sen. Certificate) Tests not in school subject areas Performance on SC remains important Impact on individuals (importance of not creating another barrier to access) Writing of tests voluntary Candidates assessed on EITHER SC results OR project test results The tests need to do something different (avoid ‘predicting the past’) Develop and implement innovative ways of ‘getting at’ underlying ability - scaffolding The tests must not The tests need to be able to perpetuate the ‘revolving inform curriculum development – door’ tendency seen in many must have some diagnostic power. access projects In a higher education context, students are required to: • Make meaning from what they read; • Understand and interpret conceptual and metaphorical language; • Identify and track academic argument; • Follow discourse structure in text; • Make inferences about and extrapolate on what they read; • Demonstrate familiarity with and understanding of the conventions of visual literacies, such as reading and interpreting graphs, flow-charts and diagrams; and • Cope with basic numeracy demands. Getting to grips with these requirements is seriously impeded by approaches to texts and epistemic practice such as: • A tendency towards verbatim reproduction or plagiarism in essays • A tendency to describe rather than analyse, and to offer tautologies in place of justification • A tendency to focus on examples (tokens) rather than on principles (types), and the relation between them • A tendency to write from a highly subjective viewpoint without depersonalising • A tendency to be prescriptive or normative when asked to be analytic (Slonimsky and Shalem 2005) Assessing the effectiveness of the scaffolding approach Did the approach provide a greater range of scores (spread), so that capable students could be more clearly differentiated from weaker students? Did the approach increase the predictive validity of the test (that is, did the test correctly distinguish between weaker and stronger students)? Did the approach improve (raise) the level of stronger students’ scores? ELPT and ALPT Scores 600 No. of Candidates 500 400 300 ELPT ALPT 200 100 0 10 20 30 40 50 60 Test Results (% ) 70 80 90 100 Task Preparation Study Question 6.1: In note form (i.e. not full sentences), list the main points the author makes about the Japanese educational system. For this exercise, do not include comparisons or references to the American system. Question 6.2: Drawing on the text of the whole article, but using your own words, summarise the points the author makes about the Japanese system of education. Note: your summary should not include comparisons or references to the American system. Points made in notes and carried over to summary Top 30% of candidates (n=114) Bottom 30% of candidates (n=114) .62 - 0.35 Semester-specific hazard rates for the Admissions Project group (511) and the control group (563) 0.18 0.16 hazard rate 0.14 0.12 Admissions Group 0.1 Control group 0.08 0.06 0.04 0.02 0 6 12 18 24 semesters 30 36 42 % items at challenge levels 'Challenge levels' of papers 70 60 50 40 30 20 10 0 Level 1 Level 2 Level 3 Item challenge levels 1992 JMB 2001 1st National Paper 2003 2nd National Paper ASSUMPTIONS UNDERLYING EXIT-LEVEL TESTING PROJECT 1 By and large, and with many notable exceptions, academics …. do not take seriously/ are not well informed about / are not very interested in …. information about the learning needs of incoming students. 2 Academics are more likely to take information about the learning needs of their students seriously if these needs are demonstrably still present in the students they are about to graduate. 3 In order to demonstrate these needs, a valid and credible (and feasible) instrument is needed. 1. By and large, academics …. do not take seriously/ are not well informed about / are not very interested in …. information about the learning needs of incoming students. Why might this be so? • They believe it would be lowering standards to ‘pander to’ needs? • They believe it’s the responsibility of the schooling sector to prepare students, not theirs? • They assume that massification will entail higher failure rates, it’s “only natural”? • They are really only interested in students who will be majoring in their disciplines, and so it’s the survivors from the first-year shakedown that they are focused on? • Because doing so requires a shift from an inward looking, introjected orientation, to an outward looking, ‘projective’ orientation (Barnett 2000), and a willingness to negotiate curricula across departmental boundaries? • The outsider status of those providing the information? DESCRIPTIVE STATISTICS Entering Group N = 95 Mean Graduating group N = 67 36.9 45.1 1.3 1.2 Median 35.4 46.1 Mode 39.9 39.3 Standard Deviation 13.0 9.5 Minimum 11.8 27.0 Maximum 71.9 65.2 Range 60.1 38.2 Standard Error Quintile boundaries: Entering and graduating 'disadvantaged schools' groups 60 Percentage 50 40 Entering 30 Graduating 20 10 0 1 (top) 2 3 4 Quintile lower boundaries 5 (bottom) Skill cluster descriptions (examples) Skill Assessed Explanation of Skill Area Extrapolation, application and inferencing Students’ capacities to draw conclusions and apply insights, either on the basis of what is stated in texts or is implied by these texts. Understanding the communicative function of sentences Students’ abilities to ‘see’ how parts of sentences / discourse define other parts; or are examples of ideas; or are supports for arguments; or attempts to persuade Separating the essential from the non-essential Students’ capacities to ‘see’ main ideas and supporting detail; statements and examples; facts and opinions; propositions and their arguments; being able to classify, categorise and ‘label’ Understanding basic numerical concepts Students’ abilities to make numerical estimations; comparisons; calculate percentages and fractions; make chronological references and sequence events / processes; do basic computations Cliff, A., Hanslo, M. and Visser, A-J. (2003). Faculty of Engineering and the Built Environment: AARP tests and first-year students’ academic performance for the 2002/3 cycles. Unpublished report: University of Cape Town, December 2003. st to ta l % ua l se nt i N um al O wn er ic Vo O al ic wn e To Vo ta ic e l Co O nt wn Vo ent O ic wn e O Vo rg ic e O Vo wn O ca wn Vo b ic Vo e ic La e ng M ec ha ni cs Es Vi s Vo ca M b et ap ho Ex r tr ap ol at Se e nt en ce Re s la ti on s Ge nr e Te Performance (%) on skills clusters Skills clusters: graduating group 'disadvantaged schools' 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 Skills clusters REFERENCES • Barnett, R. (2000). Supercomplexity and the curriculum. Studies in Higher Education, 25, 255-2665. • Cliff, A., Hanslo, M. and Visser, A-J. (2003). Faculty of Engineering and the Built Environment: AARP tests and first-year students’ academic performance for the 2002/3 cycles. Unpublished report: University of Cape Town, December 2003. • Slonimsky, L. and Shalem, Y. (2004). Pedagogic responsiveness for academic depth. In Griesel, H. (Ed.)(2004). Curriculum responsiveness – case studies in higher education. South African Universities ViceChancellors’ Association: Pretoria, 81 - 101.