Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC 12 February 2013 Dr Mafu S Rakometsi WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to context, principles, approaches and processes Dr Mafu Rakometsi - CEO of Umalusi Regulatory Framework Quality Assurance of Assessment • NQF Act Section 27 (h) The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework. Section17 of the GENFETQA Act (5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has— (i) conducted the assessment free from any irregularity that may jeopardise the integrity of the assessment or its outcomes; (ii) complied with the requirements prescribed by the Council for conducting assessments; (iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and (iv) complied with every other condition determined by the Council. Framework for QA of Learner Achievement • Based on established and existing practices in assessment for certification Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment • Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies. Framework for Quality Assurance of Assessment Evaluation and /or accreditation of assessment bodies Periodic inspection of assessment systems Ongoing monitoring of assessment systems Quality assurance of external examinations through: Moderation of examination question papers Monitoring and moderation of SBA Monitoring the conduct of examinations Moderation of marking Standardization of assessment outcomes Approval for the release of Results Approval is based on the following requirements: • The examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations • At the time of approval, there is no serious irregularity which could undermine the credibility of the examinations. Quality Assurance of the DBE 2012 National Senior Certificate Examination Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments Moderation of question papers PURPOSE To ensure that the question papers are of the required standard (standard captured in the NCS and SAG’s) To ensure that the question papers are relatively: - fair - reliable - representative of an adequate sample of the curriculum - representative of relevant conceptual domains - representative of relevant levels of cognitive challenge External moderators Moderation of the question papers Approach Question papers set by panel of examiners – DBE Internally moderated by DBE Externally moderated by Umalusi Subsequent moderations and approval. Moderation of Question papers Number of NSC 2012 question papers moderate Number of subjects Number of papers Approved/ Conditionally approved at 1st moderation Approved/ Conditional ly approved at 2nd moderation Approved/ Conditiona lly approved at 3rd moderatio n 4th moderatio n and beyond 62 (including the nonofficial languages and LO) Nov 2012 -132 27.3% (36) 56.8% (75) 15.2%(20) 0.7% (1) Mar 2013 -130 27,7% (36) 54.6 %(71) 13.9% (18) 3.8% (5) Moderation of the question papers Criteria Technical Criteria Internal moderation – quality of internal moderator reports. Content coverage Cognitive Demand Marking Guidelines Language and Bias Adherence to Policies and guidelines Predictability Moderation of the question papers Findings: Areas of Good Practice Percentage of question papers and memoranda approved after first and second moderation ( Nov 2012- 79,6% ; Mar 13 – 72,3%) Simultaneous moderation of final and supplementary question papers. Standard of Internal moderation Moderation of the question papers Findings: Areas of Concern Adherence to timeframes and impact on quality of setting and moderation. Question papers requiring more than four moderations. Role of the external moderator. Moderation of internal /continuous assessment Internal assessment refers to any assessment conducted by the provider, the outcomes of which count towards the achievement of the qualification panels of moderators / subject specialists Moderation of internal assessment (cont.) Purpose of Umalusi’s verification: To verify the rigour and appropriateness of the DBE moderation process – linked to DBE plans Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across Ascertain the standard and quality of the tasks Determine the extent and quality of internal moderation and feedback. Determine the reliability and validity of the assessment outcomes Moderation of Internal Assessment Phase 1 (June/July 2012)- focus on assessment instruments (teacher files) PROVINCIAL EDUCATION DEPARTMENT Subjects Eastern Cape KwaZulu- Natal Limpopo Mpumalanga Mathematics, Life Sciences, Physical Sciences Accounting, English FAL, History, Geography, Life Orientation, Moderation of Internal Assessment Phase 1: Findings • Quality and Standard of Assessed tasks Content Coverage – mostly appropriate Cognitive demand and difficulty levels –standards variable mainly at lower cognitive levels, Over reliance on previous question papers. Projects and practical investigations- not cognitively balanced, more theory than practice. Marking tools – not applied consistently Teachers subject knowledge often inadequate Physical Education Task (PET) inappropriately assessed. Moderation of Internal Assessment Phase 1: Findings • Internal Moderation Moderation instruments – compliance vs. quality School moderation – seriously neglected Monitoring vs. Moderation • Feedback and Support – general support by subject advisors. Very little developmental feedback. Moderation of Internal Assessment Phase 1: Areas of Good Practice • Use of Common Tasks • Diagnostic Analysis to improve performance • Compliance with Internal Assessment protocols. Moderation of Internal Assessment Phase 2 (October/November 2012)- focus on learner evidence and teacher files. PROVINCIAL EDUCATION DEPARTMENT Subjects All 9 PED’s Mathematics, Life Sciences, Physical Sciences Accounting, English FAL, History, Geography, Life Orientation, Moderation of Internal Assessment Phase 2: Findings • Quality and Standard of Moderation Brief unclear – focus on marking only and not on quality of task. Shadow marking vs. moderation • Marking of the Tasks Marking tools – not applied consistently Teachers subject knowledge often inadequate Challenges with marking extended writing and research projects. Marking too lenient Moderation of Internal Assessment Phase 2: Findings • Learner performance Difficulty with higher-order questions Ability to express/respond in English a determining factor Extremely poor performance in certain subjects e.g. Maths, LO Common Assessment Task, Evidence suggests that very little effective teaching and learning is taking place Moderation of Internal Assessment Phase 2: Findings • Internal Moderation at School, Cluster and District Levels. Varying quality of Moderation Limited moderation of candidates work – shadow marking Little evidence of moderation of tasks. Lack of mark sheets or incorrect recording of marks. Moderation of Internal Assessment Phase 2: Areas of Good Practice • Use of Common Tasks • Diagnostic Analysis to improve performance • Compliance with Internal Assessment protocols. • Evidence of moderation at the various levels. • Introduction of National Moderation Monitoring of Examinations • State of readiness • Conduct of examinations • Marking State of Readiness Monitored the DBE audit visits (March/April) and evaluation visits to all Comprehensive approach that monitors exam systems (including SBA) Monitoring of Examinations State of readiness Findings • All provinces have examination systems in place • Vacant posts and use of contract staff is a major concern. Monitoring of the writing phase. No of exam centres No of exam centers No of Umalusi monitors monitored by Umalusi per province 6623 207 36 Findings • General Management of the examinations Generally examinations conducted in line with policy Isolated instances of non-compliance (suitability of venue, noise, security, contingency plans) Shortage of question papers and answer books Provincial Monitoring Monitoring of the marking phase Total No of marking centers (2010) No of marking centers (2011) No of marking centers (2012) No of marking centers monitored by Umalusi 127 123 118 87 Monitoring of the marking phase Findings • In general marking processes in line with policy • Concern with lack of security at some marking centres • Concern with the appointment and training of markers • Concern with the quality and standard of marking and internal moderation. • Number of marking centres – security risk Verification of marking PURPOSE • • Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices Umalusi engages the following during the moderation of marking 1. Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres 2. Moderation of marking (off-site and on-site) Verification of marking Memoranda discussion meetings Approved and signed off finalized memoranda/ marking guidelines for all the NSC subjects Concern with non attendance and pre- memo discussion preparations Concern with timing and scheduling of memo discussions. Practice of marking and training in the memo discussions impacts positively on standard of marking. Verification of marking Provinces Subjects Centralised All 9 Gateway subjects + 4 others On-site Eastern Cape; Gauteng; KZN; Limpopo; Northern Cape 13 in total Findings Findings • Adherence to Marking memo • Marking generally accurate – problem with marking extended writing and essays Verification of marking • Quality of marking varied. Question marking improved consistency. Competency of markers to mark interpretation type questions. Internal moderation No unauthorised changes to marking guidelines reported Training of chief markers and internal moderators at memo discussions having a positive impact on marking. Standardisation and Verification of Resulting • Provision of GENFETQA – Council may adjust raw marks. • International practice – large scale assessment systems • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance. • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions Statistical moderation Standardisation 58 subjects standardised Raw marks accepted: 41 subjects Moderated Upward : 5 subjects Moderated Downward : 12 subjects Verification of the Resulting Process Planned Subject structures Candidate registration Status Subject structures verified. System verified during state of readiness visits. Generation of mark Sheets Capturing of Marks Standardisation data & Booklets Capturing of adjustments Monitored Monitored Data sets received and verified Verified Statistical moderation & resulting Verified Examination Irregularities The majority of irregularities were of a technical nature and these were reported to Umalusi according to the established channels. Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects. Umalusi represented on NEIC Major irregularities: Alleged leakage of Question Papers in KZN WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2012 SUBJECT Life Sciences Physical Sciences Mathematics Mathematical Literacy Mathematics: Probability; Data Handling ADJUSTMENT CA( 0 at 0 scale to -8 at 23. scale to 0 at 70, scale to +3 at 119, scale to 0 at 171, scale to -7 at 239, scale to -1 at 291, scale to -7 at 300) 0 at 0, scale to -12 at 102, block -12 from 102 to 252, scale to 0 at 300 Raw 0 at 0, raw to 114, scale 0 to +9 from 114 to 240, scale to 0 at 300 Raw SUBJECT ADJUSTMENT Geography 0 at 0, scale to -6 at 96, block -6 from 96 to 180, scale to 0 at 240, raw to 300 Life Orientation Hospitality Studies Raw Raw Tourism ½ CA( 0 at 0, scale to -12 at 121, scale to 0 at 300) Agricultural Science Consumer Studies Raw Raw SUBJECT Agricultural Management Practices Raw Agricultural Technology Raw Music Raw Dance Studies Raw Design 0 at 0, scale to -12 at 102, scale to -6 at 246, scale to 0 at 300 Dramatic Arts ADJUSTMENT ½ CA(o at o, scale to -15 at 66, scale to 0 at 300) SUBJECT Visual Arts ADJUSTMENT Raw Religion Studies Raw Information Technology Raw Computer Applications Technology Raw Civil Technology Electrical Technology 0 at 0, scale to -9 at 90, block -9 from 90 to 180, scale to 0 at 300 ½ CA(0 at 0, scale to -15 at 211, scale to 0 at 300) SUBJECT ADJUSTMENT Mechanical Technology Raw Engineering Graphics and Design Accounting Business Studies Economics Raw 0 at 0, scale to -9 at 78, block 9 from 78 to 222, scale to 0 at 300 Raw Raw SUBJECT ADJUSTMENT History Raw Afrikaans HL 0 at 0, scale to +4 at 116, block of +4 from 116 to 247, scale to 0 at 300. Afrikaans FAL Afrikaans SAL Raw Raw English HL 0 at 0, scale to +8 at 112, block of +8 from 112 to 240, scale to 0 at 300. SUBJECT English FAL ADJUSTMENT 0 at 0, scale to -6 at 51, block of -6 from 51 to 189, scale to 0 at 300. English SAL Raw IsiNdebele HL Raw IsiNdebele FAL IsiZulu FAL Raw 0 at 0, scale to +8 at 112, block of +8 from 112 to 232, scale to 0 at 300. Raw IsiZulu SAL Raw IsiZulu HL SUBJECT ADJUSTMENT Setswana HL Raw Setswana FAL Raw Setswana SAL Siswati HL Raw Raw Siswati FAL Raw Siswati SAL Raw IsiXhosa HL Raw IsiXhosa FAL Raw IsiXhosa SAL Raw SUBJECT Xitsonga HL ADJUSTMENT ½ CA (0 at 0, scale to -13 at 74, scale to 0 at 300) Xitsonga FAL Raw Tshivenda HL Tshivenda FAL ½ CA(0 at 0, scale to -12 at 115, block -12 from 115 to 153, scale to 0 at 300) Raw SUBJECT ADJUSTMENT Sepedi HL Sepedi FAL Raw Raw Sesotho HL Raw Sesotho FAL ½ CA(0 at 0, scale to -15 at 300) Sesotho SAL Raw Quality Assurance of the IEB 2012 National Senior Certificate Examination Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments Moderation of Question papers Number of IEB NSC 2012 question papers moderated Number of subjects Number of papers Approved/ Conditionally approved at 1st moderation Approved/ Conditional ly approved at 2nd moderation Approved/ Conditiona lly approved at 3rd moderatio n 4th moderatio n and beyond 39 Nov 2012 -69 39.7% (28) 54.4% (37) 4.4%(3) 1.5% (1) Moderation of the question papers Findings: Areas of Good Practice Percentage of question papers and memoranda approved after first and second moderation ( Nov 2012- 94.1% ) Standard of Internal moderation Moderation of Internal Assessment Phase 2 (November 2012)- focus on learner evidence and teacher files. IEB Subjects Mathematics, Maths Lit, Life Sciences, Physical Sciences Accounting, English HL, History, Life Orientation, Business Studies, Economics Moderation of Internal Assessment Findings: Areas of Good Practice • Regional and national moderation is in place and well executed • The marking of tasks of a highly acceptable standard Findings: Areas of Concern • Feedback to the learners • Assessment of Physical Education Tasks in LO – distance education learners Monitoring of the writing phase. Findings • General Management of the examinations Generally examinations conducted in line with policy Isolated instances of non-compliance (candidate identification) Lack of Irregularity Registers Assessment Body Monitoring Verification of marking 8 Subjects Findings • marking and moderation of scripts is extremely thorough • Marking generally accurate • practice of double marking is highly commendable • Moderation doesn’t always involve full moderation of a script Statistical moderation Standardisation 58 subjects standardised Raw marks accepted: 44 subjects Moderated Upward : 9 subjects Moderated Downward : 4 subjects WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS IEB NSC 2012 SUBJECT Accounting Business Studies Computer Application Technology Consumer Studies ADJUSTMENT 0 at 0, scale to -6 at 96; block 6 from 96 to 212, scale to -5 at 245, back to 0 at 300) ½ CA(0 at 0, scale to -15 at 139, block of -15 from 139 to 208, scale back to 0 at 300) ½ CA (Raw from 0 to 101, scaled from 0 at 101 to +7 at 219, scaled to 0 at 300) Raw Design Raw Dance Studies Raw SUBJECT Dramatic Arts Economics Engineering Graphics and Design Geography ADJUSTMENT Raw ½ CA(0 at 0, scaled to +10 at 45, block +10 from 45 to 102, scaled to 0 at 300) ½ CA ( 0 at 0, scale to +6 at 41, block of +6 from 41 to 79, scale to 0 at 300) CA(0 at 0, scale to +12 at 36, scale to -7 at 59,scale to -1 at 121, scale to +8 at 173, block +8 from 173 to 200, scale to 0 at 300) SUBJECT History ADJUSTMENT CA (0 at 0, scale to -18 at 56, scaled to -25 at 109, scaled to 0 at 300 Information Technology Raw Hospitality Studies Raw Tourism Raw Visual Arts ½ CA ( 0 at 0, scale to -15 at 89, block -15 from 89 to 222, scale to 0 at 300) Music Raw SUBJECT ADJUSTMENT Religion Studies Raw Life Orientation Raw Life Sciences Raw Mathematical Literacy Raw Mathematics Mathematics P3 Raw Raw SUBJECT Physical Science English FAL English HL Afrikaans FAL ADJUSTMENT ½ CA ( 0 at 0 scale to -7 at 28, scale to 0 at 68, block +1 from68 to 74, scale to 0 at110, scale to -7 at 241, scale to 0 at 300) ½ CA (0 at 0, scale to +15 at64, scale to +6 at 221, scale to +10 at 258, scale to 0 at 300) Raw ½ CA (0 at 0, scale to -10 at 147, block of -10 from 147 to 175, scale back to 0 at 300) SUBJECT ADJUSTMENT Afrikaans HL Raw Sepedi FAL Raw Sesotho FAL Raw Setswana FAL Raw Siswati FAL Raw Siswati HL Raw IsiXhosa FAL IsiZulu HL Raw Raw SUBJECT IsiZulu FAL IsiZulu SAL Arabic SAL ADJUSTMENT CA( 0 at 0, scale to -16 at 146, scale to -6 at 166, block -10 from 166 to 242, scale to 0 at 300) Raw French SAL Raw ½ CA( 0 at 0, scale to -13 at 51, scale to -1 at 174, scale to 5 at 206, block -5 from 206 to 279, scale to 0 at 300) German HL Raw German SAL Raw SUBJECT ADJUSTMENT Gujurati SAL Raw Hebrew SAL 0 at 0, raw to 180, scale to -6 at 246, scale to 0 at 300 Hindi FAL Raw Hindi SAL Raw Italian SAL Raw Latin SAL Raw SUBJECT ADJUSTMENT Modern Greek SAL Raw Portuguese HL Raw Portuguese SAL Raw Spanish SAL Raw Tamil SAL Raw Urdu FAL Raw Urdu SAL Raw Serbian HL Raw SUBJECT ADJUSTMENT Equine Studies* Raw Sport and Exercise Science* Raw Maritime Economics* Raw Nautical Science Raw Advanced Programme Mathematics Advanced Programme English ½ CA( Block +5) Raw Annexure to Approval Letter to DBE Specific concerns regarding the quality assurance of the NSC examination and assessment: Non-adherence to timeframes for the submission of question papers must be addressed. It is important to note that for the credibility of the NSC examination it is vitally important that every effort is made to adhere to agreed deadlines for setting and moderation of question papers. Any delays affect Umalusi in its quality assurance exercise. In addition to what is said above, every effort should be made to set question papers of an appropriate quality and standard so that they can be ideally approved at first moderation. Low levels of performance in key subjects. This issue need to be looked into and addressed so as not to compromise the standard of the NSC. • Umalusi is concerned with the problem in Life Orientation with awarding of marks at the 80% level where there is clear evidence of tampering with the marks. The DBE is urged to send a strong message of warning to the provinces indicating that should this problem persist in the future all learners at the 80% mark will be adjusted downward. The quality of School-Based Assessment (SBA) continues to be a concern. There may be several plausible reasons for this poor quality of SBA, but it is clear that there is too little control over work that is done. It is evident from our reports that progress has been made over the years with regard to the standard of marking. There is however room for improvement in the selection and training of markers, chief markers, senior markers and internal moderators. Competency tests will go a long way in the identification of competent markers. Conclusion The findings of the quality assurance processes are a clear indication of a maturing system that has, on the one hand, made positive strides towards improvement in certain areas of assessment and examination, but, on the other hand, still has a few challenges that need to be addressed. The quality assurance of each of these processes presented above was conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations. Conclusion… In general Umalusi is pleased with the manner in which the 2012 NSC examination was administered. Umalusi acknowledges that a number of technical irregularities were reported, but these were addressed in a fitting manner. Umalusi takes this opportunity to express appreciation to the national & provincial departments of education for their concerted effort in ensuring a credible examination. Umalusi expresses appreciation also to all the relevant stakeholders for the necessary support given in line with Umalusi quality assurance initiatives. Thank you!