Assessing Confidence in the Chinese Learner Stephen Bruce Napier University, Edinburgh Assessing Confidence in the Chinese Learner Chinese students at Napier University Rationale for confidence-based assessment Student responses to the test Academic self-concept, confidence and achievement research English Foundation Programme Set up in 2001 to prepare Chinese students for entry to Napier University, both in terms of linguistic and study skills. The needs of these students were complex, the “learning curve” for them was extremely steep, and that the demands they faced on a linguistically complex and culturally unfamiliar one-year Masters would be considerable, even given the requisite IELTS pass. Proportion of Chinese students on Napier Business Masters programmes has increased in recent years, which has resulted in staff and the institution facing a very different cohort with different strengths, and also needs, to their predecessors. Presents a major challenge to lecturers as they struggle to adapt their teaching approaches to suit both the increased number of overseas students and also the home students on their modules. Dr Lesley Gourlay, Napier University (2004). Crossing Boundaries: A Case Study Masters Level Chinese Students, LTSN (In Press). Linguistic and Cultural Issues Challenges for Students LANGUAGE Extreme difficulties in understanding / taking notes in lectures Difficulties with tutorial participation due to lack of confidence in English Some difficulties with tutorial tasks due to slow reading speeds Fear of failing exams due to difficulties in writing English in a time limit EDUCATIONAL CULTURE Feeling “lost” in semester 1 Some students unclear about expectations in UK-style coursework & reading Experience extremely stressful for some Lesley Gourlay, Napier University (firstname.lastname@example.org) Linguistic and Cultural Issues Challenges for Staff LANGUAGE Unsure if students are understanding lectures Required to spend extra time explaining lecture content at the end Some difficulties with tutorial participation and integration EDUCATIONAL CULTURE Some staff faced with numerous examples of plagiarism in coursework Some difficulties with tasks involving critical appraisal Some students struggled with application of theory to practice Lesley Gourlay, Napier University (email@example.com) Potential responses by the university LECTURE C O M P R E H E N S IO N & N O T E T A K IN G : TU TO R IAL P A R T IC IP A T IO N IS S U E S : PLAG IAR ISM AS LAN G U A G E CO PIN G STRATEGY: S P E C IF IC S E S S IO N S F O R IN T E R N A T IO N A L STUDENTS? S U P P O R T & ID E A S F O R LEC TU R ER S ? S P E C IF IC S E S S IO N S F O R IN T E R N A T IO N A L STUDENTS? M O R E O P P O R T U N IT IE S FOR S T U D E N T P R E P A R A T IO N ? S P E C IF IC S E S S IO N S F O R IN T E R N A T IO N A L STUDENTS? M O R E “U S E R -F R IE N D L Y ” P L A G IA R IS M A V O ID A N C E M A T E R IA L S ? GROUP R O T E -LE A R N IN G A S LAN G U A G E CO PIN G STRATEGY: STUDENT IS O LA T IO N & A N X IE T Y : IN C R E A S E D W O RKLO AD FO R STAFF: M O R E “B U D D Y IN G ” A N D IN T E G R A T IO N IN IT IA T IV E S ? M O R E IN T E R N A T IO N A L STUDENT C O U N S E L L IN G ? E X T R A T IM E G R A N T E D IN W O R K A L L O C A T IO N S ? R E D U C T IO N IN E X A M P R E D IC T A B L IT Y ? C L E A R P O L IC Y A B O U T GRAM M AR ERRORS AN D MARKS? Lesley Gourlay, Napier University (firstname.lastname@example.org) IN T E R N A T IO N A L A C A D E M IC S U P P O R T O F F IC E R S ? Confidence and achievement Chinese learners show a preference for rote-learning and memorisation. However their responses to learning strategy questionnaires indicate preference for deep learning. Learner self-esteem has been found to be positively associated with academic achievement (Brookover et al., 1964; Prendergast & Binder, 1975; Song & Hattie, 1984) Comparative studies support a view that the self-esteem of young Chinese students is lower than UK and American (Chan, 2000). – – – self-effacing and modest values in Chinese culture, strongly influenced by the Confucian tradition of a ‘humble’ character traditional authoritarian style of education or the highly competitive pressures created by schools, families and society gap in living standards in the UK and Chinese cultures Recent advances in academic self-concept and achievement research Assessing Confidence in the Chinese Learner This short pilot study is interested in two questions: 1. Would the Chinese students responses to a confidence-based test differ from UK students ? • 2. A good cultural test of this format Would their use of confidence levels in the test relate to their academic self-concept, or attitudes to academic life ? • Improving academic self-concept is often posited as mediating other desirable attributes Rationale for confidence-based scoring A student’s ability in answering may fall into a number of categories – I know it – I’m not quite sure, but I think I know it – Perhaps I can identify the answer by a deductive process on the distracters – If I guess I’ve a 25% chance of being correct (for 4 answer choices) – I really haven’t a clue and perhaps worst of all – I really know it …… what do you mean I’ve got it wrong !! Davies (2002) The standard MCQ cannot distinguish between the above – – – – possible to pass exams with knowledge only half learned or poorly understood the student certain of their knowledge should be rewarded the student should not be rewarded for guesswork encourage students to reflect on the reliability of their answers Would the student use this knowledge to make a decision or perform an action (usable knowledge) ? Hassmen and Hunt (1994) Confidence-based MCQ Select an answer in the usual way for a multiple choice question Indicate your confidence that you are correct. C=1 (low) Gardner-Medwin and Gahan (2003) C=2 (mid) C=3 (high) Scoring confidence-based MCQ What is my score if my answer is Confidence level C=2 (Mid) C=3 (High) Score when 1 2 3 Score when 0 -1 -4 < 50% >50% >75% Confidence level decision is governed by 2 judgements: o o ? C=1 (Low) Probability correct or estimated probability that the chosen answer will be correct the impact of the reward / benefit for a right / wrong answer Students rarely discuss their decisions in terms of explicit probabilities Gardner-Medwin and Gahan (2003) Encouraging good confidence judgement Students are encouraged to reflect on the reliability of their answers. Good confidence judgement means more marks ! If you are sure that your answer is correct …. o select C=2 (mid) or C=3 (high) confidence level get the marks your confidence deserves! If your are unsure (or guessing !) …. o select C=1 (low) confidence level don’t lose marks through misplaced confidence! Confidence-based MCQ feedback A 100% confidence-based score is 40 answers at C=2 (mid) confidence …. so that 40 correct at C=3 would be 150% A summary of confidence levels…. Gardner-Medwin and Gahan (2003) Confidence-based MCQ feedback Is the confidence-based score higher than the % correct score? No 1. Check the instances of INCORRECT answers at C=3 (high) and C=2 (mid). These can indicate areas of knowledge where you are misinformed. A large number of these perhaps indicates general overconfidence. 2. Check your number of CORRECT answers at C=1 (low). A large number indicates that you know more than you are willing to admit and are perhaps underconfident. Confidence-based MCQ feedback Is the confidence-based score higher than the % correct score? Yes You have shown good confidence judgement! You are willing to express an appropriate level of confidence when considering your knowledge. This is important when making decisions and performing actions based on that knowledge. The student group 31 Chinese students (54% sample) o o o Current on English as a Foreign Language (EFL) Foundation Programme o o 35 weeks of 20 hours of EFL study Gateway and Starter routes dependant upon English proficiency Students recently completed their IELTS exam (International English Language Testing System). o o 18 male, 13 female 24 postgraduate and 7 undergraduate Ages: 18-25 (19), 26-32 (8), 33-40 (4) Entry on to a Napier Programme conditional on a student’s IELTS score At the time the students did not know their results Students complete a confidence-based academic vocabulary test (40 Qs) Chinese student responses to confidence-based MCQ test No indications that Chinese students (red) behave differently from home based students (blue) o o Questions appear quite difficult, since low marks are more represented. 930 ‘Blue’ data from UCL including 40 first time responses. 150% Confidence-Based Mark 100% 50% 0% 20% 40% 60% -50% % correct 80% 100% Chinese student responses to confidence-based MCQ test No indications that Chinese students are reluctant to select high confidence levels (C=3) o Higher test scorers showed better confidence judgement o on average, high confidence was selected most often Lowest test-scorers tended to be over-confident No significant difference between genders observed. Proportion of confidence levels 40.0 37.7 30.0 33.4 29.0 % 20.0 10.0 0.0 C=3 C=2 C=1 Chinese student responses to confidence-based MCQ test Instances of C=3 (high) confidence increased as the question difficulty decreased. ‘Difficult’ language comprehension test as some small grammatical differences separated answers and distracters. C = 3 (high) 30 No. of students 25 20 15 10 5 0 0.0 Harder 0.2 0.4 0.6 Question difficulty 0.8 1.0 Easier Correlating confidence-based score and final exam (IELTS) Achievement in test scores Confidence-based score IELTS exam IELTS exam 0.91 (p=0.000) insignificant insignificant - only 7 students (of 31) improved on their % correct score with good confidence judgement. o % correct 6 of these students were the highest test scorers in the class. no apparent correlation between IELTS exam and confidence-based test o IELTS exam measuring larger scope of competencies Student opinions of the test It is appropriate for a test to measure the confidence I have in my knowledge 18 Strongly agree Agree Neutral Disagree Strongly disagree No. of students 1. 2. 3. 4. 5. 15 12 9 6 3 0 1 2 3 4 5 Relative agreem ent I sometimes took a risk by selecting a higher level of confidence than I really felt I thought the confidence-based test gave me valuable feedback about my knowledge 18 10 No. of students No. of students 12 8 6 4 2 15 12 9 6 3 0 0 1 2 3 Relative agreement 4 5 1 2 3 Relative agreement 4 5 Student comments “It is a real good interesting test and I can receive different aspects of my knowledge. However, it will cost lots of time to finish this exam. As a result, I suggest we can do the exam at a regular time such an once a week.” “It is an interesting test and I would like to do it at my university. I think that is obviously fair for students.” “It is interesting and a little difficult to understand the result. I think I should get a higher confidence-based score because I chose (c3) nine times, and the number answered correct is 7. Maybe I have not understood this well.” “I think it is quite interesting and helpful.it is also a good way to show me the link between confidence and academic study.” Academic self-concept and confidence judgement Improving academic self-concept is often posited as mediating other desirable attributes such as persistence on academic tasks, motivation, and self-efficacy. Is good confidence judgement a measure and a positive mediator o o The locus of control refers to how people explain events that happen to themselves and others. o o o an improved measure than the standard % correct score informed interventions for enhancing academic self-concept can make use of recent advances in theory Craven (1996) internal locus - guided by his/her personal decisions and efforts. external locus - guided by fate, luck, or other external circumstances a more 'surface' approach to learning is associated with an external locus of control, Watkins & Biggs (1996) Scoring scheme & feedback helps to equate internal expectation (probability answered correctly) with external performance (score) o desirable attribute that is associated with an internal locus of control A model for self-concept Posited to be multifaceted and hierarchical in nature Shavelson, Hubner and Stanton (1976) Numerous studies now based or extended from this model Hattie (1992) Self-concept and academic achievement From the literature… Support for the multifaceted nature…. o academic self-concept and academic achievement strongly correlated Marsh, Byrne and Shavelson (1988), Hattie (1992) o little correlation between social self-concepts and academic achievement Song & Hattie (1984), Waugh (1999) Support for the hierarchical nature less clear cut…. o more support for a hierarchical model for adolescents and a unitary structure for younger children Hattie (1992) o achievement and academic self-concept deserve special interpretations at the level of specific subjects Marsh (1990) o standard self-description questionnaires based on the Shavelson model for preadolescents, adolescents and late adolescents Marsh (1992a, b, c) Measuring academic self-concept 1. Your perceptions of your ability and achievement adapted from Marsh (1992), Song & Hattie (1984) and Waugh (2001) indicate your relative agreement with each of the 20 statements: All the time, or nearly all the time Most of the time Some of the time None of the time, or almost none of the time 1st 10 general university experience eg. I am capable of getting good marks at university I am proud of my achievements at university 2nd 10 experience of English Language classes only eg. I am sure of myself in English Language classes I am achieving at a high level in English Language classes Questionnaire feedback For your general university experience, your responses total 18 points (30 maximum). For your English Language classes experience, your responses total 19 points (30 maximum). Measuring academic self-concept 2. Your attitudes to academic life locus of control inventory adapted from Trice (1985) select True or False to indicate your agreement with each statement: eg. My academic marks most often reflect the effort I put into classes. I came to university because it was expected of me Questionnaire feedback The closer your score is to 0, the more you believe that your academic experience is determined by your own abilities, efforts and attitudes (internal factors). The closer your score is to 28, the more you believe that your academic experience is determined by external factors such as chance, other people, fate or luck. Correlating academic self-concept, confidence and achievement Questionnaire responses Perception of ability & achievement EFL Locus of control General EFL 0.61 (p=0.000) - - 0.54 (p=0.002) - 0.61 (p=0.000) perception of ability and achievement scores (general and EFL scales), increase in tandem. as locus scores become increasingly internal (towards 0), perception of ability and achievement scores increase accordingly Gender observations in the questionnaire responses The responses indicating the most internal of locus of control scores, were male students. Perception of ability & achievement - Generally 10 No. of students Scores from questionnaire: Females tended to respond with lower scores than males for the perception of ability and achievement (general and EFL) scales. 8 6 Male 4 Female 2 0 10-14 15-19 20-24 25-30 Range of scores (max is 30) Locus of control Perception of ability & achievement - EFL 6 8 6 Male 4 Female 2 No. of students No. of students 10 5 4 3 Male 2 Female 1 0 0 0-7 10-14 15-19 20-24 Range of scores (max is 30) 25-30 8-11 12-14 15-18 Range of scores (0 <------ internal : external ------> 28) 19-22 Correlating academic self-concept, confidence and achievement Questionnaire response v. IELTS exam Perception of ability & achievement Locus of controls General EFL insignificant insignificant 0.43 (p=0.024) Student’s perception of ability and achievement in EFL shows a moderate correlation with their IELTS exam result. o in line with academic self-concept and achievement research – strongly subject based: support for the hierarchical structure. No significant gender differences in the confidence-based test scores. Females attained higher scores in the IELTS exam. Correlating academic self-concept, confidence and achievement Questionnaire responses v. confidence profile Perception of EFL ability Perception of General ability at C=3 (high) 0.35 (p=0.052) insignificant at C=2 (mid) insignificant -0.37 (p=0.038) at C=1 (low) Locus of control insignificant insignificant Perception of EFL ability & achievement scores are moderately correlated with the number of correct answers at high confidence. These figures may indicate a trend, but may be noise o one confidence-based test is insufficient to identify genuine correlations Conclusions Indications that Chinese students perform in a similar manner to confidence-based assessment as their UK counterparts. o Provide a scheduled series of confidence-based tests for Chinese students on the English Foundation programme Examine academic self-concept and confidence-based assessment o o o potentially valuable role for the enhancement of academic self-concept, and the development of other positive academic behaviours Effective with younger learners who may enjoy a game perspective The scoring scheme and test feedback seems applicable to recent research that posits a reciprocal relationship between academic self-concept and academic achievement (Marsh, 2003). Acknowledgments Lecturing staff in the Centre for Business Languages, Napier University Nicola Beasley (Napier University) for initial software development Tony Gardner-Medwin (UCL) for valued discussions and analysis Bibliography 1. Brookover, W.B., Thomas, S. and Paterson A. (1964) Self-concept of ability and school achievement, Sociology of Education, 37, pp. 271-279. 2. Craven, R. (1996) Enhancing Academic Self-Concept: A Large-Scale Longitudinal Study in an Educational Setting, PhD thesis, University of Sydney 3. Davies, P. (2002) There’s no confidence in multiple-choice testing, Proceedings of the 6th International CAA conference, Loughborough, pp. 119-130. 4. Echternacht, G.J. (1972) The use of confidence testing in objective tests, Review of Educational Research, 42:2, pp. 217-237. 5. Gardner-Medwin, A.R. and Gahan M. (2003) Formative and summative confidence-based assessment, Proceedings of the 7th International CAA conference, Loughborough, pp. 147-155. 6. Gourlay, L. (2004) Crossing Boundaries: A Case Study Masters Level Chinese Students, LTSN (In Press) 7. Hassmen, P. and Hunt, D.P. (1994) Human self-assessment in multiple-choice testing, Journal of Educational Measurement, 31, pp. 149-160. Bibliography 8. Hattie, J. (1992) Self-Concept, Lawrence Erlbaum Associates, New Jersey 9. Khan, K.S., Davies, D.A. and Gupta, J.K. (2001) Formative self assessment using multiple true-false questions on the Internet: feedback according to confidence about correct knowledge. Medical Teacher, 23, pp. 158-163. 10. Marsh, H.W (1992a) Self-Description Questionnaire (SDQ)III: A theoretical and empirical basis for the measurement of multiple dimensions of preadolescent selfconcept: A test manual and research monograph, MacArthur, New South Wales, Australia, University of Western Sydney, Faculty of Education. 11. Marsh H.W., Byrne, B.M. and Shavelson, R.J. (1988) A multifaceted academic self-concept; Its hierarchical structure and its relation to academic achievement. Journal of Educational Psychology, 80, 366-380. 12. Marsh H.W. (1990b) The structure of academic self-concept: The Marsh/Shavelson model, Journal of Educational Psychology, 82, 623-636. 13. Marsh, H.W. (2003) A reciprocal effects model of the causal ordering of academic self-concept and achievement, Paper presented at NZARE AARE, Auckland, New Zealand. Bibliography 14. Prendergast, M.A. and Binder D.M. (1975) Relationships of selected self-concept and academic achievement measures, Measurement and Evaluation Guidance, 8, pp. 92-95. 14. Song, I.S. and Hattie, J. (1984) Home environment, self-concept and academic achievement measures, Journal of Educational Psychology, 76, pp. 1269-1281. 15. Watkins, D.A. and Biggs, J.B. (Eds, 1996) The Chinese Learner: Cultural, Psychological and Contextual Influences, CERC & ACER, (ISBN 0 86431 182 6). 16. Waugh, R.F. (2001) Measuring ideal and real self-concept on the same scale, based on a multifaceted, hierarchical model of self-concept, Educational and Psychological Measurement, 61:1, pp. 85-101.