ASSESSMENT AS SCHOLARSHIP:
IMPROVING PROGRAMS, INSTITUTIONS,
AND DISCIPLINES
Thomas P. Pusateri, Ph.D.
Florida Atlantic University
[email protected]
Closing Plenary Session
13th Annual Georgia Conference on
College & University Teaching
March 25, 2006
RESOURCES FROM JOSSEY-BASS
1990
1997
2001
2004
Scholarship
Reconsidered
Scholarship
Assessed
Scholarship
Revisited
Assessment
Clear and Simple
(Boyer)
(Glassick,
Huber, &
Maeroff
(Kreber)
(Walvoord)
ASSESSMENT IN THE NEWS
The Atlantic Monthly (November 2005)
• How College Affects Students (Pascarella & Terenzini)
Going to college makes a difference
But no differences among colleges, controlling for student quality
• Hersh: Current assessment measures may not be picking up differences
• Who will control the agenda for assessing the worth of college?
U.S. Congress: Reauthorization of the Higher Education Act
40 State legislators: FL Board of Governors–Academic Learning Compacts
Bush administration: Commission on the Future of Higher Education
• Hersh: Institutions ought to take charge to assess cumulative learning
Formative assessment: Carleton College (MN); Alverno College (WI)
Value-added assessment: Wabash College (IN) study of 16 institutions
ASSESSMENT AS SCHOLARSHIP
• This presentation focuses on assessment
for program improvement
– Accountability: Track individual students
– Program improvement: Aggregate the data
• Assessment is “action research” for
“informed intervention”
– Models from social science and education
Sampling; Pre-post differences;
Comparisons
– Quantitative and Qualitative data
GOALS OF THIS PRESENTATION
• Provide practical examples of
assessment that you might adapt at your
institution
– Examples from Loras College and FAU
• Suggest strategies for communicating
findings to colleagues (“Lessons
learned”)
– Graphing data; Conducting/Reporting audits
• Identify venues for presenting and
publishing research on assessment
– SoTL organizations, conferences, journals
EXAMPLE 1
MEASURING STUDENT ATTITUDES
• Loras College: Attitudes towards general education
– Faculty developed 41 outcomes statements
“Loras graduates should be able to….”
(“understand” or “appreciate” disciplines)
– Survey requested ratings on a 5-point scale from
“Extremely important” to “No importance”
– Most students completed the survey twice
Early in the first year and late in the senior year
• Comparison groups
– Longitudinal (First year – Senior) comparisons
Value added approach: Did students’ values change?
– Comparison groups: FTICs to Transfer students
Did completing four years at Loras matter?
1st
4th
DIFF
Items on the First-Year/
Senior Surveys
20: Understand foreign cultures
2.87
3.52
0.65
32: Skills in foreign language
2.60
3.24
07: Appreciate visual arts
2.51
11: Understand other cultures
Items on the First-Year/
Senior Surveys
1st
4th
DIFF
13: Skills in writing
4.20
4.57
0.37
0.64
38: Understand social structures
3.34
3.71
0.37
3.09
0.58
15: Understand human behavior
3.20
3.57
0.37
3.82
4.38
0.56
04: Understand scientific
method
2.97
3.33
0.36
12: Skills in speaking
3.65
4.21
0.56
30: Appreciate ancient culture
2.31
2.67
0.36
16: Open to controversies
3.89
4.43
0.54
05: Skills in persuasion
3.88
4.23
0.35
06: Volunteer service
3.03
3.52
0.49
03: Physical fitness & health
3.13
3.44
0.31
14: Appreciate literature
2.98
3.47
0.49
22: Understand biology's role
3.05
3.34
0.29
36: Appreciate musical styles
2.43
2.91
0.48
3.74
0.47
3.57
3.85
0.28
34: Understand/interpret history
3.27
18: Relate ethics/faith to
morality
29: Skills in reading
4.22
4.50
0.28
33: Understand chemistry’s role
2.72
2.99
0.27
08: Conduct library research
4.36
4.62
0.26
17: Skills in technology
4.39
4.63
0.24
26: Understand political systems
3.51
3.70
0.19
01: Evaluate others' opinions
3.80
4.26
0.46
24: Appreciate theatrical arts
2.91
3.36
0.45
39: Understand artists's works
2.37
2.81
0.44
40: Familiar with several fields
3.90
4.32
0.42
28: Skills in critical thinking
4.04
4.46
0.42
02: Work effectively with others
4.41
4.58
0.17
31: Interpret human behavior
3.47
3.89
0.42
10: Understand Catholicism
3.18
3.33
0.15
19: Concern for good of society
3.24
3.65
0.41
23: Skills in mathematics
3.49
3.63
0.14
27: Can reason scientifically
3.14
3.55
0.41
09: Appreciate mathematics
3.23
3.32
0.09
21: Open to different values
3.69
4.09
0.40
35: Understand economics
3.79
3.86
0.07
41: Understand historical events
3.50
3.89
0.39
37: Appreciate computers
4.02
4.05
0.03
LESSON 1
MEANS CAN BE MEANINGLESS
• Recommendation: Graph frequency distributions
– In Microsoft Excel, choose “100% Stacked Bar”
Based on 380 students who completed both versions of the survey.
Based on 441 non-transfers and 43 transfer students.
Differences in FTIC & Transfers
Differences in 1st year & Senior year
EXAMPLE 2
ASSESSING WRITING SKILLS
Loras College: Sophomore writing portfolios
• Phase 1: Assessment committee (2 summers)
– Collected portfolios (3 papers from each student)
– Read a subset of portfolios; wrote a scoring rubric
– Evaluated remaining portfolios (interrater reliability)
– Provided students feedback
• Phase 2: Faculty workshops (2 summers)
– Training on using the scoring rubric
– Evaluating portfolios (interrater reliability)
– Discussing findings and applying to courses
– Provided students feedback
Based on assessment by 21 faculty who read a subset of 49 student portfolios
LESSON 2
MISSED OPPORTUNITIES
Carleton College (MN): AAC&U News (December 2004)
Sophomore writing portfolio requirement
Students submit papers from at least 2 of 4 divisions
Faculty ratings: “Exemplary” “Pass” “Needs work”
14%
78% 8% (resubmit)
Evaluation criteria:
KEY WRITING SKILLS
QUALITY OF WRITING
Report on observation
Attend to audience
Analyze complex information
Clarity of prose
Provide interpretation
Clear organization
Use and document sources
Use of evidence
Articulate and support thesis
Distinctive voice
Appropriate diction
Control of error
LESSON 2
MISSED OPPORTUNITIES
REVISITED
University of South Carolina – Aiken
“Communicating Assessment Results Online to Promote
Curricular Change” at SACS-COC 2005 Annual Meeting
Rising Junior portfolio
WRITING OUTCOMES/EVALUATIVE RUBRIC
Clarity of Purpose
Quality of Thought
Organization of Content
Use of Sources
Language and Style
Grammar and Mechanics
LESSON 2
MISSED OPPORTUNITIES
RECLAIMED?
CARLETON COLLEGE LORAS COLLEGE
Attend to audience Creativity/Voice/Audience
and purpose
USC AIKEN
Clarity of purpose
Distinctive voice
Clarity of prose
Demonstrates critical
thinking
Quality of thought
Clear organization
Organizes the writing
Use of evidence
Supports ideas with
evidence
Organization of
content
Use of sources
Appropriate diction
Control of error
Uses standard English
and effective prose
Language and style
Grammar and
mechanics
EXAMPLE 3
USING STANDARDIZED TESTS
• Loras College: Assessment of General Education
CAAP Tests in selected sophomore-level courses
– Reading
– Essay Writing
– Science Reasoning
– Mathematics
• Loras College: Senior Majors in Psychology
– ETS Major Field Test in Psychology
LESSON 3
STANDARDIZED TESTS CAN BE USEFUL
(IF SELECTED & USED APPROPRIATELY)
• Comparisons to other institutions
Essay writing: Provided additional evidence for faculty
concerns
Mathematics: Supported the math department’s
recommendation for a curriculum change
Science reasoning & Reading: Added little of value
Psychology: Provided useful information concerning
the quality of content knowledge among majors
EXAMPLE 4
FLORIDA’S ACADEMIC LEARNING COMPACTS
• 2004 Florida Board of Governors resolution to
implement Academic Learning Compacts for each
baccalaureate degree program at 11 universities
– Discipline-specific knowledge/skills
– Communication skills
– Critical thinking skills
• Each baccalaureate program must also
– Identify where and how students are assessed
– Provide students access to this information
– Monitor performance for program improvement
Implementing Academic Learning Compacts
• Review of past assessment plans
• Develop flexible definitions of outcomes
• Assist departments to
– Identify discipline-specific outcomes
– Identify location of student assessments
• Courses (Core requirements; Gateway; Capstone)
[Shared learning outcomes/syllabi/assignments]
• Standardized examinations for some programs
– Develop methods to track student achievement
• Embedded questions on examinations
• Scoring rubrics on shared assignments
• Performance on standardized examinations
SCHOLARSHIP OF ASSESSMENT
• Presentations at conferences
– Using Psychological Expertise in College-Wide
Assessment and Departmental Program Review
presented at MACTOP ’97
– Assessing Classes, Courses, and Curricula
presented at MACTOP ’02, MISTOP ’01, ITOP ’00
– Aligning Assessment for Program Improvement with
Accountability for Individual Student Learning
presented at SACS ’05; NCSU ’06 Assessment Institute
• Publications
– Designing and Implementing Psychology Program
Reviews (2004 Measuring Up book chapter)
– A Decade of Changes since the St. Mary’s Conference
(2002 Teaching of Psychology interview)
DISCIPLINE-BASED JOURNALS ON TEACHING PEDAGOGY
Accounting: Issues in Accounting Education
Anthropology: Anthropology and Education
Art: Art Education
Biology: American Biology Teacher
Business: Business Education
Chemistry: Journal of Chemical Education
Communication: Communication Education
Computer Science: Computer Science Education
Economics: The Journal of Economic Education
Education: Journal of Teacher Education
English: Research in the Teaching of English
Finance: Journal of Financial Education
Geography: Journal of Geography in Higher Education
Geology: Journal of Geological Education
History: Teaching History - A Journal of Methods
International Business: Journal of Teaching in International Business
DISCIPLINE-BASED JOURNALS ON TEACHING PEDAGOGY
Journalism: Journalism Educator
Management: Journal of Management Education
Marketing: Journal of Marketing Education
Mathematics: Journal for Research in Mathematics Education
Modern Languages: Modern Language Journal
Music: Journal of Research in Music Education
Nursing: Journal of Nursing Education
Philosophy: Teaching Philosophy
Physics: Physics Teacher
Political Science: Political Science Teacher; Teaching Political Science
Psychology: Teaching of Psychology
Social Work: Journal of Teaching Social Work
Sociology: Teaching Sociology
Statistics: The American Statistician
Theater: Theater Topics
Women's Education: The Feminist Teacher
Descargar

Slide 1