ITiCSE'09 – Keynote
Analyzing CS Competencies
using The SOLO Taxonomy
Claus Brabrand
((( [email protected] )))
((( http://www.itu.dk/people/brabrand/ )))
Associate Professor,
IT University of Copenhagen
Denmark
Claus Brabrand
ITiCSE 2009 – Keynote
Paris, France (July 06, 2009)
Outline

1) Introduction
Constructive Alignment
 The SOLO Taxonomy


2) From Content to Competence
Advocate a shift in perspective
 Elaborate The SOLO Taxonomy


3) Analyzing CS Competencies
…using The SOLO Taxonomy
 Compare: CS vs NAT vs MAT

Claus Brabrand
ITiCSE 2009 – Keynote
[ 2 ] Paris, France (July 06, 2009)
Introduction to…:

Constructive Alignment & SOLO Taxonomy:
John Biggs’ popular and heavily cited book:
“Teaching for Quality Learning
at University
- What the student does”
Note: 3rd Edition now available [J.Biggs & C.Tang, 2009]
“Teaching Teaching
&
Understanding Understanding”
19 min award-winning short-film on Constructive Alignment
(available on DVD in 7 languages, epilogue by John Biggs)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 3 ] Paris, France (July 06, 2009)
Activation Exercise

T
Discuss with your neighbour:
What are the ‘main messages’
of the film (which did YOU find
particularly relevant, …if any)?
Claus Brabrand
ITiCSE 2009 – Keynote
[ 4 ] Paris, France (July 06, 2009)
Outline

1) Introduction
Constructive Alignment
 The SOLO Taxonomy


2) From Content to Competence
Advocate a shift in perspective
 Elaborate The SOLO Taxonomy


3) Analyzing CS Competencies
…using The SOLO Taxonomy
 Compare: CS vs NAT vs MAT

Claus Brabrand
ITiCSE 2009 – Keynote
[ 5 ] Paris, France (July 06, 2009)
From Content to Competence

My old course descriptions (Concurrency 2004):

Given in terms of a 'content description':

Essentially:
Goal is…:
To understand:
deadlock
interference
synchronization
...
This is a bad idea
for two reasons...!
Claus Brabrand
ITiCSE 2009 – Keynote
[ 6 ] Paris, France (July 06, 2009)
Problem 1 !

Problem with 'content' as goals !
analyze ...
theorize ...
analyze systems
explain causes
define deadlock
describe solutions
Goal is…:
Teacher
Stud. C
To understand:
deadlock
interference
synchronization
...
analyze systems
explain causes
name solutions
recite conditons
Stud. B

Stud. A
Censor
Claus Brabrand
ITiCSE 2009 – Keynote
P.S.: even if it were
possible to agree,
we know that the
exam will dictate the
learning anyway.
[ 7 ] Paris, France (July 06, 2009)
Problem 2 !

Problem with 'understanding' as goals !
Goal is…:
To understand:
deadlock
interference
synchronization
...
'concept of deadlock' ?!
The answer is simple:
It cannot be measured !
Claus Brabrand
ITiCSE 2009 – Keynote

[ 8 ] Paris, France (July 06, 2009)
Competence !

'Competence' as goals !

Competence := knowledge
+ capacity to act upon it
Have the student do something;
and then "measure" the product and/or process
Objective !
To learn how to:
analyze systems for...
explain cause/effects...
prove properties of...
compare methods of...
...
Note: 'understanding'
is of course
pre-requisitional !

Note': inherently operational (~ verbs)
'SOLO' = Structure of the Observed Learning Outcome
Claus Brabrand
ITiCSE 2009 – Keynote
[ 9 ] Paris, France (July 06, 2009)
SOLO Advantages

Advantages of The SOLO Taxonomy:
Linear hierarchical structure
 Aimed at evaluating student learning
 Converges on research (at SOLO 5)

Research:
Production of
new knowledge
Claus Brabrand
ITiCSE 2009 – Keynote
[ 10 ] Paris, France (July 06, 2009)
Note: the list is non-exhaustive
Graphic Legend
SOLO (elaborated)
QUANTITATIVE
SOLO 2
SOLO 3
”uni-structural”
“multi-structural”
define
identify
count
name
recite
paraphrase
follow (simple)
instructions
…
combine
structure
describe
classify
enumerate
list
do algorithm
apply method
…
SOLO 4
“relational”
problem / question / cue
known related issue - given!
hypothetical related issue - not given!
R student response
Q
QUALITATIVE
SOLO 5
“extended abstract”
analyze
compare
contrast
integrate
relate
explain causes
apply theory
(to its domain)
…
theorize
generalize
hypothesize
predict
judge
reflect
transfer theory
(to new domain)
…
R
Q
Claus Brabrand
R
Q
R
Q
ITiCSE 2009 – Keynote
R
Q
R'
[ 11 ] Paris, France (July 06, 2009)
Using SOLO in Practice

Recommendations on course descriptions:
Intended Learning Outcomes
[Algorithms 101]
After the course, the students are expected to be able to:
identify V and formulate algorithmic problems N ;
V
classify V and compare V algorithms N ;
construct V and analyzeV algorithms N using standard paradigms;
implement V algorithms N for simple problems.
N
V
Claus Brabrand
ITiCSE 2009 – Keynote
[ 12 ] Paris, France (July 06, 2009)
T
Activation Exercise
Which do you predict are
key CS competences ?
Claus Brabrand
ITiCSE 2009 – Keynote
[ 13 ] Paris, France (July 06, 2009)
Outline

1) Introduction
Constructive Alignment
 The SOLO Taxonomy


2) From Content to Competence
Advocate a shift in perspective
 Elaborate The SOLO Taxonomy


3) Analyzing CS Competencies
…using The SOLO Taxonomy
 Compare: CS vs NAT vs MAT

Claus Brabrand
Joint work with Bettina
Dahl
[ 14 ] Paris, France (July 06, 2009)
ITiCSE 2009
– Keynoteat Aarhus University
Grade Scales
Conversion (between EU countries):
7 steps:
4 steps
4 steps
8 steps
8 steps
ECTS
10 steps
A, B, C, D, E, Fx, F
...
...
...
...
21 steps
10 steps
SCALE
21 steps
All Universities:
Explicit ILO's
The SOLO Taxonomy!
Claus Brabrand
ITiCSE 2009 – Keynote
[ 15 ] Paris, France (July 06, 2009)
Massive DATA set

Unique Opportunity…:
5,608 competencies
734 courses
21
institutes
TWO universities
Systematically formulated ILO's for all courses
 Quantifiable (analyzable) via The SOLO Taxonomy

Claus Brabrand
ITiCSE 2009 – Keynote
[ 16 ] Paris, France (July 06, 2009)
SOLO Mapping
Mapped by:
 B. Dahl & C. Brabrand
With help from:
 3 Educational research
colleagues (medicine)
 J. Biggs & C. Tang
Claus Brabrand
ITiCSE 2009 – Keynote
[ 17 ] Paris, France (July 06, 2009)
Top 10 Competencies

Top 10 Competencies:
Natural Sciences
"
Claus Brabrand
"
:=
{ Physics, Chemistry, Biology, Molecular Biology }
ITiCSE 2009 – Keynote
[ 18 ] Paris, France (July 06, 2009)
Histogram of Top Competencies

If we look closer (comparative visualization)...:
More than 2x
More than 3x
MAT
More than 3x …also:
%
 program
 construct
 structure
NAT
NAT
MAT
CS
CS
MAT
CS
MAT
MAT
MAT
CS
CS:
15 %
Legend: 1.0 %
NAT:
MAT:
0.3 %
Computer Science
Claus Brabrand
CS:
NAT:
MAT:
Natural Science
ITiCSE 2009 – Keynote
14 %
14 %
60 %
with apply
CS:
NAT:
MAT:
Mathematics
4.5 %
4.4 %
40 %
[ 19 ] Paris, France (July 06, 2009)
SOLO Distribution

SOLO distribution:
 15% 
E[X] = 3.7
E[X] = 3.4
E[X] = 3.1

The 15% "programming competences" (all at SOLO 4):

Legend:
Claus Brabrand
{ implement, program, design, construct, structure }
SOLO 2
SOLO 3
ITiCSE 2009 – Keynote
SOLO 4
SOLO 5
[ 20 ] Paris, France (July 06, 2009)
Assumptions
Assumptions:
 SOLO is an appropriate competence measure
[Biggs’ studies]
(we refer to [J.Biggs & K.F.Collis, 1982] )
 Context independence of SOLO mapping
[approximation]
(for each competence we inspected several goals)
 Subject independence of SOLO mapping
[approximation]
(we limit ourselves to a 'science context')
 Equal weight assumptions
[approximation]
(Competences in a goal & goals in a course have equal weight)
 Outcomes: intended  formulated  achieved
[implicational]
(we “analyze” formulated, but “reason about” achieved)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 21 ] Paris, France (July 06, 2009)
Conclusions

Most frequent CS Competences are:


"Programming-related" skills:


15% of CS-curriculum
The "Essence of Math" is:


describe (13%), explain (10%),
apply method (9%),
implement (7%), analyze (6%), …
reproducing, formulating,
proving, solving, argueing,
(and applying)
SOLO-levels of subjects:

Claus Brabrand
CS
>SOLO
NAT
>SOLO
15%
MAT
ITiCSE 2009 – Keynote
[ 22 ] Paris, France (July 06, 2009)
Outline

1) Introduction
Constructive Alignment
 The SOLO Taxonomy


2) From Content to Competence
Advocate a shift in perspective
 Elaborate The SOLO Taxonomy


3) Analyzing CS Competencies
…using The SOLO Taxonomy
 Compare: CS vs NAT vs MAT

Claus Brabrand
ITiCSE 2009 – Keynote
[ 23 ] Paris, France (July 06, 2009)
Keynote Points

Constructive Alignment

…addresses many teaching / learning problems; e.g.:



The SOLO Taxonomy

…is good for reasoning about competencies:


Esp. student motivational issues (learning incentives)
...and student performance issues (learning support)
Esp. for designing courses and curricula
DATA

Study, analyze, and reflect on teaching / learning

Claus Brabrand
…using (objective) DATA!
ITiCSE 2009 – Keynote
[ 24 ] Paris, France (July 06, 2009)
Questions...
The Short-Film
?
recite
generalize
Claus Brabrand
ITiCSE 2009 – Keynote
[ 25 ] Paris, France (July 06, 2009)
Thank You!
Film's homepage:
((( http://www.daimi.au.dk/~brabrand/short-film/ )))
Claus Brabrand
ITiCSE 2009 – Keynote
Paris, France (July 06, 2009)
Related References





”Teaching for Quality Learning at University (what the student does)”
John Biggs & Catherine Tang
Society for Research into Higher Education, 2007. McGraw-Hill.
”Evaluating the Quality of Learning: The SOLO Taxonomy”
John Biggs & Kevin F. Collis
London: Academic Press, 1982
”Teaching Teaching & Understanding Understanding”
Claus Brabrand & Jacob Andersen
19 minute award-winning short-film (DVD)
Aarhus University Press, Aarhus University, 2006
”Using the SOLO Taxonomy to Analyze Competence
Progression of University Science Curricula”
Claus Brabrand & Bettina Dahl
Higher Education, 2009
"Constructive Alignment & The SOLO Taxonomy:
a Comparative Study of University Competencies
in Computer Science vs. Mathematics"
Claus Brabrand & Bettina Dahl
CRPIT, Vol. 88, ACS 3-17, R. Lister & Simon, Eds., 2007
Claus Brabrand
ITiCSE 2009 – Keynote
[ 27 ] Paris, France (July 06, 2009)
Implementing Alignment

Alignment Implementation Process:
1) Think carefully about:
overall goal of course
(what are the stud. to learn?)
2) Operationalize these goals
and formulate them as SOLO
intended learning outcomes
alignment
learning incentive
3) Choose carefully the
form(s) of assessment
(~ intended learning outcomes)
Claus Brabrand
learning support
4) Choose carefully the
form(s) of teaching
Think of teaching
(~ intended learning outcomes)
activities as
”training for exam”
ITiCSE 2009 – Keynote
[ 28 ] Paris, France (July 06, 2009)
SOLO Progression

SOLO Progression:

Claus Brabrand
Computer Science vs. Mathematics vs. …
ITiCSE 2009 – Keynote
[ 29 ] Paris, France (July 06, 2009)
Conclusion (Progression)

What have we really shown?!?
A) SOLO has "proved" that progression exists in curricula
(since we "believe" in SOLO as a measure)
xor
B) SOLO has "been proven" to be a good tool for
analyzing competence progression
(since we "believe" in the existence of progression)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 30 ] Paris, France (July 06, 2009)
Progression Assumptions
Extra assumptions wrt. Progression:
 Numeric quantification of SOLO
[assumption]
(i.e., numeric step from 2-3 is comparable to 3-4 and 4-5)
 Progression manifests itself as competences
[assumption]
(i.e., in 'verb'-, not 'noun'-dimension)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 31 ] Paris, France (July 06, 2009)
SOLO Calculation Method

Calculation Example (for a course):
identify (2) and formulate (3) algorithmic problems;
classify (3) and compare (4) algorithms;
construct (4) and analyze (4) algorithms using standard paradigms;
implement (4) algorithms for simple problems.

"SOLO average":


"double weight averaging"
[ (2+3)/2 + (3+4)/2 + (4+4)/2 + 4 ] / 4 = 3.50
"SOLO distribution":

Claus Brabrand
ITiCSE 2009 – Keynote
[ 32 ] Paris, France (July 06, 2009)
T
Neighbour Discussion
Discuss with neighbour:
"does this make sense ?!?"
(content  competence)
E.g.: ("Learning about programming" vs. "Learning to program" )
Claus Brabrand
ITiCSE 2009 – Keynote
[ 33 ] Paris, France (July 06, 2009)
T
Activation Exercise III

Discuss with your neighbour:
Discuss what you predict we
would find in the DATA set ?
Questions:
a) most frequent CS competences?
b) percentage of "programming-related" competences?
c) CS v. NAT v. MAT (wrt. SOLO levels)?
Claus Brabrand
ITiCSE 2009 – Keynote
[ 34 ] Paris, France (July 06, 2009)
T
Post-It exercise
Write down 1-2 key
competences (i.e., verbs)
(for your course)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 35 ] Paris, France (July 06, 2009)
Tips'n'Tricks (activation)

Neighbour discussions:

Post-It exercise:
 focus: zoom in
 anonymous (!)
 swap'able
 everyone will engage
 empathetic control
 shared knowledge pool
 more questions
(students dare
ask them)
 better questions
(students had a
chance to discuss)
[Phil Race]
1-2 min timeout
Form variation:

lecturing blended with in-class activation exercises
Claus Brabrand
Frequent breaks:
pulse reader measurements:

ITiCSE 2009 – Keynote
[ 36 ] Paris, France (July 06, 2009)
Tips'n'Tricks (cont'd)

Use many examples:

Explicit structure:
(build on student pre-knowledge)
 self evident to you [ teacher ]
 not to a learner
[ student ]
(esp. during learning process)

"Less-is-more":
 analyze
 compare
 relate
vs.

Student 'recap' at end:
common deadlock, uncommon
deadlock, A-synchronization, Bsynchronization, hand-shake,
multi-party synchronization,
multi-party hand-shake, binary
semaphores, generalized
semaphores, blocking
semaphores, recursive locks, ...
now
after 1 day
after 1 week
after 2 weeks
after 3 weeks
Emphasize depth over breadth (coverage)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 37 ] Paris, France (July 06, 2009)
Now, please: "3-minute recap"

Please spend 3' on thinking about and writing down
the most important points from the talk – now!:
Immediately
After 1 day
After 1 week
After 2 weeks
After 3 weeks
Claus Brabrand
ITiCSE 2009 – Keynote
[ 38 ] Paris, France (July 06, 2009)
Problematic Courses

E.g. course: ”Databases” (at RUC/Roskilde):
 obtain knowledge about the structure of database systems;
 be familiar with design of databases by use of special notations like
E/R and analysis through normalization;
 get an overview of the most important database models and a
detailed knowledge about the most important model - the relational
model as well as the language SQL;
 get an overview of database indexing and query processing;
 obtain knowledge about application programming for DB systems.

Note: almost entirely
non-operational(!)

i.e. measure how?!
Familiar with ?!
Claus Brabrand
ITiCSE 2009 – Keynote
[ 39 ] Paris, France (July 06, 2009)
BONUS SLIDES
Claus Brabrand
ITiCSE 2009 – Keynote
Paris, France (July 06, 2009)
Based on John Biggs' Theories
 "Teaching for Quality Learning at University", John Biggs

2nd edition

Claus Brabrand
(3rd edition expected this fall)
ITiCSE 2009 – Keynote
[ 41 ] Paris, France (July 06, 2009)
UNALIGNED COURSE
Teacher’s
intention

e.g.
- explain
- relate
- prove
- apply
Student’s
activity
"Dealing with the test"
Exam’s
assessment
e.g.
- memorize
- describe
Claus Brabrand
ITiCSE 2009 – Keynote
[ 42 ] Paris, France (July 06, 2009)
ALIGNED COURSE

Teacher’s
intention
e.g.
- explain
- relate
- prove
- apply
Student’s
activity
Exam’s
assessment
e.g.
- explain
- relate
- prove
- apply
Claus Brabrand
ITiCSE 2009 – Keynote
[ 43 ] Paris, France (July 06, 2009)
Top 10 Competencies

Top 10 Competencies:
Natural Sciences
"
Claus Brabrand
"
:=
{ Physics, Chemistry, Biology, Molecular Biology }
ITiCSE 2009 – Keynote
[ 44 ] Paris, France (July 06, 2009)
Note: the list is non-exhaustive
Graphic Legend
SOLO (elaborated)
QUANTITATIVE
SOLO 2
SOLO 3
”uni-structural”
“multi-structural”
define
identify
count
name
recite
paraphrase
follow (simple)
instructions
…
Q
Claus Brabrand
R
combine
structure
describe
classify
enumerate
list
do algorithm
apply method
…
Q
R
Q
R1
R2
R3
SOLO 4
“relational”
problem / question / cue
known related issue - given!
hypothetical related issue - not given!
R student response
Q
QUALITATIVE
SOLO 5
“extended abstract”
analyze
compare
contrast
integrate
relate
explain causes
apply theory
(to its domain)
…
theorize
generalize
hypothesize
predict
judge
reflect
transfer theory
(to new domain)
…
R
Q
ITiCSE 2009 – Keynote
R
Q
R'
[ 45 ] Paris, France (July 06, 2009)
T
Exercise

Buzz Session:
1) Discuss w/ neighbour:
"which film messages
did you find
particularly relevant?"
2) Write it on a Post-It
Just Keep
3) Swap Post-Its… Swapping…
Claus Brabrand
ITiCSE 2009 – Keynote
[ 46 ] Paris, France (July 06, 2009)
Student Motivation
 Susan: (”intrinsic motivation”)
- wants to…: learn !
 Robert: (”extrinsic motivation”)
- to…: pass exams !
Claus Brabrand
ITiCSE 2009 – Keynote
[ 47 ] Paris, France (July 06, 2009)
Constructivism
 ”Transmission is Dead…” :
!
active
teacher
&
(lectures risk
= passive students )
 Knowledge is…
Actively Constructed !
Claus Brabrand
ITiCSE 2009 – Keynote
[ 48 ] Paris, France (July 06, 2009)
SOLO Taxonomy
 Hierarchy for Competences:
5:
4:
3:
2:
generalize, theorize, predict, …
explain, analyze, compare, …
describe, combine, classify, …
recite, identify, calculate, …
 Deep learning (not surface) !
Claus Brabrand
ITiCSE 2009 – Keynote
[ 49 ] Paris, France (July 06, 2009)
Stud Learning Focus
 Focus on Student Learning !
(instead of ”what teacher does”
& labelling students: ’good/bad’)
 Student activitation  learning
Claus Brabrand
ITiCSE 2009 – Keynote
[ 50 ] Paris, France (July 06, 2009)
Alignment
 Make explicit ILO’s
(Intended Learning Outcomes):
 Exam = ILO’s = Teaching
 (…and tell this to students)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 51 ] Paris, France (July 06, 2009)
The Role of the Exam

Alignment:
A theory of planning (over the course of a course)
 A theory of motivation (and incentive)

"The exam does not come after, but before the course!"

The exam as a...:
"Necessary evil"
application
of alignment
Motivational and learning-guiding
pedagogical tool for the teacher(!)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 52 ] Paris, France (July 06, 2009)
Di-Transitive Verbs

Mono-Transitive verbs:

Di-Transitive verbs:
Claus Brabrand
ITiCSE 2009 – Keynote
[ 53 ] Paris, France (July 06, 2009)
Data Set (XML and XQuery)
(: Extracts all mathematics courses w/
maximum 1 goal and 2 competencies :)
xquery version "1.0";
<result>
{
for $course in fn:doc("data-au.xml")
//institute[@name = "MAT"]//course
let $goals := $course/goal
where (fn:count($goals) le 1) and
(fn:count($goals/competence) eq 2)
order by [email protected]
return $course
}
</result>
XQuery
Data set:
[ http://www.itu.dk/people/brabrand/solo.xml ]
[ http://www.itu.dk/people/brabrand/data-au.xml ]
[ http://www.itu.dk/people/brabrand/data-sdu.xml ]
Claus Brabrand
XML
ITiCSE 2009 – Keynote
[ 54 ] Paris, France (July 06, 2009)
The BLOOM Taxonomy (1956)
The BLOOM Taxonomy:

Analysis
Synthesis
Evaluation
Application
Comprehension
Knowledge
”
”[…] really intended to guide the selection of items for a test rather
than to evaluate the quality of a student’s response to a particular item”
Claus Brabrand
-- (Biggs & Collis, 1982)
ITiCSE 2009 – Keynote
[ 55 ] Paris, France (July 06, 2009)
CS vs Math Distributions

Computer Science:
( = 3.68,  = 0.39)

Mathematics:
( = 3.06,  = 0.24)
Claus Brabrand
ITiCSE 2009 – Keynote
[ 56 ] Paris, France (July 06, 2009)
Descargar

Slide 1