Effective methods for the use, creation,
analysis, and interpretation of shortanswer student conceptual evaluations.
Ronald Thornton
Professor of Physics and Education
Center for Science & Math Teaching
Tufts University
R.K. Thornton
What was I thinking?

I’ll paint your house and walk your dog as
well.
R.K. Thornton
In Defense of Thoughtful
Multiple Choice Conceptual
Assessment
Ronald Thornton
Professor of Physics and Education
Center for Science & Math Teaching
Tufts University
R.K. Thornton
Modest Suggestions from a
Chemically Illiterate Physicist.
Ronald Thornton
Professor of Physics and Education
Center for Science & Math Teaching
Tufts University
R.K. Thornton
Center for Science and Math Teaching
Tufts University
Curriculum
Development
Educational
Research
Computer Tool
Development
Teacher & Professor
Education
R.K. Thornton
Funding



NSF
National Science Foundation
FIPSE
Fund for the Improvement of Post
Secondary Education
US Department of Education
R.K. Thornton
Wouldn’t it be nice if teachers
could understand what students
know from a simple conceptual
evaluation?
and they knew what to do to help the
student learn
R.K. Thornton
What use might this talk be?
If you intend to develop a chemistry concept
inventory these suggestions may help you
make it more useful.
 If you intend to use a chemistry concept
inventory these ideas should help you pick a
useful one.

R.K. Thornton
We have spent years


Creating effective learning environments for
introductory science(physics) courses (curricula,
tools, pedagogical methods, group structures)
And developing methods of conceptual evaluation to
measure student learning and guide our progress.
R.K. Thornton
Why Multiple Choice?






More easily administered to large numbers of
students.
Evaluation takes less time.
Student responses can be reliably evaluated even
by the inexperienced.
Can be designed to guide instruction.
With proper construction, student views can be
evaluated from the pattern of answers, changes
over time can be seen, frequency of student views
can be measured.
Multiple choice combined with open response can
help the teacher/researcher explicate the students
response.
R.K. Thornton
Why not?
Every “good” educator knows multiple
choice questions are no good.
 Badly constructed multiple choice can give
misleading results.
 Unless very carefully constructed, multiple
choice will not identify student thinking.
 The choices may be inappropriate when
used with different audiences

R.K. Thornton
First steps
Why do you want to make (use) a
conceptual evaluation?
 In what conceptual area do you want to
know how students think?

R.K. Thornton
Why?

There are pre-requisite areas of conceptual
knowledge that students need to know to
actually understand chemistry.
R.K. Thornton
What? Three modest suggestions.



Explore student beliefs in the atomic nature of
matter. (students may say atoms exist but few
believe it in any functional matter)
Explore student beliefs the dynamic nature of
equilibrium. (Most students seem to have a static
model)
Explore student beliefs about the difference
between heat energy and temperature. (Most
students do not clearly make this distinction.)
R.K. Thornton
Our research has shown.






Student conceptual responses can be context dependent.
Student domains of applicability can be different from those
of a scientist.
Students (and scientists) can hold apparently inconsistent
views simultaneously. (and it doesn’t mean they are stupid.)
Conceptual transitions are not instantaneous.
There is statistical evidence of a hierarchy of student
conceptual views.
You can do more with large-scale conceptual evaluation than
just generating a single number.
R.K. Thornton
Good Practice for the Construction
of Conceptual Multiple Choice
All answers, "right or wrong," should help evaluate
student views.
 Derive the choices in the questions from from
student answers to free response questions and from
student interviews.
 Check to see students almost always find an answer
that they are satisfied with. Random answers should
be few.
 Ask similar questions in different representations.
 Check results with different student populations.
(more)

R.K. Thornton
Good Practice (continued)





Look at correlations among questions and use
patterns to understand student thinking.
Understand the implications of “correct” and
“incorrect” answers to their performance on other
tasks.
Check for gender differences
Identify circumstances for “false positive” answers
If at all possible, construct the evaluation so it is
useful to guide instruction.
R.K. Thornton
Multiple Choice Conceptual Evaluation

Conceptual evaluation for
 kinematics
(description of motion) and
 dynamics (force and motion which is well characterized
by Newton’s Laws).
Force & Motion Conceptual Evaluation (FMCE)
 Conceptual evaluation for heat energy and
temperature
The Heat and Temperature Conceptual Evaluation
(HTCE)
Both developed by the Center for Science and Math Teaching
at Tufts
R.K. Thornton
Using the FMCE as an example


Student answers correlate well (well above 90%)
with written short answers in which students
explain the reason for their choices
Almost all students pick choices that we can
associate with a relatively small number of student
models. (Conceptual Dynamics, R.K. Thornton in ICUPE
proceedings edited by Redish)

Testing with smaller student samples shows that
those who can pick the “correct” graph under these
circumstances are almost equally successful at
drawing the graph correctly without being
presented with choices.
R.K. Thornton
FMCE as example


Because we are able to identify statistically most
student views from the pattern of answers (and
because there are very few random answers), we
are also able to identify students with less common
beliefs about motion and follow up with
opportunities for interviews or open-ended
responses to help us understand student thinking.
The use of an easily administered and robust
multiple choice test has also allowed us and others
to track changes in student views of dynamics and
to separate the effects of various curricular
changes on student learning.
R.K. Thornton
FMCE as example

Use multiple representations
 The
Force Graph questions require explicit
knowledge of coordinate systems and graphs
but require little reading.
 The Force Sled questions use natural language
and make no explicit reference to a coordinate
system or graphs.
R.K. Thornton
A sled on ice m oves i n the way s de scri bed in ques ti on s 1 -7 be low . F riction is so sma ll
tha t it can be i gnored . A per son w ea ring sp iked sho e s s tand ing on the i ce c an app ly a
forc e to th e sl ed and push it a long t he ice . Choo se th e one fo rce (A th rough G ) wh ich
wou ld k ee p th e sled mov in g as desc ri bed i n e a ch st atem en t be low .
You m ay u se a cho ice m o re than once or no t a t all bu t choos e on ly on e an swe r fo r ea c h
b lank . I f you t h ink tha t none is co rr ec t, an swe r cho ice J.
A.
Direction of Force
The force is toward the
right and is
increasing in strength (magnitude).
B. The force is toward the
right and is of
constant strength (magnitude).
C. The force is toward the
right and is
decreasing in strength (magnitude).
D. No applied force is needed
E.
The force is toward the
left and is
decreasing in strength (magnitude).
F.
The force is toward the
left and is of
constant strength (magnitude).
G.
The force is toward the
left and is
increasing in strength (magnitude).
Direction of Force
1 . Wh ich fo rce wou ld k e ep the sled m ov ing towa rd the righ t and speed ing up at
a steady r ate (con st an t a c ce lerati on)?
2 . Wh ich fo rce wou ld k e ep the sled m ov ing towa rd th e ri gh t a t a st eady
(cons tan t) v el oc ity ?
3 . T he sl ed is m ov ing tow a rd the ri gh t. Wh ich fo rce wou ld sl ow it do w n at a
ste ady rate (c ons tan t a cce lerati on )?
4 . Wh ich fo rce wou ld k e ep the sled m ov ing towa rd th e left a nd sp e ed ing up at a
ste ady rate (c ons tan t a cce lerati on )?
5 . T he sl ed w a s s tart ed fr o m res t and pu shed unt il i t r eached a steady (c ons tan t)
ve loc it y to w ard the ri gh t. W h ich for c e wou ld keep t he sl ed m ov ing at th is
ve loc it y?
6 . T he sl ed is slo w ing down a t a stea dy rate and ha s a n ac c elerati on to th e ri ght .
Wh ich fo rce wou ld a ccoun t fo r th is m o ti on?
7 . T he sl ed is m ov ing tow a rd the left . W h ich fo rce w ould s low it down at a
ste ady rate (c ons tan t a cce lerati on )?
R.K. Thornton
Questions 14-21 refer to a toy car which
can move to the right or left along a
horizontal line (the positive part of the
distance axis).
A
+
F
o
r
c
e
0
Time
-
0
Assume that friction is so small that it
can be ignored.
A force is applied to the car. Choose the
one force graph ( through
A
) forH
each
statement below which could allow the
described motion of the car to continue.
+
B
+
F
o
r
c
e
0
Time
-
C
+
F
o
r
c
e
You may use a choice more than once
or not at all. If you think that none is
correct, answer choice . J
0
Time
-
__14.
The car moves toward the right
(away from the origin) with a
steady (constant) velocity.
D
__15. The car is at rest.
__16. The car moves toward the right
and is speeding up at a steady rate
(constant acceleration).
__17.
E
The car moves toward the left
(toward the origin) with a steady
(constant) velocity.
__18. The car moves toward the right
and is slowing down at a steady rate
(constant acceleration).
__21. The car was pushed toward the
right and then released. Which
graph describes the force after
the car is released.
F
o
r 0
c
e
-
Time
+
F
o
r
c
e
0
Time
-
F
__19. The car moves toward the left and
is speeding up at a steady rate
(constant acceleration).
__20. The car moves toward the right,
speeds up and then slows down.
+
+
F
o
r
c
e
0
Time
-
G
+
F
o
r
c
e
0
Time
-
H
F
o
r
c
e
+
0
-
J
None of these graphs is correct.
Time
R.K. Thornton
Comparison with short answer
As with all the questions on the test students
who answered correctly were also able to
describe in words why they picked the
answers they did.
 Statistically one of the last questions to be
answered in a Newtonian manner is the
force on a cart rolling up a ramp as it
reverses direction at the top (question 9).

R.K. Thornton
Back to best practices. Consider




All answers, "right or wrong," should help
evaluate student views.
Derive the choices in the questions from from
student answers to free response questions and
from student interviews.
Check to see students almost always find an
answer that they are satisfied with. Random
answers should be few.
Look at correlations among questions and use
patterns to understand student thinking.
R.K. Thornton
An example from the H&T
Conceptual Evaluation

Distinguishes different student models for
the relationship between heat and
temperature.
R.K. Thornton
Questions 1 through
4 refe r to tw o cups, A and B , which contain diff erent amounts of
water. The wate r in each cu p is heated as described. In questions 1 through 3 the cups
are in a room wher e the temperat ure is 25 °C. I n question 4 the cups are in di fferent
environments.
For each question choose one of the fou r answers A through D .
A) Cup A ha d more hea t energy
B) Cup B had m
ore heat energ y transferre
C) Both cups had th
D) not enoug
____1.
•
____2.
____3.
____4.
transferred
e same amount
h inf ormation
d
of hea t energ y transferred
is give n to dete rmine the answer
Cup A contains 100 grams
of wate r and
cup B contains twice as much water
. The
water in bot h cups was initially at roo m
temperature.
Cup A was heated to 75°C
an d cup B was heated to 50°C
. Whic h cup
had more heat energ
y transferre d to it?
Cup A contains 100 grams
of wate r and
cup B contains 50 grams
of wate r. The
water in bot h cups was initially at roo m
temperature.
Cup A was the n heated to
45°C and cup B was heated to 90°
C.
Which cup had mor
e heat energy
transferred t o it?
Cup A contains 100 grams
of wate r and
cup B contains 80 grams
of wate r. The
water in bot h cups was initially at roo m
temperature.
Cup A was then heated to
45°C a nd cup B was heated to 50°
C.
Which cup had mor
e heat energy
transferred t o it?
Cup A contains 100 grams
of wate r and is
initially at 10°
C in a refrig erator . Cu p A is
heated until its temperat
ure is 20°C .
Cup B contains 50
grams of wate r
initially at 70°
C in an ove n. Cup B is
heated until its temperat
ure is 90°C .
Which cup had mor
e heat energy
transferred t o it?
A
B
100 g
75°C
A
100 g
Room
Temperature
25°C
Room
Temperature
25°C
45°C
A
100 g
10°C
50°C
B
50 g
90°C
Room
Temperature
25°C
45°C
Refrigerator
Tem perature
200 g
B
80 g
50°C
A
B
100 g
50 g
20°C
90°C
Oven
Tem perature
70°C
Results by category
100
natural language
90
Before Instruction
After RealTime Physics
graphical
70
U n d erst an d in g
A v era g e % o f St u d en t s
80
60
50
40
30
20
10
0
Acceleration
1st & 2nd(n)
1st & 2nd(g)
Coin Toss
Cart on Ramp
3rd Collisions
Force and Motion Conceptual Evaluation
R.K. Thornton
What about 1 number results
Not my favorite, but useful in some
situations
 Let’s compare the performance of 350 RPI
students in the beginning physics course on
the FMCE and the FCI

R.K. Thornton
Comparison of FMCE Gains
Oregon Traditional Algebra 1988-1989 (N=236)
SUNY Albany Traditional Calculus F1998 (N=73)
Sydney Traditional Calculus 1995 (N=472)
RPI Studio Physics S1998 (N=145)
Sydney Calculus + ILDs 1999 (N=60)
Mt. Ararat H.S. ILDs S1998 (N=33)
RPI Studio Physics + ILDs S1999 (N=311)
Muhlenberg Col. Calculus + ILDs F1997 (N=87)
CU Calc +Peer & UW Tutorial S2004 (N=391)
Joliet Junior College Calculus RTP labs1997-2003 (N=199)
Dickinson Workshop Physics F1999-2000 (N=104)
Oregon Algebra + ILDs F1991, Pre from 1989 (N=79)
Oregon Algebra RTP labs F1991-94, Pre from 1989 (N=613)
Tufts Agebra + ILDs 1994, 1996, 1997 (N=325)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
R.K. Thornton
<g> (% Normalized Gain)
.
100%
Still one number

Let’s compare the performance of 350 RPI
students in the beginning physics course on
the FMCE and the FCI
R.K. Thornton
Correlation Coefficient 0.791
R.K. Thornton
Correlation Coefficient 0.8309
R.K. Thornton
Are the evaluations the same?






Yes? Very high correlations (about 0.8 pre and
post with different instructional methods)
Yes? A high score on one implies a high score on
the other.
No? FCI fractional scores are almost always
higher than FMCE scores
No? Evaluations are measuring different things
No? A low score on the FMCE (non-Newtonian
student) does not imply a low score on the FCI
Lets look at a group of non-Newtonian students
R.K. Thornton
FMCE Conceptual Categories for Low FM CE/High FCI Students
FM CE<0.4, FCI•
0 .6 (N=54)
10 0%
90 %
80 %
70 %
60 %
50 %
40 %
30 %
20 %
10 %
0%
Velocity
Acc ele ration
Coin Acc
1s t & 2nd (nl)
1s t & 2nd (g)
Coin Tos s
Car t Ramp
3r d Conta ct
3r d Collision
R.K. Thornton
The conceptual threshold effect
(looking at pre-post correlations)
R.K. Thornton
Pre/Post Evaluation--The Threshold Effect
Tufts University Calculus-based Physics (N=181)
FMCE Post vs. Pre
1.00
0.80
0.60
0.40
Spring 1994 (N=48)
Spring 1995 (N=37)
Spring 1997 (N=43)
Spring 1998 (N=53)
0.20
0.00
0.00
0.20
0.40
0.60
Before Instruction
0.80
1.00
R.K. Thornton
University Physics Courses
Before Instruction
Average College and University Results
Force
Before Instruction
Acceleration
Velocity
0
20
40
60
80
100
% of Students Understanding Concepts
R.K. Thornton
University Physics Courses
After Normal Instruction
Average College and University Results
After Traditional Instruction
Force
Before Instruction
Acceleration
Velocity
0
20
40
60
80
100
% of Students Understanding Concepts
R.K. Thornton
Physics & Science Courses
Using New Methods
We have evidence of substantial, persistent learning
of such physical concepts by a large number of
students in varied contexts in courses and
laboratories that use methods I am about to
describe.
Such methods also work for students who have
traditionally had less success in physics and
science courses: women and girls, minority
students, and those who are badly prepared.
R.K. Thornton
University Physics Courses
After New Methods
Average College and University Results
After New Methods
Force
After Traditional Instruc.
Before Instruction
Acceleration
Velocity
0
20
40
60
80
100
% of Students Understanding Concepts
R.K. Thornton
Our Instructional and Assessment Philosophy
“I still don’t have all of the
answers, but I’m beginning to ask
the right questions.”
R.K. Thornton
Descargar

Effective methods for the use, creation, analysis, and