Developing Tests for
Departmental Assessment
Deborah Moore, Assessment Specialist
Institutional Research, Planning, & Effectiveness
University of Kentucky
1
Session Overview
Steps in Test Development and Test
Administration
 Process for Evaluating Test and Items
 Common Difficulties
 Strategies for Analysis and Reports

2
Improve Student Learning
3
Exercise 1
(10 minutes)
Under What Conditions Do You Give a Test?
How do you decide you are going to
give a test?
 What do you do to develop a new test?
 How do you decide what types of item
formats to use?
 How do you develop items?
 How long does it take to put the test
together?

4
Using a Test for
Classroom Assessment

Generally, instructors develop their own classroom
tests, making all decisions about when and how to
construct, administer, score, and report results of
tests.

Construction is often done without formality or
documentation.

The most frequent use of tests by instructors is to
assign grades related to individual student learning.
5
A Test Is a Good Choice When…

the student must demonstrate acquisition of knowledge
or ability to process and use knowledge

the student's knowledge about a wide range of content is
to be evaluated (e.g., survey and capstone courses)

multiple observations of the content-related knowledge
are needed (e.g., math and foreign languages)

more resources are available for constructing the test
than for scoring and reporting

a large group is being assessed
6
Using a Test for
Departmental Assessment

Planning, implementing, and using results become a
group effort – a shared set of decisions and
responsibilities. Consensus is emphasized.

Additional planning time, communication, and record
keeping will be needed.

When used for program assessment, test
performance is generally used along with other
information to describe group achievement and is
independent of grading.
7
Advantages of
Selected-Response Item Formats

Better content coverage

Higher reliability

Greater efficiency

Objectivity

Mechanical scoring
8
Test Development Process







Define purpose/use
Outline curriculum & consult goals/objectives
Create test plan (aka blueprint or specifications)
Create a pool of items
Critique and revise items
Pilot/field test (e.g. item analysis, reliability/validity studies)
Set guidelines for test administration, scoring
procedures, interpretation of scores (e.g. develop
norm tables and standard setting procedures)
9
Exercise 2
(10 minutes)
Review a Sample Departmental Packet

What role does the exam play in the
assessment plan for this department?

What type of item formats should be
used on the comprehensive exam?
10
Objectives
Respond in an informed way to
the form, structure and aesthetic
qualities of artistic and literary
works.
Explore the interrelationships among
historical events and intellectual,
artistic, literary and philosophical or
religious movements and works.
Identify and analyze similarities,
differences and interrelationships
among the fine arts.
Articulate central philosophical and
religious questions and the varying
responses to them within different
cultures.
Apply appropriate vocabulary and
concepts for the description and
analysis of artistic, literary,
historical and philosophical or
religious works.
Explain how artistic and literary
works from the past and present
civilizations are individual
expressions of cultural, historical
and intellectual forces.
11
Test Blueprint
(Content X Objective)
Obj 1
Obj 2
Obj 3
Obj 4
Obj 5
T h e a te r
1
1
11
3
2
A rt
5
3
3
4
M u sic
5
6
1
P h ilo s o p h y /
R e lig io n
1
2
3
2
1
A n th ro p o lo g y
Obj 6
T o ta ls
18
3
18
1
14
9
8
23
1
1
2
6
L ite ra tu re
5
4
6
1
4
T o ta ls
17
6
30
12
20
20
14
99
12
Test Plan
(Items Within Cells)
T h e a te r
A rt
M u sic
P h ilo s o p h y /
R e lig io n
Obj 1
Obj 2
Obj 3
Obj 4
Obj 5
7
18
1 -6 ,9 12, 16
8, 13,
17
1 4 -1 5
47, 49,
61
5 3 -5 4 ,
59
52, 57,
60, 63
3 5 -4 0
33
85, 8990, 99
2 4 -2 5
26, 7879
65, 676 8 , 7 2 -7 7
1 9 -2 3 ,
6 9 -7 1
18
29, 32
31
27
28, 30
6
47, 49,
61
5 3 -5 4 ,
59
52, 57,
60, 63
5 6 -5 7 ,
62
20
30
12
14
99
4 8 ,5 0 -5 1 ,
58, 64
4 2 -4 6
41
66
A n th ro p o lo g y
L ite ra tu re
T o ta ls
8 6 -8 8 ,
94, 98
17
6
20
Obj 6
T o ta ls
18
5 6 -5 7 ,
62
18
14
13
Administrative Approaches

Common exams

Course-embedded testing

Assessment Center/Assessment Days
14
Sources for Items
Research
 Curriculum
 Real life
 Professional
development
materials &
textbooks

Own final exam
files
 Comprehensive
exams from other
universities
 Retired test/item
banks

15
Common Difficulties with Test Quality

Push for higher order thinking often weak

Insufficient consensus on test
plan/blueprint

Discomfort with various item-writing
approaches (writer’s block)

Insufficient editing

Little or no piloting of items
16
Critiquing Items
Match with test
plan
 Item value
 Item quality

17
Typical Item Review Questions







Has each item received a quality appraisal?
Has each item’s content been verified?
Has each item been classified?
Has the key been identified?
Have you edited the items?
Have items been checked for bias/insensitivity?
Have you field tested the items?
18
Planning A Departmental Test
Develop test
 Plan administration
 Analyses test data
 Report and use results
 Agree on leadership/support roles

19
Exercise 3
(30 minutes)
Developing a Planning Checklist

Review the headings of the checklist form

List additional headings needed

List activities that need to be completed
under the various headings

Add a number to the activities to represent
the order of completion
20
Analyses

Examine quality of your assessment
tools and procedures

Describe the test results

By test plan features

By meaningful subgroups

Combine with other datasets

Execute and summarize planned
comparisons linked to research design
21
Reports

Determine stakeholders/constituents



Identify specific interest
Create a timeline and share
Include a vetting procedure
22
Testing in the Future

Computer based testing
Adaptive (CAT)
 Self-Adaptive (SAT)

Computer assisted item generation
 Automated essay scoring

23
Thank you for your attention.

Deborah Moore, Assessment Specialist

101B Alumni Gym
Office of Institutional Research, Planning, &
Effectiveness

[email protected]

859/257-7086

http://www.uky.edu/LexCampus/
24
Descargar

Anatomy of an Item - University of Kentucky