Using Surveys To Assess
Student Learning
A Simple Guide
Office of Assessment and Accreditation
Division of Academic Affairs
Indiana State University
What Is A Survey?
• A survey is a method of
measurement by which
respondents offer their
opinions in respect to specific
questions
• Answers are usually limited to
certain questions and are
measured on a specific scale
• Surveys can also contain
open-ended questions that
lead to qualitative analysis,
which can then lead to a
comparison with closed-ended
responses
Why Use Surveys to Assess
Student Learning?
•
•
•
•
•
•
It is usually convenient
Allows departments to limit
questions to certain objectives,
allowing for more efficient analysis
Several surveys can be
comparatively analyzed to detect
themes (triangulation)
Results can be easily summarized
Results can be compared with
open-ended responses and/or
interviews to detect themes
Can be used with direct
assessments of student learning
to comparatively analyze for
themes (surveys are not
considered methods of direct
assessment of student learning)
Outline of This Presentation
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Survey Types
Knowing Purpose of Survey
Designing Structure
Designing Scale(s)
Writing Questions
Defining Respondents
Identifying Audience
Analyzing Results
Presenting Results
Defining Periodicity
Triangulating
Plan for Effective Use
Types of Surveys
1.
2.
3.
4.
5.
6.
Student Surveys
Alumni Surveys
Employer Surveys
Performance Surveys
Field Experience Surveys
Internship Surveys
Example of Employer Survey
Student Teacher Survey
Indiana State University
Demographic Data. Please check the box that most accurately represents your current student teaching
position.
1.
Your student teaching position?
1 Early Childhood Education
2 Middle Childhood Education
3 Junior High/Middle School Education
4 Secondary Education
5 Special Education
6 Speech Language Pathology
7 Other ___________________________
2.
3.
During your student teaching experience, did you
teach the subject area for which you were
prepared at ISU?
4.
Demographic and
program information
1
2
5.
Questions are
designed on a
four-point scale.
Notice that the
questions begin
with verbs. The
faculty expressed
concerns in an
assessment retreat
about the fourpoint scale,
preferring instead
to insert a neutral
point between 2
and 3
Your college-based preparation for teaching?
Check all that apply.
1
Early Childhood Education
2
Middle Childhood Education
3
Junior High/Middle School Education
4
Secondary Education
5
Special Education
6
Speech Language Pathology
7
None of these
Please indicate the year you plan to graduate
from ISU.
__________
Yes
No
What is your teaching area(s)? Check all that apply.
1 Art
9 Music Education
2 Computer Education
10 Reading
3 Driver’s Education
11 School Media Services
4 Elementary Education
12 Science Education
5 English Education
13 Social Studies Education
6 Foreign Languages
14 Special Education
7 Family & Consumer Science
15 Speech Communication and
Theater
8 Mathematics Education
16
17
18
19
20
21
Business Education
Health and Safety Education
Physical Education
Technology Education
Vocational Trade/Industrial
Technology
Other __________________
Please circle the number that most closely represents your assessment about how well Indiana State
University prepared you for each item listed below.
Poorly Prepared
Somewhat Prepared
Well Prepared
Very Well Prepared
6.
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
7.
8.
9.
Design units of instruction
that focus on the content
standards for schools in
Indiana
Design classroom
assessments that are
aligned with Indiana
content standards
Create meaningful learning
experiences based on
content knowledge
Assess your own
professional growth needs
to develop a meaningful
professional development
plan
Poorly Prepared
Somewhat Prepared
Well Prepared
Very Well Prepared
10.
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
11.
12.
13.
Use technology as an
integral part of your
instruction to support and
extend student learning
Design and use formative
assessments that give
purposeful feedback to
students
Use teaching approaches that
are sensitive to diverse
learners
Use information about the
community and student
backgrounds to design
relevant learning
experiences
Know Your Purpose
1.
2.
3.
4.
5.
Is there a reason for
performing the survey?
Given that the use of
surveys can be timeconsuming and expensive,
is this the best method of
assessing student learning?
Can survey questions be
aligned with program
objectives?
Can survey questions be
aligned with student learning
objectives?
Can survey questions be
aligned with accreditation
standards?
Designing Survey Structure
• Know the general
format. Is it
convenient, or is it an
eyesore?
• Location of question
items
• Location of scales
• Location of
boxes/sections
Designing Scales
Choose Type of Scale
–
–
–
–
Likert or likert-type
scale
Rankings
Importance Scale
Link to what kind of
analysis is desired
Write Survey Questions
• Usually good to begin
with active verbs, or at a
minimum words other
than “the,” “a,” etc.
• Beware of general
concerns associated with
survey question writing
–
–
–
–
–
–
Double-barreled questions
Double negatives
Loaded terms
Assumed knowledge
Vagueness
Lingo/use of unfamiliar
terminology
Questions 1 through 6 ask
respondents to share
information about
themselves; analysis can
then focus on questions
averages and means in
respect to each. Faculty
suggested inserting
“elementary education” in
#2, and deleting some
questions altogether.
Faculty also suggested a
question asking students
to identify how far through
a program they had
progressed.
Example: Survey Prior to Revision
Graduate Student Survey
Indiana State University
Demographic Data. Please check the box that most accurately represents your situation.
1.
Your present teaching position?
1 Early Childhood Education
2 Elementary Education
3 Junior High/Middle School Education
4 Special Education
5 Other ____________________
6 Not involved in education right now
3.
Were you formally prepared
for all areas in which you are
teaching?
1
2
6.
4.
2. Your college-based preparation for teaching?
Check all that apply
1
Early Childhood Education
2
Middle Childhood Education
3
Junior High/Middle School Education
4
Special Education
5
None of these
Are you teaching on a limited
license?
1
2
Yes
No
5.
Please indicate the year you
graduated from ISU
__________
Yes
No
What is your teaching area(s)? Check all that apply
1 Art
7 Music Education
2 Computer Education
8 Reading
3 Elementary Education
9 School Media Services
4 English Education
10 Science Education
5 Foreign Languages
11 Social Studies Education
6 Mathematics Education
12 Special Education
13
14
15
16
Health and Safety Education
Physical Education
Technology Education
Other___________________
Please circle the number that most closely represents your assessment about how well Indiana State
University prepared you for each item listed below.
Several faculty were
concerned about the 4-point
scale and suggested the
insertion of a middle point.
Poorly Prepared
Somewhat Prepared
Well Prepared
Very Well Prepared
7.
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
8.
9.
10.
11.
1
2
3
4
1
2
3
4
12.
Use knowledge of child
development and relationships
1
2
with students and families to plan
instruction based on abilities,
interests, and needs.
Use knowledge of subject matter
and curriculum to design
1
2
developmentally appropriate
lessons within and across the
disciplines.
Establish caring, inclusive,
stimulating and safe environment
1
2
that facilitates learning,
collaboration, independence, and
intellectual risk taking.
Help students learn to respect
1
2
individual and group differences.
Use a variety of methods,
strategies, and materials to
1
2
promote development, learning,
and cooperation.
Work with colleagues to improve
1
2
schools, programs, and practice.
Please continue on next page
Not Very Important
Somewhat Important
Important
Very Important
How important is this skill?
3
4
3
4
3
4
3
4
How important is this skill?
How important is this skill?
How important is this skill?
3
4
How important is this skill?
3
4
How important is this skill?
Example: Survey After First Revision
Graduate Student Survey
Indiana State University
Elementary education
was included in #2,
question #4 included a
question about on-line
courses, and #5 about
how far students had
progressed through the
graduate program.
Demographic Data. Please check the box that most accurately represents your situation.
1.
Your present teaching position is in what area?
1 Early Childhood Education
2 Elementary Education
3 Junior High/Middle School Education
4 Special Education
5 Other ____________________
6 Not involved in teaching right now
3. Are you currently enrolled in
the Transition to Teaching
program?
1 Yes
2 No
2.
Your undergraduate preparation for teaching
was in what areas? Check all that apply
1
Early Childhood Education
2
Elementary Education
3
Junior High/Middle School Education
4
Secondary Education
5
Special Education
6
None of these
4.About what percentage of
5. About how many graduate
your courses are taken oncredit hours will you have
line?
completed by the end of this
semester?
1 Less than 25%
1 Less than 15 graduate
2 About 50%
credits
3 More than 50%
2 More than 15 graduate
credits
6. What is your teaching area(s)? Check all that apply
1
Art
8
Music Education
2
Computer Education
9
Reading
3
Early Childhood Education
10 School Media Services
4
Elementary Education
11 Science Education
5
English Education
12 Social Studies Education
6
Foreign Languages
13 Special Education
7
Mathematics Education
14
15
16
17
Health and Safety Education
Physical Education
Technology Education
Other___________________
Please circle the number that most closely represents your assessment about how well Indiana State
University prepared you for each item listed below.
Note insertion
of mid point
and changes
in meaning of
1 through 5 (in
black box).
Also note
inclusion of –
ing words to
begin each
sentence in
each question.
1
2
3
4
5
Very Poorly Prepared
Poorly Prepared
Somewhat Prepared
Well Prepared
Very Well Prepared
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
7. Using knowledge of child/adolescent
development and relationships with
students and families to plan instruction
based on abilities, interests, and needs.
8. Using knowledge of subject matter
and curriculum to design
developmentally appropriate lessons
within and across the disciplines.
9. Establishing caring, inclusive,
stimulating and safe environment that
facilitates learning, collaboration,
independence, and intellectual risk
taking.
10. Helping students learn to respect
individual and group differences.
11. Using a variety of methods,
strategies, and materials (including
technology) to promote development,
learning, and cooperation.
Not At All Important
Not Important
Somewhat Important
Important
Very Important
1
2
3
4
5
How important is
this skill?
1
2
3
4
5
How important is
this skill?
1
2
3
4
5
1
2
3
4
5
How important is
this skill?
1
2
3
4
5
How important is
this skill?
How important is
this skill?
Define Respondents
1.
2.
3.
Who are the respondents?
How will they be contacted?
How to maximize response
rate?
Answers to these questions will
provide guidelines for
problem solving. Identifying
problems after attempting
assessment might provide
for “lessons learned,” but will
not quench a need to
generate useful assessment
data.
Identify Audience
1.
2.
3.
Who will use this
information?
Will questions on
survey enable audience
to create meaning?
In what ways will the
survey facilitate focus
on findings, and
thereby encourage
discussion about how
to improve student
learning in respect to
program objectives?
Analyze Results
• Survey questionnaires
often pre-determine the
kinds of analysis that can
be conducted
• Types of analyses:
– Descriptive statistics
(means, medians, etc.)
– Correlation analysis
– Regression and logistic
regression
– ANOVA
– Graphs: Bar, Boxplots,
ANOVA, etc.
More than 180
surveys were
inputted onto
Excel. Results
show minimum
and maximum
scores, means
and standard
deviations for
faculty review.
Means that had
medians of 4 (the
highest value on
the survey) were
noted with
asterisk (*); all
others had
median of 3.
Results were
shared with
faculty which
then discussed
findings.
Example of Tabular Information
N
Minimum
Mean[2]
Maximum
Std. Deviation[3]
Q6: Designing units of instruction
182
1
4
3.29*
.825
Q7:Designing classroom assessment
182
1
4
3.09
.839
Q8:Creating meaningful learning
experiences based on content
knowledge
183
1
4
3.34*
.675
Q9: Assessing own professional growth
needs
183
1
4
3.10
.822
Q10: Using technology
183
1
4
3.00
.908
Q11: Designing and using formative
assessments
183
1
4
2.98
.795
Q12: Using teaching approaches that
are sensitive to diverse learners
182
1
4
3.23
.820
Q13: Using information about
community and student backgrounds
183
1
4
2.90
.842
Q14: Using multiple teaching strategies
181
1
4
3.34*
.724
Q15: Designing learning environment to
actively engage students
181
2
4
3.40*
.672
Q16: Serving as a respected role model
181
2
4
3.66*
.508
Q17: Designing developmentally
appropriate learning tasks
181
1
4
3.40*
.697
Q18: Working collegially with peers
181
1
4
3.32*
.728
Q19: Mediating student learning
181
1
4
3.20
.736
Q20: Designing and using classroom
assessments
180
1
4
3.07
.759
Q21: Reflecting on teaching
181
2
4
3.41*
.641
Q22: Engaging students in problembased learning
179
1
4
3.22
.729
Q23: Developing productive
relationships with parents or guardians
179
1
4
2.85
.925
Valid N (listwise)
176
Example of Graphical Information
3.5
3.4
3.3
Mean of Q9
When
respondents note
themselves by
year, question
averages can be
placed on graph.
This is especially
helpful when
faculty are
interested in
viewing trends
over years. In this
case, survey were
originally placed
in an Excel
spreadsheet, then
imported on
SPSS, where a
“means plots”
(under ANOVA)
was run.
3.2
3.1
3
2.9
2.8
2002
2003
2004
GRAD_YR
2005
Present Results
1.
2.
3.
4.
5.
Limit results to interest of your
audience
Limit results to significant
findings; often statistical
analysis will help with
identification of these findings,
although familiarity with
institutional history and program
objectives is helpful
Focus on both strengths and
weaknesses revealed from
findings
Keep in mind that purpose of
presentation is not to discuss
statistics and methodology, but
to encourage discussion about
what to do about findings
Keep it simple!
Define Periodicity
• It is very important
that faculty, program
directors, and
department chairs
manage assessment
on a regular basis
• Collect, summarize,
present, and produce
action items (based
on assessment) on a
regular schedule
Triangulate
• It is often helpful to use
several different sources
of information (surveys
and other assessments)
so that common themes
can be detected from
different sources
• Triangulation means that
these themes are
detected from different
sources (preferably
detected in faculty
meetings)
Example: Summaries
Assessment: Graduate
Student Surveys
Graduate
student,
alumni, and
employer
survey results
in respect to a
specific
standard are
listed. When
placed on a
grid, general
comparative
themes may
be detected by
faculty.
Assessment: Graduate
Alumni Surveys
Assessment:
Graduate Employer
Surveys
Assessment: Expert Panel
NBPTS 1: Teachers Are Committed to Students and Their Learning
1.1: Teachers recognize
individual
differences in their
students and
adjust their
practice
accordingly
q21, "providing multiple paths
to learning school
subjects, themes and
topic" (4-"well
prepared")
q8: "Using a variety of methods,
strategies, and materials
to promote
development, learning
and cooperation" (3.29-satisfactory);
q5: "Using a variety of
methods, strategies,
and materials to
promote development,
learning and
cooperation" (4.14-good);
Graduate students know about
a wide range of
research-based
pedagogies--expert
panel average was 3.7
(satisfactory/well
prepared)
1.2: Teachers have an
understanding of
how students
develop and learn
q7: "Using knowledge of
child/adolescent
development and
relationships with
students and families
to plan instruction
based on abilities,
interests and needs"
(3.0--satisfactory)
q4: "Using knowledge of
child/adolescent
development and
relationships with
students and families to
plan instruction based
on abilities, interests and
needs" (3.14-satisfactory)
q1: "Using knowledge of
child/adolescent
development and
relationships with
students and families
to plan instruction
based on abilities,
interests and needs"
(4--well prepared)
Graduate students know how
to teach based upon
what is
developmentally
appropriate--expert
panel average was 3.8
(satisfactory/well
prepared)
1.3: Teachers treat
students equitably
q9: "Establishing caring,
inclusive, stimulating
and safe
environments that
facilitate learning,
collaboration,
independence, and
intellectual risk taking"
(4.0--"well prepared")
q6: "Establishing caring,
inclusive, stimulating
and safe environments
that facilitate learning,
collaboration,
independence, and
intellectual risk taking"
(3.0--satisfactory)
q3: "Establishing caring,
inclusive, stimulating
and safe environments
that facilitate learning,
collaboration,
independence, and
intellectual risk taking"
(3.0--satisfactory)
Graduate students have
developed a
recognition of
professional ethics of
their discipline--expert
panel average was 4
(well prepared)
1.4: Teachers' mission
extends beyond
developing the
cognitive capacity
of their students
q19: "Helping students
explore important life
issues, and world
issues" (3.0--satisfactory)
q19: "Helping students explore
important life issues,
and world issues" (2.57-satisfactory)
q16: "Helping students explore
important life issues,
and world issues"
(3.43--satisfactory)
Graduate students appreciate
how knowledge is
linked to other
disciplines--expert
panel average was 3.4
(satisfactory).
Example: Summary Statements Based on Triangulation
Assessment Summary: Teacher Candidate Strengths and Weaknesses
Strengths:
These general
themes are then
listed on a chart
of strengths and
weaknesses
and shared with
program or
department
faculty, who
then discuss
actions that
might be taken
to further
enhance
student learning
outcomes
and/or program
objectives.
Weaknesses:
Teacher candidates can clearly explain important
principles and concepts delineated in professional, state,
and institutional standards
Teacher candidates need more instruction and practice in
working with parents and guardians
Teacher candidates reflect on their practice and make
necessary judgments
Teacher candidates should learn more about and practice
pedagogical skills
Teacher candidates know how their students learn
Teacher candidates need more training on behavior
management and student discipline
Teacher candidates recognize when their own
dispositions may need to be adjusted
Teacher candidates need to know more about their
communities, and working within communities to enhance
student learning
Teacher candidates need more knowledge of assessments,
and in accurately using those assessments in the classroom
Creating Meaning and Closing the
Loop!
• Ideally, faculty play a role in
designing surveys, reviewing
their use, analyzing data, and
producing findings
• Important note: Survey findings
can only be useful in
assessment if they lead to
general findings about student
learning and program
objectives
• Important also: Any
assessment can only be useful
if it leads to documented action
that leads to enhancement of
student learning and/or
program objectives
Concluding Remarks
1.
2.
3.
4.
Survey assessments are considered indirect assessments. Therefore, it
is best to compare findings with direct assessments of student learning
Survey assessments can be very useful for observing what students
believe they are learning, what alumni feel that have learned, and how
well employers feel graduates have been prepared
Survey assessments create very useful findings if a program or
department is concerned about the quality of student preparation (i.e.,
employer, mentor, or internship surveys).
Open-ended survey responses can also be analyzed to detect trends or
concerns. Although not a focus of this analysis, useful information can be
gained through the systematic analysis of open-ended questions.
Descargar

Using Surveys To Assess Student Learning