Evaluation Designs and Methods
Program Evaluation Basics Webinar Series
Mary E. Arnold, Ph.D.
Professor and Youth Development Specialist
Oregon State University
4-H Professional Development Webinar
February 14, 2013
Webinar Agenda
Building on previous month’s topic of focusing and planning
your evaluation we will:
• Explore the concept of rigor in evaluation, and its particular place
in Extension and 4-H evaluation
• Learn the role that evaluation questions play in determining
evaluation design and data collection methods
• Explore common evaluation designs
• Explore common and innovative evaluation methods
• Learn of resources available to support evaluation efforts in 4-H
and Extension
Elements of Rigor
• Evaluation design
• Conceptualization of program constructs & outcomes
• Measurement strategies
• Timeframe of the evaluation study
• Program integrity
• Program participation and attrition
• Statistical analyses
Braverman, M. T., & Arnold, M. E. (2008). An evaluator’s balancing act: Maintaining rigor while being responsive to multiple
stakeholders. In M. T. Braverman, M. Engel, R. A. Rennekamp, & M. E. Arnold (Eds.) Program evaluation in a complex
organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 71-86.
Rigor and the 4-H Organization
• Who determines standards of rigor?
• How do decisions about evaluation
methods get made?
• How, and to what extent, is the quality
of a completed evaluation determined?
Post Only Design
Evaluation Question Example:
What skills do campers report developing at science camp?
TIME
(X
O = “Observation” (data collection)
O)
X = “intervention” (program)
Never
Sometimes
Usually
Always
I can use scientific knowledge to form a question
1
2
3
4
I can ask a question that can be answered by collecting data
1
2
3
4
I can design a scientific procedure to answer a question
1
2
3
4
Post Only Control Group Design
Evaluation Question Example:
Do youth who attend 4-H summer science camp have better
science skills than youth who do not attend?
E (X
C (X
O)
O)
TIME
O = “Observation” (data collection)
X = “intervention” (program)
E = Experimental group (program participants)
C = Control group (non-participants)
Never
Sometimes
Usually
Always
I can use scientific knowledge to form a question
1
2
3
4
I can ask a question that can be answered by collecting data
1
2
3
4
I can design a scientific procedure to answer a question
1
2
3
4
What is the level of rigor? What cannot be said?
One Group Pre-Test/Post-Test
Evaluation Question Example:
Do youth have higher levels of positive youth development
at the end of the program than they did at the beginning?
(O
X
TIME
O)
Strongly
Disagree
Disagree
Agree
Strongly
Agree
I feel good about my scholastic ability
O
O
O
O
I feel accepted by my friends
O
O
O
O
I can figure out right from wrong
O
O
O
O
I can do things that make a difference
O
O
O
O
What is the level of rigor?
What cannot be said?
Retrospective Pre-Test
Evaluation Question Example:
Do youth have higher levels of positive youth development
at the end of the program than they did at the beginning?
TIME
(O
X
TIME
O)
For each of the following items, please indicate how you felt before
participating in this program, and how you feel now after participating in
this program.
1 = Strongly disagree
2 = Disagree
3 = Agree
4 Strongly Agree
Before
I feel accepted by my friends
I can figure out right from wrong
I can do things that make a difference
After
1
2
3
4
1
2
3
4
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
Control Group Pre-Test/Post Test
Evaluation Question Example:
Do youth in program develop higher levels of PYD than
youth who do not participate?
E (O
C (O
X
---
O)
O)
Strongly
Disagree
Disagree
Agree
Strongly
Agree
I feel good about my scholastic ability
O
O
O
O
I feel accepted by my friends
O
O
O
O
I can figure out right from wrong
O
O
O
O
I can do things that make a difference
O
O
O
O
LOW -
Level of PYD -
HIGH
E
C
TIME
HIGH
Time Series Design with Control Group
LOW -
Level of PYD -
E
C
O
O
O
O
X
O
O
O
Choosing an Evaluation Data Collection Method
Some Common Methods
• Archival data (records and documents)
• Surveys (mailed, electronic, phone)
• Interviews (phone, face to face, group)
• Focus group interviews
• Observation
• Tests (scenarios or skill/knowledge tests)
Important Steps
What is the key concept that must be measured in each evaluation question?
Did youth participants in the YA4-H! Teens as Teachers program pilot
increase their own consumption of fruits and vegetables?
Who has knowledge of this potential change?
Several sources may emerge: Youth,
parents, leaders, teachers, friends, records,
observations.
What sources of data will be acceptable to
stakeholders?
What expertise and funding is available to
make a particular method practical?
Existing Documents and Records
Have you ever considered meeting notes,
minutes, videos, registrations, test scores,
forms, records, reports as possible sources
of data?
• Considerably more cost effective than
original data collection
• Data are not affected by the act of
collecting it
• Programs collect lots of information that
is never used, and too often we forget to
look for existing data that can answer the
question
What existing information could answer the question of increased fruits
and vegetable consumption?
Surveys (Mailed, Electronic, Phone)
Designing Surveys: A Guide to Decisions and Procedures.
Ronald Czaja and Johnny Blair (2005)
Internet, Mail, and Mixed-Mode Surveys: Tailored Design Method (3rd Ed.)
John Dillman, Jolene Smyth, & Leah Melani Christian (2008)
How to Conduct Surveys: A Step by Step Guide
Arlene Fink (2013)
Survey Research Methods (4th Ed.)
Floyd Fowler (2009)
How could surveys help us assess increase in youth consumption of
fruits and vegetables?
Interviews (Phone, Face to Face, Group)
Designing and Conducting Your First Interview (text book)
Bruce Friesen (2010)
Qualitative Interviewing: The Art of Hearing Data (3rd Ed.)
Hebert and Irene Rubin (2011)
Focus Groups; A Practical Guide for Applied Research (2nd Ed.)
Richard Krueger (1994)
Interviewing as Qualitative Research: A Guide for Researchers in Education
and the Social Sciences (4th Ed.)
Irving Seidman (2012)
Why might interviews be a good method for collecting data
about youth consumption of fruits and vegetables?
Direct Observation
Situations where you want direct information
• May be more reliable than asking people if they are using new practices
When you are trying understand an ongoing behavior, process or unfolding
situation or event
• Observing camp counselors before, during and after a training program
When there is physical evidence, products or outcomes that can be readily seen
• Inspecting project records, newsletters, signs
When written or other data collection procedures seem appropriate
• Programs to vulnerable or underserved audiences, when language or
literacy is a problem
Key Resource
Collecting Evaluation Data: Direction Observation
Ellen Taylor-Powell and Sara Steele- Booklet available on the State 4-H Website
Could we use direct observations to measure youth consumption of
fruits and vegetable?
Tests and Scenarios
Useful when assessing learning that requires specific knowledge that must be
turned into action for the program to be considered successful.
Requires the ability to do a real pre –post test process
Can be very creative! Think skits and role plays
• Camp counselor training
• Youth leadership programs
• Risk management training
Can you think of how a scenario evaluation might be a useful
method for assessing youth consumption of fruits and
vegetables?
Summary
Your Evaluation Question
Determines Your Design
Determines Your Methods
Stay tuned on March 6th for our next webinar, which will focus more in
depth on creating high quality questionnaires!
That’s all for now!
Join in next month for:
Creating High Quality Questionnaires
Don’t forget to complete an evaluation of today’s webinar at:
http://www.surveymonkey.com/s/4HEvaluationwebinar
Descargar

Slide 1