Partnering with an Evaluator
Presented by Joanne Kahn
1
Session goals
2

Understand how to hire and partner with a
consultant to assist you with evaluation.

Set the stage for an effective evaluation.
Steps in the partnership
1 - Project definition
2- Vendor selection
3- Contract negotiation
4- Data collection
5- Reporting
3
Project Definition: what to include?
Things you could do in-house:
 Library use statistics
 Point-of-use inquiry by staff
 Skills tests
 Feedback forms/surveys (but could get design help)
Outside firm helpful for:
 Survey design
 Focus groups
 Interviews
 Observation
 Instructor assessments
4
Steps in the partnership
Project definition
Vendor selection
Contract negotiation
Data collection
Reporting
5
Vendor selection
Does your library have an existing relationship with a
particular research firm?
To find an evaluator, ask peers or check the American
Evaluators Association website www.eval.org.
Members commit to:
 Systematic inquiry
 Competence
 Integrity/honesty
 Respect for people
 Responsibilities for general and public welfare
6
Vendor selection
Interview for:
 Compatibility with your goals
 Flexibility
 Experience with methods you want to use
 Staff who speak/write in needed languages
 Schedule availability
 Willing to work within your budget constraints
 A portfolio of similar work
 References — are they easy to work with?
See handout: article from “Usable Knowledge.”
www.usablellc.net/White_Papers/Hiring%20an%20Evaluation%20Consultant.pdf
7
Points to discuss with the evaluator
8

Timing and frequency

Who will do each task?

Which participants? How many?

Protecting the confidentiality of participants’
personal information

How will participants be recruited?

Incentives and perhaps transportation for
participants
Evaluator’s perspective*

Focus on the story you want to tell stakeholders,
and collect the data you need to tell it.

If you change staff (or delegate), make sure
everyone knows the goals.

Plan how you will handle unexpected or negative
feedback. The evaluator will take the approach
that all information is useful.
*Special thanks to Juliette Mackin of NPC Research.
9
Steps in the partnership
Project definition
Vendor selection
Contract negotiation
Data collection
Reporting
10
Contract negotiation

Ask for a written proposal and fixed-price bid, and
understand how the total was reached (hourly rates
and line-item costs).

If bid exceeds budget, look for ways to reduce costs:
 Junior staff members may be less expensive than
partners
 You may be able to reduce their time by:



11
Arranging the venue, incentives, and refreshments
Providing pre-screened participants with phone/email
contact info
Bundling this project with another
Contract negotiation
12

Set a schedule that includes time for review and
revisions.

Agree up front:
 What intermediate deliverables will be provided?
 Who needs to sign-off on the report?
Steps in the partnership
Project definition
Vendor selection
Contract negotiation
Data collection
Reporting
13
Data collection — surveys
14

You may draft questions for the evaluator to review
and include.

Have you made the process as participant-friendly
as possible?

Measure before/after results.

Are the questions clear? For multiple choice
questions, are the choices adequate?

Are there open-ended questions?
Data collection — focus groups
15

How will you recruit/screen participants?

Have you made the logistics as participant-friendly
as possible?

The evaluator will probably want you out of the
room.

Review script for completeness, especially follow-on
questions.

Will there be a transcript or recording?
Steps in the partnership
Project definition
Vendor selection
Contract negotiation
Data collection
Reporting
16
Reporting
Respect the firm’s objectivity!
But it’s fair to ask them to have a positive outlook.
17
Reporting: make it readable!
o Executive summary conveys the main points
o Outline is easy to follow
o Visually attractive – photos!!
o Do tables/graphs improve the report?
o Are patron comments or anecdotes included?
o Patterns — do data sources corroborate each other?
o Conclusions should follow from the data
18
Descargar

Smart Investing @ your library Orientation