Performance Measurement
in Academic Libraries
Martha Kyrillidou, ARL
Steve Hiller, University of Washington
Jim Self, University of Virginia
EBLIP 4 Workshop
North Carolina
May 11, 2007
ARL Tools and R&D
Martha Kyrillidou
Director, ARL Statistics and Service Quality Programs
Association of Research Libraries
“Life is not measured by the breaths we
take...but by the moments that take our
breath away.”
ARL
www.arl.org
. . . chart a course for every endeavor
that we take the people's money for, see
how well we are progressing, tell the
public how we are doing, stop the things
that don't work, and never stop
improving the things that we think are
worth investing in.
– President William J. Clinton, on signing the Government
Performance and Results Act of 1993
ARL
www.arl.org
University of Hawaii
• Goal 1: Educational Effectiveness and Student
Success
• Goal 2: A Learning, Research, and Service
Network
• Goal 3: A Model Local, Regional, and Global
University
• Goal 4: Investment in Faculty, Staff, Students, and
Their Environment
• Goal 5: Resources and Stewardship
» Measuring Our Progress Report 2006,
http://www.hawaii.edu/ovppp/mop/mop06_webaccesible.html#
satisfaction
ARL
www.arl.org
University of Hawaii (cont’d)
Student Engagement
How engaged are University of Hawaii students in
their educational experience at upper division/fouryear campuses?
Benchmark #1 Level of Academic Challenge
Benchmark #2 Active and Collaborative Learning
Benchmark #3 Student-Faculty Interaction
Benchmark #4 Enriching Educational Experiences
Benchmark #5 Supportive Campus Environment
ARL
www.arl.org
University of Hawaii (cont’d)
Information and Technology Resources: Library
How does U H’s major library compare on a national basis?
U H Manoa ranks 68th among the 113 ranked university
libraries that are members of the Association of Research
Libraries (A R L).
Source: 2003–2004 A R L Membership and Statistics
The indexed ranking is based on the number of volumes held,
number of volumes added in the last fiscal year, number of
current serials, number of permanent staff, and total
operating expenditures.
The library aspires to regain its previous higher standing
which was significantly impacted by budget cuts in the
mid- to late 1990s, and from which the library has been
slowly recovering.
ARL
www.arl.org
Mission:
Shaping the future of research libraries in the
changing environment of public policy and scholarly
communication.
Members:
123 major research libraries in North America.
Ratios:
4 percent of the higher education institutions
providing 40 percent of the information resources.
Users:
Three million students and faculty served.
Expenditures: 37 percent is invested in access to electronic
resources.
ARL
www.arl.org
Library Assessment in an
Electronic Era
What are some of the current developments with library
assessments efforts?
ARL StatsQUAL™
E-Metrics
LibQUAL+®
DigiQUAL™
MINES for Libraries™
Where are the most critical needs and opportunities?
What are the lessons learned?
ARL
www.arl.org
Thinking Strategically About
Libraries’ Futures
• What is the central work of the library and how can we do
more, differently, and at less cost?
• What important set of services does the library provide that
others can’t? What new roles are needed?
• What advantages does the library possess?
• What will be the most needed by our community of users in
the next decade? How is user behavior changing?
• What should our libraries aspire to be ten years from now?
What are the implications of technology driven change?
• What are the essential factors responsible for the success
of the library?
ARL
www.arl.org
Defining Success in a Digital
Environment
• Crafting new measures of success.
• Moving from measuring inputs to outputs and outcomes
• Understanding impact of library roles and services.
• Agreeing on qualitative measures of success: user
perceptions, user success, creating value, advancing HE
goals.
• Reallocating and managing capabilities to focus on new
definitions of success.
ARL
www.arl.org
Updating the Traditional ARL
Statistics
• E-Metrics = ARL Supplementary Statistics
– On going efforts to update and refine core data.
– Exploring feasibility of collecting e-metrics.
• ARL Task Force on New Ways of Measuring
Collections :
– Growing concern with utility of membership index.
– Study ARL statistics to determine relevance.
– Develop Profile of Emerging Research Libraries.
ARL
www.arl.org
The LibQUAL+® Update
• The LibQUAL+® premise, dimensions, and
methodology
• LibQUAL+® Results
• LibQUAL+® in Action
ARL
www.arl.org
The LibQUAL+® premise,
dimensions, and
methodology
The Need for LibQUAL+®
• Underlying need to demonstrate our worth
• The reallocation of resources from
traditional services and functions
• Rapid shifts in information-seeking behavior
• Need to keep abreast of customer demands
• Increasing user demands
ARL
www.arl.org
Why Use LibQUAL+®?
Feedback from LibQUAL+® Users
“Why did you choose to use LibQUAL+®?”
• LibQUAL+® was recommended to us as offering a
well designed, thoroughly Library-focused set of
survey tools
• Opportunity to benchmark
• Cost-effectiveness
• Automated processing & fast delivery of results
• Respectability and comparability (with others and
historically)
ARL
www.arl.org
The LibQUAL+® Premise
PERCEPTIONS
SERVICE
“….only customers judge quality; all other
judgments are essentially irrelevant”
Zeithaml, Parasuraman, Berry. (1999).
Delivering quality service. NY: The Free Press.
ARL
www.arl.org
Multiple Methods for Listening
to Customers
•
•
•
•
•
•
•
•
•
•
•
Transactional surveys*
Mystery shopping
New, declining, and lost-customer surveys
Focus group interviews
Customer advisory panels
Service reviews
Customer complaint, comment, and inquiry capture
Total market surveys*
Employee field reporting
Employee surveys
Service operating data capture
*A SERVQUAL-type instrument is most suitable for these methods
ARL
www.arl.org
World LibQUAL+® Survey 2005
ARL
www.arl.org
LibQUAL+® Languages
American English
French Canadian
Swedish
British English
Afrikaans
Dutch English
Dutch
Continental French
Swedish
(British English)
German
Norwegian
Finnish
Danish
Over 1,000 institutions
1,000,000 respondents
ARL
www.arl.org
Survey Instrument – “22
items…
ARL
www.arl.org
and A Box.”
• Why the Box is so Important:
– About 40% of participants provide open-ended
comments, and these are linked to
demographics and quantitative data
– Users elaborate the details of their concerns
– Users feel the need to be constructive in their
criticisms, and offer specific suggestions for
action
ARL
www.arl.org
LibQUAL+® Results
Understanding LibQUAL+®
Results
• For the 22 items LibQUAL+ asks users’ to rate
their:
• Minimum service level
• Desired service level
• Perceived service performance
• This gives us a ‘Zone of Tolerance’ for each
question; the distance between minimally
acceptable and desired service ratings
• Perception ratings ideally fall within the Zone of
Tolerance
ARL
www.arl.org
General Findings
• Highly desired
• Making electronic resources accessible from my
home or office
• Print and/or electronic journals I require for my
work
• A haven for study, learning or research
• Lowest
• Library staff who instil confidence in users
• Giving users individual attention
• Space for group learning and group study
ARL
www.arl.org
LibQUAL+® in Action
Using LibQUAL+® Results
• Strategic Service Developments
• Data to support service development
• Ability to identify where not meeting expectations
• Measure if change has met need
• Budget Discussions
• Data to support bid for increased funding
• Data to support case for change in emphasis (towards
e-provision)
• Marketing Position
• Status of the library within the University
• Importance of national & international benchmarking
ARL
www.arl.org
In Closing, LibQUAL+®:
• Focuses on the users’ point of view
(outcomes)
• Requires limited local survey expertise and
resources
• Analysis available at local, national and
inter-institutional levels
• Offers opportunities for highlighting and
improving your status within the institution
• Can help in securing funding for the Library
ARL
www.arl.org
"Each organization must create and
communicate performance measures
that reflect its unique strategy."
Dr. Robert S. Kaplan, Harvard Business School
ARL
www.arl.org
Developing the DigiQUAL™ Protocol for
Digital Library Evaluation
• Building on the LibQUAL+® experience
• Secures feedback on user’s perceptions of library’s web site
• Five questions on services, functionality, and content
• Goal is to determine utility, reliability, and trustworthiness
DigiQUAL™ Dimensions
Accessibility
Navigability
Interoperability
Collection building
Resource Use
Evaluating collections
DL as community: users, developers, reviewers
Copyright
Role of Federations
DL Sustainability
ARL
www.arl.org
Outstanding Issues and
Challenges
• Unique DLs: niche market, critical mass, both?
• Balance:
– custom vs. generic content  results
– flexible vs. standard implementation  scaling
• Mixed methods
– Preserving user privacy
– Collecting truly useful data
• Moving target: digital libraries as… it depends.
ARL
www.arl.org
Assessing the Value of
Networked Electronic Services
The MINES Survey
Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™
ARL
www.arl.org
What is MINES for Libraries™?
• A research methodology consisting of a webbased survey form and a sampling plan.
• Measures who is using electronic resources,
where users are located at the time of use, and
their purpose of use.
• Adopted by the Association of Research Libraries
(ARL) as a part of the “New Measures” toolkit May,
2003.
• Different from other electronic resource usage
measures that quantify total usage or measure
how well a library makes electronic resources
available.
ARL
www.arl.org
Questions Addressed
• How extensively do sponsored researchers use
the new digital information environment?
• Are there differences in usage of electronic
information based on the user’s location (e.g., in
the library; on-campus, but not in the library; or offcampus)?
• What is a statistically valid methodology for
capturing electronic services usage both in the
library and remotely through web surveys?
• Are particular network configurations more
conducive to studies of digital libraries patron use?
ARL
www.arl.org
Library User Survey
ARL
www.arl.org
What are the most critical assessment
needs and opportunities?
• Complementing LibQUAL+® with additional measures.
• Developing impact studies on user success, economic
value, and community return on investment.
• Moving target: what is a digital library?
• E-Resources: understanding usage.
• Gaining acceptance and use of standard measures for eresources.
• Building a climate of assessment throughout library.
ARL
www.arl.org
What is the lesson learned?
• Building standardized assessment
methods and tools are a key component
of understanding users, performance
measurement, and improvement of
services.
ARL
www.arl.org
In Closing
• As higher education is challenged on
accountability and effectiveness issues so
will libraries.
• A growing appreciation of need for fresh
assessment measures, techniques, and
processes - old arguments don’t work.
• Basic questions of role, vision, and impact
must be answered by library community.
ARL
www.arl.org
What’s in a word?
LIBRARY
ARL
www.arl.org
A word is not crystal, transparent and
unchanged; it is the skin of a living
thought, and may vary greatly in color
and content according to the
circumstances and time in which it is
used.
--Justice Oliver Wendell Holmes
ARL
www.arl.org
User Needs Assessment or Through
the Looking Glass
Steve Hiller
Director of Assessment and Planning
University of Washington Libraries
User Needs Assessment or Through
the Looking Glass
How have things
changed as far as
getting your
information in
the past 5 years?
“We never
have to go
to the
library.”
(sounds of laughter and lots of paper ripping noise on audio tape)
Faculty Focus Group 2000 (UW College of Education)
ARL
www.arl.org
Traditional Library Core Business
• Physical Collections
– Print (primarily)
– Microforms
– Other (minor)
• Facilities
– House collections
– Customer service and work space
– Staff work space
• Services
– Reference
– Instruction
– Access
ARL
www.arl.org
Use of UW Physical Collections
1995-96 To 2004-05 DOWN
2121895
2100000
1831768
1853964
1900000
1653298
1700000
1453632
1500000
1300000
1100000
1306986
1286490
1166147
1018721
1040719
1155552
1167863
1092525
997831
1006856
1032155
1048437
900000
799325
616683
700000
534266
500000
1995-96
96-97
97-98
98-99
CHECKOUT
ARL
99-00
00-01
2001-02
2002-03
2003-04
2004-05
IN-LIBRARY USE
www.arl.org
UW Libraries Gate Counts DOWN
Gate Count Large
Libraries
Gate Count Branch Libaries
2,000,000
300,000
1,800,000
250,000
1,600,000
1,400,000
200,000
1,200,000
FY2002-03
FY2003-04
150,000
1,000,000
FY2004-05
800,000
100,000
600,000
400,000
50,000
200,000
0
Arch
ARL
Art
East
Asia
Engrg
FishOcean
Foster
Bus
Math
Music
Physic
Astro
Social
Wk
0
Health Sci
Undergrad
Main
www.arl.org
UW Libraries In-Person Reference
Queries DOWN
150,000
141,618
140,000
134,525 (-5%)
130,000
120,746 (-10%)
120,000
106,456 (-12%)
110,000
100,000
2002-03
ARL
2003-04
2004-05
2005-06
www.arl.org
Time for a New Business Model?
Try the Customer-Centered Library
• All services & activities viewed through the eyes of
customers
• Customers determine quality
• Library services and resources add value to the customer
Not better libraries . . . Not better customers . . . but
Assess/Measure the Value the Library Provides the
University Community
“Documenting the libraries contributions to quality teaching,
student outcomes, and research productivity will become
critical.”
(Yvonna Lincoln 2006)
ARL
www.arl.org
The Value of User Needs Assessment
• Decisions based on data not assumptions “assumicide”
Fundamental to User-Centered Library
• Users determine quality, importance and success
• Evaluation and assessment focus on user outcomes
• Align collections, resources and services with user needs
• Identify differences/similarities in needs and use by
academic areas/groups
• Data that can help tell our story
Ensure libraries are responsive to their communities
ARL
www.arl.org
What We Need to Know to Support
Our Communities
• Who are our customers (and potential customers)?
• What are their teaching, learning, clinical and research
interests? How do they work? What’s important for their
work?
• What are their library and information needs?
• How do they currently use library/information services?
• How would they prefer to do so?
• How do they differ from each other in library use/needs?
• How does the library add value to their work?
ARL
www.arl.org
Understand Differences in Your
Community
• Library and information needs and use may differ
substantially by academic area, groups, and culture
• Identifying and understanding these differences enables
libraries to target and market services that add the most
value for each group or area
• Multiple assessment methods, including both
quantitative and qualitative data, can identify
differences and provide the most comprehensive picture
of these communities
ARL
www.arl.org
Multidimensional Library Assessment:
Beyond Counts and Satisfaction Surveys
• Data based decision making needs good data source
• Use multiple assessment methods
• Focus on user work and their information seeking and
using behavior
• Increased reliance on qualitative data to identify issues
from the perspective of users
• Learning from our users
• Partnering with other campus programs
• Repurposing existing data when possible
ARL
www.arl.org
How Do We Measure Value?
UW Assessment Priorities
Customer Needs, Use and Success
•
•
•
•
•
•
•
ARL
Information seeking behavior and use
Patterns of library use
Value of library
User needs
Library contribution to customer success
User satisfaction with services, collections, overall
Data to make informed and wise decisions that lead to
resources and services that contribute to user success
www.arl.org
University of Washington Libraries
Assessment Methods Used
• Large scale user surveys every 3 years (“triennial survey”): 1992,
1995, 1998, 2001, 2004, 2007
– All faculty
– Samples of undergraduate and graduate students
– Research scientists, Health Sciences fellow/residents 2004-
•
•
•
•
•
•
•
ARL
In-library use surveys every 3 years beginning 1993
LibQUAL+™ from 2000-2003
Focus groups/Interviews (annually since 1998)
Observation (guided and non-obtrusive)
Usability
Use statistics/data mining
Information about assessment program available at:
http://www.lib.washington.edu/assessment/
www.arl.org
Our Latest Assessment Method
ARL
www.arl.org
UW Triennial Library Use Survey
Number of Respondents and Response Rates 1992-2004
Large number of respondents allows for analysis within groups
2004
2001
1998
1995
1992
Faculty
1560
40%
1345
36%
1503
40%
1359
31%
1108
28%
Grad
Student
627
40%
597
40%
457
46%
409
41%
560
56%
Undergrad
502
25%
497
25%
787
39%
463
23%
407
41%
ARL
www.arl.org
Reasons for In-Person Library Visits 2001
Faculty and Undergrads Visiting Weekly or More Often
100%
Faculty
90%
80%
Undergrads
70%
60%
50%
40%
30%
20%
10%
0%
Use Collections
ARL
Use space or Only Collections Only space or
services
serviceswww.arl.org
Changes in UW Libraries Use Patterns
1998-2004 (% of each group who use library at least weekly)
Faculty
Visit
1998
Visit
2001
Visit
2004
47%
40%
-15%
29% 73%
-28%
79%
+8%
91%
+15%
78%
59%
-24%
52% 63%
-12%
75%
+19%
87%
+16%
67%
61%
-9%
61%
54%
+26%
57%
+6%
(78%)
(72%)
(76%)
Change
Grad
Change
Undergrad
Change
“Visit library for”
ARL
Remote Remote Remote
2004
1998
2001
43%
www.arl.org
UW Faculty Mode of Use by Academic Area 1998/2004
Non- Weekly
17%
Rem ote
Only
45%
Rem ote &
Visit
32%
Visit Only 6%
Health Sci
1998
ARL
Non- Weekly
8%
Rem ote
Only
74%
(w eekly+)
Non- Weekly
10%
Non- Weekly
25%
Rem ote
Only
26%
Rem ote
Only
23%
Rem ote
Only
61%
Rem ote &
Visit
51%
Rem ote &
Visit
39%
Rem ote &
Visit
16%
1%
Health Sci
2004
Non- Weekly
15%
Rem ote &
Visit
27%
Visit Only10%
2%
Visit Only
10%
Science-Engin Science-Engin Hum-Soc Sci
1998
2004
1998
Non- Weekly,
9%
Rem ote
Only
41%
Rem ote &
Visit
45%
3%
Hum-Soc Sci
2004
www.arl.org
E-Journals Drive Remote Use
Look for E-Journals at Least 2x week Faculty by Area
70%
70%
Health Sciences
60%
60%
ScienceEngineering
50%
50%
40%
40%
Journal Article Downloads
4,761,704 (2004-05)
30%
HumanitiesSocial Sciences
20%
10%
20%
10%
2001
ARL
30%
2004
www.arl.org
2004 Resource Type Importance
Faculty By Selected Colleges
5.00
4.80
Journals>1985
4.60
4.40
4.20
4.00
Bib Databases
Books
3.80
3.60
3.40
Journals<1985
3.20
3.00
Humanities
ARL
Social Sci
Business
Science
Engineering Public Health
www.arl.org
2004 Overall Collections Satisfaction
By Group in Selected Colleges
4.60
4.50
Grad
4.40
4.30
Grad
Undergrad
4.20
Undergrad
Faculty
4.10
Faculty
4.00
3.90
3.80
3.70
3.60
Humanities
ARL
Social Sci
Business
Science
Engineering Public Health
www.arl.org
Overall Satisfaction by Group
1995-2004
4.6
4.6
4.5
Faculty 4.44
4.4
4.4
4.33
4.3
4.33
Grad 4.34
Faculty 4.25
4.26
Undergrad 4.32
4.2
4.1
4.2
Grad 4.18
4.11
4.1
4
Undergrad 3.97
3.99
3.9
3.8
3.8
1995
ARL
4.3
4.22
4
3.9
4.5
1998
2001
2004
www.arl.org
2004 Top Priorities by UW Group
More E-Journal titles
Older journals online
Print Coll Quality
Quiet work areas
Undergrads
Grads
Faculty
E-Reserves
Extend wireless
Increase hours
0%
ARL
20%
40%
60%
80%
100%
www.arl.org
UW Faculty Top Priorities by
Academic Area (2004)
More E-Journal titles
Older journals online
Print Collection
Scholarly Pub
Changes
Health Sciences
Science-Engineering
Humanities-Social Sciences
Preservation
Increase hours
0%
ARL
20%
40%
60%
80%
100%
www.arl.org
UW Libraries 2007 Triennial Survey –
Outcomes Question for Faculty
9. What contribution does the UW Libraries make to:
(Likert scale of 1 (minor) to 5 (major)
•
•
•
•
•
•
•
ARL
Keeping current in your field
Finding information in related fields or new areas
Being a more productive researcher
Being a more effective instructor
Enriching student learning experiences
Helping you make more efficient use of your time
Recruiting colleagues and students to UW
www.arl.org
Use of Physical Library
2005 In-Library Use Survey (Triennial)
• One page survey distributed to people entering 15 campus
libraries during 4-6 two hour time blocks
• 3861 surveys returned (minimum of 50 for any library)
• Surveys asked:
– What did you do in this library today*
– How often do you use this library
– How important are these services to you*
– How would you rate these library services and resources*
– What can we do to make this library better for you (write in)
*Option for local library question
ARL
www.arl.org
Respondents by User Group for
Selected Libraries
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Main
Art
Business
Undergrad
ARL
Health Sci
Grad
Faculty
Math
Music
Undergrad
Other
www.arl.org
What Do They Do in the Library?
Activities by Group UW 2005 In-Library Use Survey
(Undergrads 70%, Grads 25%, Faculty/Staff 5%)
70%
60%
50%
Undergrads
Grad Students
40%
Faculty/Staff
30%
20%
10%
0%
Ask for help
ARL
Look for
material
Copy
Work alone
Work in groups
Use Lib
Computer
Use printer
www.arl.org
The Qualitative Provides the Key
• Increasing use of such qualitative methods as, comments
interviews, focus groups, usability, observation
• Statistics often can’t tell us
– Who, how, why
– Value, impact, outcomes
• Qualitative provides information directly from users
– Their language
– Their issues
– Their work
• Qualitative provides understanding
ARL
www.arl.org
Information Literacy Focus Groups:
Findings
• The information environment is too complex
• General search engines (e.g. Google) are preferred
over library licensed/provided interfaces
• Undergrads have difficulty determining which library
sources to use
• Faculty “dumbing down” library research
assignments
• Ubiquity of library research – any place, any time
has changed research patterns
• Availability online is more efficient way to research
• The personal connection with a librarian is important
ARL
www.arl.org
Guided Observation 2003
Bibliographic Database Searching
• Faculty and graduate students search very differently
than we think they should
• Common observations included:
–
–
–
–
–
Prefer to use single keyword search box
Little use of Boolean commands
Limits or format changes rarely employed
Commands need to be on first page or lost
Visible links to full-text critical
• Important features for librarians are not necessarily
important to faculty and students
ARL
www.arl.org
Understanding How Researchers
Work: Four Recent Studies
Focus first on work of faculty/grad students and then
on connection to library
• University of Minnesota Humanities/Social Science
research (2005 Mellon funded)
• New York University 21st Century Library Study (2006)
• Ithaka Faculty survey and interviews on scholarly
communication (2006)
• University of Washington Biosciences Review (2006)
ARL
www.arl.org
Initial Results
• University of Minnesota
– Extremely comfortable with electronic sources
– Inadequate methods for organizing their research materials
• New York University
– Researchers (all disciplines) no longer tied to physical library
– Expectations for info shaped by Web and commercial sector
• Ithaka
– Prefer publishing in widely circulated journals at no cost to them
– Biologists frequently search for information outside their area
• University of Washington
– Start info search outside library space (virtual and physical)
– Could not come up with “new library services” unprompted
ARL
www.arl.org
Biosciences Review Task Force
(2005-06): Reasons for Review
•
•
•
•
Better understand how bioscientists work
Growing interdisciplinarity
Significant change in use patterns
Libraries organizational structure in subjectbased silos
• Value of research enterprise to the University
• Strengthening library connection to research
ARL
www.arl.org
Biosciences Review Process
• Define scope
• Mine existing data
• Acquire new information
–
–
–
–
Environmental scan
Interviews (biosciences faculty)
Focus groups (biosci faculty & students)
Peer library surveys
• Synthesis and first draft
• Final report and recommendations
• Incorporate into Libraries plan
ARL
Jan-May 2006
Feb 2006
Mar-Apr 2006
Apr 2006
May-Aug 2006
Sep-Dec 2006
2007www.arl.org
UW Students, Faculty and Doctorates
Awarded by Academic Area
Undergraduate Majors
Grad/Professional Students
Bioscience
21%
Health Sci
5%
Other 52%
Phy Sci Eng 22%
Faculty
Bioscience
10%
Other 20%
Phy Sci Eng 18%
ARL
Bioscience,
12%
Other 38%
Health Sci
47%
Health Sci
31%
Phy Sci - Eng
18%
Doctorates Awarded
Other 31%
Bioscience
19%
Health Sci,
20%
Phy Sci Eng 30%
www.arl.org
FY 2005 External Funding By Source and UW Faculty Area
Awards by Source (in millions)
Awards by Area (in millions)
Other Non-Fed 105
Other Programs 45
Hum/SocSci/Arts39
State of Wash 21
Industry-Found 79
Other Federal 81
Sciences & Natural
Resources 173
US Dept of Educ 46
US Dept of Def 40
National Science
Foundation 88
Other Health
Sciences 198
Health and Human
Services 536
ARL
Engineering 67
School of Medicine
470
www.arl.org
External Funding Source by Academic Area UW Faculty 2004 Survey
Health Sciences (728)
Sciences-Engineering (389)
None
17%
None
16%
Other Only
11%
Other Only 8%
Humanities/Soc Sci/Arts (365)
100%
90%
80%
70%
60%
50%
Fed and Other
43%
None
62%
Fed and Other
42%
40%
30%
20%
10%
ARL
0%
Fed Only
28%
Fed Only
34%
Other Only
21%
Fed and Other 9%
Fed Only 8%
www.arl.org
Physical Library Use by Group and
Academic Area
(2005 In-Library Use Survey)
Libraries Used
Biology Undergrads (n=126)
Other
Other
Sci
Chem
9%
OUGL
44%
Health
Sci
Health Sci Library Use - Grad by Area (n=186)
Unknown
Main
33%
Other
Health Sci
0%
ARL
20%
40%
60%
80%
100%
www.arl.org
2004 - Top 4 Faculty Priorities
by Academic Area
90%
More ejournal
titles
80%
70%
60%
EJournal backfiles
50%
Print collection
quality
40%
Scholarly
Communications
30%
20%
Health Sci Faculty
ARL
Bio Sci Faculty
Hum-Soc Sci Faculty
www.arl.org
Faculty Interview Themes
•
•
•
•
•
•
ARL
Library seen primarily as E-Journal provider
Physical library used only for items not available online
Start information search with Google and PubMed
Too busy for training, instruction, workshops
Faculty who teach undergrads use libraries differently
Could not come up with “new library services”
unprompted
www.arl.org
Focus Group Themes Print Is Dead, Really
Dead; Our Virtual Space Not Yours
• Google, PubMed, Web of Science starting points for all
• Faculty identify library with E-journals
• Want more online, including older materials; if not online
deliver digitally
• Faculty/many grads go to physical library as last resort
• Too many physical libraries
• E-Science emerging as a new priority
• Lack understanding of many library services, resources
• Undergrads rarely use print unless assigned by faculty
• Increasing overlap between “bio” research and other science
research
ARL
www.arl.org
Task Force Recommendations
• Consolidate collections and service points
– Reduce print holdings; focus on online resources
• Reorganize libraries around broad user communities
• Integrate search/discovery tools into users workflow
• Expand/improve information/service delivery options
• Use an integrated approach to collection allocations
• Increase integration of librarians with user workflow
• Lead scholarly communication and e-science work
• Partner with broader science/technology community
• Provide more targeted communication and marketing
ARL
www.arl.org
What We’ve Learned about the UW
Community
• Libraries are still important source of information used for
teaching, learning and research but lessening in value
• Library needs/use patterns vary by and within academic areas
and groups
• Remote access is preferred method and has changed the way
faculty and students work and use libraries
• Faculty and students find information and use libraries
differently than librarians prefer them too
• Library/information environment is perceived as too complex;
users find simpler ways (Google) to get info
• Customers cannot predict the Libraries future
ARL
www.arl.org
From Data to Outcomes
Jim Self
Director, Management Information Services
University of Virginia Library
The next topics
• Using data for improvement and
reassurance
– LibQUAL+ as a case study
– Corroboration
– Benchmarking
• The Balanced Scorecard
ARL
www.arl.org
“…but to suppose that the facts, once
established in all their fullness, will ‘speak for
themselves’ is an illusion.”
Carl Becker
Annual Address of the President of the
American Historical Association, 1931
LibQUAL+®
Finding the right numbers
Initial Examination
• Focus on the separate user categories
– Faculty
– Undergraduates
– Graduate students
• Tally the number of responses
– 100+ needed in each category
ARL
www.arl.org
UVa 2006
• Responses per category
– Faculty -- 219
– Undergraduates -- 210
– Graduate students -- 244
ARL
www.arl.org
The 22 core questions
• Examine the notebooks
– Scan results for each user category
– Scan the summary charts by dimension
– Rearrange the dimension charts
• Create thermometer graphs
– Question by question
– 22 bars for each user category
• Identify the red zones
ARL
www.arl.org
Dimension Scores
Undergrads
UVa 2006
Grad Students
Library Staff
Faculty
ARL
www.arl.org
LibQUAL+ 2006
Ratings of Affect of Service
9.00
Library Staff
Faculty
Grads
8.00
Undergrads
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
Faculty Affect
ARL
Grad Affect
Undergrad Affect
Library Staff Affect
www.arl.org
LibQUAL+ 2006
Ratings of Information Control
9.00
Faculty
Grads
Library Staff
Undergrads
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
Faculty Info Control
ARL
Grad Info Control
Undergrad Info Control
Library Staff InfoControl
www.arl.org
LibQUAL+ 2006
Ratings of Library as Place
9.00
Undergrads
8.00
Grads
Library Staff
Faculty
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
Faculty Place
ARL
Grad Place
Undergrad Place
Library Staff Place
www.arl.org
LibQUAL+ 2006
University of Virginia Faculty
9.00
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9
ARL
IC-1
IC-2
IC-3
IC-4
IC-5
IC-6
IC-7
IC-8
LP-1
LP-2
LP-3
LP-4
LP-5
www.arl.org
LibQUAL+ 2006
University of Virginia Graduate Students
9.00
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9
ARL
IC-1
IC-2
IC-3
IC-4
IC-5
IC-6
IC-7
IC-8
LP-1
LP-2
LP-3
LP-4
LP-5
www.arl.org
LibQUAL+ 2006
University of Virginia Undergraduate Students
9.00
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9
ARL
IC-1
IC-2
IC-3
IC-4
IC-5
IC-6
IC-7
IC-8
LP-1
LP-2
LP-3
LP-4
LP-5
www.arl.org
Red Zones at UVa
• Journal Collections
– Faculty
– Grad Students
• Website
– Faculty
• Remote access
– Grad students
ARL
www.arl.org
LibQUAL+ 2006
IC-1: Making electronic resources accessible from my home or office
9.00
Faculty
Grads
Library Staff
Undergrads
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
Faculty IC-1
ARL
Grad IC-1
Undergrad IC-1
Lib Staff IC-1
www.arl.org
LibQUAL+ 2006
IC-8: Print and/or electronic journal collections I require for my work
9.00
Faculty
Grads
Library Staff
Undergrads
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
Faculty IC-8
ARL
Grad IC-8
Undergrad IC-8
Lib Staff IC-8
www.arl.org
LibQUAL+ 2006
IC-2: A library Web site enabling me to locate information on my own
9.00
Library Staff
Faculty
Grads
Undergrads
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
Faculty IC-2
ARL
Grad IC-2
Undergrad IC-2
Library Staff IC-2
www.arl.org
Following up with Journals
• Who is unhappy?
– Drill down by discipline and user category.
• Why are they unhappy?
– Read the comments
– Conduct targeted interviews
ARL
www.arl.org
LibQUAL+ 2006
UVA Faculty and Graduate Student Ratings of Journal Collections
9.00
8.00
7.00
Top of Blue Bar = Desired Level of Service
6.00
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
Architecture
Faculty
ARL
Education
Faculty
Engineering
Faculty
Humanities
Faculty
Science/Math
Faculty
Social
Science
Faculty
Architecture
Grads
Education
Grads
Engineering
Grads
Humanities
Grads
Science/Math
Grads
Social
Science
Grads
www.arl.org
Among the 22 core questions,
the most desired are:
• Faculty
– Journal collections 8.60
– Web site 8.49
• Grad Students
• Journal collections 8.61
• Remote access 8.53
• Undergrads
– Modern equipment 8.35
– Comfortable and inviting location 8.29
– Space that inspires study and learning 8.29
ARL
www.arl.org
Among the 22 core questions,
the least desired are:
• Faculty
– Community space for group learning 6.56
– Quiet space for individual activities 7.03
• Grad Students
• Community space for group learning 6.86
• Giving users individual attention 7.27
• Undergrads
– Giving users individual attention 7.06
– Employees who instill confidence 7.30
ARL
www.arl.org
Benchmarking with Peers:
General Satisfaction Questions
• In general, I am satisfied with the way I am
treated at the library.
• In general, I am satisfied with library support
for my learning, research, and/or teaching.
• How would you rate the overall quality of the
service provided by the library?
ARL
www.arl.org
LibQUAL+ 2006
Overall Quality of the Service Provided by the Library
38 ARL Libraries
7.87
7.87
UVA
7.63
7.63
UVA
7.52
7.48
UVA
6.61
Undergraduates
6.51
Graduates
5.87
Faculty
5.5
ARL
www.arl.org
LibQUAL+ 2006
Faculty Ratings of Journal Collections
ARL Libraries
9.00
UVA
8.00
7.00
6.00
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Red Square = Perceived Service Performance
5.00
1
ARL
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
www.arl.org
…beyond LibQUAL+®
• Corroboration
• Data mining
• Balanced Scorecard
ARL
www.arl.org
Corroboration
• Data are more credible if they are supported
by other information
• John Le Carre’s two proofs
ARL
www.arl.org
Analyzing U.Va. Survey Results
• Two Scores for Resources, Services,
Facilities
– Satisfaction = Mean Rating (1 to 5)
– Visibility = Percentage Answering the Question
• Permits comparison over time and among
groups
• Identifies areas that need more attention
ARL
www.arl.org
Reference Activity and Visibility
in Student Surveys
7,000
6,008
Reference Questions
Recorded per Week
in Annual Sample
64%
Visibility
39%
Visibility
34%
Visibility
Reference Visibility
among Undergraduate
75%
Visibililty
1,756
10%
1,000
1993
ARL
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
www.arl.org
Investment and Customer Activity
University of Virginia Library
1993-2006
Acquisitions Expenditures by Format
Customer Activities
$3,000,000
2,400,000
1,800,000
E-Journals
$2,000,000
Circulation
Reference
1,200,000
$1,000,000
600,000
Electronic Resources
Print Monographs
Print Serials
$0
0
FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
ARL
FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
www.arl.org
Benchmarking
Comparisons with Peers
• Within the University
• Within ARL
ARL
www.arl.org
Total Expenditures at UVA
1989-2003
300%
% Change since 1989
Research (+219%)
Other Academic Support
(+200%)
Total Academic Division
(+140%)
Libraries (+81%)
Instruction (+80%)
19
89
19
90
19
91
19
92
19
93
19
94
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
-50%
Fiscal Year
ARL
www.arl.org
Collections Expenditures
UVA vs. ARL
•
A RL Pr o je c tio n
( +1 8 0 % )
190%
A RL Pe r c e n ta g e
% C h an g e Sin ce 1991
140%
G r o w th Th r o u g h 2 0 0 3
( +1 0 2 % )
90%
UV a Pr o je c tio n
( +1 0 4 % )
40%
UV a Pe r c e n ta g e
G r o w th
( +4 8 % )
-10% 1991
2002
2012
Fis c a l Y e a r
ARL
www.arl.org
Median Faculty Salaries
University of Virginia Library
Compared to ARL Median
$55,000
U.Va.
ARL
$50,000
$45,000
$40,000
$35,000
$30,000
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
Fiscal Year
ARL
www.arl.org
The Balanced Scorecard
• A layered and categorized instrument that
– Identifies the important statistics
– Ensures a proper balance
– Organizes multiple statistics into an intelligible
framework
ARL
www.arl.org
The Balanced Scorecard
• Reflects the organization’s vision
• Clarifies and communicates the vision
• Provides a quick, but comprehensive,
picture of the organization’s health
ARL
www.arl.org
The scorecard measures are
“balanced” into four areas
•
•
•
•
ARL
The user perspective
The finance perspective
The internal process perspective
The learning and growth perspective
www.arl.org
Metrics
• Specific targets indicating full success,
partial success, and failure
• At the end of the year we know if we have
met our target for each metric
• The metric may be a complex measure
encompassing several elements
ARL
www.arl.org
The BSC at the U.Va. Library
•
•
•
•
•
ARL
Implemented in 2001
Reports for FY02 to FY06
Completing metrics for FY08
Will tally FY07 in July and August
A work in progress
www.arl.org
Core Questions
• User Perspective
– How well is the library meeting user needs?
• Internal Processes
– Do the library’s processes function efficiently?
• Finance
– How well are the library’s finances managed?
• Learning and Growth
– Is the library well positioned for the future?
ARL
www.arl.org
Metric U.1.A: Overall rating in
student and faculty surveys
• Target1: An average score of at least
4.00 (out of 5.00) from each of the
major constituencies.
• Target2: A score of at least 3.90.
FY06 Result: Target1
– Graduate students 4.08
– Undergraduates 4.11
ARL
www.arl.org
Metric U.3.A: Circulation of new
monographs
• Target1: 60% of newly cataloged
monographs should circulate within two
years.
• Target2: 50% of new monographs should
circulate within two years.
• Result FY06: Target 1.
– 61.2% circulated (15,213 out of 24,852)
ARL
www.arl.org
Metric U.4.B: Turnaround time
for user requests
• Target1: 75% of user requests for new
books should be filled within 7 days.
• Target2: 50% of user requests for new
books should be filled within 7 days.
• Result FY06: Target1.
– 79% filled within 7 days.
ARL
www.arl.org
Metric F.1.B.: Library spending
compared to University expenditures
• Target1: : The University Library will
account for at least 2.50% of the
University’s academic division
expenditures.
• Target2: : The Library will account for at
least 2.25% of expenditures.
• Result FY06: Target1.
– 2.57% ($25.2M of $972M)
ARL
www.arl.org
Metric F.2.A: Unit Cost of
Electronic Serial Use
• Target1: There should be no increase in unit
cost each year.
• Target2: Less than 5% annual increase in
unit cost.
• Result FY06: Target not met.
– 8.8% increase ($2.10 vs. $1.93)
ARL
www.arl.org
Metric I.1.A: Processing Time for
Routine Acquisitions
• Target1: 90% of in-print books from North
America should be processed within one
month.
• Target2: 80% should be processed within
one month.
• Result FY06: Target2.
– 87.2% processed.
ARL
www.arl.org
Metric I.2.A.: Internal
Communications
• Target1: Positive scores (4 or 5) on internal
communications statements from 80% of
respondents in the biennial work-life survey.
• Target2: Positive scores from 60%.
• Result FY06: Target not met.
– 48% gave positive scores.
ARL
www.arl.org
Metric L.3.A.: Expenditures for
digital materials
• Target1: Rank in the top 25% of ARL
libraries in percentage of collections dollars
spent on digital materials.
• Target2: Rank in top 33%.
• Result FY06: Target not met.
– Ranked 74 of 109.
ARL
www.arl.org
Metric L.2.C.: Compare staff
salaries to peer groups.
• Target1: Library faculty salaries should rank
in the top 40% of salaries at ARL libraries.
• Target2: Rank in top 50%.
• Result FY06: Target1.
– Ranked 33 of 113. (Top 28%)
ARL
www.arl.org
Reporting BSC Results
University of Virginia Library
Balanced Scorecard
FY06 Results
Target1
Target2
Not Met
ARL
www.arl.org
Descargar

Slide 1