usability testing
Summon Discovery Layer
SerialsSolutions partnership
About Karen
About the University of Manitoba Libraries
advice about partnering
impact on teaching
impact on services
over 5 years as information literacy coordinator at University of Winnipeg
at the moment, Head of Libraries Electronic Technologies and Services at the University of Manitoba
moving to University of New Brunswick at Saint John in August as Director of Information Services and Systems
ARL size library
multi-campus / multi-library
Before and After
Partnering with the Vendor:
Results of Summon Usability Testing
Karen (Hunt) Keiller
University of Manitoba Libraries
Summon Discovery Layer Usability partnership
Lessons Learned
Developing a better feedback loop
June 2009 - picked Summon Discovery layer
Fall 2009 - implemented
December 2009 - started usability testing in partnership with SerialsSolutions
January 2010 - completed usability testing
February 2010 - soft launch of “One Stop Search”
April 2010 - hard launch
9 students, December 09 and January 10 before soft launch
humanities, sciences, architecture, recruited from notice on library web site and help from liaison librarians
wanted novice students, but got more experienced students
to be eligible they needed a research project they were working on
incentive of $50 paid by SerialsSolutions

How?
UserVue - subscription paid by SerialsSolutions
Morae - software used to view and analyze video

students working remotely, recorded, audio with phone
expensive and has a learning curve
http://bit.ly/cu256P
Goals of the Discovery Layer
Goals of Usability Research
address problems in consultants report (2008/09) on website usability
one search for books and articles
EASY
does the tool work for novice students
find out what can be improved
help liaison librarians

Role of Serials Solutions

scheduling of students
setup and subscription to UserVue
payment of incentive
research assistant (recent library school grad in Seattle) took notes
Role of Me
recruit students
facilitate the session
analyze results
disseminate
What’s in it for SS?

What's in it for the Library
access to students
exposure of Summon to librarians at conferences and in the literature
research assistance in handling details of recruitment
access to SS management and developers for input
What didn’t go so well?

I didn’t analyze results quickly enough 
I couldn’t seem to take off my librarian hat
Real life question meant that I wanted to provide real life solutions
Relevant topics meant that students would get sidetracked with problems that I didn’t care about (e.g. link resolver),  or they started to read interesting things they were finding
This kind of usability testing should be in the toolkit of more librarians, especially in larger libraries
“I could have told you that”
front line librarians know a lot about what students are doing, but how do we open up effective channels
Think about how usability testing and feedback could be integrated into teaching and services
What did we learn about Summon and student searching behaviour?

it took about a nanosecond for students to take it for granted that we were searching books and articles at the same time
this means that we can’t rest here, we’ve climbed up a hill, but we haven’t arrived at the summit
it seemed that science and humanities students approached using a new system differently, but that’s an observation based on one student
there is something in the design of Summon that caused most students to not notice the limiting options
the premise of the design of Summon (that students start out with broad general searches and then narrow down) was validated
In an agile development world . . .
How do get in the feedback loop?
usability testing in parterships with vendors is one way
what are some other ways?
What is agile development?
Customer satisfaction by rapid, continuous delivery of useful software
Working software is delivered frequently (weeks rather than months)
Working software is the principal measure of progress
Even late changes in requirements are welcomed
Close, daily cooperation between business people and developers
Face-to-face conversation is the best form of communication (co-location)
Projects are built around motivated individuals, who should be trusted
Continuous attention to technical excellence and good design
Simplicity
Self-organizing teams
Regular adaptation to changing circumstances
there are newer and better tools out there
http://remoteusability.com/avalanche/