Better than Google Scholar?

Usability Testing

University of Manitoba Libraries

• ARL member
• 25,000+ students
• Multi-campus, multi-library
• Acquisitions budget over $9 million

Background

2008/09 - Consultant's report on website usability

Recommendations:

• make it easier
• single search for books and articles

Sent out RFP for a "discovery layer"

Usability Research Goals

• Does Summon address consultant's findings
• How do novice students fare
• What do librarians need to know

Methodology

• December 2009, January 2010
• 9 students (Arts, Science, Architecture), recruited via website
• Wanted novices, but got more experienced
• Needed to have a research assignment
• SerialsSolutions paid $50 incentive
• UserVue and Morae - captured and analyzed video and audio of students working remotely

What did we learn about Summon and students?

• Took roughly a nanosecond for students to take it for granted that they could search books and articles simultaneously
• Nearly all students failed to notice Summon's limiting options until prompted
• Summon's premise validated: students start out with broad, general searches and then narrow down
• This sort of testing should be in toolkit of more librarians, especially in larger libraries
• Front-line librarian response: "I could have told you that." They know a lot about student behaviour; how to open up communication channels

Google Scholar vs. Summon

We compared ...

• 35 citations used in an earlier study
• Study compared Google Scholar, Windows Live Search and Scirus
• This time, Google Scholar, Scirus and Summon

What we found ...

• April 2006 - Google Scholar retrieved 25, Scirus found 20
• November 2006 - Google Scholar retrieved 28, Scirus found 19
• September 2010 - Google Scholar retrieved 31; Summon retrieved 20; Scirus retrieved 18

Conclusions

• Our citations are a mix of books, articles, etc. May be that GS is more inclusive of format.
• Clear difference between GS and other two.
• Need to repeat the search at intervals.
• Both GS and Scirus claim to be stronger in science. Summon designed to be broader.
• Our test belies all three claims.

Journal Usage

We compared ...

SUSHI-compliant platforms using journal successful requests (COUNTER Journal Report 1)

• Relied on USTAT and SUSHI-compliant platforms that had 2010 stats available
• Collected usage stats from May-Aug 2010 when available

9 platforms fit criteria - used top 5 journals in each

• Available comparable dates of pre- and post-Summon stats
• Checked that each title was available in Summon

Journal Usage

usage changes 2009 - 2010

Average Usage by Platform

Sorted by Platform

Some reflected the overall trend of decreased usage in 2010

• ACS (Chemistry)
• Chicago Journals (varied)
• Highwire Press (Medicine, including Science)

Increased usage in 2010

Platforms

Some showed increased usage in 2010, but accounted for a small number of successful requests

• Project Euclid (Mathematics)
• ACM Digital Library (Engineering, Computer Science)

Some showed no pattern

• Liebert (Medicine)
• CJO (Medicine)
• MIT Press (varied subjects)
• Highwire Press (Medicine, includes Science)

No consistent pattern

EBSCO Journals in Summon

Conclusion

• Definitely needs more study...
• Summon implementation doesn’t seem to have affected successful journal uses at all
• Interesting that EBSCO journal usage is increasing – again further investigation needed