Discovery systems have been a welcome innovation for library users and staff. The ability to draw information from multiple sources simultaneously represents a big step forward for the busy user. For the busy member of library staff, they also offer advantages (such as fewer interfaces to maintain and less training needed).

At the National University of Ireland, Galway (NUI Galway), we were excited by the promise of a discovery system, and were willing to work at making it perfect. In addition to financial investment, we invested in resources in the form of a lot of staff getting involved in every step, and staff themselves invested many working hours in the project. We had the opportunity to use the experience and perspectives of a large body of staff to weave one robust interface that would answer the needs of a large, varied and demanding user population.

The Library had purchased Primo from Ex Libris, though most of the industry-standard discovery systems on the market offer the same set of core functionality and operational possibilities. We had a number of Ex Libris products, and there was a belief that Primo would work well with other services from the same stable. Primo gathers information from Aleph (catalogue), MetaLib (cross-searching interface), ARAN (our institutional repository), and later, Primo Central (centralized scholarly index).

In early 2009, the Primo Implementation Group was established, a large group intended to reflect all relevant areas of the Library, allowing those areas the chance to contribute at a granular level to the project. A number of implementation sub-groups were formed; for the purposes of this article, we will be looking at the Interface Group. The Group had a clear remit to develop the interface for our discovery layer. It had a broad membership to allow for different perspectives to be brought to bear on the development. The Group's members had a responsibility to liaise with colleagues on progress, and to garner and summarize opinions from those colleagues for the Group. Again, the idea was to get as many perspectives as possible to develop a balanced view.

“we were excited by the promise of a discovery system, and were willing to work at making it perfect.”

To begin, we gave demos of the out-of-the-box version to staff to show the functionality and explain the philosophy of discovery systems. We showed examples of live sites to give staff a feel for what we hoped to achieve. Group members also began to liaise with colleagues to go into more depth regarding what we were hoping the system would accomplish. As communication filtered back, we could immediately identify common concerns.

A significant worry from staff was the idea that the discovery layer would be a radical departure from what went before it. This is understandable – any new way of accessing data would need consideration from all concerned to ensure all needs were being accounted for. Primo would be vastly different from Aleph (our catalogue) in terms of cosmetic appearance as well as function, which introduces another step in the user's learning curve. A period of reintegration to understand a new system can be frustrating for users when they are in a hurry.

During implementation, we tried hard to incorporate these issues. In retrospect though, these concerns heavily shaped our vision of Primo. For example, our old catalogue offered a ‘browse’ function that staff felt was very important to bring to the discovery layer, and this we duly did.

Another concern that raised a lot of debate was the idea of ‘dumbing down’ what we offered users. Some staff felt the idea of a single search interface powered by strong search algorithms was bad for information literacy. One staff member said, “Primo should be a flagship model of good pedagogical practice” – a view shared by quite a few. The debates on this issue were long and tedious, but staff felt strongly about either side of the debate. As a compromise, we incorporated three tabs on the Primo homescreen to allow users to be more specific about what they were searching (see Figure 1).

Our design and implementation was based heavily on tenets of librarianship and information literacy. While we adopted these strategies robustly, in retrospect, they were essentially the wrong ones. We were well-intentioned but misguided. We tried hard to integrate the past into our new system for legitimate reasons. However, the innovative possibilities offered by discovery systems started to become compromised, and we turned a revolutionary idea into an evolutionary one.

“We were well-intentioned but misguided. … we turned a revolutionary idea into an evolutionary one.”

The Interface Group worked on assembling the complex jigsaw pieces, and in 2009 came up with a version that managed to achieve a balanced version of what staff felt it should do. Happily, this model also followed closely with other Primo sites globally, which was encouraging.

The next step was to thoroughly check all aspects of Primo's operation to make it as polished as possible. It may be a stereotype, but librarians are detail-oriented, and a lot of librarians testing our implementation meant a lot of errors were found and fixed. In retrospect, our pursuit of perfection also meant we got bogged down in detail. Did this form negative opinions sub-consciously? Perhaps.

Figure 1 

The adapted Primo homescreen incorporating three search tabs

For example, we were attempting to identify every possible sub-collection to apply to our holdings (Special Collections, Special Collections Reference, Strong Room, etc) instead of merely indicating whether a title was freely available or available on request.

An important lesson we learnt is that librarians want perfection, but users just want good enough. The level of perfection we were aiming for was not necessary in the real world. We are all of us tasked to do the best job we can on a daily basis, but sometimes the ‘best job’ means doing less and spending the saved time doing other things.

Pursuit of perfection meant we spent a lot of time customising Primo's interface. The cosmetic and functional options of discovery systems are very open. It can be tempting to create a completely local feel to an interface. This can also be time-consuming, and runs the risk of being over-written during a subsequent upgrade. We learnt that just because you can push a button, it doesn't mean you have to.

“We learnt that just because you can push a button, it doesn't mean you have to.”

We launched Primo in September 2009; usage was predictably high and it followed the Pareto Principle1 quite closely. In the first couple of years, users did 1.5 million basic searches and 50,000 advanced searches. Some of Primo's rich offerings such as reviews and tagging were not really used at all – possibly due to lack of awareness.

In November 2010, the Library implemented its first LibQual survey. We were quietly confident of good results in relation to our website and discovery system. We had invested a lot of money, time and experience into getting our vision right, and we felt we had a discovery system that was both intuitive and robust.

To our surprise and disappointment, the survey showed that for question IC-2, ‘A Library website enabling me to find information on my own’, we didn't even meet our users' minimum expectations. For IC-1, ‘Making electronic resources accessible from my home or office’, we barely met their minimum expectations. For both questions, we were way off what we were hoping to score (see Figure 2).

A Task & Finish Group was quickly formed to get to the bottom of the bad feedback and identify means to fix the problem. There was clearly a problem, but we didn't know what it was.

Survey comments were analysed in depth; these were revealing but at times provided conflicting feedback. Some recurring themes were dissatisfaction with website usability, confusing navigation and a desire for more user-friendly access to e-resources. However, contradicting these were comments that the website was easy to use and that our online services were great. Similarly, we had many positive comments on the ease of using e-resources off-campus, which contradicted the poor scores given in the survey for off-campus access.

In order to better understand our users' needs and in what way we were not meeting their expectations, we carried out a user observation study during which we observed them undertaking typical tasks. The aim was to ascertain how our users go about information seeking and to identify features of our discovery system that were causing them to be dissatisfied. We hoped to find out when users were becoming confused and whether there were interface features that they didn't notice that might have helped. We wanted to establish whether and when users logged in, and whether their experience would have been more seamless if login had happened sooner.

Figure 2 

Disappointing initial LibQual survey results

Eight users took part in the study from a variety of subject areas. We interviewed each user to establish whether they had received library training, how they would normally search for information and their overall perceptions of the Library's discovery system. They were then asked to carry out some typical tasks and to think aloud while they were doing so, explaining why they chose particular options and any points of frustration or pleasure experienced.

This study provided a really deep insight into the usability of our discovery system. Having been immersed in its design and implementation for more than a year, we realized that our in-depth knowledge had blinkered us to the experience of a normal user. Observing users in action was like those blinkers being removed. We learnt that confusion was arising in understanding and navigating to our discovery tools. We had not succeeded in presenting a simple representation of the complexity and power of the software behind our discovery system.

“… our in-depth knowledge had blinkered us to the experience of a normal user.”

By observing users we realized that our efforts to integrate the past by providing multiple routes to information had only served to confuse users. Our knowledge of the fantastic range of choices we could offer had stopped us from selecting wisely what we should offer. Our expectation that users would simply search for the resource they needed was incorrect, as many users tried to find a way to browse to those resources. Users therefore navigated to web pages about particular resource types (e.g. journals) rather than searching for what they needed. Many users didn't understand the terminology we used. Links like ‘Get It’ and ‘Online Access’ tended not to be noticed by the users. When they did notice them they were confused about what the wording ‘Get It’ or ‘Currently available’ meant.

The design premise of Primo is that the online version of a resource is preferred by users if both print and online are available. This meant that print holdings were somewhat hidden, if a resource had both print and online holdings. Users clicked on ‘Get it’ or ‘Online Access’ and were brought straight into the online holdings of a journal, for example. None of the users we observed found the long back-runs of print journal holdings that predated our e-access.

Users were unsure how to search for journal articles. Some navigated to our page about journals as a resource and didn't know how to go further. Others accessed our list of e-journals, derived from SFX, and then searched by article name with no success as this is searchable only by journal title. Some users typed the article name into our main search box, again with no success, as this only allowed searching at the level of book or journal.

Even when users navigated to our journal articles search, none managed to find the article they were seeking. The journal articles search used MetaLib functionality to cross-search databases. Because cross-searching can be very slow, we had grouped our databases by subject so as to limit cross-searching to eight databases. This meant the user needed to select a subject area from a drop-down list before searching. None of our users noticed the drop-down box, and none selected a subject area. Unless the user is logged in, only free databases are included in their search. Despite two large login buttons within the search tile, none of our participants signed in! This meant that they were all searching on databases that were multi-disciplinary and free. The search results were very limited and none of the users managed to find the article they were seeking.

“Users were unsure how to search for journal articles.”

We found a strong reliance on Google by all participants. When given a general topic to research, they were unclear how to start with library resources, and reported that they would normally use Google to establish context. Prior attendance at library training hugely improved the users' search success, indicating that despite the promise discovery systems offer of Google-style searching, our design clearly hadn't achieved the intuitiveness of Google.

To get a wider perspective, focus groups were held with students and academic staff. These confirmed the findings of the user observation. It was clear that to meet our users' expectations we needed to simplify the search experience. We planned to upgrade to v.3 of Primo and saw this upgrade as an opportunity to redesign based on what we had learnt.

One of the main tasks was to declutter and simplify the search tile on the website and the Primo homepage. We replaced the three search tabs with a single Primo search box, which searches across Aleph, MetaLib and our institutional repository. We removed four links from the search tile and hid that functionality behind a ‘more search options’ button. This brings the user to the Primo homepage where there are links to ‘Find Databases’ and our institutional repository. We have improved the visibility of the login button on Primo and have enabled login from the Library homepage. We believe that dissatisfaction with off-campus access may have been due to users not logging in. We believe we have not only reduced confusion but now draw the user's eye to the things we need them to notice, e.g. login and search box (see Figure 3).

“We believe we've not only reduced confusion but now draw the user's eye to the things we need them to notice …”

The difficulties with journal article searching required a completely new approach. We have therefore implemented Primo Central, a massive index of metadata harvested from primary and secondary publishers, aggregators and open access repositories. Adding an index of this magnitude leads to huge increases in results and can lead to difficulty finding specific known items. Our default search therefore excludes articles (i.e. Primo Central), but the user can add articles into their results by choosing ‘Including Articles’ from a drop-down box.

One function we have removed entirely is the ability to browse. Browsing is not available in Primo v.3 and to reintroduce browsing would mean linking the user to our underlying catalogue, Aleph. Primo v.3 has brought OPAC functionality such as real time availability, renewals, etc. into the discovery layer. We have therefore been able to reduce user confusion by avoiding sending them to the completely different search experience of Aleph. Removing browsing remains controversial, however, and we have had requests for its reintroduction. Primo v.4 includes a browse function and we look forward to reintroducing it when we upgrade.

Primo v.3 brought other enhancements that addressed some of our findings. It allows clearer links to both the online and print holdings of a resource and displays available items more clearly. ‘Find Database’ functionality has brought MetaLib into the discovery system, reducing the need for users to be sent to a separate system.

To establish whether we had succeeded in simplifying the search experience and to spot anything further we needed to do, we repeated the user observation study post-implementation. We also elicited feedback from academic staff; we wanted to ensure that we were meeting their needs as well. We were delighted to find that users had significantly less difficulty completing the set tasks and were a lot less confused. Some areas were identified for further work, for example easier access to ‘My Account’. However, overall feedback from students, academic staff and library staff has been very positive.

This will be an ongoing process and we intend to make user observation a regular practice to ensure we keep up with evolving user needs.

It is not enough to design according to your belief of what is right for your users; it is absolutely essential to check in with those users to make sure what you are doing meets their needs.

Figure 3 

Making the login and search box much clearer to improve search results