Indexed-based discovery services (now referred to in this article simply as discovery services) rely on a just-in-case approach to indexing, whereby a large number of items are indexed in a central index and are then discoverable by end users via a single search. Indexed scholarly materials include content from journals, e-books and other electronic information of a scholarly nature as well as local collections such as library catalogues and digital repositories.
Discovery services first emerged in 2009 with the launch of Summon by Serials Solutions (now ProQuest). Summon was quickly followed by a number of other services, most notably Primo by Ex Libris, EBSCO’s Discovery Service (EDS) and OCLC’s WorldCat Local (to be renamed WorldCat Discovery). These services have been adopted at a rapid pace, and thousands of institutions globally are making their collections available to their users through such systems. As Marshall Breeding stated in his 2013 Library Journal Library Automation Marketplace 2013 report, ‘Discovery services continue to represent a major component of the industry. Web-scale, or index-based, discovery services now are must-have products for libraries with large collections of electronic resources’. According to Breeding’s Library Systems Report 2014 for American Libraries, 9,409 libraries globally had implemented one of the four major discovery services by the end of 2013.
Libraries expect their entire collection, including licensed and free electronic content, to be made available to their users within the discovery service of their choice. When acquiring licensed content, libraries expect a clear explanation of the degree of availability of that content in their discovery service. For example, libraries may wish to know which databases are indexed by the service, what is the breadth and quality of the metadata, whether the content includes the types of material needed by the library, e.g. full text, citations and journal back-files, and if the full text is searchable.
Furthermore, libraries would like to understand how the discovery service ranks the search results and whether the service enables libraries to modify the ranking algorithm as required locally. In addition, libraries want to make sure that the ranking of the search results is objective and that there is no bias towards or against specific information provided.
To date, proprietary agreements have been made between the discovery service providers and content providers for harvesting the providers’ collections and indexing them in the central indexes of the discovery systems. These agreements are largely opaque to the library customers in terms of the breadth and depth of data received from specific content providers and how this data should be indexed and made available to the end user.
The Open Discovery Initiative (ODI) was first launched at the end of 2011. It set out to improve the ecosystem for the main discovery service stakeholder groups, namely the content providers, discovery service providers and libraries, and it included a clear goal to better serve the end user.
The National Information Standards Organization (NISO) published the ODI Recommended Practice in June 2014 as NISO RP-19-2014. It is already having an impact on the industry and is referenced in a number of library discovery service requests for price quotation (RFPs) as well as in the Texas State Library and Archives Commission’s white paper on discovery services.
History of the ODI
The ODI was initiated in June 2011 by Ex Libris together with Marshall Breeding, a library industry consultant (previously at Vanderbilt University) who, at the 2011 ALA Annual Conference in New Orleans, invited senior industry managers to meet and explore areas of mutual interest related to library discovery services. Following this initial meeting, a proposal was submitted to NISO; later in the year, the NISO Discovery to Delivery Topic Committee accepted the proposal as a new NISO work item. The co-chairs of the ODI working group were Marshall Breeding and the author, initially representing Ex Libris and latterly working as an independent consultant. The working group members were selected to represent, in equal numbers, the various stakeholder groups of librarians, content providers and discovery service vendors, with representation from individual libraries and consortia as well as different types of information providers, including primary publishers, abstracting and indexing (A&I) publishers and aggregators.
Aiming to improve communication and clarity around the new expectations for industry practices related to discovery, the ODI working group set the following goals:
- create ways for libraries to assess the level of content providers’ participation in discovery services
- help streamline the process by which content providers work with discovery service vendors to enable their collections to be indexed and accessed
- define models for ‘fair’ linking from discovery services to publishers’ content
- determine what usage statistics should be collected.
Based on the input from a survey done early in the project – and after much discussion amongst the working group members – it was agreed to develop recommended practices in the following five areas:
- technical recommendations for data format and data transfer, including methods of delivery and ongoing updates
- recommendations for the communication (automated or through reporting) of libraries’ rights for their users to access specific content (e.g. restricted to users from subscribing libraries versus open to all users)
- clear descriptors regarding the level of indexing performed for each item or collection of content and the availability of the content
- determination of what usage statistics should be collected and for whom, and how these data should be disseminated
Further, the working group agreed to develop mechanisms to evaluate conformance with the recommended practice.
ODI published recommendations
The ODI recommendations were targeted only at content providers and discovery service vendors, but with a clear goal to better serve libraries and their users.
Recommendations for content providers
The ODI recommends that content providers should make their whole corpus of content available for indexing by the discovery service providers. The corpus of content includes the core metadata and, where applicable, additional descriptive metadata plus the underlying content item. Wherever possible, for each item submitted by the content provider, compliance should be made with the detailed ODI core metadata standard which stipulates the provision of comprehensive item citation metadata, item type and format, the item URL, an open access designation and an indicator of full text availability. Content providers should also comply, where relevant, with the recommended enhanced metadata standard which covers descriptive metadata such as an abstract or description as well as any keywords, plus the underlying content item such as the full text or transcript.
The ODI further recommends that content providers disclose to library subscribers their level of participation in discovery services. For each market product (journal collection, A&I database, etc.), content providers should disclose the content coverage and content depth provided, as described in the recommendations.
In addition, the ODI further recommends that any agreements between content providers and discovery service providers do not prevent disclosure to libraries of the key terms of the agreement. Transparency is strongly encouraged.
The ODI has considered the transmission of data from content providers to discovery service providers for the purpose of building a central index. Although the number of current providers of index-based discovery services remains fairly limited, and mechanisms are already in place for discovery services to deal with the data from these providers, it was felt that recommendations should be put in place sooner rather than later, as the number of potential content providers is immense. Given the vast amounts of data being exchanged, it is in the interest of all stakeholders to employ the most standardized and efficient transfer mechanisms available. The ODI, therefore, recommends that the transfer of data from content providers to discovery service providers should make use of existing standards where applicable. Some of the standards and protocols most directly applicable include the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and Knowledge Bases And Related Tools (KBART). ResourceSync, a new standard also published by NISO in 2014, has strong potential as a mechanism for data transfer for discovery services.
Recommendations for discovery service providers
Central to the recommendations for discovery service providers is the disclosure to libraries, on request, of a content listing detailing all the metadata elements that are indexed by the discovery service. The recommendations include the data elements and format of the content listing that would make it easy for libraries to compare and evaluate the content coverage of the different discovery services. Recommendations are also given for suitable distribution mechanisms for the content listings to ensure ease of access by the libraries.
A number of recommendations are designed to ensure that relevance ranking of the search results, the presentation of results and the linking to a full-text version do not introduce bias towards a particular content provider. Libraries should be able to customize links to the full text by setting their own preferences for the number and type of links presented, the order in which links are presented for a given item, and the way in which libraries label and brand the ‘get the full text’ link.
Discovery service providers should offer, on a regular basis, a set of simple usage metrics that are relevant to most content providers and to library customers.
The recommendations specify that the following usage reports should be made available by discovery systems to the contributing content providers:
- Number of searches
- Number of results clicked (selected)
- Number of clickthroughs (that is, the number of times a user requested the full text of an item).
The recommendations specify that the following usage reports should be made available by discovery systems to libraries:
- Number of searches per month
- Number of unique visitors per month
- Number of clickthroughs per month
- Top 500 search queries for the last period
- Top 100 referring URLs to the discovery service for the last period.
It is recommended that the metrics above be incorporated by COUNTER in a future version of the Code of Practice that specifies details of format and distribution of usage data. The definition of more complex metrics should also be considered for a future phase of the ODI and should be developed in conjunction with COUNTER.
The final recommendation for discovery service providers is that they support standard data formats and transfer methods for the flow of content from the content provider to the discovery service vendor.
Assessing ODI compliance
Conformance checklists were developed by the ODI working group as tools for communicating the high-level ODI recommendations. These checklists also provide a framework for the discovery service providers and content providers to engage with their library customers on ODI issues. See Appendices B and C of the ODI report, which should be completed by content providers and discovery service vendors respectively and made available to libraries on request.
A number of issues identified by the ODI working group could not be addressed given the proposed time constraints that were set for the completion of the ODI recommendations. These issues were listed in the ODI recommendations as possible next steps, once the baseline recommendations are in place. The issues include the establishment of a collaborative forum for discussion among all discovery stakeholders, closer collaboration with COUNTER to align usage reporting, the management of ‘restricted’ content, i.e. content that is stipulated by its owner to be accessible only to users from subscribing institutions, and how to manage usage rights when the user searches for content and accesses it through an application programming interface (API).
The work of the ODI is now being continued by an ODI standing committee, led by Laura Morse (Director, Library Systems, Harvard Library) and Lettie Conrad (Online Publishing/Product Manager, SAGE Publications). Both the new co-chairs were active participants in – and substantial contributors to – the ODI. The ODI standing committee is now focused on the promotion and implementation of the recommended practices as well as providing a forum for ongoing discussion on issues related to discovery services. This group will determine a suitable timetable to address next steps in the development of recommended practices related to discovery services, including a review of items that were declared out of scope in the recommended practice.
To ensure that their data and services meet the needs of customers and end users, content providers and discovery service vendors are strongly urged to implement the ODI recommendations.
Libraries are encouraged to engage in conversations with content providers and discovery service vendors on a regular basis to make sure that discovery services best serve their needs and those of their users, now and into the future. It is recommended that libraries use the conformance checklists from the content providers and discovery service providers to inform their decisions. It is still relatively early days since the release of the ODI recommendations; but with the promotional efforts of the ODI standing committee, it is hoped that there will be a wider awareness of these recommendations and a commitment to implement them. Further guidance for stakeholders on implementing the ODI recommendations is expected from the standing committee in 2015. The first content provider to announce full compliance with the ODI recommendations was Credo Reference in October 2014.
The ODI standing committee will be sending out regular updates about their activities on the ODI mailing list. Those interested in the ongoing work of the ODI should visit the NISO ODI web page for instructions on how to subscribe.
The author has declared no competing interests and has openly stated that she was co-chair of the ODI working group.