Background and context
Academic libraries are spending a growing proportion of their increasingly stretched budgets on e-books each year. Within this context, demonstrating a return on investment is imperative, but gathering data about e-resource usage, particularly the use of e-books, is not always easy. Libraries require access to consistent and reliable data and effective tools to help analyse the value and impact of e-resources.
Jisc,1 the UK higher, further education and skills sectors’ not-for-profit organization for digital services and solutions, provides access to library analytics tools to support communities in accessing, analysing, evaluating and reporting on e-resource data. Services include IRUS-UK2 that improves reporting and supports benchmarking through access to COUNTER-compliant usage statistics for content downloaded from participating institutional repositories, and JUSP, a portal that aims to save libraries time and duplicated effort by offering a single point of access to COUNTER-compliant usage statistics from participating publishers. The Jisc Library Analytics team conduct research, offer advice and support individuals and organizations involved in creating and managing usage data throughout the world.
Initially focused on journal usage data, JUSP started to include e-book usage data in 2016. However, this development highlighted challenges around the development, delivery and management of consistent e-book usage data that are of relevance to the global community. Implementation highlighted a general lack of clarity and consistency around treatment of usage data for e-books. At the same time, it was abundantly clear that there were significant opportunities for greater standardization, communication and collaboration in this area.
In July 2016 Jisc co-ordinated an e-book discussion forum event in London, inviting representatives from publishers, aggregators, libraries and standards bodies to attend. This group met to discuss current challenges involved in development, delivery and utilisation of consistent, reliable usage statistics for e-books. The meeting was intended as an initial step towards:
- informing the future release of a COUNTER standard
- providing greater clarity around use and analysis of e-book reports
- exploring opportunities for greater optimization of services and support mechanisms
- developing a shared understanding of challenges from all perspectives.
A report describing the outcome of the forum is available.3 During the event, the group supported the idea of further research around challenges, considering such work to be of value and interest to the community. The subsequent research, which ultimately aims to translate challenges into practical, actionable recommendations, is described below.
A literature review focused on e-book acquisition and usage was the starting point for this research. This review augmented some of the themes that emerged during the JUSP e-book discussion forum and prompted specific lines of enquiry. Themes were explored in more detail through a series of case study interviews with a small cross-section of library consortia, academic libraries, publishers and aggregators in the UK, US and Australia.
The role of e-books
A review of the literature in the context of our research clearly indicated the growing importance and use of e-books in academia, with a higher proportion of the monograph budget now being spent on e-books. Many case-study libraries had adopted policies of prioritizing e-book purchases over print if possible. Where universities are serving geographically dispersed communities and there is pressure on resources, this approach offers clear advantages. However, the area of e-books is one filled with complexity from the perspective of acquisition, management and evaluation.
In terms of acquisition, libraries in this study were using a mixture of purchasing models from both publishers and aggregators, and cited advantages and disadvantages of different approaches. Amongst libraries, collections or titles acquired directly from publishers were favoured in some cases as there were no usage caps on licences, and their platforms were generally free from digital rights management (DRM). There was also a preference for definitively owning items. Moreover, aggregators can offer books from a range of publishers on a single platform, and in some cases they are the only available source for an online version. Libraries also benefit from automated purchasing workflows and discount pricing when part of a consortium. Agreements negotiated with purchasing consortia would also determine which approach offered the best value for money.
The literature review for this study found a number of published articles from mainly US academic libraries showing a preference for the demand-driven acquisition (DDA) model, also referred to as patron-driven acquisition (PDA). The libraries and consortia considered in this study were using a variety of models, from individual title purchase, subject collections, subscriptions, to PDA and evidence-based acquisition (EBA), though a preference for PDA or EBA seemed to be emerging.
The role of usage data
Clearly, the type of purchase model influences the approach used to analyse and evaluate content. In this context, acquiring robust data is critical to support decision-making processes.
Recognizing the value and importance that libraries place on consistent and trustworthy usage statistics, publishers and aggregators participating in this study were keen to provide access to accurate and reliable data. They highlighted supporting library customers in a variety of ways, such as calculating cost per download to indicate value for money, investigating usage patterns to identify system issues or changing user behaviour, and reporting turnaways to identify titles in demand and make subscription suggestions.
Case-study libraries were seen to be collecting a range of usage statistics to support various tasks and activities. Uses included assessing titles for purchase or promotion and institutional reporting on usage of resources to demonstrate value. In the UK, submitting data for the annual SCONUL return was also important. Usage statistics were collected to evaluate the different purchase models used. This was seen by some libraries as especially important for PDA as money taken from the budget required regular reporting.
A requirement for comparable usage data indicates a preference for COUNTER-compliant reports, and the BR1 (number of successful title requests), BR2 (number of successful section requests) and BR3 (access denied to content items) reports are used where available. COUNTER reports were used to show patterns of use, either snapshots using BR2 reports or longer-term trends. The increasing use of turnaways to support reporting, new purchase decisions or upgrades was apparent from many of the library case-study interviews and from suppliers.
In consortia deals where usage may be one of the factors affecting how much members pay, data are an important element and reviewed regularly.
The role of supporting data
Usage data are being used in conjunction with other contextual information to evaluate collections. This may be data on subject classification to identify gaps in provision or relevant subject needs, or information about research requirements and reading lists used to support collection management and development. Additional requirements include information to support assessment of cost per use and cost per title. For consortia, there was interest in the number of institutions using the title and cost per purchase.
Libraries were also interested in their users and had set up ways to determine who was using a particular resource. This could be done using EZProxy or by suppliers who provided survey forms when users clicked on an item.
It was clear from the research that libraries are requesting a range of contextual information from publishers in addition to the COUNTER reports. Examples include number of titles used, usage by collection, usage by format, usage by subject, percentage of the collection used, data for use/non-use and cost per use. Clearly this work involves considerable time and effort for the publisher to collate. Some suppliers provide additional information to their customers on a regular basis, others on request. Case-study libraries found that publishers and aggregators are generally responsive and helpful in providing this additional information.
The scope of work and range of activities highlighted in the study suggests a great deal of manual and duplicated effort amongst both libraries and publishers. In addition, manually collating data from disparate sources needs care to ensure adequate interpretation and comparison.
There was broad agreement among those interviewed for the case studies on the challenges faced by those dealing with e-book usage statistics. These followed the pattern identified in the literature review and in the JUSP e-book discussion forum. The following major challenges were identified.
When COUNTER data are inconsistent
The main issues around COUNTER data were:
Definition of ‘section’
As several case-study participants pointed out, the definition of a ‘section’ for a dictionary or encyclopaedia will refer to an individual entry, which is very different from a section of a textbook. This means that the BR2 report may produce misleading overall results, which will also impact on attempts to match with other data, for example to determine cost per use. This difficulty was recognized by both libraries and suppliers interviewed for this study.
Comparing results from BR1 and BR2
The lack of consistency among suppliers, with some providing the BR1 report and some the BR2, meant that case-study libraries could not always make direct comparisons between different suppliers. While the choice of report depended on the platform or the way the content was provided, this was particularly a problem when making decisions about changing platforms or trying to compare one supplier with another in usage reports.
The optional COUNTER BR7 report (number of successful unique title requests by month and title in a session) has been introduced to allow comparable usage of e-books, but it is rarely applied.
Not enough suppliers are COUNTER compliant
Despite the challenges noted above, access to comparable COUNTER-compliant data is a priority for all. However, the number of suppliers who are not providing COUNTER e-book reports continues to be a problem and leads libraries to search for any statistics they can find that are roughly comparable. The lower number of e-book suppliers providing COUNTER reports via SUSHI (compared to e-journal suppliers) makes the task of collecting usage statistics much more difficult and time-consuming than for e-journals. This was seen by one case-study interviewee as a major reason that the e-book market presented far more challenges than the e-journal market in terms of collecting usage statistics.
Difficulty of dealing with non-COUNTER data
Case-study libraries pointed to the difficulties caused when vendors were supplying non-COUNTER data or no data at all. This meant a large amount of effort in getting usage statistics from multiple sources and the need to justify the effort involved in terms of the results gained. Libraries had to decide what data they could use when no COUNTER statistics were available. One case-study library noted they tried to work with the most comparable data supplied by the publisher. Another took whatever usage data they could get, for example, click-throughs from the link resolver. Libraries found it difficult and laborious to compare e-books from more than one source, particularly because of the lack of consistency in non-COUNTER reports from different suppliers. When reporting on e-book usage, it was sometimes necessary to give a ‘health warning’ to show that results were not strictly comparable.
Where libraries were looking for evidence from turnaways to inform future purchases, the varying vendor interpretation of turnaways meant that these often gave inadequate data for decision making.
Lack of common identifiers
The lack of a common identifier for an e-book title is one of the major challenges. The same ISBN or e-ISBN is not used consistently, with e-books available through aggregators, publishers or as part of a package often having slightly different titles and different ISBNs or e-ISBNs. When trying to track usage of titles according to whether they are outright purchases, subscriptions or unowned content, matching up the title lists with usage reports is particularly difficult when the data are not presented in a consistent fashion. One case-study library explained how the already laborious process of matching titles from different sources was more problematic due to the inconsistent use of identifiers.
This is a particular problem with aggregators, who are harvesting and presenting data from multiple sources. Tracking titles available across different platforms is challenging. Library interviewees also gave examples of how inconsistent identifiers caused problems when loading records for titles on the EBA model.
It is evident therefore that problems of metadata quality not only apply to usage data but also affect the discoverability of content through the library’s discovery system. The large amount of metadata inevitably leads to inaccuracies, and there are no industry systems in place, although this is something KBART Phase II4 is looking to address.
Publishers and aggregators in the study recognized the challenge of inconsistent ISBNs between publisher and aggregator. This created problems when a customer wanted to avoid duplicate purchase of a title on an aggregator and publisher platform.
Even where common identifiers were used, the variety of formats in which contextual data were presented to the libraries by different suppliers caused libraries a good deal of work matching up data on subject classification, costs or entitlements with the usage reports. Comparison between suppliers was difficult when these contextual data were presented in different ways.
Exclusion of nil use titles
The COUNTER BR reports do not require the inclusion of nil use titles so libraries need to compare titles in usage reports with publisher title lists and, for the reasons described above, this is not a simple task. Nil use information is important for libraries not only for assessing the value of a collection, but because it may sometimes reveal titles that have been purchased but have not had access switched on.
E-book reports do not distinguish between purchases, subscription or unowned content
Another difficulty is that the usage reports do not generally identify which titles have been purchased, which are on subscription and which are unowned. This means going through title reports line by line and relying on individual and team knowledge. Such work is time-consuming and labour-intensive, particularly given the problems of lack of common identifiers explained above.
Tracking titles available in different models, with content moving in and out of the library catalogue throughout the year was also arduous. Once the subscription had been agreed, titles in the deal might be subject to change, making it difficult to see what titles were available, and difficult to interpret annual usage figures when titles had changed. Faculty members need to know that content will continually be available, and for the EBA model it is important to know whether the library already has the title, to avoid duplicate purchase.
While these problems of matching up lists may make it difficult and challenging to avoid duplicate purchases, there was also a concern that e-books that had been acquired through a DDA or PDA model might not always be available. In one case, a bad experience where an e-book was removed and could not be replaced on a particular model had made some academics cautious about using e-books rather than print.
For one publisher, enhancing the metadata by providing package information was considered an important part of the business model. This publisher would like to see a way of combining information about the business model with the usage data so that the two reports did not have to be viewed separately. While all publishers and aggregators aimed to provide this information as part of their service to customers, providing reports to individual libraries showing usage by collection may take considerable effort.
It was obvious from all the case-study libraries and suppliers that dealing with the challenges listed above was both time-consuming and labour-intensive, involving a lot of manual work to provide context for usage statistics and to make comparisons between different suppliers, models and subscriptions.
Some libraries gave examples of the workarounds they used to help overcome some of these challenges. One library was using a standard calculation for comparing BR1 and BR2 statistics, as is also a requirement in the UK when completing the SCONUL return. They also use Excel VLOOKUP to match titles or ISBNs to combine usage reports with other contextual information. However, the publisher may send several different spreadsheets to be matched up, and this is still a very time-consuming process. Some libraries have also experimented with using only download figures to compare usage from different aggregators, but as aggregators count items in different ways and not many users actually download, this was not felt to be a very reliable method. One library interviewee provided a document illustrating a large piece of work done for a report to evaluate different purchasing models, looking at a range of key performance indicators based on the BR1 or the BR2. This type of work takes a long time and could not be done on a regular basis. Representatives from a library consortium provided a spreadsheet showing the usage statistics that they hoped to get from suppliers with whom they had negotiated deals.
Requirements for change
For both libraries and suppliers, the sheer number of e-book titles and the variety of usage models makes finding solutions to the challenges outlined above a daunting, but necessary, task. This research aimed to uncover challenges while identifying solutions through a set of recommendations and an action plan included in the final report. Several initiatives with the potential to support change are outlined briefly below.
Proposed changes to COUNTER, as part of the move to Release 5 of the Code of Practice, will address some of the incompatibilities noted in BR1 and BR2.
There is a clear requirement for more suppliers to become fully COUNTER compliant. COUNTER itself has the leading role here in encouraging more suppliers to join and help shape the standard, and it will continue to actively pursue this aim. Other international organizations such as the National Information Standards Organisation (NISO)5 and Usus,6 the community forum for discussion of issues relating to usage data, provide opportunities for discussion and understanding of the issues involved.
In the UK, JUSP, in working with libraries and suppliers and offering COUNTER and SUSHI expertise, has a role to play in supporting development and adherence to the standard. Librarians themselves can use their influence to question suppliers that do not yet provide COUNTER usage data. Purchasing consortia and library consortia that provide deals to their members can help in defining expectations for COUNTER compliance during negotiations.
Need for consistent use of common identifiers
The issue of titles not matching up, either because of different ISBNs or e-ISBNs, or variations in the form of the title itself, is problematic, and the need to combine usage data with contextual information is rather arduous. Though much of this information is library-specific and can only be provided by the library itself or through an individual supplier, the lack of consistent metadata makes joining together information from different sources particularly difficult, and comparisons between data from different suppliers can be especially challenging.
However, KBART Phase II has recognized the importance of accurate metadata for e-books and is working on the basis of this as a priority for future development. Initiatives such as Knowledge Base+ (KB+)7 and the National Bibliographic Knowledgebase (NBK)8 run by Jisc aim to tackle these challenges on a national scale by providing shared services for UK academic and research libraries. KB+ is an established service for managing journal collections, and from 2017 it has started to provide additional standardized, verified and accurate titles lists for e-book packages. NBK will include catalogue data from 200+ libraries, aiming to improve overall quality of bibliographic data through a co-ordinated approach to authority controls and identifier frameworks, and it hopes to be able to aggregate bibliographic data with availability and usage data. NBK will also explore provision of master identifiers to address the types of challenges indicated above.
Initiatives such as the KB+/Global Open Knowledgebase (GOKb) e-books decision support and availability tracking tools, following a pilot with libraries in the UK, are key. These tools support pooling of collective knowledge about e-book platforms and enable tracking of e-books as they enter or leave packages.
Ongoing dialogue between librarians and suppliers
The research highlighted the value of surfacing issues and challenges through regular and ongoing communications. Events such as the Jisc e-book discussion forum were thought to provide a good model. Proposals made at the forum included the sharing of templates and subject classifications used by suppliers to explore areas for greater co-operation.
Suppliers often receive requests for information and data without a clear understanding of background and context. Work to clarify and communicate requirements could be particularly beneficial.
Clearly, opportunities for e-resource librarians and supplier representatives to convene and share ideas and approaches are key. This will help to support understanding from various perspectives and to develop a consensus on approaches towards solutions.
Solutions and next steps
This article summarizes the role of e-books and usage data in the context of library and publisher workflows as well as the challenges and their impact. A report describing the outcome of the research, together with recommendations and an action plan to address the challenges, is available.9