The research that underpins this paper was inspired by concern that the purchase of resources to support arts and humanities and some social science (AHSS) subjects is more threatened by the worldwide squeeze on library budgets than the purchase of resources for scientific, technical and medical (STM) subjects. In the UK, especially, the threat is increased by the withdrawal of government funding for arts and humanities subjects. More universally, librarians consulted in several of the Anglophone countries suggested that the problem may be exacerbated by the type of metrical analysis conventionally employed to justify financial outlay. Librarians are now required to prove how much value is contributed by expenditure on each paid-for resource. However, using traditional quantitative metrics to assess the worth to the institution of outlay on journals and specialized research databases for AHSS subjects can be less conclusive than for STM subjects, because often the number of recorded transactions is lower. Each resource may be used only by a small number of students and faculty, and their departments may attract fewer major research awards.
In August 2012, ProQuest commissioned an extensive piece of research to investigate how resources for AHSS subjects are discovered, used, promoted and valued by academics and librarians across a wide number of institutions in Anglophone countries. The University of Huddersfield agreed to act as the lead supporting institution by providing in-depth information via focus groups.
The ultimate aim of this project was to analyse and disseminate alternative and more appropriate ways of measuring the value of online resources for the AHSS subjects. It presents effective new methods of evaluating AHSS databases, provides ideas for increasing their visibility in the library and suggests how they can be promoted to faculty and students. It sets out to provide a cogent argument in support of continuing investment in AHSS resources, accompanied by a practical toolkit to aid librarians when presenting their case.
An extensive amount of primary and secondary analysis has been undertaken. It may be summarized as follows:
- a literature search and summary of relevant published articles
- an extensive questionnaire-based survey of the views of librarians and academics in the following Anglophone countries: Australia, Canada, Eire, New Zealand, UK and USA. (This work was carried out on behalf of ProQuest by Hanover, an independent research company)
- several ‘short’ focus group meetings conducted at academic conferences
- two ‘long’ focus group meetings conducted at the University of Huddersfield, with a group of AHSS librarians and a group of AHSS academics respectively
- a ‘control group’ (to complement the Huddersfield results) of librarians and academics from at least one university in each of the countries covered (except Canada); this group responded by semi-structured questionnaire
- analysis of usage results at a middle-sized UK university before and after the installation of the Summon discovery system.
Note: as far as possible, all the surveys and focus groups were asked to respond to the same set of questions.
The literature survey revealed that the justification of resource expenditure forms part of a wider series of challenges currently being experienced by librarians.
Creaser and Spezi, in Working Together: evolving value for academic libraries1, observe that although librarians are aware of the need to demonstrate the value of the library to teaching and research staff, there is no consensus on how this should be done. Auckland points out, in Evolving Needs of Researchers2, that ‘there is a need to translate the outcome of library interventions into terms which relate to academic outcomes, such as better grades or student retention’. Schonfeld and Housewright, in US Faculty Survey 2009: key insights for librarians, publishers and societies3, note that they found only 35% of library directors/associate directors agreed with the statement: ‘My library has a well-developed strategy to meet changing user needs and research habits’, yet 84% agreed with the statement: ‘It is strategically important that my library be seen by its users as the first place they go to discover content’.
Tenopir et al, in ‘Perceived Value of Scholarly Articles’4, were able to demonstrate that ‘… the most valued … characteristics … for [researchers] choosing articles to read are, in order of importance, 1) online accessibility, 2) author, and 3) type of journal. When the reader is able to access the article online without a cost to them personally (even though their institution may bear the cost), readers … are more likely to read and cite the article than if the article appeared in print only or if there is a direct personal cost to the reader associated with online access’. This survey also found that users are less likely to source an article from a non-peer-reviewed article than from a non-journal source; that younger faculty members are more likely to read on-screen [a significant pointer to publishers for planning future strategy]; and that source of article is more important to North Americans, while online accessibility is more important to researchers in other parts of the world.
“… paid-for resources to support AHSS subjects are particularly vulnerable to being cut.”
The observation about availability of alternatives was corroborated by the current co-author, Bennett, in an earlier study: The Potential Effect of Making Journals Free After a Six Month Embargo5. This survey of 210 libraries worldwide found that if all journals articles were to be made available via open access after a six-month embargo period, 56% of libraries would continue to subscribe to STM journals, 10% would cancel all STM subscriptions and 34% would cancel some STM subscriptions; while 35% of libraries would continue to subscribe to AHSS journals, 23% would cancel all AHSS subscriptions and 42% would cancel some AHSS ones. This suggests that paid-for resources to support AHSS subjects are particularly vulnerable to being cut.
In How Readers Discover Content in Scholarly Journals6, Inger and Gardner focus on three main forms of behaviour exhibited by readers of journals: citation searching, core journal browsing and subject searching. They show that readers given a citation seem to know their subject areas well enough to go directly to the website of the relevant journal to follow up on it. Consequently, the use of library web pages for discovery is in steady decline. Both they and Tenopir suggest that users are adopting a ‘quicker, smarter’ approach towards accessing resources; the amount of time spent on locating and reading a resource may have declined in some cases, but the benefit accruing to the user may have increased.
Stone and Ramsden's research in Library Impact Data Project: looking for the link between library usage and student attainment7 focused particularly on the importance of library resource provision to students. It aimed to support the hypothesis that ‘there is a statistically significant correlation across a number of universities between library activity data and student attainment’, and concluded that there was strong evidence to support it.
The Hanover survey
The Hanover survey obtained 570 complete responses. Respondents were segmented by country and by occupation (academic engaged in teaching and research, academic mainly engaged in research, academic mainly engaged in teaching, librarian/other). Key findings included:
- of respondents, 72.8% indicated that they used/referred to academic journals on a weekly basis at least. 63.6% agreed with the statement that ‘online journals are essential for my work/for helping others to carry out their work’. The great majority (89%) said that they used online journals for their own research projects and/or for citation in their own published articles (see Figure 1)
- asked how much time they spent searching for and reading online journal articles in order to write a research paper, 84.2% of respondents said that they spent at least five hours on this, 35.7% said 5–10 hours, 18.3% said 11–20 hours, 17.4% said 20–40 hours and 12.8% said more than 40 hours. Of the librarians in the sample, 80% said that they used/referred to online journals on a daily basis. Of academics, 39.5% mainly engaged in research, 26.2% engaged in teaching and research and 16.3% mainly engaged in teaching also said this.
- of academics, 19.4% mainly engaged in teaching said that they do not regularly use/refer others to online journals but still consider them important. Of the librarians in the sample, 46% said that they spent more than six hours each week searching for and reading online journals. Of the academics, 18.2% engaged mainly in research, 17% engaged in teaching and research and 6.1% mainly engaged in teaching also said this. 82.4% said that they believed that the ability to access journals improved their chances of obtaining funds for research and/or getting their own articles published.
- during vacations, 17% of academics mainly engaged in teaching and research said that they spent more than six hours each week using online journals. Of academics, 12.5% engaged mainly in research and 6.1% engaged mainly in teaching, along with 3.3% of librarians said this. 66% of academics engaged in teaching and research said that online journals were essential to their work, 57.6% of academics engaged mainly in research and 42.9% of academics engaged mainly in teaching agreed. Of academics, 93.3% engaged in teaching and research said that they were likely to use online journals for their own research projects and 90.9% of academics mainly engaged in research, 74.5% of academics engaged in teaching and research and 87.5% of librarians agreed.
It is not possible to list here all the country-by-country findings (though they will be supplied with the other documentation contained within the toolkit) with one exception, because it indicates how much paid-for AHSS resources ‘pay for themselves’ (and supports the conclusions of Carol Tenopir on this topic):
- of academic respondents from New Zealand, 94.1% said that the ability to access journals improved their chances of obtaining funds for research or getting their own articles published. Percentages were also high elsewhere: 86.7% of Australian respondents, 84.9% of Canadian respondents, 79.5% of respondents from the US, 79.3% of respondents from the UK and 76.5% of respondents from the Irish Republic agreed.
The ‘short’, Huddersfield and ‘control’ focus groups
Space restrictions make it impossible to include more than a few of the most important findings from the focus group work.
Librarians were unanimous in agreeing that most library resources are chosen by reaching consensus among all interested parties. Value for money is assessed by examining a range of variables, including usage, perceived relevance and match with course/research requirements, as well as price. Most agreed that conventional metrics have limited benefit when value for money is being assessed. Some say that they cannot be used to compare value across disciplines. One of the most interesting findings to emerge from the focus groups was that, although librarians try hard to forge good communications with academics, there are gaps (on both sides) in the communications process, which means that sometimes resources are not always discovered and therefore used to their fullest potential.
“… conventional metrics have limited benefit when value for money is being assessed.”
Most of the librarians had a discovery system, though many said that it had not been in place for long enough to make gathering and assessing statistics meaningful. Time was also a factor; few said that they could spare the time – or indeed worked within an infrastructure – that made the collection of Web 2.0 statistics viable.
Most confirmed that budget squeezes meant that when a new resource was purchased, an existing subscription would have to be cancelled. As cancelling the wrong publications could result in disaster, having the ability to access and interpret correctly a relevant set of statistics would be invaluable. The Huddersfield librarians, in particular, could supply evidence that after the installation of their discovery system the usage of more specialist resources had increased.
When asked why they might choose one resource over another, the librarians cited the following reasons: academic demand (sometimes including student demand); relevance to what is being taught/researched; cost; quality; ease of use/flexibility (of both navigability and the business model) and compatibility with discovery systems.
Librarians have developed a comprehensive range of methods to promote electronic resources, including training sessions (for lecturers, students and researchers); teaching sessions ‘embedded’ in lectures; programmes of workshops; one-to-one sessions with researchers; links in the online public access catalogue (OPAC) or virtual learning environment (VLE), or in reading lists; changing plasma screens in the library that illustrate resources; LibGuides; online and printed subject guides; and e-mail alerts, newsletters and ‘latest news’ announcements, often now delivered via Twitter or other social media. Some have developed portals or subject blogs that link in to discovery services.
“Some universities record a drop in choice of certain AHSS subjects …”
Resources most favoured by students were full-text primary works and journals, database products, criticism and reviews and information on research methods. Although some respondents said that students are format-agnostic, most said that they value highly cross-searchability of databases.
Resources most favoured by academics were full-text primary works and journals, database products and abstract and indexing (A & I) products. The latter seems indicative that some academics are not yet aware of the potential of discovery systems. Younger academics seem to prefer electronic to print. There was some anecdotal evidence that the quality and range of electronic resources held by the university is instrumental in attracting academics, and in subsequent staff retention.
Asked about the effect of government spending cuts, the UK respondents said that there is some indication that more students are choosing vocationally-orientated courses such as business and health sciences. Some universities record a drop in choice of certain AHSS subjects, e.g., history and archaeology; there is some evidence of polarization of ‘winner’ and ‘loser’ universities for specific subjects.
Individual comments from librarians included:
‘Humanities resources can be quite narrow in scope and therefore relevant to a smallish number of people; so when contrasted with broader coverage science or social science resources, they might be at risk of being seen to be less influential and as a consequence more vulnerable to cuts.’ (UK: ‘old’ university)
‘The increasing dependence of libraries on numbers/statistics over quality is worrying. Humanities are not as easily categorised by numbers as the sciences. This approach removes the opportunities for the ‘quick sound-bite’ that is useful in hall talk or publicity to advertise the Humanities.’ (USA: Ivy League university)
‘There is some evidence from student panels that they've come for the resources. We now have categorised evidence that the library adds value to the courses and helps attainment. There is positive correlation between use of the library and achievement.’ (UK: ‘new’ university)
Individual comments from academics included:
‘I can't overstate the difference [now that we have] online collections. It still seems miraculous that we have so many texts available to us … being able to read and search these texts has transformed research possibilities.’ (UK: ‘old’ university)
‘Repositories are all very well, but we need a way of identifying things that are available online. There is a patchiness [lack of match] between what is available and what is in the repository.’ (UK: ‘new’ university)
‘It is very important that assessment of critical thinking comprise a component when assessing the value of Humanities resources.’(USA: Ivy League university).
Usage results at a middle-sized UK university before and after the installation of the Summon discovery system
The final part of the study looked at how data obtained from ProQuest databases support the findings of the surveys and focus groups mentioned above. Statistics were prepared to ascertain the use of Periodicals Archive Online via the Summon discovery service at a middle-sized English university over the three-and-a-half year period July 2009 – December 2012 (see Figure 2). Summon was installed at this university in July 2010. Usage of Periodicals Archive Online increased modestly over the next few months (unsurprisingly, given that Summon was installed during the summer vacation) and then surged dramatically in the peak viewing month of November. Views in November 2010 were four times as great as the same period in 2009. They dropped slightly in November 2011 (a statistical blip which was more than compensated for by a much higher view rate during the summer of that year). There was then another massive hike, to almost 1,400 views, in November 2012. The December data saw a usage increase of over 1200% over the three-year period.
Another interesting feature concerning customer satisfaction was that it improved yet further when a large number of pages were viewed by the same patron. Further work would need to be carried out in order to understand the exact reasons for this; but anecdotal evidence suggests that this might be that power users that spend more time on the platform know how to obtain the best out of it.
The discovery expectations of patrons in all arts and humanities subjects were higher than the average for all disciplines by an average of 10%, with history (at 15% higher than the average for all subjects) achieving particularly high satisfaction ratings. Most patrons said that they had been able to accomplish the task that they set out to do, with those working in social and political sciences being particularly satisfied with the results that they were able to achieve.
Taking into account all of the above research, it can be concluded that:
- librarians develop ingenious and thoughtful ways of measuring value for money of AHSS resources; however, they need a new paradigm for interpreting metrics as access methods increase in sophistication
- a stronger communication loop between librarians and academics would be beneficial (though may be difficult to achieve)
- new measurement methods will help to address the perennial problem that AHSS transactions are less numerous than STM transactions
- providing a new approach to metrics may contribute towards development of resources strategies to meet changing user needs and demands
- usage statistics alone do not give the full view of the value to the institution from access to online journals in the humanities, but it is possible to use supplementary return on investment (ROI) measures to make them more meaningful.
In order to disseminate the research more widely and to provide others with the tools used, ProQuest is making available a toolkit containing the key documents used. The toolkit contains a PDF of the Hanover survey, complete with charts by country and occupation; the PowerPoint presentation given at the UKSG 2013 conference; a short guide to methods to use for assessing value, based on the research outcomes; a short guide to methods of increasing usage, based on the research outcomes; a set of the questionnaires used for the online survey, the focus groups and control surveys; and two ProQuest white papers on getting the most from usage statistics.
“usage statistics alone do not give the full view of the value to the institution …”
If you would like to receive the toolkit, please contact firstname.lastname@example.org