Usage, engagement and impact: evaluating the usage of and measuring impact and engagement with library resources at Leeds Beckett University Library

In 2017–18 Leeds Beckett University Library undertook a project to assess the feasibility of using OpenAthens and student record data to enable in-depth analysis of learning resources usage and costs, student engagement and library impact. This article describes the different approaches taken to evaluate usage of the Library’s e-resources to identify levels of engagement at school and course level. These include evaluating the impact of induction attendance on e-resource usage, reviewing school e-resource usage and return on investment, analysing usage trends at school and year level and examining the correlation between National Student Survey (NSS) results and e-resource usage for specific courses. The project confirmed the feasibility of using OpenAthens and student record data for in-depth analysis of learning resources usage, student engagement and library impact. Successful analysis was performed for schools, courses and levels of study and provided significant insight that informed resource provision and NSS action. Library managers, academic librarians and academics have consequently engaged with and benefited from the project and its outputs. This article builds on a breakout session presented at the 42 nd UKSG Annual Conference in April 2019.


Background
In 2017-18 Leeds Beckett Library undertook a project to assess the feasibility of using OpenAthens and student record data to enable in-depth analysis of learning resources usage and costs, student engagement and library impact. OpenAthens is an access and identity management system used to authenticate users to access subscribed e-resources. Our e-resource usage statistical data, where a user accesses a particular resource, was obtained via the administration site of the OpenAthens system. The student record data was a weekly snapshot report run by our IT Services and Registry departments and saved in a secure folder where permission was needed to access it.
The aim of the project was to evaluate the usage of e-resources by students to identify levels of engagement at school and course level and to inform resource provision.
There were existing monthly e-resource usage reports from OpenAthens, but they only showed the number of users per provider, and it had not been previously possible to analyse the data beyond that level. The creation of the new Electronic and Data Services Librarian role in May 2017, whose remit was to combine OpenAthens data with student record snapshot data at the user ID level, meant we could get a more granular level of usage by course and school level. The role holder had previously worked as an information data analyst in Registry so had experience of combining student record data with other data sets for performance monitoring. This project was a piece of operational analysis to provide evidence and insight to support collections management and decision-making so that our users have appropriate and value-for-money learning resources. We were not aware of anyone else in the sector who had undertaken this type of analysis, so we were not influenced in how we carried out the feasibility study.
Prior to commencement, we ensured that the analysis we were undertaking complied with Leeds Beckett University's Data Protection policy and the Student Privacy Notice 1 which is validated by the Student Contract signed by all Leeds Beckett students as part of registration. 2

Proof of concept
In May 2017 the new Electronic and Data Services Librarian developed a proof of concept which combined OpenAthens, stock circulation and reading list data with student record data at the user ID level. This demonstrated that it was possible to provide more granular usage data on book loans, user e-resource accesses through OpenAthens and reading-list accesses for analysis at course and school level.
Data for 338 students on the Childhood Studies course was used. We were able to combine student data with reading-list data to see if students on the course were accessing their reading lists, and with e-resource access data from OpenAthens ( Figure 1a) and book loans from the library management system (Figure 1b) to analyse resource usage. 'it was possible to provide more granular usage data … for analysis at course and school level' 'We were not aware of anyone else in the sector who had undertaken this type of analysis' The initial analysis showed that students were using e-resources more than physical stock and that the most used e-resources had content relevant to the course, for example Wiley Online Library, Taylor & Francis Online and the digitized articles and book chapters that had been uploaded to the Copyright Licencing Agency's Digital Content Store. However, we noticed that the items on the reading lists did not appear in the book loans. This raised the question whether students were reading beyond the items recommended on the reading lists or simply not using the reading lists. This would have required further investigation of reading list usage, but this was out of the scope of the project, which prioritized OpenAthens data, due to the capacity constraints of working in Access and Excel.
The proof of concept also highlighted multiple combinations of titles for the same e-resources in the OpenAthens data, which in some cases also differed from the title of the resource used locally. It was therefore necessary to enhance the OpenAthens data by combining the multiple entries under the local e-resource name, thus making it more meaningful for stakeholders, as shown in Figure 2. The final column shows the final resource name used in reports to Libraries and Learning Innovation (LLI). The focused analysis showed that the total number of unique users (students) declined over the period, there had been an in increase in the number of e-resources accessed by those students, the total number of accesses of e-resources peaked in September and November, and there had been usage in August prior to the start of term and in September prior to the library induction.
The early usage was accesses to the NUS extra Student Discount card and UNiDAYS Perks of Student Life websites, probably due to students having received their University information packs around that time. From September, students were using library e-resources like Cengage Learning, Kanopy, Box of Broadcasts and Wiley. Their course inductions would have taken place by then, which may account for that usage. The library induction was delivered in November and, following induction, the breadth of e-resources used increased. This may have been due to the promotion of the library's search engine (EBSCO Discovery Service) and the demonstration of searching for online journals. For example, Taylor & Francis Online was only used afterwards and the usage of Wiley increased again after the induction. Kanopy and Box of Broadcasts, also promoted as part of the induction, continued to be used. More e-resources were accessed by those who attended the induction compared to those who did not. For example: Oxford University Press e-resources were only accessed following the induction and only by students who had attended it, and Oxford Reference Online had been promoted.
It was too early to see any long-lasting impact and the numbers were too small to conclude a direct relationship, but it may be inferred that the library induction encouraged the students who attended to use the e-resources promoted. However, we did acknowledge that in order to make more tangible conclusions we would have needed to gather further qualitative information through liaison with the academic librarians (who work closely with students), staff and students.

Annual reports
In August 2018 a full academic year of data was analysed and data tables and visualizations were included in the Library's annual reports for each School. The School annual reports are based around the Schools' key performance indicators and provide data on staff and student engagement with Library resources, teaching and support. Some of the data that was provided included e-resource usage, resource list engagement and resource provision for specific Schools. The reports highlight the return on investment (ROI) for resource provision and usage while linking to the National Student Survey (NSS) results for the School.
'More e-resources were accessed by those who attended the induction' An institutional overview of e-resource accesses by School from August 2017 to July 2018 ( Figure 4) was provided. The purple line is the institutional percentage of all student users accessing e-resources through OpenAthens. This can be compared with the lilac bars which are the Schools' percentage of all student users accessing e-resources through OpenAthens. The green line is the institutional average number of accesses per active user and is for comparison with the green diamonds which are the Schools' average numbers of accesses per active user; active users being those students who have used e-resources through OpenAthens, as opposed to those who have not. The blue line is the institutional average number of accesses per all users and is for comparison with the blue diamonds which are the Schools' average numbers of accesses per all users; all users being all student users including those students who have not used e-resources through OpenAthens, as opposed to only those who have. Further analysis showed that the spike referred to a particular e-resource, and usage mainly came from nine students across four Schools and seven courses (Table 1).

School
Course title Anonymized student With academic librarian insight, the consensus was that the e-resource usage was legitimate. For example: some Events Tourism and Hospitality Management courses run on a three-semester year, continuing teaching through the summer, so it was possible that their deadlines fell then; also the Digital and Technical Solutions course has students working on unusual topics in quite distinctive ways, so one of them producing an unusual access pattern would not be that unexpected.

Number of accesses
Tables showing the top ten e-resources by School were also provided.   Table 2 shows usage for the School of Sport and indicates that students were accessing a range of e-resources. Their top ten accounted for 88 per cent of total e-resources accessed and the top three made up 71 per cent. The high usage of Taylor & Francis Online was a positive outcome for the library as an example of an ROI, because it was licensed following requests from academic staff in that School.

Bespoke reports
The annual reports were presented to a University Teaching and Learning Steering Group; consequently, a bespoke report was created for the School of Sport, which provided a deeper level of analysis.
'high usage of Taylor & Francis Online was a positive outcome … because it was licensed following requests from academic staff'    Academics were also interested in annual trends. Figure 9 shows school e-resource usage was higher in October and November following library inductions and for submission deadlines in March and April.
The annual trend was broken down by level (Figures 10a, 10b, 10c and 10d), which highlighted different patterns of usage, for example less usage in vacation periods for undergraduate years 1 and 2, and in January and February for undergraduate year 3 and taught-postgraduates following dissertation work or exam revision over Christmas.
'Academic staff were particularly interested in students' e-resource usage at different levels of study' These patterns supported findings from discussions with undergraduate students which suggested high engagement in year 1, particularly after inductions and at submission deadlines, coasting in year 2 and re-engagement in year 3, especially at dissertation time. For postgraduate students, there was similar engagement to year 1 undergraduates. The spikes in March and May were potentially attributed to dissertation reference checking.   Figure 11 shows courses that scored question 19 at the lower and upper end of the NSS analysis. The courses codes, for example BGAME for BSc (Hons) Games Design Course and BSSLT for BSc (Hons) Speech and Language Therapy Course, were used as course identifiers. The red line reflects the institutional average score (90 per cent). The red highlights are the low-scoring courses. The orange highlights are falling courses with drops of more than two percentage points in the NSS. The green highlights are the improving and high-scoring courses.  Figure 12 shows the low-scoring NSS courses compared with the NSS response rate, the percentage of active users of e-resources and the course population. Figure 12 indicates that some low-scoring courses (below the NSS average shown by the red line) had low percentages of active users but a good NSS response rate as well as a larger course population. Two of these courses, picked out by the purple dashed line in Figure 12, were examined in more detail: BSc (Hons) Computing (CMPXX) and BA (Hons) Accounting and Finance (ACCOU), and the resource usage for students on these courses was investigated. (See Tables 3 and 4

Percentage of overall e-resource usage
Box of Broadcasts 23 10% Elsevier 21 9% ProQuest and Chadwyck-Healey databases 12 5% It was determined that the resources used by students on these courses were relevant to their subject areas but had low levels of usage -see Figure 12 -which raised the questions: Are they the right resources? Do they need more promotion? Would improving engagement with e-resources increase NSS scores? Figure 13. Falling NSS courses: Sport Business Management (BHSBM) and Sport and Exercise Science (SPEXS) compared across NSS response rate, percentage of active users and course population Figure 13 shows some falling courses also had a good NSS response rate and a larger course population but a higher percentage of active users. These were BA (Hons) Sport Business Management (BHSBM) and BSc (Hons) Sport and Exercise Science (SPEXS), picked out by the purple dashed line. The resource usage for students on these courses was investigated (Tables 5 and 6).

Sport Business Management students
Resources accessed

E-resource accesses
Percentage of overall e-resource usage The e-resource usage of these courses showed a reasonable number of accesses. Again, this raised questions: were the students dissatisfied with existing e-resource provision? Were they unaware the library provided them? Insight from the academic librarians suggested that students do not necessarily associate the Library with e-resources, due to using Google Scholar, reading lists or online module handbooks that link through to Library e-resources, and score accordingly. Figure 14. High-scoring NSS courses, focusing on Human Geography (HGEOG) and Speech and Language Therapy (BSSLT), compared across NSS response rate, percentage of active users and course population Figure 14 shows the high-scoring courses. They had different NSS response rates and different percentages of active users, but most had a smaller course population. A flatter line would have been expected for the percentage of active users if there was any correlation because as the NSS score was 100 percent for the last 15 courses, the percentage of active users should have followed the same trend. The courses concentrated on were: BA (Hons) Human Geography (HGEOG) and BSc (Hons) Speech and Language Therapy (BSSLT), picked out by the purple dashed line and the resource usage for students on these courses was again investigated (See Tables 7 and 8).

Human Geography students
Resources accessed

E-resource accesses
Percentage of overall e-resource usage 'the courses with high NSS scores did appear to make more use of e-resources' chapters linked directly from the reading lists or the virtual learning environment (VLE). One explanation for the high NSS score could be that students value the more tailored, personalized experience of linking directly to the specific chapters and articles they needed for their course rather than expecting them to access a book or journal and search for the specific chapter or article.
Overall, there was no clear correlation between e-resource usage and high NSS scores. Some low-scoring NSS courses had relatively high e-resource usage, while some high-scoring courses had relatively low usage. However, the courses with high NSS scores did appear to make more use of e-resources and over 90 per cent of students on those courses were active users.

Challenges
Analysis and visualization were restricted by the software available, because Access and Excel are limited in terms of the amount of data that can be processed and the lack of charts and dashboard functionality. That meant complex time-consuming workarounds were necessary and therefore technical and analytical expertise was needed, which only the Electronic and Data Services Librarian had. So there was also limited resource and a potential single point of failure.
Also, OpenAthens e-resource usage is only part of the picture, as only 75 per cent of the e-resources use OpenAthens authentication; another 23 per cent are authenticated through EZproxy. To achieve a more holistic view of e-resource usage, the EZproxy logs would also need to be analysed, but the files were so large that only very focused analyses could be performed.
Some risks identified also become issues, due to loss of access and changes to OpenAthens and student record data while they were being enhanced.

Conclusions and beyond
The project confirmed the feasibility of using OpenAthens and student record data for in-depth analysis of learning resources usage, student engagement and library impact. Successful analysis was performed for Schools, courses and levels of study and provided significant insight that informed resource provision and NSS action. These included renewal and retention decisions and an insight into user resource retrieval behaviour to inform our future resource provision projects to see if we can influence user engagement and satisfaction. Library managers, academic librarians and academics have consequently engaged with and benefited from the project and its outputs.
While providing some interesting insights, the project highlighted that further investigation and data analysis is required to deliver more tangible conclusions. There is great potential in this area and future analyses may include widening the scope to cover book loans, EZproxy, reading lists, the Digital Content Store and VLE usage to obtain a more holistic view, as well as the inclusion of impact, longitudinal and predictive analysis. We will be looking to build on work already done to compare data year on year, to take particular actions like working with a particular School or promotion of specific resources to make projections on whether that might increase engagement with resources and improve NSS scores.
It also needs to be considered whether there is dissatisfaction with current provision and if so, why, and whether more promotion of the Library and its resources or a more personalized, tailored approach to resource provision is needed. Quantitative data tells only part of the story and continuing to work closely with academic librarians who engage with academic and student stakeholders is imperative to gather qualitative information to complete the picture.

Abbreviations and Acronyms
A list of the abbreviations and acronyms used in this and other Insights articles can be accessed here -click on the URL below and then select the 'full list of industry A&As' link: http://www.uksg.org/publications#aa