Start Submission Become a Reviewer

Reading: Usage, engagement and impact: evaluating the usage of and measuring impact and engagement wi...

Download

A- A+
Alt. Display

Case Studies

Usage, engagement and impact: evaluating the usage of and measuring impact and engagement with library resources at Leeds Beckett University Library

Authors:

Julie Cleverley ,

Journal and Electronic Resources Manager, Leeds Beckett University, GB
X close

Samantha Heeson

Electronic and Data Services Librarian, Leeds Beckett University, GB
X close

Abstract

In 2017–18 Leeds Beckett University Library undertook a project to assess the feasibility of using OpenAthens and student record data to enable in-depth analysis of learning resources usage and costs, student engagement and library impact. This article describes the different approaches taken to evaluate usage of the Library’s e-resources to identify levels of engagement at school and course level. These include evaluating the impact of induction attendance on e-resource usage, reviewing school e-resource usage and return on investment, analysing usage trends at school and year level and examining the correlation between National Student Survey (NSS) results and e-resource usage for specific courses. The project confirmed the feasibility of using OpenAthens and student record data for in-depth analysis of learning resources usage, student engagement and library impact. Successful analysis was performed for schools, courses and levels of study and provided significant insight that informed resource provision and NSS action. Library managers, academic librarians and academics have consequently engaged with and benefited from the project and its outputs. This article builds on a breakout session presented at the 42nd UKSG Annual Conference in April 2019.

How to Cite: Cleverley, Julie, and Samantha Heeson. 2019. “Usage, Engagement and Impact: Evaluating the Usage of and Measuring Impact and Engagement with Library Resources at Leeds Beckett University Library”. Insights 32 (1): 25. DOI: http://doi.org/10.1629/uksg.474
221
Views
37
Downloads
12
Twitter
  Published on 18 Sep 2019
 Accepted on 17 Jul 2019            Submitted on 11 Jun 2019

Background

In 2017–18 Leeds Beckett Library undertook a project to assess the feasibility of using OpenAthens and student record data to enable in-depth analysis of learning resources usage and costs, student engagement and library impact. OpenAthens is an access and identity management system used to authenticate users to access subscribed e-resources. Our e-resource usage statistical data, where a user accesses a particular resource, was obtained via the administration site of the OpenAthens system. The student record data was a weekly snapshot report run by our IT Services and Registry departments and saved in a secure folder where permission was needed to access it.

The aim of the project was to evaluate the usage of e-resources by students to identify levels of engagement at school and course level and to inform resource provision.

There were existing monthly e-resource usage reports from OpenAthens, but they only showed the number of users per provider, and it had not been previously possible to analyse the data beyond that level. The creation of the new Electronic and Data Services Librarian role in May 2017, whose remit was to combine OpenAthens data with student record snapshot data at the user ID level, meant we could get a more granular level of usage by course and school level. The role holder had previously worked as an information data analyst in Registry so had experience of combining student record data with other data sets for performance monitoring. This project was a piece of operational analysis to provide evidence and insight to support collections management and decision-making so that our users have appropriate and value-for-money learning resources. We were not aware of anyone else in the sector who had undertaken this type of analysis, so we were not influenced in how we carried out the feasibility study.

Prior to commencement, we ensured that the analysis we were undertaking complied with Leeds Beckett University’s Data Protection policy and the Student Privacy Notice1 which is validated by the Student Contract signed by all Leeds Beckett students as part of registration.2

Proof of concept

In May 2017 the new Electronic and Data Services Librarian developed a proof of concept which combined OpenAthens, stock circulation and reading list data with student record data at the user ID level. This demonstrated that it was possible to provide more granular usage data on book loans, user e-resource accesses through OpenAthens and reading-list accesses for analysis at course and school level.

Data for 338 students on the Childhood Studies course was used. We were able to combine student data with reading-list data to see if students on the course were accessing their reading lists, and with e-resource access data from OpenAthens (Figure 1a) and book loans from the library management system (Figure 1b) to analyse resource usage.

Figure 1a 

E-resource usage for students on Childhood Studies course (April–June 2017)

Figure 1b 

Book loans for students on Childhood Studies course (April–June 2017)

The initial analysis showed that students were using e-resources more than physical stock and that the most used e-resources had content relevant to the course, for example Wiley Online Library, Taylor & Francis Online and the digitized articles and book chapters that had been uploaded to the Copyright Licencing Agency’s Digital Content Store. However, we noticed that the items on the reading lists did not appear in the book loans. This raised the question whether students were reading beyond the items recommended on the reading lists or simply not using the reading lists. This would have required further investigation of reading list usage, but this was out of the scope of the project, which prioritized OpenAthens data, due to the capacity constraints of working in Access and Excel.

The proof of concept also highlighted multiple combinations of titles for the same e-resources in the OpenAthens data, which in some cases also differed from the title of the resource used locally. It was therefore necessary to enhance the OpenAthens data by combining the multiple entries under the local e-resource name, thus making it more meaningful for stakeholders, as shown in Figure 2. The final column shows the final resource name used in reports to Libraries and Learning Innovation (LLI).

Figure 2 

Example of multiple labels for the same OpenAthens resource and the final LLI description

Prototype

A prototype was then developed to evaluate e-resource usage following library induction, to see if there was a positive impact on usage as a result of attendance.

It combined five months of OpenAthens data from August to December 2017 with a snapshot of student record data from November 2017. The analysis, which included visualizations (Figures 3a, 3b and 3c), focused on e-resource usage by students on Year 1 (Level 4) of the Childhood Studies course, who had their induction in November 2017.

Figure 3a 

Total number of unique student users of e-resources and accesses to e-resources (August – December 2017)

Figure 3b 

E-resources by number of unique users (August – December 2017)

Figure 3c 

E-resources by number of accesses (August – December 2017)

The focused analysis showed that the total number of unique users (students) declined over the period, there had been an in increase in the number of e-resources accessed by those students, the total number of accesses of e-resources peaked in September and November, and there had been usage in August prior to the start of term and in September prior to the library induction.

The early usage was accesses to the NUS extra Student Discount card and UNiDAYS Perks of Student Life websites, probably due to students having received their University information packs around that time. From September, students were using library e-resources like Cengage Learning, Kanopy, Box of Broadcasts and Wiley. Their course inductions would have taken place by then, which may account for that usage.

The library induction was delivered in November and, following induction, the breadth of e-resources used increased. This may have been due to the promotion of the library’s search engine (EBSCO Discovery Service) and the demonstration of searching for online journals. For example, Taylor & Francis Online was only used afterwards and the usage of Wiley increased again after the induction. Kanopy and Box of Broadcasts, also promoted as part of the induction, continued to be used. More e-resources were accessed by those who attended the induction compared to those who did not. For example: Oxford University Press e-resources were only accessed following the induction and only by students who had attended it, and Oxford Reference Online had been promoted.

It was too early to see any long-lasting impact and the numbers were too small to conclude a direct relationship, but it may be inferred that the library induction encouraged the students who attended to use the e-resources promoted. However, we did acknowledge that in order to make more tangible conclusions we would have needed to gather further qualitative information through liaison with the academic librarians (who work closely with students), staff and students.

Annual reports

In August 2018 a full academic year of data was analysed and data tables and visualizations were included in the Library’s annual reports for each School. The School annual reports are based around the Schools’ key performance indicators and provide data on staff and student engagement with Library resources, teaching and support. Some of the data that was provided included e-resource usage, resource list engagement and resource provision for specific Schools. The reports highlight the return on investment (ROI) for resource provision and usage while linking to the National Student Survey (NSS) results for the School.

An institutional overview of e-resource accesses by School from August 2017 to July 2018 (Figure 4) was provided. The purple line is the institutional percentage of all student users accessing e-resources through OpenAthens. This can be compared with the lilac bars which are the Schools’ percentage of all student users accessing e-resources through OpenAthens. The green line is the institutional average number of accesses per active user and is for comparison with the green diamonds which are the Schools’ average numbers of accesses per active user; active users being those students who have used e-resources through OpenAthens, as opposed to those who have not. The blue line is the institutional average number of accesses per all users and is for comparison with the blue diamonds which are the Schools’ average numbers of accesses per all users; all users being all student users including those students who have not used e-resources through OpenAthens, as opposed to only those who have.

Figure 4 

Annual institutional overview of users by school and e-resource accesses (August 2017–July 2018)

Key to Schools: BW: Languages; DU: Education; DW: Sport; DY: Events, Tourism and Hospitality Management; GA: Built Environment; GB: Art, Architecture and Design; GC: Cultural Studies and Humanities; GE: Film, Music and Performing Arts; GF: Computing, Creative Technologies & Engineering; ML: Law; MN: Business; RE: Clinical and Applied Sciences; RG: Social Sciences; RH: Health and Community Studies

Charts visualizing the annual trend of usage at an institutional level showed a large spike in e-resource usage in June (Figure 5), which prompted further investigation. The annual trends in each School were therefore examined and this highlighted the fact that four of the Schools had the same spike in usage: Events, Tourism and Hospitality Management, Computing and Creative Technology, Education, and Cultural Studies and Humanities.

Figure 5 

Annual institutional trend of users and e-resource accesses

Further analysis showed that the spike referred to a particular e-resource, and usage mainly came from nine students across four Schools and seven courses (Table 1).

Table 1

E-resource usage spike investigation

School Course title Anonymized student Number of accesses

DU Education EDUCATION STUDIES A 244
DY Events, Tourism and Hospitality Management TRAVEL BUSINESS MANGT (TU) B 275
DY Events, Tourism and Hospitality Management INT TOUR & HOSP MAN C 6,939
DY Events, Tourism and Hospitality Management RESPONSIBLE TOURISM MGT D 220
DY Events, Tourism and Hospitality Management LEISURE RETAIL MANAGT WSPOONS E 208
GC Cultural Studies and Humanities SOCIAL HISTORY F 1,780
GC Cultural Studies and Humanities SOCIAL HISTORY G 658
GC Cultural Studies and Humanities SOCIAL HISTORY H 219
GF Computing, Creative Technologies and Engineering DIGITAL & TECH SOLUTIONS (DA) I 245

With academic librarian insight, the consensus was that the e-resource usage was legitimate. For example: some Events Tourism and Hospitality Management courses run on a three-semester year, continuing teaching through the summer, so it was possible that their deadlines fell then; also the Digital and Technical Solutions course has students working on unusual topics in quite distinctive ways, so one of them producing an unusual access pattern would not be that unexpected.

Tables showing the top ten e-resources by School were also provided.

Table 2 shows usage for the School of Sport and indicates that students were accessing a range of e-resources. Their top ten accounted for 88 per cent of total e-resources accessed and the top three made up 71 per cent. The high usage of Taylor & Francis Online was a positive outcome for the library as an example of an ROI, because it was licensed following requests from academic staff in that School.

Table 2

School top 10 e-resources (August 2017–July 2018)

Pos School of SportTop ten e-resources Total e-resources accesses No. e-resources accesses No. active users

  School Total 100.00% 31,195 1,550

1 EBSCO Information Services 50.95% 15,895 780
2 Taylor & Francis Online 13.62% 4,248 430
3 Elsevier Products 6.38% 1,991 424
4 Wiley Online Library 5.28% 1,647 351
5 SAGE Journals 2.35% 734 209
6 Learning on Screen (Box of Broadcasts) 2.35% 734 119
7 ProQuest and Chadwyck-Healey databases 2.29% 714 225
8 Wolters Kluwer Health – Ovid 1.95% 609 83
9 Atypon SP 1.67% 520 209
10 Ovid Technologies, Inc. 0.97% 302 61
Total T10 Total 87.82% 27,394 2,891
Total Rest Total 12.18% 3,801 1,618

Bespoke reports

The annual reports were presented to a University Teaching and Learning Steering Group; consequently, a bespoke report was created for the School of Sport, which provided a deeper level of analysis.

Figure 6 shows e-resource usage in the Schools, with DW representing the School of Sport. The blue bars show the number of active users in the Schools and the orange bars show the total number of e-resources accesses through OpenAthens made by those active student users for the year.

Figure 6 

Schools users and e-resource accesses (August 2017–July 2018)

Key to Schools: BW: Languages; DU: Education; DW: Sport; DY: Events, Tourism and Hospitality Management; GA: Built Environment; GB: Art, Architecture and Design; GC: Cultural Studies and Humanities; GE: Film, Music and Performing Arts; GF: Computing, Creative Technologies & Engineering; ML: Law; MN: Business; RE: Clinical and Applied Sciences; RG: Social Sciences; RH: Health and Community Studies

Figure 7 shows the number of unique e-resources accessed. Sport students accessed 109 out of a possible 208 (the orange bar in the figure), showing use of a broad range of e-resources.

Figure 7 

Schools e-resources accesses (August 2017–July 2018)

Key to Schools: BW: Languages; DU: Education; DW: Sport; DY: Events, Tourism and Hospitality Management; GA: Built Environment; GB: Art, Architecture and Design; GC: Cultural Studies and Humanities; GE: Film, Music and Performing Arts; GF: Computing, Creative Technologies & Engineering; ML: Law; MN: Business; RE: Clinical and Applied Sciences; RG: Social Sciences; RH: Health and Community Studies

Academic staff were particularly interested in students’ e-resource usage at different levels of study, i.e. undergraduate (UG), taught postgraduate (PG) and research postgraduate (RP).

Figure 8 shows undergraduates used a greater range of e-resources, which declines at taught-postgraduate level and again at research level. It was noted that this pattern of usage would be expected as students move from undergraduate to postgraduate courses and focus their area of study.

Figure 8 

School of Sport unique e-resources accessed by year

Key: UGY1: Undergraduate Year 1; UGY2: Undergraduate Year 2; UGY3: Undergraduate Year 3; TPY1: Taught Postgraduate Year 1; TPY2: Taught Postgraduate Year 2; RPY?: Research Postgraduate undefined year; RPY1: Research Postgraduate Year 1; RPY2: Research Postgraduate Year 2; RPY3: Research Postgraduate Year 3; RPY4: Research Postgraduate Year 4; RPY5: Research Postgraduate Year 5; RPY6: Research Postgraduate Year 6

Academics were also interested in annual trends. Figure 9 shows school e-resource usage was higher in October and November following library inductions and for submission deadlines in March and April.

Figure 9 

School of Sport’s annual trend for active users and e-resource accesses

The annual trend was broken down by level (Figures 10a, 10b, 10c and 10d), which highlighted different patterns of usage, for example less usage in vacation periods for undergraduate years 1 and 2, and in January and February for undergraduate year 3 and taught-postgraduates following dissertation work or exam revision over Christmas.

Figure 10a 

School of Sport undergraduate first-year trend for active users and e-resource accesses

Figure 10b 

School of Sport undergraduate second-year trend for active users and e-resource accesses

Figure 10c 

School of Sport undergraduate third-year trend for active users and e-resource accesses

Figure 10d 

School of Sport undergraduate taught postgraduate trend for active users and e-resource accesses

These patterns supported findings from discussions with undergraduate students which suggested high engagement in year 1, particularly after inductions and at submission deadlines, coasting in year 2 and re-engagement in year 3, especially at dissertation time. For postgraduate students, there was similar engagement to year 1 undergraduates. The spikes in March and May were potentially attributed to dissertation reference checking.

National Student Survey analysis

A correlation analysis was performed on NSS results and e-resource usage. The NSS is an independent questionnaire sent to all final-year undergraduates asking for feedback on all aspects of the student experience including the course and university services.3 Our analysis focused on the 2018 scores for question 19: ‘The library resources have supported my learning well’ and looked at the e-resource usage of the survey population during their final year (2017–18, level 6), which is when they completed the NSS. It also concentrated on courses that had the lowest scores, significant decreases or increases in score and the highest scores. The intention of the analysis was to identify if the levels of engagement with e-resources directly correlated with the scoring of question 19 in the NSS.

Figure 11 shows courses that scored question 19 at the lower and upper end of the NSS analysis. The courses codes, for example BGAME for BSc (Hons) Games Design Course and BSSLT for BSc (Hons) Speech and Language Therapy Course, were used as course identifiers. The red line reflects the institutional average score (90 per cent). The red highlights are the low-scoring courses. The orange highlights are falling courses with drops of more than two percentage points in the NSS. The green highlights are the improving and high-scoring courses.

Figure 11 

NSS question 19 – course performance against the NSS average of 90% (shown by red line)

Figure 12 shows the low-scoring NSS courses compared with the NSS response rate, the percentage of active users of e-resources and the course population. Figure 12 indicates that some low-scoring courses (below the NSS average shown by the red line) had low percentages of active users but a good NSS response rate as well as a larger course population. Two of these courses, picked out by the purple dashed line in Figure 12, were examined in more detail: BSc (Hons) Computing (CMPXX) and BA (Hons) Accounting and Finance (ACCOU), and the resource usage for students on these courses was investigated. (See Tables 3 and 4.)

Figure 12 

Low-scoring NSS courses, focusing on Computing (CMPXX) and Accounting and Finance (ACCOU), compared across NSS response rate, percentage of active users and course population

Table 3

E-resource accesses for students on the Computing course (August 2017-July 2018)

Computing students

Resources accessed E-resource accesses Percentage of overall e-resource usage

Box of Broadcasts 23 10%
Elsevier 21 9%
ProQuest and Chadwyck-Healey databases 12 5%

Table 4

E-resource accesses for students on the Accounting and Finance course (August 2017–July 2018)

Accounting and Finance students

Resources accessed E-resource accesses Percentage of overall e-resource usage

Marketline 62 13%
Lexis Nexis 59 12%
Taylor & Francis 30 6%

It was determined that the resources used by students on these courses were relevant to their subject areas but had low levels of usage – see Figure 12 – which raised the questions: Are they the right resources? Do they need more promotion? Would improving engagement with e-resources increase NSS scores?

Figure 13 shows some falling courses also had a good NSS response rate and a larger course population but a higher percentage of active users. These were BA (Hons) Sport Business Management (BHSBM) and BSc (Hons) Sport and Exercise Science (SPEXS), picked out by the purple dashed line. The resource usage for students on these courses was investigated (Tables 5 and 6).

Figure 13 

Falling NSS courses: Sport Business Management (BHSBM) and Sport and Exercise Science (SPEXS) compared across NSS response rate, percentage of active users and course population

Table 5

E-resource accesses for students on the Sport Business Management course (August 2017–July 2018)

Sport Business Management students

Resources accessed E-resource accesses Percentage of overall e-resource usage

Taylor & Francis 378 18%
Wiley 344 16%
Elsevier 184 9%

Table 6

E-resource accesses for students on the Sport and Exercise Science course (August 2017–July 2018)

Sport and Exercise Science students

Resources accessed E-resource accesses Percentage of overall e-resource usage

EBSCO 11,066 73%
Taylor & Francis 750 5%
Elsevier 644 4%

The e-resource usage of these courses showed a reasonable number of accesses. Again, this raised questions: were the students dissatisfied with existing e-resource provision? Were they unaware the library provided them? Insight from the academic librarians suggested that students do not necessarily associate the Library with e-resources, due to using Google Scholar, reading lists or online module handbooks that link through to Library e-resources, and score accordingly.

Figure 14 shows the high-scoring courses. They had different NSS response rates and different percentages of active users, but most had a smaller course population. A flatter line would have been expected for the percentage of active users if there was any correlation because as the NSS score was 100 percent for the last 15 courses, the percentage of active users should have followed the same trend. The courses concentrated on were: BA (Hons) Human Geography (HGEOG) and BSc (Hons) Speech and Language Therapy (BSSLT), picked out by the purple dashed line and the resource usage for students on these courses was again investigated (See Tables 7 and 8).

Figure 14 

High-scoring NSS courses, focusing on Human Geography (HGEOG) and Speech and Language Therapy (BSSLT), compared across NSS response rate, percentage of active users and course population

Table 7

E-resource accesses for students on the Human Geography course (August 2017–July 2018)

Human Geography students

Resources accessed E-resource accesses Percentage of overall e-resource usage

Digimap 228 26%
Taylor & Francis 122 14%
SAGE 72 8%

Table 8

E-resource accesses for students on the Speech and Language course (August 2017–July 2018)

Speech and Language students

Resources accessed E-resource accesses Percentage of overall e-resource usage

Digital Content Store 539 30%
EBSCO 277 16%
Lexis Nexis 59 3%

E-resource usage was lower than for the falling courses, but that may be due to smaller cohorts. It was noted that the Speech and Language accesses to the Digital Content Store accounted for 30 per cent of all e-resource accesses. These are digitized articles or book chapters linked directly from the reading lists or the virtual learning environment (VLE). One explanation for the high NSS score could be that students value the more tailored, personalized experience of linking directly to the specific chapters and articles they needed for their course rather than expecting them to access a book or journal and search for the specific chapter or article.

Overall, there was no clear correlation between e-resource usage and high NSS scores. Some low-scoring NSS courses had relatively high e-resource usage, while some high-scoring courses had relatively low usage. However, the courses with high NSS scores did appear to make more use of e-resources and over 90 per cent of students on those courses were active users.

Challenges

Analysis and visualization were restricted by the software available, because Access and Excel are limited in terms of the amount of data that can be processed and the lack of charts and dashboard functionality. That meant complex time-consuming workarounds were necessary and therefore technical and analytical expertise was needed, which only the Electronic and Data Services Librarian had. So there was also limited resource and a potential single point of failure.

Also, OpenAthens e-resource usage is only part of the picture, as only 75 per cent of the e-resources use OpenAthens authentication; another 23 per cent are authenticated through EZproxy. To achieve a more holistic view of e-resource usage, the EZproxy logs would also need to be analysed, but the files were so large that only very focused analyses could be performed.

Some risks identified also become issues, due to loss of access and changes to OpenAthens and student record data while they were being enhanced.

Conclusions and beyond

The project confirmed the feasibility of using OpenAthens and student record data for in-depth analysis of learning resources usage, student engagement and library impact. Successful analysis was performed for Schools, courses and levels of study and provided significant insight that informed resource provision and NSS action. These included renewal and retention decisions and an insight into user resource retrieval behaviour to inform our future resource provision projects to see if we can influence user engagement and satisfaction. Library managers, academic librarians and academics have consequently engaged with and benefited from the project and its outputs.

While providing some interesting insights, the project highlighted that further investigation and data analysis is required to deliver more tangible conclusions. There is great potential in this area and future analyses may include widening the scope to cover book loans, EZproxy, reading lists, the Digital Content Store and VLE usage to obtain a more holistic view, as well as the inclusion of impact, longitudinal and predictive analysis. We will be looking to build on work already done to compare data year on year, to take particular actions like working with a particular School or promotion of specific resources to make projections on whether that might increase engagement with resources and improve NSS scores.

It also needs to be considered whether there is dissatisfaction with current provision and if so, why, and whether more promotion of the Library and its resources or a more personalized, tailored approach to resource provision is needed. Quantitative data tells only part of the story and continuing to work closely with academic librarians who engage with academic and student stakeholders is imperative to gather qualitative information to complete the picture.

Abbreviations and Acronyms

A list of the abbreviations and acronyms used in this and other Insights articles can be accessed here – click on the URL below and then select the ‘full list of industry A&As’ link: http://www.uksg.org/publications#aa

Competing interests

The authors have declared no competing interests.

References

  1. “Data Protection: Partners – Compliance and Legal,” Leeds Beckett University: https://www.leedsbeckett.ac.uk/partners/compliance-legal/data-protection/ (accessed 22 July 2019). 

  2. “Student Regulations,” Leeds Beckett University: https://www.leedsbeckett.ac.uk/public-information/student-regulations/ (accessed 22 July 2019). 

  3. “About the NSS,” The National Student Survey: https://www.thestudentsurvey.com/about.php (accessed 22 July 2019). 

comments powered by Disqus