Introduction

As this article goes to press, we are entering what is being referred to, with the sort of hyperbole usually reserved for a Star Wars trailer, as ‘TEF Year 3’. We have much more information than we did in ‘TEF Year 2’, but this is a fast-changing landscape with considerable uncertainty over the impact and further development of the Teaching Excellence Framework (TEF), let alone what it may mean for academic libraries and learning support.

Times Higher Education continues to publish opinion pieces about the relevance of the TEF,1, 2, 3 with vice-chancellors from some institutions (often Gold rated) making grand claims for TEF’s ability to measure teaching excellence and others arguing that proxy measures provide poxy results.

Uncertainty and opinion aside, the TEF is clearly here to stay. Of course, it does not exist in a policy or pedagogic vacuum and it can be seen as both a reaction to and a symptom of discourse about the broader higher education (HE) landscape. Jo Johnson, the Minister of State for Universities and Science, has long bemoaned the ‘patchiness in the student experience’ in relation to teaching,4 with the TEF being seen as a way of exposing the inconsistency he believes to be a weakness in the sector. Since the introduction of student fees, the Higher Education Policy Institute Student Academic Experience Survey has published declining student satisfaction with value for money – decreasing from 52% in 2012 to 35% in 2017, a decrease that is seen as a reflection of how the consumerization of HE affects the views of its customers.5

Approaches to learning and teaching, be they blended or flipped, are already impacting on learning resource provision and support, with greater need for personalized and packaged electronic resources. The introduction of student fees and the application of consumer rights legislation to universities have already had a significant effect on student appetite to purchase core texts and the need to address the so-called ‘hidden costs’ of study, which include the purchase of learning resources.

At the very centre of these concerns is the student. Inevitably, and with a certain lack of imagination, those born after 1995 are being referred to as Generation Z. This generation was born into a world where the internet, social media and mobile technology have always existed. According to Eric Stoller, universities that understand how Generation Z think will ‘win the day’ by concentrating on student-focused practice which understands students to be co-producers in the education process.6 Anne Kingston describes Generation Z as ‘smarter than Boomers and way more ambitious than Millenials’.7 She poses the challenge: ‘Are we ready for this influx of industrious, collaborative and entrepreneurial learners?’ In many ways these comments about collaborative learners align very closely with the TEF’s focus on student engagement.

The Teaching Excellence Framework

The Department of Education8 states that the TEF was introduced to better inform student choice on what and where to study; to raise esteem for teaching; to recognize and reward excellent teaching and to better meet the needs of employers, industry and the professions. The TEF uses three core ‘aspects of quality’ as indicators of teaching excellence. These are:

• teaching quality, assessed by metrics of student satisfaction for ‘teaching on my course’ and ‘assessment and feedback’ in the National Student Survey (NSS)
• academic support, assessed by metrics for ‘academic support’ in the NSS and by retention data
• learning gain, assessed by metrics of employment outcomes or further study in the Destination of Leavers in HE survey.

All the data are benchmarked to account for local context, including student characteristics such as gender, ethnicity and age. This approach seeks to allow very different institutions to be compared on a like-for-like basis.

When the TEF metrics were first announced, it was believed that they would be the major driver in determining the outcome results of Gold, Silver or Bronze. Although the metrics are critical to the final outcome, the 15-page institutional supporting statement has proved to have a much greater impact on the final result than was anticipated. Some institutions have had an initial Silver rating, based purely on metrics, changed to Gold on the evidence of the supporting statement. Others have found a Silver result changed to Bronze on the same analysis.

Learning resources and support do not feature at all in the metrics. However, institutions were instructed to include reference to them in the supporting statement, with a criterion for Resources included in the ‘Learning Environment’ aspect of quality. The HEFCE guidance criterion for Resources is focused on provision of content, with the definition: ‘Physical and digital resources are used effectively to aid students’ learning and the development of independent study skills’. In setting out how submissions will be judged in relation to this aspect of quality, the guidance states that Gold will be awarded for ‘outstanding’ provision that is ‘actively used … to enhance learning’ by students; Silver for ‘high quality’ provision ‘used … to enhance learning’ by students and Bronze for provision ‘used by students to further learning’.9

It is important to place this perceived role of libraries and learning resources in the context of the much broader range of activity relating to teaching excellence. There are ten reference areas, of which Resources is one. The others are:

• Student engagement
• Valuing teaching
• Rigour and stretch of the curriculum
• Quality of assessment and feedback
• Exposure to scholarship and innovative practice
• Level of personalisation in the learning experience
• Employment outcomes
• Transferable skill development
• Value added attainment.

By far the most important assessment criterion within the TEF is that of student engagement. Chris Husbands, Chair of the TEF Panel and Vice-Chancellor of Sheffield Hallam University, has said that Gold-rated institutions come from all parts of the sector, with very different missions and approaches. However, the TEF Panel identified common and compelling practices across these institutions which ‘… take(s) students’ interests, needs, aspirations and trajectories seriously (and) see(s) them as real partners’.10

A reading of the written submissions also suggests that those institutions doing very well in TEF (in addition to having good results in the metrics) are able to evidence actions and student engagement that impact the whole institution as opposed to there being excellent practice in one or two disciplines. Submissions from institutions that fared well in the TEF tend not to use data to simply impress the reader with quantity (we had x thousand visitors to the library) but to use the data to both evidence quality and to show how it has informed the enhancement of practice (we had an increase of x visitors year on year, they did this and we changed this in response …).

The submissions from Gold-rated institutions also tend to demonstrate cross-institutional alignment. For example, in one Gold submission the section on Teaching Quality described student feedback approaches and then linked this to the development of enhanced library opening hours – rather than waiting to include the opening hours reference in the Resources section. Another Gold submission referenced the importance of the Customer Service Excellence charter, gained by its library, in the introduction to the submission as evidence of the institutional support for empowering people.

A straw poll of SCONUL members11 suggests that many, but certainly not all, directors of information services were consulted to some extent in order to inform the supporting statements. This ranged from full membership of working groups that developed the whole submission, to providing data on library and learning support activity, to a request to comment on a final submission.

It is inevitable that this range of involvement reflects local politics and status. However, the findings of research carried out for the Booksellers Association (BA) by Linda Bennett and Annika Bennett12 point to a troubling level of ignorance amongst some senior administrators (those responsible for developing the TEF submission) about their own libraries and, by default, about the ways in which libraries can play an important part in the quality of teaching and learning. The BA report was written following a series of semi-structured telephone interviews with publishers, librarians, senior administrators and academics. It identified that communication between these groups was patchy and fractured and that this was particularly the case between key staff within universities. The authors stated: ‘… with some notable exceptions the university administrators knew less about their own libraries and resource provision generally than had been expected. It came as something of a surprise that some university administrators had barely considered resource provision when submitting their TEF applications.’13

The statement by Bennett and Bennett might well lead some more mature librarians to wonder what has happened since the Follett Report was published.14 In the report Follett advised that ‘… there is scope for giving more attention to library and related issues in the overall assessment of teaching quality’. This recommendation led to the development of the eLib national training programme for librarians who delivered teaching.

It could be argued that the TEF provides a fresh opportunity for libraries to demonstrate their value beyond the day-to-day delivery of services and to revisit and restate the role of the modern academic library in the institution and, specifically, in learning and teaching. The challenge to those leading information services is that of understanding the sphere of influence in their institution – who is making key decisions and who is influencing those decisions? If your Academic Registrar was blissfully unaware of the library service, why was this?

Metrics and analytics

The TEF is a metrics-based scheme which rewards evidence-based narrative statements. The current intention is to introduce subject-level ratings by TEF Year 5 (2019/2020) – with a need for data that can demonstrate quality at this more granular level. Library and learning support services will clearly need to be able to evidence their own practice in the same way. This is especially so if future submissions are to better demonstrate the impact of learning resources and support on teaching excellence and student engagement. This is not simply a way of ensuring librarians protect their jobs. Students and academic staff rely heavily on resources and on the support provided by information professionals. If there is a failure to demonstrate that value there is a risk of a lower institutional priority being placed on them and a resulting reduction in funding, with those core users being disadvantaged over time.

When asked what data source they used in the TEF supporting statement on Resources, library directors highlighted the reliance placed on the annual SCONUL statistics to report on information such as visits to libraries, use of resources and attendance by students at information skills sessions. In addition, service directors referred to data on library investment, Customer Excellence reports and data from institutional repositories. Feedback results from major surveys such as the NSS, the Postgraduate Teaching Experience Survey and the Postgraduate Research Experience Survey were also used. It is notable that the questions most pertinent to the library from such satisfaction questionnaires are concerned with resources provided rather than broader teaching support and this begs the question already apparent from the definition of Resources in the TEF of whether the library’s role in teaching and learning outcomes is recognized.

One significant development in HE is the growing use of learning analytics, which are being seen as a useful proxy for student engagement. By mining data from university corporate systems, it is possible to assess whether students are accessing teaching and learning. If required, targeted support can then be put into place before a student fails or withdraws.

An influential report, commissioned by the Higher Education Commission,15 analysed the potential of data and its analytics about learners to enhance the student experience. The report suggests that analytics can be used effectively to contextualize the metrics that underpin the TEF. The authors define learning analytics as the ‘measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs’.16

The authors argue that routine use of both ‘static’ and ‘fluid’ data in various combinations has the ability to transform HE by aiding higher education institutions (HEIs) individually and collectively to understand and uncover student behaviour and attributes in order to measure impact. This is made possible by the (now) ubiquitous online environment present in HEIs and beyond, which students engage with in a variety of ways as they learn with their university. This of course includes the library.

Of relevance to TEF, Norton and Porter argue that ‘Institutions should be encouraged to use the information from learning analytics systems to identify and foster excellent teaching within their institutions and to consider using this information in their submission to the TEF’.17

Norton and Porter signal a key role for libraries, suggesting that the level of use of the university library could indicate or predict likely student outcomes. They argue that by unlocking static data held in multiple systems (library management systems, reading list systems, e-resources accesses, gate counts) and manipulating these results with data held in other systems outside the library (student management systems, virtual learning environments [VLEs], attendance systems), student success can be observed, predicted and managed and fluid data is created as a result. They recommend that all institutions ‘should consider introducing an appropriate learning analytics system to improve student support and performance’.18 Student data footprints are exposed by such systems, providing a rich personal portrait of learners’ preferences, learning styles and level of engagement that universities and their libraries can act upon.

The use of learning analytics is not new. For example, pioneering library metrics work was conducted at the University of Huddersfield in 2011, working with Jisc and eight other universities in 2011.19 This work demonstrated a link between library use and retention rates, indicating that students were 7.53 times more likely to drop out of their programme if they did not use the library’s electronic resources. Although a direct cause and effect cannot be proven (since other factors are bound to be relevant), this work was the first of its kind to show how student activity might be used to track success.20

The latest work by Jisc examined the state of play of learning analytics in the USA and Australia21 where learning analytics systems and projects are more mature. The report highlights the difficulty of isolating the influence of the use of analytics – a known problem for academic libraries where there are so many factors (including the library) that contribute to improving student outcomes. One possible response to help understand this issue better has been the rise of user experience (UX) studies,22 which build a picture of the student from the ground up, using ethnographic approaches that involve the learners themselves. A much greater array of techniques are deployed to find new and meaningful connections between what students feel and do when engaging with library services. It remains to be seen whether initiatives of this kind are sufficiently developed and can be used effectively to explain the TEF split metrics at subject level.

These issues point clearly to the need for library professionals to think ahead to subject-level TEF assessments, maintaining a continuing focus on the subject of study and the collection of impact data at a subject level to evidence how their teaching and provision has impacted on student success and attainment. We need to ensure data about support and teaching activity is available in institutional information systems at a granular level and that it can be easily extracted and manipulated.

Learning resources

What type of learning resources and support are needed to align with the approaches to learning that the TEF highlights? A recent Higher Education Academy Conference23 coined the term ‘educating the TEF Generation’ to frame the expectations of students studying within HE in a post-TEF environment. Arguably, students will not see themselves in this way, but the focus on differentiated teaching and the ‘Connected Curriculum’24 that TEF promotes provides a useful construct to think about modern teaching and learning techniques underpinned by research-informed practice.

The notion of the ‘flipped classroom’, for example, has found resonance in some academic libraries, largely in the USA. The timing and use of a variety of learning resources (for example digital and tutorial materials) are ‘flipped’ so that they become available before teaching takes place, allowing the session to focus on the application of the resources, not the resources themselves.

Cohen’s25 case study at the City University of New York illustrates how flipped classroom techniques turned ‘one-shot’ library instruction – a typical challenge for academic libraries – into embedded academic practice. Whilst this may be seen as resource-intensive for librarians, the positive emphasis on learning outcomes, on academic staff and student engagement is in line with TEF expectations for impact. Cohen’s view is that the flipped classroom model ‘… could be a prototype for engaging discipline faculty in the process of teaching students to become twenty-first century learners who think critically, research deeply and solve problems collaboratively’.26

A strategic emphasis on learning outcomes and the new role of libraries post-TEF is also suggested by Chad and Anderson in a recent HELib briefing.27 They too confirm views expressed earlier in this paper that there is a challenge for library leaders in being able to articulate the value of the library in terms of the new HE agenda. This goes beyond the pedagogy present in the design of library spaces, which is well established, to assessing the nature of learning resources, and taking an institutional lead on how increasingly available open educational content such as open e-textbooks can be provided in a sustainable manner, including at an above-campus level. EDUCAUSE defines ‘above-campus’ as open materials and platforms on which much of HE worldwide can be constructed or enhanced.28

The reference to ‘fractured relationships’ by Bennett and Bennett29 included an argument that publishers and librarians should be working more closely together. There is a good argument that library leaders should be taking a strategic lead on the further development of open access resource implementation, seeking to influence new commercial publishing models to position their institutions well and sustainably, with Jisc, content providers and publishers.

Steve Rose’s editorial in a recent issue of SCONUL Focus puts it well: ‘The continued development and exploitation of resources, whether digital or physical, would appear to tick the boxes in supporting all of the elements of TEF’s assessment criteria.30 Or to put it another way: “Accessing and engaging with the right resources is one of the fundamental bridges between teaching and learning whilst facilitating this as the central role of a library service”’.31

Staff skills

Academic librarians’ skills have evolved to reflect changes in learning and teaching. A recent article in SCONUL Focus considers the skills required of information professionals in the future, as viewed by three UK ‘library schools’, and describes the types of attributes that will be required. They centre firmly on data and its analysis and on large-scale digital services and their associated advocacy skills.32 These can be summarized as:

• knowledge of digital pedagogies and being able to engage with students using a range of technologies, tools and platforms
• Big Data knowledge – both management of and methods
• repository management and digital asset management
• social media data analysis
• advocacy and demonstrating the impact of information.

A merging of library and educational technology is already taking place through the development of (for example) massive open online courses (MOOCs) and VLEs that embed library discovery features. The role of the conventional library management system (LMS) is becoming less significant as a stand-alone system. Next-generation learning services platforms, that integrate with a wide range of systems and learning content, are assuming more importance.33

Consequently, library-dedicated instructional roles will be required with a specific focus on library learning technologies and their application. These roles will focus on embedding open and subscribed-to content which impacts on student learning outcomes and enabling a holistic view of learning resources and support to be taken. Whilst it varies from institution to institution, such roles are currently and typically developing in central learning technology units and the library may not have dedicated time or expertise.

Conclusion

The TEF has been introduced at a time when higher education is being seen by many as a consumer product (namely, teaching), with a requirement for evidence for the consumer of the comparative quality of that product. Although the TEF is causing a necessary reaction from HE providers, it is not a significant driver for change in itself – it is a symptom of much wider disruption. Student engagement is at the heart of the TEF, reflecting the context of a new generation of students for whom partnership and engagement, often through digital channels, is the norm.

There are clear possibilities for the role of those responsible for library and learning support, be it in taking a strategic role in leading on the full transition to open resources by exerting influence on new commercial publishing models, or by aligning teaching of information skills towards the future split metrics so that learning gain and impact can be measured and direct correlations made in Annual Provider Review statements. Thinking ahead to subject-level assessments, a continuing focus on the subject of study and the collection of impact data at a subject level is also needed to enable evidence of how library teaching has impacted on student success. The continued focus on qualitative measures of excellence such as UX for specific groups of students could be used powerfully if an embedded approach to academic and student engagement is adopted specifically in a format to enable reporting via the TEF. It is up to the profession to debate issues such as these and work together to focus on what supports our host institutions and the engagement of the students we serve.

None of these activities has any great relevance unless library leaders engage in the politics and strategic discourse of their institutions. By the time the next TEF submissions are drafted, it is to be hoped that there is not an academic registrar in the country who can claim ignorance of the direct role his or her library service plays in ensuring teaching excellence.