Influencing the changing world of research evaluation

Elizabeth Gadd

Commentaries

Influencing the changing world of research evaluation

Authors

Elizabeth Gadd

Abstract

This article provides an overview of the current research evaluation landscape, the problems inherent within it and some of the ways a newly formed INORMS (International Network of Research Management Societies) working group on research evaluation is seeking to address them. These include developing a way of rating university ranking bodies and developing a set of briefing materials on responsible research evaluation for senior managers.

A correction article relating to this publication can be found here: http://doi.org/10.1629/uksg.491

Keywords:

Year: 2019

Volume 32

Page/Article: 6

DOI: 10.1629/uksg.456

Submitted on Dec 14, 2018

Accepted on Jan 15, 2019

Published on Feb 13, 2019

Peer Reviewed

CC BY 4.0

Publisher's Note

A correction article relating to this publication can be found here: http://doi.org/10.1629/uksg.491

Whatever scholarly forum you find yourself in at the moment, the critical issues usually boil down to problems with research evaluation. Want to encourage greater take-up of open access? Change the way research is evaluated to prize open scholarship. Want to improve the reproducibility and rigour of research? Change the way research is evaluated to value high quality data and negative results as much as sensational findings more likely to be welcomed by ‘top’ journals. Want to improve equality and diversity in academia? Change recruitment and promotion criteria to include a wider range of qualities, skills and outcomes – not primarily cash and citations. These are all sensible conclusions. The way research is evaluated has a significant effect on the whole of the scholarly endeavour. This is called Campbell’s Law: often summarized as ‘the way you measure someone affects the way they behave’. However, identifying the problem is far more straightforward than solving it – especially when it is so deeply embedded in existing, global, scholarly cultures. This short commentary seeks to provide an overview of the current research evaluation landscape, the problems inherent within it and some of the ways a newly formed INORMS (International Network of Research Management Societies) working group on research evaluation is seeking to address them.

More evaluation

The first thing to say about the changing world of research evaluation is that there is just more and more of it. Not a week goes by without another global, national or subject ranking being published and these are mainly based on research indicators as they are the only consistent, global data sources we have. There is also an increase in national research evaluation exercises – both in the volume of them (most EU countries now have some form of national exercise) and the sophistication (read cost). An increasing number are also linked with funding.

An increase in external evaluation leads predictably to an increase in internal evaluation. We see universities’ research ambitions stated and evaluated from whole university key performance indicators through to departmental targets and, of course, most controversial of all, individual-level evaluations through annual appraisal, recruitment and promotion. Although academics complain most bitterly about the latter, they are often equally complicit in the evaluation culture by engaging in what I call ‘measure for pleasure’ activities: watching their h-index creep up and comparing themselves to peers – or, should I say, competitors?

More competition

An evaluation culture necessarily leads to an increase in competition. As soon as you can put a number on something, you can rank it. And, where there is a ranking, you can bet a university, department, or academic will want to compete to get to the top of it. Indeed, they may feel as though they have little choice – their finances and reputation depend on it. Of course, once you have made it to the top, a whole new level of anxiety kicks in as you seek to stay there. It is one thing to be a university on the rise, quite another to be a university in decline.

National research evaluation exercises also keep raising the bar as to what level of brilliance is needed to win prizes. After the 2014 UK Research Excellence Framework (REF), four-star (world leading) research was funded at a ratio of 4:1, with three-star (internationally excellent) research perhaps inadvertently sending a message that ‘excellence is not good enough’. For REF 2021, due to the change in methodology (submit all papers, but with an average of 2.5 papers per person), there is speculation that four-star research will get a much higher proportion of the available funding and that perhaps three-star research will not get funded at all.

In other forms of competition, we repeatedly hear in the research press that there are more and more researchers chasing fewer and fewer funding opportunities, and that there are too many PhDs for the academic jobs available. Indeed, in a blog post earlier this year, Imperial College professor Chris Jackson stated that were he attempting to move into academia from industry now, ‘there is no way I’d get a job’.

More at stake

Not surprisingly, with more competition, there is more at stake. A university’s ranking position has always (inevitably) been linked to its reputation and indirectly to funding through students who may or may not choose to attend that university based on their ranking. However, rankings are now directly linked to funding as more and more global funders are using university ranking position as a barrier to entry. Financial institutions also seem to be using a university’s ranking position as shorthand for credit-worthiness.

There is also more at stake for academics now with, at best, recruitment and promotion based heavily on measurable publication performance (how many, what journals, who co-authored, how cited) and, at worst, direct cash prizes associated with publications in ‘top journals’. According to a People’s Daily news story last year, some 90% of Chinese universities have policies of rewarding publications, and the practice is far from unique to China. (However, some Chinese funders are now seeking to back Plan S, which may change this situation).

Of course, this sort of activity puts pressure on journals to become ‘top journals’ and we are now seeing artificial intelligence services such as Meta Bibliometric Intelligence offering publishers the opportunity to predict ‘high impact’ publications based on historical publication and citation data in order ‘to control the future impact factor of the journal’.

More pressure on the academy

Inevitably, all this evaluation-based competition has led to more pressure on the academy. A frequently cited case is that of Professor Stefan Grimm, formerly of Imperial College London, who committed suicide allegedly after he was told he was not meeting the expected income-generation target of £200,000 p.a. Over the summer of 2018 another academic, at Cardiff University, took his own life due to pressure of work. Higher education is a sector with a known, significant mental health problem, and when we consider appropriate forms of research evaluation, this can never be far from our minds.

More calls for ‘responsible metrics’

Having outlined all that, it is hardly surprising that there have been increasing calls for so-called ‘responsible metrics’. In 2012 a group established by the American Society of Cell Biology wrote the Declaration on Research Assessment (DORA). This is principally a backlash against the use of Journal Impact Factors (JIF) to evaluate individual scholars and individual articles. Both individuals and institutions are invited to sign the declaration. In April 2015 a group of scholars from the CWTS research group at Leiden University published The Leiden Manifesto. This is formed of ten principles for the responsible use of bibliometrics in research evaluation, so a much broader focus than DORA. Perhaps of most significance to the UK was the publication of The Metric Tide in July 2015, which was an independent review of the use of metrics in UK research assessment. This established five principles for the responsible use of all metrics (not just bibliometrics) and called on universities to establish their own principles for the use of research metrics.

The Lis-Bibliometrics Forum has run annual ‘state-of-the-art’ surveys around responsible metrics since 2015 to map the sector’s engagement with these calls. Figures 1 and 2 show the uptake of DORA and the development of responsible metrics principles by higher education institutions (HEIs) over time. They show fairly equal levels of engagement with both. Thirty have signed and six are likely to sign DORA (36 in total), whilst 11 already had, and 24 were currently developing, their own responsible metrics principles (35). It is interesting to note that more had actually signed DORA than had already developed their own principles. This may be because it is easier to sign an existing protocol than develop your own. However, the tide appears to be turning, with fewer actively considering DORA in 2018 (19) than actively considering their own principles (33). Similarly, a higher proportion had rejected DORA by 2018 (11) than had rejected the idea of developing their own principles (3).

Figure 1

Engagement by HEIs with DORA over time

Figure 2

HEI development of responsible metrics principles over time

Under Plan S it may be that funders seek to push this agenda even further by mandating engagement with responsible metrics. Indeed, the Wellcome Trust has recently announced that as part of its new open access policy guided by Plan S, from 1 January 2020 it will only fund organizations that have made a public commitment to responsible research evaluation. It is unclear as yet how other funders will interpret the cOAlition S requirement for members ‘to sign DORA and implement those requirements in their policies’.

Who is responsible for responsible metrics?

It should be noted that most of the pressure to deliver on responsible metrics seems to be focused on universities. However, the truth is that there are many other players in this space. Universities measure because they are measured.

In the UK, one of the biggest external evaluators is the REF. The REF prides itself on being a peer-review exercise, and not a controversial metrics-based exercise such as that in Italy. Citation metrics are supplied to some panels, but essentially it is based on peer review. However, whilst peer review is held up as the ‘gold standard’ of research evaluation, evidence suggests that peer review can be just as problematic as metrics in terms of bias and poor practice. After the 2014 REF, Professor Alan Dix, who sat on the Computing panel, ran a citation-based correlation with the peer-review ratings in his panel and found significant bias against female authors and applied papers. He concluded that ‘metrics are rubbish, but people are worse’. Perhaps the focus on responsible metrics should expand to include responsible peer review?

Of course, when it comes to reporting the findings of this big qualitative peer-review exercise, the outcomes are 100% quantitative: sheets and sheets of numbers – 120,000 data points, to be precise. Thus, if anyone wants to assess the quality of civil engineering research at Loughborough, for example, they can only find out in a quantitative way. Is that responsible?

Secondly, we have research funders. A 2017 study sought to understand what the level of understanding was of bibliometrics within National Institute for Health Research panels. Responses included ‘I am not at all an expert in bibliometrics; I just have a general idea of what it is’. In fact, most panel members described their understanding of bibliometrics in similar terms, as ‘rudimentary’, ‘cursory’ and ‘limited’, even though they were using bibliometrics to inform funding decisions. Interestingly, even if funders sign up to Plan S and seek to evaluate outputs based only on their intrinsic merit rather than by journal brand, unless they eradicate the use of bibliometrics completely we may still see situations like this. Is that responsible?

Tackling the world university rankings

Then we have ranking organizations. There is no shortage of evidence for the lack of methodological credibility for the rankings. Indeed, any research study which claimed to be able to identify the ‘top’ or ‘best’ universities would be rejected at peer review for a badly formed research question. (Top at what?) The truth is that there is no governance around the activity of rankings and, considering the power they have (as outlined above), is that responsible? Indeed, this is one of the greatest challenges facing research evaluation simply because of their reach. National research evaluation exercises can be tackled at national level. Funders are also, to an extent, regional. However, the global university rankings are, well, global.

It is for this reason that the INORMS Research Evaluation Working Group has decided to try and tackle this problem through one of its work packages. Attempts have been made before to expose the flaws in the world rankings, but these have tended to be done by national groups, not by an international collective. The Group plans to develop a scoring mechanism for ranking organizations – perhaps treating them to a ranking of their own. It is hoped that by joining forces to call for more critical understanding of the rankings, we may see fewer universities outsourcing their values to them.

More data and indicators

Given the growth in evaluation activity, it should not come as a surprise that the metrics industry has grown alongside it. In addition to the standard range of bibliographic databases (Web of Science and Scopus), all of which now include citation metrics, we have altmetric services (Altmetric.com and Plum Analytics) and dedicated citation benchmarking tools (Incites and SciVal). In 2018 alone we also saw three new tools: Dimensions, 1finder and Wizdom, all of which contain citation indicators and, increasingly, other forms of data such as grants, patents and clinical trials.

In addition to the increase in data sources, we are seeing bibliometric scholars develop more and more indicators. In 2014 Lorna Wildegaard and colleagues analysed 108 individual author metrics. The vast majority of these had been developed post-2000. We see a similar picture with journal metrics. The 2016 Handbook of bibliometric indicators lists in excess of 50 journal citation metrics. However, whilst we are seeing huge growth in the number of metrics being developed, we are seeing only a small increase in the number of metrics available to us in our citation tools. This might be a good thing in terms of overload, but all of these indicators are the result of many research projects, and who knows how much funding, to try and create something better than we have. Some of them may be better than the indicators we have, but we would never know about it because they are not adopted by our citation tools. There is a question then as to how metrics get into our products. Indeed, a recent study into bibliometric scholarship concluded that ‘commercial providers have gained a powerful role in defining de-facto standards of research excellence without being challenged by expert authority’. There is also a question as to how these metrics get designed, and whether academics are ever involved in the development of the sticks used to measure them.

More cost

Of course, with more products comes more cost. It is difficult to get any figures from suppliers on the take-up of their products. However, I know from keeping a watching brief on this area that we are seeing a growth in subscriptions to citation benchmarking tools. Data from a survey of 114 ACRL libraries in the US found that 80% had a ‘faculty activity reporting (FAR)’ tool. We are also seeing growth in the number of research impact or analytics-type posts in universities (mostly in libraries) and more recently (as universities are struggling to recruit) I have observed an increase in the seniority of these posts. Again, the ACRL library survey found that 70% had an individual or group assigned to research impact.

More skills

With all these tools and people, we are seeing more and more sophisticated analyses. A Lis-Bibliometrics survey which asked the community what messages they would like to send to suppliers reported many calls for open data, increased download limits and standard identifiers. This was to enable them to combine data from their citation tools with other data in visualization tools such as Tableau and PowerBI and network analysis tools such as VosViewer. They are also keen to run their own statistical analyses and calculate confidence intervals (perhaps because suppliers don’t do it for them). All of these analyses require more skills. However, a survey in 2017 found that only 29% of librarians who were supporting their institutions with bibliometrics had actually covered bibliometrics in their LIS diploma or degree. One of the outcomes of that project was a set of bibliometric competencies by which practitioners can assess their own skill levels and seek to improve on any gaps. Perhaps not surprisingly, at a ‘Statistics for Responsible Bibliometrics’ course run by the Lis-Bibliometrics group, the largest tranche of delegates (50%) described their skills as ‘entry level’.

Skilling the decision makers

Of course, it is not just the analysts that need upskilling. As we saw from the NIHR analysis of grant reviewers, there is a distinct lack of skill amongst those relying on bibliometric data for decision-making and policy setting. Arguably, it is this group that has the most power to do harm through a lack of understanding. It is for this reason that the INORMS Research Evaluation Working Group has decided to make this the focus of its second work package and to develop a set of briefing materials for senior managers. The plan is to develop a standard, globally relevant, slide set that gives our policymakers a better understanding of the strengths and weaknesses of various research evaluation methods. Ultimately, responsible metrics is not just about our duty of care but about good decision-making. Decisions made using poor indicators, incomplete data, or inappropriate methods are simply not going to lead to good outcomes. All senior managers should care about that, and it is this group that the responsible metrics lobby most urgently needs to reach. As I have said before, although we use the expression ‘responsible metrics’, metrics cannot be responsible, only people can be responsible. Ensuring we skill up responsible people is our best bet for engendering a responsible research evaluation culture in the future.

You can keep up to date with the work of the INORMS Research Evaluation Working Group by joining their mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=INORMS-RES-EVAL

Abbreviations and Acronyms

A list of the abbreviations and acronyms used in this and other Insights articles can be accessed here – click on the URL below and then select the ‘full list of industry A&As’ link: http://www.uksg.org/publications#aa

Competing interests

The author has declared no competing interests.

References

Donald T. Campbell, “Assessing the impact of planned social change”, Evaluation and Program Planning, 2(1), (1979), 67–90; DOI: https://doi.org/10.1016/0149-7189(79)90048-X (accessed 16 January 2019).
INORMS Research Evaluation Working Group (2018): https://inorms.net/activities/research-evaluation-working-group/ (accessed 16 January 2019).
IREG Observatory, IREG Inventory of International University Rankings 2014–17. Brussels: Perspektywy Education Foundation, (2018): http://ireg-observatory.org/en/pdfy/IREG-Inventory-of-International%20University%20Rankings.pdf (accessed 18 January 2019).
Andrejs Rauhvargers, Global University Rankings and Their Impact – Report II (2013): https://www.eua.eu/downloads/publications/global%20university%20rankings%20and%20their%20impact%20-%20report%20ii.pdf (accessed 19 January 2019).
Erik Arnold et al., Review of the Research Excellence Framework (October 2018): https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/768162/research-excellence-framework-review-evidence-report.pdf (accessed 18 January 2019).
Benedetto Lepori et al., “Comparing the evolution of national research policies: what patterns of change?”, Science and Public Policy, 34 (July 2007), 372–388, https://doi.org/10.3152/030234207X234578 (accessed 16 January 2019).
Ulf Sandström and Peter Van den Besselaar, “Funding, evaluation, and the performance of national research systems”, Journal of Informetrics, 12(1), (2018), 365–384, https://doi.org/10.1016/j.joi.2018.01.007 (accessed 16 January 2019); ANVUR (2018): http://www.anvur.it/en/activities/vqr/ (accessed 16 January 2019).
Rauhvargers, Global University Rankings ref. 4.
Jenny Rohn, Stephen Curry and Andrew Steele, “UK research funding slumps below 0.5% GDP”, The Guardian, March 13, 2015: https://www.theguardian.com/science/occams-corner/2015/mar/13/science-vital-uk-spending-research-gdp (accessed 16 January 2019).
[Anonymous academic], “I struggle when hiring academics – because the candidates are too good”, The Guardian, June 1, 2018: https://www.theguardian.com/higher-education-network/2018/jun/01/academics-candidates-outstanding-application-recruitment (accessed 16 January 2019).
Christopher Jackson, A researcher’s perspective on the recent event “The turning tide: A new culture of responsible metrics for research”, (HEFCE blog), 19 March 2018: https://webarchive.nationalarchives.gov.uk/20180416090255/http://blog.hefce.ac.uk/2018/03/19/a-researchers-perspective-on-responsible-metrics/ (accessed 21 January 2019).
See for example: Russia Global Education Program: https://educationglobal.ru/en/info/ (accessed 21 January 2019).
See for example: Karin Erlander and Sood Ratul, “University of Nottingham Rating Affirmed At ‘A+’ On Solid Performance; Outlook Stable”, 2017: https://www.nottingham.ac.uk/fabs/finance/documents/spcr-2017.pdf (accessed 24 January 2019).
[Anon.], Don’t pay prizes for published science, Nature, 547 (7662), (2017): https://www.nature.com/news/don-t-pay-prizes-for-published-science-1.22275 (accessed 18 January 2019).
Quirin Schiermeier, “China backs bold plan to tear down journal paywalls”, Nature New, December 5, 2018, https://doi.org/10.1038/d41586-018-07659-5 (accessed 18 January 2019).
Meta.com 2016. Meta for publisher: http://cdn.meta.com/assets/downloads/Pub_Bibliometric.pdf (accessed 18 January 2019).
Chris Parr, Imperial College professor Stefan Grimm “was given grant income target”, Times Higher Education, December 3, 2014: https://www.timeshighereducation.com/news/imperial-college-professor-stefan-grimm-was-given-grant-income-target/2017369.article (accessed 18 January 2019).
Rachael Pells, Lecturer’s suicide a “wake-up call” on overworking in academia, Times Higher Education, June 4, 2018: https://www.timeshighereducation.com/news/lecturers-suicide-wake-call-overworking-academia (accessed 18 January 2019).
Holly Else, Academics “face higher mental health risk” than other professions, Times Higher Education, August 22, 2017: https://www.timeshighereducation.com/news/academics-face-higher-mental-health-risk-than-other-professions (accessed 18 January 2019).
Declaration in Research Assessment (DORA). [n.d.]: https://sfdora.org/ (accessed 18 January 2019).
Diana Hicks et al., “The Leiden Manifesto for Research Metrics”, Nature 520: 429–31, (2015), https://doi.org/10.1038/520429a (accessed 18 January 2019).
Wilsdon, James, et al., The Metric Tide, (2015): https://responsiblemetrics.org/the-metric-tide/ (accessed 18 January 2019).
Elizabeth Gadd, “Who will welcome Wellcome? Results of the 2018 responsible metrics state-of-the-art survey”, The Bibliomagician (blog), November 8, 2018: https://thebibliomagician.wordpress.com/2018/11/08/who-will-welcome-wellcome-results-of-the-2018-responsible-metrics-state-of-the-art-survey/ (accessed 18 January 2019).
The Wellcome Trust. 2018. Open Access Policy 2020: https://wellcome.ac.uk/sites/default/files/wellcome-open-access-policy-2020.pdf (accessed 15 January 2019).
cOAlition S, Plan S Implementation and feedback, 2018: https://www.coalition-s.org/feedback/ (accessed 18 January 2019).
Jonathan P. Tennant et al., “A Multi-Disciplinary Perspective on Emergent and Future Innovations in Peer Review”, F1000Research, (2017): https://doi.org/10.12688/f1000research.12037.3 (accessed 18 January 2019); Tony Ross-Hellauer, “What Is Open Peer Review? A Systematic Review [Version 2; Referees: 4 Approved]”. F1000Research (May 2017): https://doi.org/10.12688/f1000research.11369.1 (accessed 18 January 2019); Claire Le Goues et al., “Effectiveness of Anonymization in Double-Blind Review.” Communications of the ACM 61 (6), June 2017: 30–33, https://doi.org/10.1145/3208157 (accessed 18 January 2019).
Alan Dix. Metrics are rubbish but… Slides presented at ARMA Conference 2017, June 8, 2017: https://www.slideshare.net/alanjohndix/metrics-are-rubbish-but (accessed 18 January 2019).
Salil Gunashekar, Steven Wooding and Susan Guthrie, “How do NIHR peer review panels use bibliometric information to support their decisions?” Scientometrics 112 (3), (2017): 1813–35, https://doi.org/10.1007/s11192-017-2417-8 (accessed 18 January 2019).
See for example: Henk F. Moed, “A Critical Comparative Analysis of Five World University Rankings,” Scientometrics 110 (2), (2017): 967–90, https://doi.org/10.1007/S11192-016-2212-Y (accessed 18 January 2019); Jill Johnes, “University Rankings: What Do They Really Show?” Scientometrics Online First, 2017, https://doi.org/10.1007/s11192-018-2666-1; CWTS, CWTS Leiden Rankings, Ten Principles for the responsible use of rankings, 2017: http://www.leidenranking.com/information/responsibleuse (accessed 18 January 2019).
Laura Bridgestock, “How do Universities Use Rankings?” Top Universities, November 7, 2014: https://www.topuniversities.com/student-info/university-news/how-do-universities-use-rankings (accessed 18 January 2019).
Dimensions: https://dimensions.ai (accessed 18 January 2019).
1findr: https://1findr.1science.com/home/ (accessed 18 January 2019).
Wizdom: https://wizdom.ai/ (accessed 15 January 2019).
Lorna Wildgaard, Jesper W Schneider and Birger Larsen, “A Review of the Characteristics of 108 Author-Level Bibliometric Indicators”, Scientometrics 101 (1), (2014), 125–158, https://doi.org/10.1007/s11192-014-1423-3 (accessed 21 January 2019).
Roberto Todeschini and Alberto Baccini, Handbook of Bibliometric Indicators: Quantitative Tools for Studying and Evaluating Research, John Wiley & Sons, 2016, https://doi.org/10.1002/9783527681969 (accessed 18 January 2019).
Arlette Jappe, David Pithan and Thomas Heinze, “Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972–2016,” Plos One 13 (6), (2018): e0199031, https://doi.org/10.1371/journal.pone.0199031 (accessed 18 January 2019).
Caitlin J. Bakker and Jenny McBurney, “Big Splashes & Ripple Effects: Research Impact Services Across University Libraries,” 2017: https://conservancy.umn.edu/handle/11299/188632 (accessed 18 January 2019).
Sabrina Petersohn, “Professional competencies and jurisdictional claims in evaluative bibliometrics: The educational mandate of academic librarians,” Education for Information, 32, (2016): 165–193, https://doi.org/10.3233/EFI-150972 (accessed 18 January 2019).
cOAlition S, Plan S, ref.25.
Elizabeth Gadd and Ian Rowlands, “How can bibliometric and altmetric suppliers improve? Messages from the end-user community,” Insights, 31, 38, (2018): https://insights.uksg.org/articles/10.1629/uksg.437/ (accessed 18 January 2019).
Andrew Cox, Elizabeth Gadd, Sabrina Petersohn and Laura Sbaffi, “Competencies for Bibliometrics,” Journal of Librarianship and Information Science, Online First, 2017, https://doi.org/10.1177/0961000617728111 (accessed 18 January 2019).
Elizabeth Gadd, The Changing world of research evaluation. Slides presented at the Jisc/CNI Leaders Conference, Oxford, July 2, 2018: https://figshare.com/articles/The_changing_world_of_research_evaluation/6754184 (accessed 18 January 2019).
Pells, Lecturer’s suicide a “wake-up call”, ref. 18.

[B1] Donald T. Campbell, “Assessing the impact of planned social change”, Evaluation and Program Planning, 2(1), (1979), 67–90; DOI: https://doi.org/10.1016/0149-7189(79)90048-X (accessed 16 January 2019).

[B2] INORMS Research Evaluation Working Group (2018): https://inorms.net/activities/research-evaluation-working-group/ (accessed 16 January 2019).

[B3] IREG Observatory, IREG Inventory of International University Rankings 2014–17. Brussels: Perspektywy Education Foundation, (2018): http://ireg-observatory.org/en/pdfy/IREG-Inventory-of-International%20University%20Rankings.pdf (accessed 18 January 2019).

[B4] Andrejs Rauhvargers, Global University Rankings and Their Impact – Report II (2013): https://www.eua.eu/downloads/publications/global%20university%20rankings%20and%20their%20impact%20-%20report%20ii.pdf (accessed 19 January 2019).

[B5] Erik Arnold et al., Review of the Research Excellence Framework (October 2018): https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/768162/research-excellence-framework-review-evidence-report.pdf (accessed 18 January 2019).

[B6] Benedetto Lepori et al., “Comparing the evolution of national research policies: what patterns of change?”, Science and Public Policy, 34 (July 2007), 372–388, https://doi.org/10.3152/030234207X234578 (accessed 16 January 2019).

[B7] Ulf Sandström and Peter Van den Besselaar, “Funding, evaluation, and the performance of national research systems”, Journal of Informetrics, 12(1), (2018), 365–384, https://doi.org/10.1016/j.joi.2018.01.007 (accessed 16 January 2019); ANVUR (2018): http://www.anvur.it/en/activities/vqr/ (accessed 16 January 2019).

[B8] Rauhvargers, Global University Rankings ref. 4.

[B9] Jenny Rohn, Stephen Curry and Andrew Steele, “UK research funding slumps below 0.5% GDP”, The Guardian, March 13, 2015: https://www.theguardian.com/science/occams-corner/2015/mar/13/science-vital-uk-spending-research-gdp (accessed 16 January 2019).

[B10] [Anonymous academic], “I struggle when hiring academics – because the candidates are too good”, The Guardian, June 1, 2018: https://www.theguardian.com/higher-education-network/2018/jun/01/academics-candidates-outstanding-application-recruitment (accessed 16 January 2019).

[B11] Christopher Jackson, A researcher’s perspective on the recent event “The turning tide: A new culture of responsible metrics for research”, (HEFCE blog), 19 March 2018: https://webarchive.nationalarchives.gov.uk/20180416090255/http://blog.hefce.ac.uk/2018/03/19/a-researchers-perspective-on-responsible-metrics/ (accessed 21 January 2019).

[B12] See for example: Russia Global Education Program: https://educationglobal.ru/en/info/ (accessed 21 January 2019).

[B13] See for example: Karin Erlander and Sood Ratul, “University of Nottingham Rating Affirmed At ‘A+’ On Solid Performance; Outlook Stable”, 2017: https://www.nottingham.ac.uk/fabs/finance/documents/spcr-2017.pdf (accessed 24 January 2019).

[B14] [Anon.], Don’t pay prizes for published science, Nature, 547 (7662), (2017): https://www.nature.com/news/don-t-pay-prizes-for-published-science-1.22275 (accessed 18 January 2019).

[B15] Quirin Schiermeier, “China backs bold plan to tear down journal paywalls”, Nature New, December 5, 2018, https://doi.org/10.1038/d41586-018-07659-5 (accessed 18 January 2019).

[B16] Meta.com 2016. Meta for publisher: http://cdn.meta.com/assets/downloads/Pub_Bibliometric.pdf (accessed 18 January 2019).

[B17] Chris Parr, Imperial College professor Stefan Grimm “was given grant income target”, Times Higher Education, December 3, 2014: https://www.timeshighereducation.com/news/imperial-college-professor-stefan-grimm-was-given-grant-income-target/2017369.article (accessed 18 January 2019).

[B18] Rachael Pells, Lecturer’s suicide a “wake-up call” on overworking in academia, Times Higher Education, June 4, 2018: https://www.timeshighereducation.com/news/lecturers-suicide-wake-call-overworking-academia (accessed 18 January 2019).

[B19] Holly Else, Academics “face higher mental health risk” than other professions, Times Higher Education, August 22, 2017: https://www.timeshighereducation.com/news/academics-face-higher-mental-health-risk-than-other-professions (accessed 18 January 2019).

[B20] Declaration in Research Assessment (DORA). [n.d.]: https://sfdora.org/ (accessed 18 January 2019).

[B21] Diana Hicks et al., “The Leiden Manifesto for Research Metrics”, Nature 520: 429–31, (2015), https://doi.org/10.1038/520429a (accessed 18 January 2019).

[B22] Wilsdon, James, et al., The Metric Tide, (2015): https://responsiblemetrics.org/the-metric-tide/ (accessed 18 January 2019).

[B23] Elizabeth Gadd, “Who will welcome Wellcome? Results of the 2018 responsible metrics state-of-the-art survey”, The Bibliomagician (blog), November 8, 2018: https://thebibliomagician.wordpress.com/2018/11/08/who-will-welcome-wellcome-results-of-the-2018-responsible-metrics-state-of-the-art-survey/ (accessed 18 January 2019).

[B24] The Wellcome Trust. 2018. Open Access Policy 2020: https://wellcome.ac.uk/sites/default/files/wellcome-open-access-policy-2020.pdf (accessed 15 January 2019).

[B25] cOAlition S, Plan S Implementation and feedback, 2018: https://www.coalition-s.org/feedback/ (accessed 18 January 2019).

[B26] Jonathan P. Tennant et al., “A Multi-Disciplinary Perspective on Emergent and Future Innovations in Peer Review”, F1000Research, (2017): https://doi.org/10.12688/f1000research.12037.3 (accessed 18 January 2019); Tony Ross-Hellauer, “What Is Open Peer Review? A Systematic Review [Version 2; Referees: 4 Approved]”. F1000Research (May 2017): https://doi.org/10.12688/f1000research.11369.1 (accessed 18 January 2019); Claire Le Goues et al., “Effectiveness of Anonymization in Double-Blind Review.” Communications of the ACM 61 (6), June 2017: 30–33, https://doi.org/10.1145/3208157 (accessed 18 January 2019).

[B27] Alan Dix. Metrics are rubbish but… Slides presented at ARMA Conference 2017, June 8, 2017: https://www.slideshare.net/alanjohndix/metrics-are-rubbish-but (accessed 18 January 2019).

[B28] Salil Gunashekar, Steven Wooding and Susan Guthrie, “How do NIHR peer review panels use bibliometric information to support their decisions?” Scientometrics 112 (3), (2017): 1813–35, https://doi.org/10.1007/s11192-017-2417-8 (accessed 18 January 2019).

[B29] See for example: Henk F. Moed, “A Critical Comparative Analysis of Five World University Rankings,” Scientometrics 110 (2), (2017): 967–90, https://doi.org/10.1007/S11192-016-2212-Y (accessed 18 January 2019); Jill Johnes, “University Rankings: What Do They Really Show?” Scientometrics Online First, 2017, https://doi.org/10.1007/s11192-018-2666-1; CWTS, CWTS Leiden Rankings, Ten Principles for the responsible use of rankings, 2017: http://www.leidenranking.com/information/responsibleuse (accessed 18 January 2019).

[B30] Laura Bridgestock, “How do Universities Use Rankings?” Top Universities, November 7, 2014: https://www.topuniversities.com/student-info/university-news/how-do-universities-use-rankings (accessed 18 January 2019).

[B31] Dimensions: https://dimensions.ai (accessed 18 January 2019).

[B32] 1findr: https://1findr.1science.com/home/ (accessed 18 January 2019).

[B33] Wizdom: https://wizdom.ai/ (accessed 15 January 2019).

[B34] Lorna Wildgaard, Jesper W Schneider and Birger Larsen, “A Review of the Characteristics of 108 Author-Level Bibliometric Indicators”, Scientometrics 101 (1), (2014), 125–158, https://doi.org/10.1007/s11192-014-1423-3 (accessed 21 January 2019).

[B35] Roberto Todeschini and Alberto Baccini, Handbook of Bibliometric Indicators: Quantitative Tools for Studying and Evaluating Research, John Wiley & Sons, 2016, https://doi.org/10.1002/9783527681969 (accessed 18 January 2019).

[B36] Arlette Jappe, David Pithan and Thomas Heinze, “Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972–2016,” Plos One 13 (6), (2018): e0199031, https://doi.org/10.1371/journal.pone.0199031 (accessed 18 January 2019).

[B37] Caitlin J. Bakker and Jenny McBurney, “Big Splashes & Ripple Effects: Research Impact Services Across University Libraries,” 2017: https://conservancy.umn.edu/handle/11299/188632 (accessed 18 January 2019).

[B38] Sabrina Petersohn, “Professional competencies and jurisdictional claims in evaluative bibliometrics: The educational mandate of academic librarians,” Education for Information, 32, (2016): 165–193, https://doi.org/10.3233/EFI-150972 (accessed 18 January 2019).

[B39] cOAlition S, Plan S, ref.25.

[B40] Elizabeth Gadd and Ian Rowlands, “How can bibliometric and altmetric suppliers improve? Messages from the end-user community,” Insights, 31, 38, (2018): https://insights.uksg.org/articles/10.1629/uksg.437/ (accessed 18 January 2019).

[B41] Andrew Cox, Elizabeth Gadd, Sabrina Petersohn and Laura Sbaffi, “Competencies for Bibliometrics,” Journal of Librarianship and Information Science, Online First, 2017, https://doi.org/10.1177/0961000617728111 (accessed 18 January 2019).

[B42] Elizabeth Gadd, The Changing world of research evaluation. Slides presented at the Jisc/CNI Leaders Conference, Oxford, July 2, 2018: https://figshare.com/articles/The_changing_world_of_research_evaluation/6754184 (accessed 18 January 2019).

[B43] Pells, Lecturer’s suicide a “wake-up call”, ref. 18.