Research evaluation is increasingly a central topic of debates and reports in and around academia.1 The process of evaluating researchers mainly on the basis of publication and citation metrics has come under fierce scrutiny because it is believed to be one of the main drivers of the documented adverse effects of the ‘publish or perish’ pressure on academic careers. These adverse effects include:
Reform of research evaluation, to a system more focused on diverse aspects of scientists’ work besides publishing alone, would have many benefits. It may help to relieve pressures on researchers’ mental health and encourage better scientific practices that put the emphasis on collaboration, like data sharing and other open science practices.7
Where do early career researchers (ECRs) stand on this issue? Many of us are strong supporters of open science,8 but there is always a gap between consciousness and action. Whether we like it or not, with a few exceptions, we all know that metrics will not disappear overnight, but will still be on the table at the time we will be (and are) evaluated. The game may be rigged, but we all are forced to play.
Here, we share reflections from the perspective of a PhD candidate, a young academic and PhD supervisor and an ECR who recently switched from academic research to science advocacy.
A striking effect of the systems of evaluation based on publication and citation metrics is on the expectations of evaluation panels towards early career researchers when they are applying for postdoctoral positions or grants. We are still evaluated on the basis of our publications – not so much on the intrinsic quality of our work as on the quartile of the Journal Citation Report (JCR)9 to which the journal where we publish belongs. Choosing to send articles to innovative field-related open access journals or platforms with interesting features such as open peer review and reasonable article processing charges (APCs) is something I would like to do more to actively support a change of culture. However, my strong perception is that there would be a substantial cost to these choices in future contexts where I am being evaluated; for example, when applying for postdoctoral funding. The system forces us all to play the game to a certain extent – it favours the conventional choice of publishing in the most prestigious and cited journals in our subject area, which tend either to have a paywall or to offer open access publication only on payment of relatively high APCs, or via expensive transformative agreements. In a way, early career researchers are ‘locked-in’ when it comes to selecting venues for publication, as we are evaluated not so much on the content of our work (and the effectiveness of the associated peer-review process) as where we publish it. Embracing innovative open access journals/platforms and the many advantages they offer in terms of quality control and transparency of the scientific debate (pre-registration, open peer review, post-publication reviews), comes at a cost: that of facing a possible backlash from peer evaluation when applying for funding and new positions.
I have been supportive of open access since early in my PhD. Although I have tried to publish openly whenever possible, I have consciously – and with the advice of my mentors – chosen journals that are perceived to be most prestigious, in order to further my career. This strategy has been successful: thanks to ample third-party funding that can be used for APCs, the majority of my published output is openly available, but at the same time, I have been able to win prestigious grants and find a tenure-track position at a fairly young age. However, although in this sense I have ‘won the academic lottery’ and my position will soon be secure, a new challenge has started to emerge – I am no longer only responsible just for my own career. I have PhD trainees and postdoctoral researchers under my supervision, whose career prospects I feel responsible to safeguard. Should I submit the key findings of a PhD project to a less prestigious but open access journal, or should I go for the top-tier ‘closed’ options to give them the best chances of succeeding, as I have succeeded? For me, the choice is unfortunate but clear: I need to make sure that the people I supervise have the best chances for career success. This dilemma shows starkly the systemic nature of the problem and also highlights the vulnerability of early career researchers – no researcher should have to martyr themselves to advance openness, given how valuable it is for science. Instead, what is urgently needed is a systematic overhaul of the entire reward and evaluation system to value research on its own merits instead of where it is published.
Anyone who looks up my publication record may think ‘This is what is wrong with our current system’. The broader research community I was brought up in taught me early that publishing a lot and in high-impact factor journals was the only way to get ahead, and I was very proud that I managed to get my name into Science and Nature-branded journals and similar prestigious venues. Although I was part of research projects adept at getting articles accepted into high-impact journals, these same teams were also where I had the most vibrant discussions around how research culture needs to change. There was a broad awareness that the current system is not working well. However, many of us (including myself) felt that the best way to change it was to play by the rules until you became ‘established enough’ and then leverage that to help change the system from within. But the core question was, when is ‘enough’? An often unspoken question was also, ‘If the current system actually helps me succeed, will my interest to change it wane over time?’ In parallel, we had lively discussions in our laboratory around how to improve day-to-day research processes, ranging from introducing GoPro cameras in our laboratory10 and (perhaps quixotic) quests to help improve bio-nano research,11 create bridges with cancer nanomedicine12 and the development of a ‘minimum information standard’.13 This passion for improving the research process is what grew into a desire to improve the broader research culture, which is now a core part of my advocacy work for CESAER, including the modernization of research careers.14 In my current advocacy work with non-academics, no one really cares about my Science or Nature articles, so this new perspective has reinforced for me that the obsession (which I also perpetuated) is largely a focus of the research community. But this ‘containment’ also gives more power to researchers to actually change the system.
Change is complex and will require the involvement of all levels and actors in academia. It is clear that those who evaluate researchers – most of whom climbed the publish or perish ladder successfully – may find it hard to reconsider the system in which they themselves excelled. It is therefore crucial to raise awareness of the fact that, while the system might indeed reward excellence (although this has been questioned15), it comes at the cost of adverse effects. Striving for a more balanced system of rewards, one that supports collaborative practices (for example in terms of data sharing), transparency (for example in terms of peer review) and effective value and impact (through article content analysis and article-level rather than journal-level metrics) will reward actual as opposed to perceived excellence, without the above-described documented adverse effects.
Dialogue is also needed to resolve how we can shift from an evaluation system that relies on metrics and which allows researchers to be easily classified – towards an evaluation system that goes back to the roots of what defines the quality of the scientists’ work: first, its content and its dissemination, at best, second. This change of culture appears to be in opposition to the broader trend of ranking (and related conditional financing) of research and higher education establishments to which the evaluation of researchers directly contributes, and which also needs a fundamental rethink.16 Reform thus entails a more systemic consideration of the issue of research evaluation within the broader topic of how research and higher education are financed and supported by public authorities, especially related to ensuring sustainable funding levels.17
A vital aspect of the debate, in our eyes, is that the evaluation system as it exists today does not empower researchers towards excellence. Instead, the present system, combined with decreasing public budgets allocated to research and thus ever-increasing competition for research grants, largely functions as a convenient ‘controlling and sorting’ tool. It fails to support ECRs in setting realistic goals that make them grow and evolve in their career. Where is the place in today’s system for enhancing the reliability and impact of research,18 focusing for example on the robustness of datasets, creation of outreach material for policymakers, exploration of new avenues with no certainty of results, or constructive criticism of other scientists’ approaches through peer review? Research is so much more than publishing articles. We should strive for an evaluation system that empowers researchers to act in all the – currently hidden – aspects of what constitutes the research ecosystem and contribute to its vitality and connections with society.
A positive note in this debate is that, unlike debates about the future of academic publishing, evaluation is much more in the hands of the academic community and its funders, with little commercial interest in maintaining the status quo. Of course, some governments fund universities based on rankings and metrics. However, we would claim that this is largely with the tacit approval (or even explicit encouragement) of the research community. Thus, provided that awareness of the problems is shared, and common solutions are agreed upon, we have the power to effect change without having to convince a broader ecosystem of external actors who might potentially have conflicting interests.
Which steps do we identify to implement this change of culture?
First, as a research community, among researchers across all career stages, we need to take a hard, realistic and honest look at the current reward system and its flaws, regardless of how well it may have served us.
Second, beyond localized examples of evolving practices of research evaluation, for example in the recruitment practices of some faculties or research institutes,19 a broader internal dialogue is needed within the research community (including research funding organizations: see the discussions held at the level of the Global Research Council in this regard20) to focus on what is important, what should be rewarded and how individuals are evaluated at different stages of their research careers.
We believe that the core motivation of all of this should be to empower ECRs, as we are the actors whose futures are at stake, and as a community we feel a passionate need to improve research culture. Momentum is building, and to unleash it we should strive to empower ECRs through changes in research evaluation as perhaps the most important lever for improving research quality and culture.
The authors are extremely grateful to Noémie Aubert-Bonn for supporting the compilation of this article and providing relevant bibliographical resources.
The authors have declared no competing interests.
American Society for Cell Biology, “San Francisco Declaration on Research Assessment,” 2013, https://sfdora.org/read/ (accessed 22 April 2021); James Wilsdon et al., “The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management,” 2015, DOI: https://doi.org/10.13140/RG.2.1.4929.1363 (accessed 22 April 2021); Diana Hicks et al., “Bibliometrics: The Leiden Manifesto for Research Metrics,” Nature 520, no. 7548 (April 2015): 429–31, DOI: https://doi.org/10.1038/520429a (accessed 22 April 2021); David Moher et al., “Assessing Scientists for Hiring, Promotion, and Tenure,” PLOS Biology 16, no. 3 (March 29, 2018): e2004089, DOI: https://doi.org/10.1371/journal.pbio.2004089 (accessed 22 April 2021); Bregt Saenen and Lidia Borrell-Damián, “Reflections on University Research Assessment: key concepts, issues and actors,” EUA Briefing, April 2019, https://eua.eu/resources/publications/825:reflections-on-university-research-assessment-key-concepts,-issues-and-actors.html (accessed 22 April 2021); David Moher et al., “The Hong Kong Principles for Assessing Researchers: Fostering Research Integrity,” PLOS Biology 18, no. 7 (July 16, 2020): e3000737, DOI: https://doi.org/10.1371/journal.pbio.3000737 (accessed 22 April 2021).
Andrew D. Higginson and Marcus R. Munafò, “Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions,” PLOS Biology 14, no. 11 (November 10, 2016): e2000995, DOI: https://doi.org/10.1371/journal.pbio.2000995 (accessed 22 April 2021); Moher et al., “Assessing Scientists for Hiring, Promotion, and Tenure.”
Colin A. Chapman et al., “Games Academics Play and Their Consequences: How Authorship, h -Index and Journal Impact Factors Are Shaping the Future of Academia,” Proceedings of the Royal Society B: Biological Sciences 286, no. 1916 (December 4, 2019): 20192047, DOI: https://doi.org/10.1098/rspb.2019.2047 (accessed 22 April 2021).
“The Cost of Salami Slicing,” Nature Materials 4, no. 1 (January 2005): 1–1, DOI: https://doi.org/10.1038/nmat1305 (accessed 22 April 2021); Vikas Menon and Aparna Muraleedharan, “Salami Slicing of Data Sets: What the Young Researcher Needs to Know,” Indian Journal of Psychological Medicine 38, no. 6 (November 2016): 577–78, DOI: https://doi.org/10.4103/0253-7176.194906 (accessed 22 April 2021).
Melissa S. Anderson et al., “The Perverse Effects of Competition on Scientists’ Work and Relationships,” Science and Engineering Ethics 13, no. 4 (December 2007): 437–61, DOI: https://doi.org/10.1007/s11948-007-9042-5 (accessed 22 April 2021); Kelly L. Wester, John T. Willse, and Mark S. Davis, “Psychological Climate, Stress, and Research Integrity Among Research Counselor Educators: A Preliminary Study,” Counselor Education and Supervision 50, no. 1 (September 2010): 39–55, DOI: https://doi.org/10.1002/j.1556-6978.2010.tb00107.x (accessed 22 April 2021); Joeri K. Tijdink, Reinout Verbeke, and Yvo M. Smulders, “Publication Pressure and Scientific Misconduct in Medical Scientists,” Journal of Empirical Research on Human Research Ethics 9, no. 5 (December 2014): 64–71, DOI: https://doi.org/10.1177/1556264614552421 (accessed 22 April 2021).
Chapman et al., “Games Academics Play and Their Consequences”; Lennart Stoy, Rita Morais, and Lidia Borrell-Damián, “Decrypting the Big Deal Landscape. Follow-up of the 2019 EUA Big Deals Survey Report,” (European University Association, October 2019), https://eua.eu/downloads/publications/2019%20big%20deals%20report.pdf (accessed 22 April 2021); Martin Paul Eve, “The Problems of Unit Costs Per Article,” 2019, https://eve.gd/2019/09/19/the-problems-of-unit-costs-per-article/ (accessed 22 April 2021).
VSNU, NFU, KNAW, NWO, and ZonMw, “Room for Everyone’s Talent. Towards a New Balance in the Recognition and Rewards of Academics,” 2019, https://www.vsnu.nl/recognitionandrewards/wp-content/uploads/2019/11/Position-paper-Room-for-everyone’s-talent.pdf (accessed 22 April 2021); “A kinder research culture is possible,” Nature 574, no. 7776 (October 3, 2019): 5–6, DOI: https://doi.org/10.1038/d41586-019-02951-4 (accessed 22 April 2021); Bregt Saenen et al., “Research Assessment in the Transition to Open Science. 2019 EUA Open Science and Access Survey Results,” (European University Association, October 2019), https://eua.eu/downloads/publications/research%20assessment%20in%20the%20transition%20to%20open%20science.pdf (accessed 26 April 2021); Noémie Aubert Bonn and Wim Pinxten, “Rethinking Success, Integrity, and Culture in Research (Part 1) – A Multi-Actor Qualitative Study on Success in Science,” Research Integrity and Peer Review 6, 1 (2021), DOI: https://doi.org/10.1186/s41073-020-00104-0 (accessed 30 April 2021); Mattias Björnmalm et al., “Advancing Research Data Management in Universities of Science and Technology,” February 13, 2020, DOI: https://doi.org/10.5281/ZENODO.3665372 (accessed 26 April 2021).
David Nicholas et al., “How is open access publishing going down with early career researchers? An international, multi-disciplinary study,” El Profesional de La Información, 29, no. 6 (November 25, 2020), DOI: https://doi.org/10.3145/epi.2020.nov.14 (accessed 26 April 2021).
Mondragon Unibertsitatea, “Publications Impact Indexes,” 2017, https://www.mondragon.edu/en/web/biblioteka/publications-impact-indexes#:~:text=Each%20subject%20category%20of%20journals,the%2075%20to%20100%25%20group (accessed 26 April 2020).
Mattias Björnmalm, Matthew Faria, and Frank Caruso, “Advancing Research Using Action Cameras,” Chemistry of Materials 28, no. 23 (December 13, 2016): 8441–42, DOI: https://doi.org/10.1021/acs.chemmater.6b04639 (accessed 26 April 2021); Mattias Björnmalm and Frank Caruso, “Robust Chemistry: The Importance of Data and Methods Sharing,” Angewandte Chemie International Edition 57, no. 5 (January 26, 2018): 1122–23, DOI: https://doi.org/10.1002/anie.201710493 (accessed 26 April 2021).
Mattias Björnmalm, Matthew Faria, and Frank Caruso, “Increasing the Impact of Materials in and beyond Bio-Nano Science,” Journal of the American Chemical Society 138, no. 41 (October 19, 2016): 13449–56, DOI: https://doi.org/10.1021/jacs.6b08673 (accessed 26 April 2021).
Mattias Björnmalm et al., “Bridging Bio–Nano Science and Cancer Nanomedicine,” ACS Nano 11, no. 10 (October 24, 2017): 9594–9613, DOI: https://doi.org/10.1021/acsnano.7b04855 (accessed 26 April 2021).
Matthew Faria et al., “Minimum Information Reporting in Bio–Nano Experimental Literature,” Nature Nanotechnology 13, no. 9 (September 2018): 777–85, DOI: https://doi.org/10.1038/s41565-018-0246-4 (accessed 26 April 2021).
Rik Van de Walle, “Reform Is Overdue,” Research Europe, September 17, 2020. https://www.cesaer.org/content/5-operations/2020/20200917-reform-overdue-rvdw.pdf (accessed 30 April 2021).
“Excellence R Us: University Research and the Fetishisation of Excellence,” n.d., https://hcommons.org/?get_group_doc=1001240/1507389514-ExcellenceRUs.pdf (accessed 26 April 2021).
Elizabeth Gadd, “University Rankings Need a Rethink,” Nature 587, no. 7835 (November 24, 2020): 523–523, DOI: https://doi.org/10.1038/d41586-020-03312-2 (accessed 26 April 2021).
CESAER, “Sustainable Funding for Universities of the Future,” March 16, 2020, DOI: https://doi.org/10.5281/ZENODO.3712083 (accessed 26 April 2021).
Jean Lebel and Robert McLean, “A Better Measure of Research from the Global South,” Nature 559, no. 7712 (July 2018): 23–26, DOI: https://doi.org/10.1038/d41586-018-05581-4 (accessed 26 April 2021).
Bregt Saenen et al., “Reimagining Academic Career Assessment: Stories of Innovation and Change,” (European University Association, January 2021), https://eua.eu/downloads/publications/eua-dora-sparc_case%20study%20report.pdf (accessed 26 April 2021).
Stephen Curry et al., “The Changing Role of Funders in Responsible Research Assessment: Progress, Obstacles and the Way Ahead,” RoRI Working Paper No.3, November 2020, https://rori.figshare.com/articles/report/The_changing_role_of_funders_in_responsible_research_assessment_progress_obstacles_and_the_way_ahead/13227914 (accessed 26 April 2021).