Case Studies

Implementing the Declaration on Research Assessment: a publisher case study

Authors: {'first_name': 'Victoria', 'last_name': 'Gardner'},{'first_name': 'Mark', 'last_name': 'Robinson'},{'first_name': 'Elisabetta', 'last_name': 'O\xe2\x80\x99Connell'}

Abstract

There has been much debate around the role of metrics in scholarly communication, with particular focus on the misapplication of journal metrics, such as the impact factor in the assessment of research and researchers. Various initiatives have advocated for a change in this culture, including the Declaration on Research Assessment (DORA), which invites stakeholders throughout the scholarly communication ecosystem to sign up and show their support for practices designed to address the misuse of metrics. This case study provides an overview of the process undertaken by a large academic publisher (Taylor & Francis Group) in signing up to DORA and implementing some of its key practices in the hope that it will provide some guidance to others considering becoming a signatory. Our experience suggests that research, consultation and flexibility are crucial components of the process. Additionally, approaching signing with a project mindset versus a ‘sign and forget’ mentality can help organizations to understand the practical implications of signing, to anticipate and mitigate potential obstacles and to support cultural change.

Keywords: DORAresearch assessmentimpact factorjournal metricschange managementplatform development 
DOI: http://doi.org/10.1629/uksg.573
 Accepted on 08 Feb 2021            Submitted on 13 Dec 2021

Background and context

Taylor & Francis Group is an international knowledge service provider with a portfolio of journals and books that span the disciplinary spectrum from anthropology to zoology. We publish journals on behalf of hundreds of learned societies and professional member organizations, work with thousands of expert academic editors and board members and support researchers to share the outputs of their work.

As a publisher, our role in scholarly communications includes both responding to the evolving priorities of our partners and customers, as well as helping to take a lead in areas such as publishing standards and ethics. This includes discussion around the use, and misuse, of metrics in research and researcher assessment, including criticism of how – rather than supporting the assessment process – metrics have replaced a qualitative review of research outcomes.1 A recurring concern raised by numerous commentators, from institutions2 and think-tanks3 to researchers themselves,4 has been about the misapplication of metrics. The most commonly levelled charge is against the impact factor – with much discussion on how this journal-level metric, originally used to inform library purchasing decisions,5 is incorrectly used to evaluate researcher performance and individual research outputs.6

There has been concerted action to address this in recent years, with many institutions,7 funders8 and publishers9 cautioning against misuse of the impact factor to assess researchers or individual research outputs. Calls have also been made for different approaches,10 including more qualitative processes to underpin researcher assessment using tools such as the narrative CV.11 The Global Research Council’s 2021 report on responsible research assessment12 advocates for researchers to be rewarded based on their contribution to the community and not just their publication record, for example peer reviewing or public engagement efforts.

Contributions to this debate have come from a range of sources and experts, for example the Leiden Manifesto for Research Metrics13 and the Hong Kong Principles.14 Arguably, the best known of these initiatives is the Declaration on Research Assessment (DORA). DORA came into being as an outcome of the 2012 American Society for Cell Biology (ASCB) meeting.15 At its heart, DORA aims to address this misapplication of the impact factor with its general recommendation, ‘Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.’16 The Declaration outlines a number of other best practices for funding agencies, institutions, publishers, metrics providers and researchers. Supporters are invited to sign up to DORA on the understanding that this means they will follow the recommended practices outlined within the Declaration.

Taylor & Francis signed up to DORA in March 2021 after almost two years of discussion, consultation and development. We could have signed sooner but took a deliberate decision to wait until we had met two key milestones. Firstly, we wanted to consult with our learned society partners, academic editors and colleagues across the globe and consider their feedback as part of a consensus-driven approach. Secondly, we needed to be sure that we could adhere to the practices outlined in the Declaration and wanted to link our signature with concrete developments to ensure that we were acting on the intentions expressed in DORA.

There has been much discussion and advocacy around DORA. Not as much has been written about implementation and the implications of signing up to the Declaration, but there is a growing body of case studies and insights. UiT The Arctic Institute of Norway has provided some reflections on their experience, noting that implementing DORA is easier for an institution than implementing open access policies but that ensuring full compliance in practice is challenging.17 The DORA site also hosts some insightful case studies, with more being added as experience is gained.18 We write from the perspective of a publisher and not an academic institution; implementation has much broader practical implications for institutions. Nonetheless, we hope this case study provides some guidance that can be used by organizations considering signing up to and implementing DORA.

Discussion

The approach taken by the team at Taylor & Francis was to consider DORA signature and implementation as a project comprising various activities and stakeholders and based on a simple project plan. We loosely followed the software development life cycle,19 though this process applied not just to technical developments but to the whole project. The process typically comprises sequential steps covering the following areas: discovery (or ideation), development, launch and review. The key elements of each stage are outlined below. Supporting this activity internally was a large cast of colleagues drawn from our Editorial, Marketing, Research and Analytics, and Technology and Product teams. The composition of the working group changed as we moved from one stage of the project to the next.

Discovery

Analysis and action plan

Although we had DORA signature in mind from the outset, we devoted some time to investigating the other initiatives. Our research focused on research assessment reform, comparing and evaluating them based on a loose set of criteria (including ease of implementation, research community buy in and community awareness). This research affirmed our original plan of working towards DORA signature because DORA 1) contains a set of practices that could be used as a checklist, 2) has a clear process for signing and 3) already has a critical mass of supporters from across the research community.

As well as reviewing the best practices that were applicable to publishers, we considered the broader set of practices.20 Our analysis sought to clarify those areas where we were already working to the stated outcome and those where we would need to develop or change policy. We were already working to best practice in many of the areas. This included the provision of article-level metrics, such as usage and Altmetric attention scores21 and opening up citations through our support of the Initiative for Open Citations (I4OC).22 We had the most work to do on recommendation number 6. This requires publishers to either cease promoting the impact factor or to present it ‘in the context of a variety of journal-based metrics … that provide a richer view of journal performance’.23 To meet this requirement, we drew up a high-level action plan that included what change or development was required, possible issues, who needed to buy in to or sign off on this work and approximately what level of effort and investment it would require to implement.

Consultation and socialization

We used the outcomes of our analysis to carry out an internal consultation, sourcing views from colleagues across the business and from different global regions. From this, we obtained a range of views and feedback. Common themes included:

  • while change is needed to how the impact factor is used, it still has value in certain contexts
  • DORA signature should be supported with clear actions to avoid ‘virtue signalling’
  • we needed to acknowledge that our editors and learned society partners aren’t obligated to take the same stance as us.

We expected to hear more concerns from colleagues in regions where impact factor is still used in research assessment. In anticipation of this, we noted that our intention was to downweight, but not remove, the impact factor and to present it in context alongside other metrics. This met with broad agreement.

On a parallel track, we engaged in invaluable discussions with the DORA team as well as seeking views from our external partners – academic editors and boards – and society and member organizations to whom we provide publishing services.24 We received general support for the principles of DORA, although there was some hesitation when we outlined the likely consequences of signature. This mainly centred on giving less prominence to the impact factor and fears that might negatively influence authors’ submission decisions. Additional concerns included whether such a development would add to the complexity of the editor’s role or their workload. Understanding these hesitations helped inform our implementation process, particularly in the guidance we created for our partners and authors.

Scoping and requirement gathering

Based on the feedback above, the working group felt confident that we could sign the Declaration as Taylor & Francis and could provide information to partners to allow them to consider whether they wished to support the Declaration as well. We wanted to make sure that our signature was supported by concrete action and to help colleagues and partners understand the practical implications of signature. From this outcome came the decision to review how we presented journal-level information on our platform, Taylor & Francis Online. Previously, the journal impact factor had been prominently displayed on website pages. The feedback we gathered made clear the importance of reducing the emphasis placed on this metric on our journal homepages and that, where present, it needed to be supported by a broader range of metrics to contextualize overall journal performance.

From the outset, we acknowledged that addressing issues around the misuse of metrics by providing more journal-level metrics might not solve the issue. With this in mind, we noted the importance of developing external communications and internal training materials which emphasized that research outputs should always be assessed on their own merits. We debated when would be an appropriate time to sign up to the Declaration and agreed to wait until we had implemented the changes on our platform to de-emphasize the impact factor and introduce the additional journal-level data.

We devoted a good deal of time to considering what additional metrics to include and decided upon these based on inputs from researchers, including the questions most asked of our customer help desks, discussions with the Society Publishers’ Coalition25 and a review of what metrics were commonly reported on other publisher sites. At the time, a pilot was running on a group of our Medical and Health Sciences journals around the display of more metrics of journal performance, such as turnaround times; this was also highly influential. More time than expected was devoted to debating whether we should display mean or median data for speed metrics. We decided to display median data, which lessens the impact of outliers and means that times are more representative of a typical manuscript journey, but this topic is still under debate. One challenge we encountered was that, particularly in terms of researcher feedback, the impact factor was considered as a ‘must-have’ metric. The group agreed that in order to encourage cultural change and a shift away from reliance on impact factor, we needed to do our part to present it in context alongside other metrics that provided more detail on journal performance.

Development

Once agreed on the metrics scope, we had to face two main challenges: how to source them and where and how to display them on the platform so that users could easily find and interpret them.

With regards to the latter, we conducted specific user interviews. The users were researchers with experience in publishing journal articles. Each user was asked to walk through several journeys, giving feedback throughout and to compare different placements for the new journal metrics on the journal landing page. Most of the users interviewed expected to find the link to the journal metrics under the ‘About this journal’ menu. They all gave positive feedback about having a dedicated page for the metrics, and they particularly liked the use of icons.

The outcome was the creation of the new ‘Journal metrics’ page, linked to from the ‘About this journal’ menu and presenting the metrics in three main groups: Usage, Citation metrics and Speed/acceptance (see Figure 1).26

Journal metrics page
Figure 1 

Journal metrics page

The sourcing of the data to feed the metrics page was more complex because the information for the fields did not reside in a single source that would easily feed the web page. After some technical discussions, it was agreed that the data would be gathered and uploaded via an API (application programming interface) to feed into the platform for display via a series of custom properties.

Launch

The project group had been interacting with colleagues and external partners for a prolonged period by this point, so we felt that there was a high level of awareness around both the planned enhancement to the platform and our plans to sign up to DORA. Once we were confident that we were technically ready to launch, we focused much of our energy on briefing colleagues about the forthcoming release, providing them with information to share with external partners and support in the form of guidance and resources. These resources included information on our Author Services site for researchers in particular27 but also signposted best practice guidance for colleagues to refer to – most importantly, ensuring that we didn’t promote the impact factor in isolation. Guidance also included an internal standard operating procedure for those journals where teams wished to opt out of the display of turnaround and acceptance rates.

A working group of Marketing and Communications colleagues representing a range of regions and subject areas mapped where current practices needed to be changed to ensure we aligned with DORA recommendations. For example, our regular author surveys consistently demonstrated the importance of the impact factor in researchers’ submission decisions and so it was previously common for this information to be featured prominently in promotions. The result of this group’s work was a global briefing and an internal guide on best practice use of journal metrics. The guide sets out a range of protocols for marketing colleagues, including ensuring metrics are always used in context, avoiding the use of single journal metrics in isolation and highlighting qualitative indicators of quality and impact alongside quantitative measures.

Once we had gone live with the platform update and made a few other tweaks to ensure that we were meeting the intention of DORA’s practices, we announced our signature of the Declaration in March 2021.28 We attracted some criticism for our perceived slowness in signing up compared to other publishers, for applying the same journal-level metrics across all disciplines and in some quarters for our choice of journal-level metrics. However, we also received positive feedback29 with the platform development signposted as an example of us making change to support the Declaration in deed as well as word.30

Post launch – lessons learned and future plans

As noted above, we were pleased by the positive response from stakeholders to our support of DORA, and we have received very few queries around, for example, impact factor promotion.

We have already updated the metrics display to include an additional metric (CiteScore Best Quartile), based on feedback from users of the platform and our academic editors. In the future we hope to develop our data sources so that management and update of metrics is automated. We would also be keen to work with other stakeholders to develop a common standard to aid comparison between journals, regardless of publisher.

We still have work to do, for example in helping prospective authors to understand and interpret the metrics we present, considering how to display some metrics in context and improving our data flows. Broader cultural change will take longer to have an effect. Although there is a trend towards de-emphasizing the impact factor, there still are parts of the world that reward researchers for publishing in journals with an impact factor. We hope that by signing up to the Declaration, and with the changes that we have made, we are playing our part in effecting change and helping to encourage our networks to review and consider their approaches to research assessment.

Conclusion

We have outlined the process that a large academic publisher underwent to sign up to and implement the Declaration of Research Assessment, including activity to downplay the impact factor on its platform. Although there was broad support for the spirit of DORA and its call to reform research assessment practices, we had to overcome some friction when implementing the changes needed to support our signature, particularly those that required a change in behaviour. We recommend that organizations considering signing DORA should approach it with an implementation mindset, considering the practical implications of signing the Declaration (including any targeted investment in technological infrastructure), how these might affect key stakeholders and how to win support for proposed changes.

Acknowledgements

The authors wish to thank the team at DORA for their support and advice, in particular Professor Stephen Curry, Chair of the DORA Steering Committee, for his review and improvement to an earlier version of this article. They would also like to thank the Society Publishers’ Coalition for their inputs and colleagues at Taylor & Francis for their careful review of this case study, as well as all those who supported the project.

Abbreviations and Acronyms

A list of the abbreviations and acronyms used in this and other Insights articles can be accessed here – click on the URL below and then select the ‘full list of industry A&As’ link: http://www.uksg.org/publications#aa.

Competing Interests

All authors are employed by Taylor & Francis Group. Victoria Gardner is a member of the editorial board for Insights on a voluntary basis.

References

  1. Diana Hicks et al., “Bibliometrics: The Leiden Manifesto for research metrics,” Nature 520, (2015): 429–431, DOI: https://doi.org/10.1038/520429a (accessed 16 February 2022). 

  2. “Recognition and rewards,” Utrecht University, https://www.uu.nl/en/research/open-science/tracks/recognition-and-rewards (accessed 16 February 2022). 

  3. David Moher et al., “Assessing scientists for hiring, promotion, and tenure,” PLOS Biology 16, no. 3 (2018): e2004089, DOI: https://doi.org/10.1371/journal.pbio.2004089 (accessed 16 February 2022). 

  4. Lesley A. Schimanski and Juan Pablo Alperin, “The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future,” [version 1; peer review: 2 approved] F1000Research 7 (2018): 1605, DOI: https://doi.org/10.12688/f1000research.16493.1 (accessed 16 February 2022). 

  5. Erin McKiernan, Juan Pablo Alperin, and Alice Fleerackers, “The ‘impact’ of the Journal Impact Factor in the review, tenure, and promotion process,” Impact of Social Sciences (blog), LSE, 26 April 2019, https://blogs.lse.ac.uk/impactofsocialsciences/2019/04/26/the-impact-of-the-journal-impact-factor-in-the-review-tenure-and-promotion-process/ (accessed 16 February 2022). 

  6. Clyde W. Holsapple, “Isn’t it about time to meet DORA?,” Journal of Organizational Computing and Electronic Commerce 28, no. 4 (2018): 287–290, DOI: https://doi.org/10.1080/10919392.2018.1522774 (accessed 21 February 2022). 

  7. Chris Woolston, “Impact factor abandoned by Dutch university in hiring and promotion decisions,” Nature 595, (2021): 462, DOI: https://doi.org/10.1038/d41586-021-01759-5 (accessed 16 February 2022). 

  8. “Making way for all aspects of quality,” The Dutch Research Council (NWO), last updated March 12, 2020, https://www.nwo.nl/en/news/making-way-all-aspects-quality (accessed 16 February 2022). 

  9. STM, The STM Report 1968–2018, Fifth Edition, October 2018, https://www.stm-assoc.org/2018_10_04_STM_Report_2018.pdf (accessed 16 February 2022). 

  10. Chris R. Triggle et al., “Requiem for impact factors and high publication charges,” Accountability in Research 29, no. 3 (2021): 133–164, DOI: https://doi.org/10.1080/08989621.2021.1909481 (accessed 21 February 2022). 

  11. “Résumé for Researchers,” The Royal Society, https://royalsociety.org/topics-policy/projects/research-culture/tools-for-support/resume-for-researchers/ (accessed 16 February 2022). 

  12. Global Research Council, Responsible Research Assessment, Global Research Council (GRC) Conference Report 2021, https://www.globalresearchcouncil.org/fileadmin/documents/GRC_Publications/GRC_RRA_Conference_Summary_Report.pdf (accessed 16 February 2022). 

  13. “Leiden Manifesto for Research Metrics,” Leiden Manifesto, http://www.leidenmanifesto.org/ (accessed 16 February 2022). 

  14. David Moher et al., “The Hong Kong Principles for assessing researchers: Fostering research integrity,” PLOS Biology 18, no. 7 (2020): e3000737, DOI: https://doi.org/10.1371/journal.pbio.3000737 (accessed 16 February 2022). 

  15. “American Society for Cell Biology,” The American Society for Cell Biology (ASCB), https://sfdora.org/resource/american-society-for-cell-biology/ (accessed 16 February 2022). 

  16. “What is DORA?,” The American Society for Cell Biology (ASCB), https://sfdora.org/ (accessed 16 February 2022). 

  17. Kenneth Ruud, “Implementing DORA at UiT The Arctic University of Norway,” The 13th Munin Conference on Scholarly Publishing 2018, No. 1 (2018), DOI: https://doi.org/10.7557/5.4614 (accessed 16 February 2022). 

  18. “Reimagining academic assessment: stories of innovation and change,” The American Society for Cell Biology (ASCB), https://sfdora.org/dora-case-studies/ (accessed 16 February 2022). 

  19. Andrew Aken, “CHUNK: An Agile Approach to the Software Development Life Cycle,” Journal of Internet Commerce 7, no. 3 (2008): 313–338, DOI: https://doi.org/10.1080/15332860802250385 (accessed 21 February 2022). 

  20. “San Francisco Declaration on Research Assessment,” The American Society for Cell Biology (ASCB), https://sfdora.org/read/ (accessed 16 February 2022). 

  21. “What are Altmetrics?,” Altmetric, https://www.altmetric.com/about-altmetrics/what-are-altmetrics/ (accessed 16 February 2022). 

  22. “About I4OC,” I4OC, https://i4oc.org/#about (accessed 16 February 2022). 

  23. The American Society for Cell Biology (ASCB), “San Francisco Declaration on Research Assessment”. 

  24. Taylor & Francis Group, “The use of metrics in research assessment: share your views,” Taylor & Francis Editor Resources, 6 January 2020, https://editorresources.taylorandfrancis.com/peersupport/the-use-of-metrics-in-research-assessment/ (accessed 16 February 2022). 

  25. “About us,” Society Publishers’ Coalition, https://www.socpc.org/about-us (accessed 16 February 2022). 

  26. “Journal Metrics,” Taylor & Francis Online, https://www.tandfonline.com/action/journalInformation?show=journalMetrics&journalCode=tres20 (accessed 16 February 2022). 

  27. “Understanding Journal Metrics,” Taylor & Francis Author Services, https://authorservices.taylorandfrancis.com/publishing-your-research/choosing-a-journal/journal-metrics/ (accessed 16 February 2022). 

  28. Taylor & Francis Group, “Taylor & Francis signs up to principles outlined in DORA supporting balanced and fair research assessment,” Taylor & Francis Group Newsroom, 3 March 2021, https://newsroom.taylorandfrancisgroup.com/taylor-francis-signs-up-to-principles-outlined-in-dora-supporting-balanced-and-fair-research-assessment/ (accessed 16 February 2022). 

  29. Hylke Dijkstra (@DijkstraHylke), “Taylor and Francis journals now have “journal metrics” on their website. And I love it” https://twitter.com/DijkstraHylke/status/1362778481873072132 (accessed 16 February 2022). 

  30. Prof David Bailey (@dgbailey), “Delighted that the journal metrics tab has been changed on the @tandfonline portal for all @regstud journals including @RegionalStudies, in line with DORA principles. It’s better to use a variety of metrics.” https://twitter.com/dgbailey/status/1368899007481847809 (accessed 16 February 2022).