Transparency versus anonymity:  which is better to eliminate bias in  peer review?

Faye Holst; Kim Eggleton; Simon Harris

Key Issues

Transparency versus anonymity: which is better to eliminate bias in peer review?

Authors

Faye Holst
Kim Eggleton
Simon Harris

Abstract

Peer review is a critical component of the scientific process. When conducted properly by dedicated and competent reviewers, it helps to safeguard the quality, validity, authority and rigour of academic work. However, bias in peer review is well documented and can skew objectivity of the review and hinder fair assessment of research. To mitigate against bias and enhance accountability, IOP Publishing has introduced two different, but complementary, approaches to all their peer-reviewed, open access (OA) journals: double-anonymous peer review and transparent peer review. Double-anonymous peer review, where the reviewer and author identities are concealed, is designed to tackle inequality in the scholarly publishing process as it reduces bias with respect to gender, race, country of origin or affiliation. Transparent peer review shows readers the full peer review history, including reviewer reports, editor decision letters and the authors’ responses alongside the published article. Making this process visible to the community increases accountability, allows reviewers to be recognized more for their work and can aid the training of aspiring reviewers. IOP Publishing is the first physics publisher to adopt both of these approaches portfolio wide. In this article we discuss how applying these methods has altered different elements of the publishing process. Early indicators show that there may be a marked difference in acceptance rates across regions.

Keywords:

Year: 2022

Volume 35

Page/Article: 16

DOI: 10.1629/uksg.584

Submitted on May 5, 2022

Accepted on Jun 9, 2022

Published on Oct 5, 2022

Peer Reviewed

CC BY 4.0

Context

(An earlier version of this article was published as a blog post in Times Higher Education. See reference 1.) There is broad consensus that conducting science more openly can accelerate scientific discovery and improve trust in the integrity of research. At the same time, funders are increasingly advocating for improved transparency and accountability, encouraging academic publishers, authors and reviewers to adopt new ways of working. This article provides insights into IOP Publishing’s experience in introducing two complementing peer review methodologies which will help inform publishers and scholars studying the merits of enhancing or altering the standard peer review process.

Shifting to new ways of working

One way to enhance transparency in the research and publication process is transparent peer review, a method where readers can see the full peer review history, including reviewer reports, editor decision letters and the authors’ responses alongside the published article. Making this process visible to the community increases accountabilities for all those involved and allows reviewers to be publicly recognized for their work.

Another benefit of transparent peer review is that it enables training and supervision. Early career researchers who are new to reviewing articles can learn by analysing the editorial process and published reports accompanying published articles in their area of research. This not only helps to train early career researchers but also means that reviewers may receive more credit for the work they do, particularly if they choose to sign their report.

Although several publishers have adopted elements of this way of working, there is great variation in how it is applied. Some journals use transparent peer review as the required form of peer review for that journal, while others give authors the choice to opt in. For instance, publisher Wiley gives authors the option to decline transparent peer review at submission, while their reviewers can choose to remain anonymous or may sign their reports if they wish. Publisher Sage offers transparent peer review to four of their journals via their Web of Science™ Transparent Peer Review Program. There is also a difference in the degree of transparency, with some journals releasing all peer review reports alongside relevant correspondence, author responses and editorial decision letters, while others only publish the reviewer reports. Variation is also seen in the naming conventions of this method, a challenge first STM’s Working Group on Peer Review Taxonomy followed by NISO is attempting to address.

At the time of writing, just over half of all authors publishing OA with IOP Publishing choose to disclose reviewer reports and 40 per cent of reviewers have opted in. Other publishers see a similar or higher appetite amongst authors. According to Nature nearly half of their authors are voluntarily opting in to publishing the reviewer reports of their article and after initial trials, Sage reported that 84 per cent of submitting authors chose to participate.

Additional research is needed to shed light on the motivations of authors and reviewers as to why they may not want to opt into new peer review methodologies.

In support of open science, and after an initial trial, IOP Publishing has recently moved all 18 of their fully OA journals to transparent peer review. The approach gives both authors and reviewers the option to opt in on a voluntary basis and includes publication of the full peer review history. The authors opt in or out when they submit their article, there is an option on the submission form, and reviewers opt in or out when they submit their report.

Double-anonymous peer review

Since April 2021, IOP Publishing has also moved all their owned journals over to default double-anonymous peer review – where the reviewer and author identities are concealed during the review process. The move is part of the publisher’s dedication to tackle inequality in the scholarly publishing process as it has the potential to reduce bias with respect to gender, sexual orientation, country of origin or affiliation. Studies show that academic reviewers are unconsciously more likely to accept work if it is written by people they consider to be like them – this seems to apply across gender, ethnicity and geography. Reputation bias also seems to thrive under the single-anonymous method. Many studies also show that double-anonymous peer review is preferred by many, especially under-represented minorities who perceive it to increase their chances of publication. This perception is made reality in early user data from IOP Publishing collected since the introduction of double-anonymous peer review, which suggests that anonymized articles are more likely to make it through peer review and acceptance compared to articles published in the same journal under single-anonymous peer review. IOP Publishing considers manuscripts to be single-anonymous reviewed if the author has left identifying information in the manuscript.

Double-anonymous peer review is not only preferred by early career researchers or people who might feel that their name or geographical area of residence would reflect negatively on the assessment of their work. Recently, Nobel laureate Novoselov anonymized his manuscript when submitting to one of IOP Publishing’s journals, demonstrating a belief in the publishing system and a trust in the quality of the research rather than relying on his established reputation.

Transparent peer review: findings so far and looking forward

IOP Publishing’s trial showed that there is surprisingly little variation in the opt-in rates for researchers in different disciplines covering physics, materials science, biomedical engineering and environmental science.

According to IOP Publishing’s user statistics collected since the introduction of transparent peer review, most reviewers do not seem to be deterred by transparent peer review as long as they are allowed to remain anonymous. Anecdotal feedback from reviewers reveals that early career researchers especially are reluctant to have their names made public to the authors, particularly if the authors are senior researchers, as this may count against them in future. Consequently, revealing the names of reviewers could make them decide to be less critical in their reports. This would explain why most reviewers (over 90 per cent) choose to publish their reviews anonymously. There is no observable impact on the willingness of reviewers to report under the model, and no change in the average time to first decision. Since implementing transparent peer review, IOP Publishing has seen a small increase in the average review quality with slightly more reports submitted under this model receiving the maximum rating compared to reviews submitted without transparent peer review.

To increase the uptake of transparent peer review, IOP Publishing is considering automatically enrolling reviewers so that just authors have to opt into transparent peer review. One of the reasons for authors not participating is the low level of awareness about it among authors and reviewers.

With transparent peer review set to become increasingly popular, IOP Publishing has deployed a method that could be scaled up. It was decided that Publons, owned by Clarivate, would be the best partner to apply transparent peer review to IOPP’s entire OA portfolio. Publons offers an article page where the peer review history is published, and each element of transparent peer review (reviewer reports, decision letters and author responses) is assigned an individual digital object identifier (DOI) which makes it easy to reference and cite content. This workflow complies with best practice data privacy regulation and ensures the individual preferences of authors, peer reviewers and journals are maintained. Publons has also built an interface that receives accepted articles automatically, removing the need to manually compile and send details to them, thus saving a significant amount of time.

Uptake of double-anonymous peer review

The feedback from authors and reviewers has generally been positive, with some IOP Publishing journals seeing nearly half of all authors opting to submit anonymized manuscripts, see Figure 1.

Figure 1

Percentage of submissions anonymized by the end of 2021 across all IOP Publishing journals offering double and single anonymity. (Full journal titles can be found in Appendix 1)

IOP Publishing is also starting to see the difference it is making for some under-represented groups. For researchers in Africa and the Middle East, their chances of having work accepted has more than doubled under the double-anonymous method, see Figure 2. It is early days in terms of data collection, but it is certainly striking. Interestingly, there are no equivalent results for gender – all genders see a similar impact on acceptance rates. This contradicts the perception that gender influences peer review outcome.

Figure 2

Data based on 12,192 articles with final decisions made between January and April 2022, across all IOP Publishing journals offering double and single anonymity, indicating the change in acceptance rate when authors choose anonymity

Some argue that double-anonymous is not particularly effective as reviewers might be able to identify someone by looking at which references are used in the manuscript or the field of study. However, early data from IOP Publishing’s research shows that 85 per cent of reviewers on anonymized manuscripts say they would not feel confident guessing author identities. This data was collected via IOPP’s reviewer survey, see Appendix 2, which, since the introduction of double-anonymous peer review in April 2021, is sent to all reviewers once the final decision about a manuscript is made. This ongoing survey asks reviewers about their experience, and specifically about their level of confidence in guessing the author identities for double-anonymous submissions.

Conclusion

Offering both double-anonymous review before acceptance for publication and transparent review post-publication allows for maximum objectivity during the review process, and maximum transparency after publication. Both methods of peer review deliver distinct benefits. Combining the two approaches will help mitigate against conscious and unconscious bias and will stimulate greater diversity and greater accountability.

Publishers can play a leading role in testing new practices in peer review. Teaching authors and reviewers the benefits of these new methods will help raise consciousness of what affects peer review judgements. In turn, this will strengthen the foundations of science and will ensure that the highest possible standards of peer review are applied to all published work.

Supplemental File

The supplemental files for this article can be found as follows:

Appendix 1

Full titles of journals included in Figure 1. DOI: https://doi.org/10.1629/uksg.584.s1

Appendix 2

IOP Publishing: Reviewer Experience Survey. DOI: https://doi.org/10.1629/uksg.584.s2

Abbreviations and Acronyms

A list of the abbreviations and acronyms used in this and other Insights articles, can be accessed here – click on the URL below and then select the ‘full list of industry A&As’ link: http://www.uksg.org/publications#aa.

Competing Interests

The authors have declared no competing interests.

References

Kim Eggleton, “Is anonymity or transparency the best solution to bias in peer review?,” Times Higher Education (THE), March 14, 2022, https://www.timeshighereducation.com/blog/anonymity-or-transparency-best-solution-bias-peer-review (accessed 21 July 2022).
“OACA List,” SPARC Europe, https://sparceurope.org/what-we-do/open-access/sparc-europe-open-access-resources/open-access-citation-advantage-service-oaca/oaca-list/ (accessed 21 July 2022).
Katrina Pickersgill, “Review of Six Months of Transparent Peer Review at SAGE,” SAGE Perspectives (blog), September 22, 2021, https://perspectivesblog.sagepub.com/blog/six-months-of-transparent-peer-review-at-sage (accessed 21 July 2022).
Rachel Pells, “Journal Transparency Rules to Help Scholars Pick Where to Publish,” Times Higher Education (THE), June 4, 2019, https://www.timeshighereducation.com/news/journal-transparency-rules-help-scholars-pick-where-publish (accessed 21 July 2022).
Alice Meadows, “What Does Transparent Peer Review Mean and Why Is It Important?,” The Scholarly Kitchen, August 3, 2017, https://scholarlykitchen.sspnet.org/2017/08/03/transparent-peer-review-mean-important/ (accessed 21 July 2022).
“Transparent Peer Review,” Authorservices.wiley.com, https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-review/transparent-peer-review.html (accessed 21 July 2021).
“Transparent Peer Review,” SAGE Publications Ltd., May 21, 2021, https://uk.sagepub.com/en-gb/eur/transparent-peer-review (accessed 21 July 2021).
“Working Group on Peer Review Taxonomy,” STM, https://www.stm-assoc.org/standards-technology/peer-review-taxonomy-project/ (accessed 21 July 2022); “Work Item Title: Developing a Standard Peer Review Taxonomy (PRT): Background and Problem Statement,” NISO, https://groups.niso.org/higherlogic/ws/public/download/25638/NISO_PeerReviewTaxonomy_for_VM_approval.pdf (accessed 21 July 2022).
“Transparent peer review: the value is clear,” Commun Phys 5, no. 108 (2022), DOI: https://doi.org/10.1038/s42005-022-00891-6 (accessed 15 August 2022); “Transparent peer review one year on,” Nature Communications 7, no. 13626 https://www.nature.com/articles/ncomms13626 (accessed 15 August 2022).
Katrina Pickersgill, “Six months of Transparent Peer Review at SAGE,” Sage Perspectives (blog), September 22, 2021, https://perspectivesblog.sagepub.com/blog/six-months-of-transparent-peer-review-at-sage (accessed 15 August 2021).
Marc Domingo and Simon Harris, “Transparent Peer Review—a Practical Solution to Implement Open Peer Review at Scale: A Case Study,” Science Editor 43, no. 3 (September 1, 2020): 72–76, DOI: https://doi.org/10.36591/se-d-4303-72 (accessed 21 July 2022).
Amber E. Budden et al., “Double-Blind Review Favours Increased Representation of Female Authors,” Trends in Ecology & Evolution 23, no. 1 (January 1, 2008): 4–6, DOI: https://doi.org/10.1016/j.tree.2007.07.008; Stefanie K. Johnson and Jessica F. Kirk, “Dual-Anonymization Yields Promising Results for Reducing Gender Bias: A Naturalistic Field Experiment of Applications for Hubble Space Telescope Time,” Publications of the Astronomical Society of the Pacific 132, no. 1009 (18 February, 2020): 034503, DOI: https://doi.org/10.1088/1538-3873/ab6ce0 (accessed 21 July 2022); Andrew Tomkins, Min Zhang, and William D. Heavlin, “Single versus Double Blind Reviewing at WSDM 2017,” ArXiv:1702.00502 [Cs] (October 2017), https://arxiv.org/abs/1702.00502 (accessed 15 August 2022).
Dakota Murray et al., “Author-Reviewer Homophily in Peer Review,” bioRxiv 400515, August 4, 2019, DOI: https://doi.org/10.1101/400515 (accessed 21 July 2022); Johnson and Kirk, “Dual-Anonymization Yields Promising Results for Reducing Gender Bias: A Naturalistic Field Experiment of Applications for Hubble Space Telescope Time.”; Tomkins, Zhang, and Heavlin, “Single versus Double Blind Reviewing at WSDM 2017.”
Mengyi Sun, Jainabou Barry Danfa, and Misha Teplitskiy, “Does Double-Blind Peer Review Reduce Bias? Evidence from a Top Computer Science Conference,” Journal of the Association for Information Science and Technology 73, no.6 (October 12, 2021): 811–819, DOI: https://doi.org/10.1002/asi.24582 (accessed 21 July 2021).
Andrew Tomkins, Min Zhang, and William D. Heavlin, “Reviewer bias in single- versus double-blind peer review,” PNAS, 114, no. 48 (October 10, 2017): 12708-12713, DOI: https://doi.org/10.1073/pnas.1707323114 (accessed 21 July 2021).
“Single-Blind Vs. Double-Blind Peer Review,” Enago Academy, last updated March 16, 2022, https://www.enago.com/academy/double-blind-peer-review-for-better-or-for-worse/ (accessed 15 August 2022).
“IOP Publishing: Reviewer Experience Survey,” Institute of Physics Publishing.
Xinyao Zhou et al., “Conformal Screen Printed Graphene 4 × 4 Wideband MIMO Antenna on Flexible Substrate for 5G Communication and IoT Applications,” 2D Materials 8, no. 4 (August 20, 2021): 045021, DOI: https://doi.org/10.1088/2053-1583/ac1959 (accessed 21 July 2022).
Laura Simonite, “What Is Publons Transparent Peer Review?” Web of Science, modified August 1, 2019, https://publons.freshdesk.com/support/solutions/articles/12000060026-what-is-publons-transparent-peer-review- (accessed 21 July 2022).

[B1] Kim Eggleton, “Is anonymity or transparency the best solution to bias in peer review?,” Times Higher Education (THE), March 14, 2022, https://www.timeshighereducation.com/blog/anonymity-or-transparency-best-solution-bias-peer-review (accessed 21 July 2022).

[B2] “OACA List,” SPARC Europe, https://sparceurope.org/what-we-do/open-access/sparc-europe-open-access-resources/open-access-citation-advantage-service-oaca/oaca-list/ (accessed 21 July 2022).

[B3] Katrina Pickersgill, “Review of Six Months of Transparent Peer Review at SAGE,” SAGE Perspectives (blog), September 22, 2021, https://perspectivesblog.sagepub.com/blog/six-months-of-transparent-peer-review-at-sage (accessed 21 July 2022).

[B4] Rachel Pells, “Journal Transparency Rules to Help Scholars Pick Where to Publish,” Times Higher Education (THE), June 4, 2019, https://www.timeshighereducation.com/news/journal-transparency-rules-help-scholars-pick-where-publish (accessed 21 July 2022).

[B5] Alice Meadows, “What Does Transparent Peer Review Mean and Why Is It Important?,” The Scholarly Kitchen, August 3, 2017, https://scholarlykitchen.sspnet.org/2017/08/03/transparent-peer-review-mean-important/ (accessed 21 July 2022).

[B6] “Transparent Peer Review,” Authorservices.wiley.com, https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-review/transparent-peer-review.html (accessed 21 July 2021).

[B7] “Transparent Peer Review,” SAGE Publications Ltd., May 21, 2021, https://uk.sagepub.com/en-gb/eur/transparent-peer-review (accessed 21 July 2021).

[B8] “Working Group on Peer Review Taxonomy,” STM, https://www.stm-assoc.org/standards-technology/peer-review-taxonomy-project/ (accessed 21 July 2022); “Work Item Title: Developing a Standard Peer Review Taxonomy (PRT): Background and Problem Statement,” NISO, https://groups.niso.org/higherlogic/ws/public/download/25638/NISO_PeerReviewTaxonomy_for_VM_approval.pdf (accessed 21 July 2022).

[B9] “Transparent peer review: the value is clear,” Commun Phys 5, no. 108 (2022), DOI: https://doi.org/10.1038/s42005-022-00891-6 (accessed 15 August 2022); “Transparent peer review one year on,” Nature Communications 7, no. 13626 https://www.nature.com/articles/ncomms13626 (accessed 15 August 2022).

[B10] Katrina Pickersgill, “Six months of Transparent Peer Review at SAGE,” Sage Perspectives (blog), September 22, 2021, https://perspectivesblog.sagepub.com/blog/six-months-of-transparent-peer-review-at-sage (accessed 15 August 2021).

[B11] Marc Domingo and Simon Harris, “Transparent Peer Review—a Practical Solution to Implement Open Peer Review at Scale: A Case Study,” Science Editor 43, no. 3 (September 1, 2020): 72–76, DOI: https://doi.org/10.36591/se-d-4303-72 (accessed 21 July 2022).

[B12] Amber E. Budden et al., “Double-Blind Review Favours Increased Representation of Female Authors,” Trends in Ecology & Evolution 23, no. 1 (January 1, 2008): 4–6, DOI: https://doi.org/10.1016/j.tree.2007.07.008; Stefanie K. Johnson and Jessica F. Kirk, “Dual-Anonymization Yields Promising Results for Reducing Gender Bias: A Naturalistic Field Experiment of Applications for Hubble Space Telescope Time,” Publications of the Astronomical Society of the Pacific 132, no. 1009 (18 February, 2020): 034503, DOI: https://doi.org/10.1088/1538-3873/ab6ce0 (accessed 21 July 2022); Andrew Tomkins, Min Zhang, and William D. Heavlin, “Single versus Double Blind Reviewing at WSDM 2017,” ArXiv:1702.00502 [Cs] (October 2017), https://arxiv.org/abs/1702.00502 (accessed 15 August 2022).

[B13] Dakota Murray et al., “Author-Reviewer Homophily in Peer Review,” bioRxiv 400515, August 4, 2019, DOI: https://doi.org/10.1101/400515 (accessed 21 July 2022); Johnson and Kirk, “Dual-Anonymization Yields Promising Results for Reducing Gender Bias: A Naturalistic Field Experiment of Applications for Hubble Space Telescope Time.”; Tomkins, Zhang, and Heavlin, “Single versus Double Blind Reviewing at WSDM 2017.”

[B14] Mengyi Sun, Jainabou Barry Danfa, and Misha Teplitskiy, “Does Double-Blind Peer Review Reduce Bias? Evidence from a Top Computer Science Conference,” Journal of the Association for Information Science and Technology 73, no.6 (October 12, 2021): 811–819, DOI: https://doi.org/10.1002/asi.24582 (accessed 21 July 2021).

[B15] Andrew Tomkins, Min Zhang, and William D. Heavlin, “Reviewer bias in single- versus double-blind peer review,” PNAS, 114, no. 48 (October 10, 2017): 12708-12713, DOI: https://doi.org/10.1073/pnas.1707323114 (accessed 21 July 2021).

[B16] “Single-Blind Vs. Double-Blind Peer Review,” Enago Academy, last updated March 16, 2022, https://www.enago.com/academy/double-blind-peer-review-for-better-or-for-worse/ (accessed 15 August 2022).

[B17] “IOP Publishing: Reviewer Experience Survey,” Institute of Physics Publishing.

[B18] Xinyao Zhou et al., “Conformal Screen Printed Graphene 4 × 4 Wideband MIMO Antenna on Flexible Substrate for 5G Communication and IoT Applications,” 2D Materials 8, no. 4 (August 20, 2021): 045021, DOI: https://doi.org/10.1088/2053-1583/ac1959 (accessed 21 July 2022).

[B19] Laura Simonite, “What Is Publons Transparent Peer Review?” Web of Science, modified August 1, 2019, https://publons.freshdesk.com/support/solutions/articles/12000060026-what-is-publons-transparent-peer-review- (accessed 21 July 2022).