Many years ago I happily received an acceptance notice for a submitted article, but I was quite taken aback; the journal accepted the article without revisions. Due to my surprise, I requested the editor send along referee reports for my examination, but I never received them. To this day I remain dumbfounded. Did the editor send the article out for review? Did the referees have no comments? (Have you ever met an academic with nothing to say?) I certainly did not write the perfect article or pose a flawless argument, so what happened? When the editor failed to respond, I dropped the issue. But the experience continued to nag at me, and consequently further shaped my scholarly inquiry. For the next five years I continued my scholarship, publishing, and service work to focus on open peer review, and achieved my tenure goal. Today peer-review practices (and how to open them up) are positioned front and center of my research.
During my 2018–2019 sabbatical leave I did a lot of listening to others’ experiences with peer review. My project, Stories of Open, an interview-based study to discover individual human experiences of peer review in Library and Information Science, uncovered narratives from ten participants that resonated with my own.1 I heard from authors who experienced frustration with peer-review processes. I heard from referees whose managing editors did not keep them in the loop as to final editorial decisions, or who decried a lack of guidance when it came to performing the reviews themselves. On the other hand, I heard stories from journal editors working to develop more transparent peer-review processes and policies, as well as their markedly human-centered communication practices with authors, referees and editorial board members.
I would like to be clear; I research and am a staunch advocate of open peer review, a peer-review process in which authors and referees know the others’ identities, among other traits.2 Based on my many conversations, one thing is apparent: open peer review will never succeed unless publications can move from transparency of peer-review process to one of praxis. Praxis is conscious action. It is ‘…the application of a theory or philosophy to a practical political, social, etc., activity or programme.’3 Transparency praxis puts front and center that transparency is a value informing the whys and hows of scholarly publishing processes, instead of relegating them as items on a to-do list.
In their report Untangling Academic Publishing, Aileen Fyfe et al.4 remind us how peer review first developed and how scholars of the 18th and 19th centuries practiced it in their communities, yet they also detail how the term and practice became co-opted by commercial publishers as scholarly publishing moved from its basis in scholarly communities to the proprietary sector since the 1960s and 1970s. In essence, peer review moved from a collective, community-oriented process to an ‘individual responsibility’ within the commercialization of scholarly publishing. Despite the equity issues endemic to historical learned societies (made up of well-to-do men of Western European descent), their work was collective. Community members understood the vetting, reviewing and publishing processes.
This brings us to reflect on the present. As technology has progressed, commercialization efforts in scholarly publishing have continued, including around peer-review processes. Journal management systems offer peer-review solutions, and companies attempt to develop networks around peer review, which they often mask as ‘free’ services. Publons, a service from Clarivate Analytics’ Web of Science Group, boasts a community where scholars can track (and publish) their refereeing efforts. Similarly, commercialization calls into question the authenticity of the relationship between an open publishing ethos and for-profit entities. For example, F1000Research, a mega-journal utilizing a completely transparent peer-review process,5 is an entirely commercial venture. The immersion of commercial interests into peer review lends itself to using transparent practices as a marketing tool, a checkbox of product functionality, not a value driving peer-review decisions and practices.
In the rest of this article, I use experience-based narratives from my Stories of Open project to argue why and how transparency-based values and a praxis thereof should overtake perfunctory transparency practices. Specifically, we should base communication practices in values of transparency, and the scholarly community must undertake efforts to better train its members in peer-reviewing tasks. This is where we most often fail: communication is murky and training is inadequate (if it exists at all). Individual names used in this article are adopted pseudonyms for interview participants, and their narratives have been lightly edited for clarity.
Jessica, an early career researcher (ECR), shared her frustration with an article rejection. For her, communication from the journal editor was lacking, leaving her and her co-author with many unanswered questions:
‘So we just kind of walked away and said, “Well we’re going to try and submit it elsewhere, thanks so much for your time.” … but we’re like, “should we have done something differently?” So we were kind of dumbfounded on how to move forward and we’d already spent a couple months just poking around, seeing what are the better journals. I sent it to one of my mentors and asked, “What do you think?” and then eventually we submitted it elsewhere…
‘So I think that there was just a lack of mentorship with the rejection. I think rejection is part of this process; it’s to be expected. Particularly with something where you’re trying to identify what’s the right venue for this. But as early career researchers we just had no idea what process it went through and where to go from there…
‘This is something no one teaches you as an early career researcher, you’re just like, “Alright I’m just going to throw something at the wall and see if it sticks.” And I think the thing that was frustrating to me, is my co-author and I had no idea what the process was. Did they send that out to reviewers and the reviewers gave them feedback, and if so could we see that feedback? Or was that just an editorial decision made (similar to desk review)? What would have made it stronger?’
The Committee on Publication Ethics (COPE), a leader in scholarly publishing, offers guidance and best practices to editors and authors, yet their work can only have so much impact. Their Principles of Transparency and Best Practice in Scholarly Publishing6 include peer review, but its instruction falls short. It would not have helped Jessica or me understand our experiences. COPE’s third principle, Peer Review Process, states: ‘Journal content must be clearly marked as whether peer reviewed or not. Peer review is defined as obtaining advice on individual manuscripts from reviewers expert in the field who are not part of the journal’s editorial staff. This process, as well as any policies related to the journal’s peer review procedures, shall be clearly described on the journal website, including the method of peer review used. Journal websites should not guarantee manuscript acceptance or very short peer review times.’7
Transparency of the peer-review process is just as much about clear and consistent communication as it is about posting information to a journal website. But it is not just authors like Jessica who may experience a lack of clear communication; poor communication also hinders refereeing work. Alma shared her experience:
‘So I signed up to be a reviewer … I received an article and when I received it … I was communicating with the editor to try and get an extension, I wasn’t hearing back so I just kind of opted to do it. I mean I was definitely frustrated when he didn’t get back to me about giving me a deadline … It also felt like a burden because it came at a time when, at the beginning of the quarter, where I had this very heavy teaching load and a lot of other responsibilities that – my work is usually skewed so that the beginning of every quarter is really exhausting – and I had to get it turned around in that time.’
But sometimes the lack of communication does not even seem to stem from a person, as in Alma’s case; rather, it comes from an automated system. Nancy dubbed this the ‘Elsevier robot’:
‘I was approached by an Elsevier journal at some point last year … It’s like the machine contacted me. I don’t know what it is but [I keep] getting these automated [e-mails]. Somebody must have put me in as a reviewer that would be potentially interested, but I never got an e-mail from the editor or anything. It was always this automated [e-mail] … and I’ve been getting them periodically, two or three times a year … The first one … it was like “by the way you haven’t accepted this review yet”. I was like “no and I will not, Elsevier robot, accept this review. I’m not even going to respond to the e-mail” … I mean it’s Elsevier so that’s already like “no I would never” … I’m like, “you know, you can’t just like feed me into an algorithm and expect that I’m going to do a review for [your journal]”. It was weird. That’s the first time that has ever happened to me; it was even more impersonal.’
In addition to clear communication, referees often seek guidance in order to provide helpful manuscript reviews. Stephanie relayed to me an experience she had reviewing for a high-impact journal:
‘The first peer review I ever got was I think within the first year of my job here … And so I got a peer review from like the big journal … and there was zero guidance. And I just had massive imposter syndrome. I mean it was on [my area of expertise] but it wasn’t a very eye-opening experience in terms of “this is what peer review actually looks like for some people …” What I got was three very general questions in text boxes to enter in.
‘And there really wasn’t particular guidance. And recognizing that, so graduate students in this discipline, often will get trained on peer review because they’re a postdoc or the Principle Investigator will get peer-review requests and they will do it with the graduate student, essentially to train them on the process. So I expected more, I guess, of a rubric … So I think probably what I would have expected were more pointed questions about the research … did the methods align with the research questions? Were the results reported clearly? Were the conclusions and the discussion within the realm of reasonable based on the data that was presented? That kind of stuff.’
We do not learn to referee by osmosis. Training in peer review certainly varies by disciplinary culture, but many assume that scholars simply know how to referee, even if we’ve never been taught. Researchers Sciullo and Duncan, corroborate my thinking. They offer referee training as one of five suggestions to improve peer review in the humanities.8 So where is the training? How can journals support reviewers to be successful? Expanding this line of questioning further: How do editors learn to do their work? How are they ‘onboarded’?
Vendors sometimes offer webcasts and training sessions. For example, Scholastica, a company providing journal management platforms and services, offers a free training course for new editors titled ‘Guide to Managing Peer Reviewers.’9 Publons Academy claims participants can become a ‘master of peer review’ by completing their training course.10 It is not a publication’s role to train referees and editors, nor is it the role of the commercial publishing industry. Rather, our refereeing work should, as it was historically practiced, occur in scholarly societies and communities. Certainly, for-profit companies are part of the scholarly publishing community, but they do not solely comprise it. Individual scholars engaging in research and in the societies in which they find community of practice, outnumber the companies. Yet in a commercialized publishing market, they are overpowered. (To say nothing of scholars’ donated labor.) Today it should remain the role of scholarly societies and communities – those positioned to embrace a transparency praxis – to train referees. This training may occur during a scholar’s formal education, or continuing education offered within scholarly societies. Too often we assume that refereeing and editorial skills are learned in formal educational settings, yet I heard numerous stories of folks who sought guidance and received none.
While it is not a journal or publication’s responsibility to provide training, these entities can support refereeing tasks by adopting transparent peer-review processes guided by their transparency praxis. In addition to the standards outlined by COPE, journals should offer referee guidelines and expectations: a contract of sorts. Clear communication should provide for referees – and the authors whose works undergo review – the following: why have they been selected to referee? What should be considered during referee reports (and what should not)? What are the expected timelines and communication practices with the managing editor? How does the editor make decisions on a manuscript from referee reports and recommendations? Lastly, publishing decisions should be communicated to referees.
Stephanie compared her first experience serving as a referee with her most recent referee request from a Frontiers journal using an open peer-review process11:
‘So I got the request and read through the abstract which is … the only information that I got to make the decision [whether or not to accept the review task]. And they had some interesting questions that I hadn’t seen before about “does this fall within your expertise” which I think is an excellent question to ask explicitly. And a link to the guidelines, which are fairly different. So I had to process through what that might look like and their process for publishing, because once you make your recommendation, as long as you don’t decide to withdraw, basically what you say about it is out there. So you can, I think, recommend the article and sort of stand behind it as a champion a little bit, or you can – there’s another sort of middle path and I can’t remember quite what the language is there – but I think it’s more of a revise and resubmit option. But it’s interesting. I haven’t really delved into the details yet … I was given a link to their guidelines which are public so it was really helpful.’
As an advocate for open peer review, I see that one of the first steps we must take in any implementation is to tackle transparency of process – the rote, the checklists – and move it into a praxis-, theory- and values-informed action. In addition to falling in line with COPE’s guidelines and best practices, publications should ensure that there are transparent and clear communication practices, as well as explicit rights and responsibilities for all roles in the scholarly publishing process: authors, referees, editors and editorial board members. When transparency of process manifests as a praxis of transparency, when we perform purposeful practices in everyday business, not just put on paper, we will be positioning experiments with open peer review for success. Kurt, an editor, reflected on what it would take to implement open peer review:
‘… what I think you have to do is come up with something that works for you and then be very transparent about what those decisions are and where … if you submit your manuscript to this, this is going to be the process, these are who will look at it and you will know about whom, does that make sense? I think that it’s not so much what you do as coming up with something that works, being consistent, and then being transparent about what the process will be.’
One of the powerful things about open peer review is that is offers a more human-centered refereeing process. Nancy generalized her experiences with it:
‘I’ve had enough open peer review experiences now that I just think that the conversations that happen are more robust. The feedback is more valuable. It just feels better… I mean thinking about what you just said about humans, right? So the humans are there in open peer review and they’re named, actual people – as opposed to the OJS [Open Journal Systems] interface [where] the users send to and get things back out of.’
However, without a transparent peer-review praxis, this valuable, robust and human-centered scholarly discourse cannot come to fruition. If we want to support this kind of peer-review experience, we must make the move from transparent peer-review processes to a transparent peer-review praxis.
So how do we make this move? First, COPE could further clarify and expand their transparency of publishing statement to incorporate clarity of communication. Next, publications should explicitly communicate peer-review guidelines as well as the rights and responsibilities of authors, referees, editors and editorial board members. Additionally, publications should expect and conduct their business with clarity in all communication practices. Finally, scholarly societies and communities need to actively engage in training and education to support peer review. If we leave it to commercial publishers to fill this gap, the training and subsequent peer-review practices will become what the commercial industry desires: profits!, efficiency!, data mining!, not what scholarly communities want: human-to-human scholarly discourse.
I am grateful to those individuals who shared their experiences, and from whom I learned during my research. Additional thanks to Jill Emery for feedback on an early draft of this article, and the referee, whose comments helped me improve this work. A grant from the Portland State University Faculty Development Fund, in part, supported research presented in this article.
A list of the abbreviations and acronyms used in this and other Insights articles can be accessed here – click on the URL below and then select the ‘full list of industry A&As’ link: http://www.uksg.org/publications#aa
The author has declared no competing interests.
Emily Ford, “Defining and Characterizing Open Peer Review: A Review of the Literature,” Journal of Scholarly Publishing 44, no. 4 (July 2013): 311–26; DOI: https://doi.org/10.3138/jsp.44-4-001 (accessed 6 August 2019); Tony Ross-Hellauer, “What is open peer review? A systematic review,” F1000Research 6 (August 31, 2017): 588; DOI: https://doi.org/10.12688/f1000research.11369.2 (accessed 6 August 2019).
“Praxis, n.,” in OED Online (Oxford University Press), https://www.oed.com/view/Entry/149425 (accessed 6 August 2019).
Aileen Fyfe et al., “Untangling Academic Publishing: A history of the relationship between commercial interests, academic prestige, and the circulation of research,” (University of St. Andrews, May 2017); DOI: https://doi.org/10.5281/zenodo.546100 (accessed 6 August 2019).
Emily Ford, “Open Peer Review at Four STEM Journals: An Observational Overview,” F1000Research 4 (2015): 6–6; DOI: https://doi.org/10.12688/f1000research.6005.2 (accessed 6 August 2019).
“Principles of Transparency and Best Practice in Scholarly Publishing” Committee on Publication Ethics, https://publicationethics.org/files/Principles_of_Transparency_and_Best_Practice_in_Scholarly_Publishingv3.pdf (accessed 6 August 2019); DOI: https://doi.org/10.6087/kcse.148
Nick J. Sciullo and Mike Duncan, “Professionalizing Peer Review Suggestions for a More Ethical and Pedagogical Review Process,” Journal of Scholarly Publishing 50, no. 4 (July 2019): 248–64; DOI: https://doi.org/10.3138/jsp.50.4.02 (accessed 6 August 2019).
“New Editor Training Course: Guide to Managing Peer Reviewers,” https://blog.scholasticahq.com/post/journal-editor-course-managing-peer-reviewers (accessed 6 August 2019).
“Publons Academy,” Publons, http://publons.com/community/academy (accessed 6 August 2019).
“Frontiers|Peer Reviewed Articles – Open Access Journals,” Frontiers, https://www.frontiersin.org (accessed 6 August 2019).