'At-risk articles': the imperative to recover lost science

Jeanette Hatherill

Opinion Pieces

'At-risk articles': the imperative to recover lost science

Authors

Jeanette Hatherill

Abstract

Deceptive publishers have been discussed and written about from a multitude of perspectives and in a variety of disciplines, but scant attention has been devoted to a particular aspect of the issue: How we as scholarly communities are dealing with the research that appears in these outlets. It is problematic that the question is not being addressed, as this research is at risk of being lost. It is at risk because articles that appear in deceptive publications are not indexed, so they are less visible, discoverable and citable. Additionally, they are not preserved and therefore likely to disappear should the publisher cease its activities or neglect to carry out basic maintenance on their archives and servers. Furthermore, it is particularly problematic because this lost science is potentially valuable. In this article, it is argued that, rather than continuing to risk the loss of this potentially important research by ignoring its existence, research disciplines should look at developments in open peer review and the increasing use of preprint servers for their potential to recover and reintegrate these at-risk articles into the scholarly record.

Keywords:

Year: 2020

Volume 33

Page/Article: 19

DOI: 10.1629/uksg.514

Submitted on Mar 24, 2020

Accepted on Jun 4, 2020

Published on Jun 24, 2020

Peer Reviewed

CC BY 4.0

Introduction

A significant amount of digital ink has been devoted to the question of deceptive publishers. It is an issue that affects all scholarly disciplines that engage with APC-based gold open access (OA), and as such elicits a broad range of commentaries, opinions, research pieces, checklists and, above all, warnings. They caution researchers to stay away, to avoid, to identify and to check lists. The good ones also ask researchers to critically engage with this subject. They question the use of the term ‘predatory’; reflect on a broken academic system that forces researchers to publish at increasingly onerous rates; interrogate the presence of these publishing outlets in the Global North/Western hegemony of for-profit academic publishing; and consider the limitations and racial overtones of ‘blacklisting’ and ‘whitelisting’. However, in all this rich literature, there is a question with which the research community is largely unengaged, which is: What do we do with the research that appears in these outlets?

Quantifying the problem

Quantifying the amount of research that appears in deceptive outlets is difficult for a number of reasons, one of which is that most of these analyses use a controversial basis for data collection and therefore any subsequent analysis of the data represents a particular view of the market. A full discussion on the various quantitative studies of the market is outside the scope of this article, but the interested reader should refer to the widely cited 2015 article by Shen and Björk and the response to their work by Crawford in Cites & Insights for an understanding of how market analyses can vary. Regardless, it may be helpful to put the available numbers in context. Shen and Björk estimated that, in 2014, 420,000 articles appeared in potentially deceptive journals. This number should be considered in light of available figures for the journal article market as a whole in the same period. In 2015, 55 million Crossref DOIs referred to journal articles, so their estimate represents approximately 0.76% of the market at the time. Crawford estimates the number at a much lower 120,000 articles, representing 0.22% of the market. In another study, Shaghei et al. looked at potentially questionable journal articles published by researchers at the University of Southern Denmark for the years 2015 and 2016. In their analysis of 6,851 articles, they found 31 possibly questionable articles, or a rate of 0.45%. There has not been a more recent large-scale analysis similar to that done by Shen and Björk, so current figures on the number of articles in potentially deceptive journals are not available. However, recent figures show that the total number of published journal articles has increased significantly since 2015, with 73 million DOIs pointing to articles in 2018. Additionally, the OA journal market has increased over this period as well, with 2.1 million articles searchable in the DOAJ in 2015, increasing to over 4.5 million in 2019. Given the overall increase in article output, it would be reasonable to assume that the volume of articles appearing in potentially deceptive venues would have increased since 2015.

Lost science

This increase is concerning for a variety of reasons, not least of which is that these articles are at risk of disappearing from the scholarly record. Clarke and Smith discuss the problem of ‘lost science’, noting that articles that appear in deceptive publications are not indexed in scholarly databases so they are less visible, discoverable and citable. Others have also noted the issue of discovery for these articles. Consider too that articles appearing in these outlets are not likely to be preserved. Should the publisher cease its activities or neglect to carry out basic maintenance on their archives and servers, articles that appear in these outlets are at risk of disappearing. This should be concerning to researchers and their communities because work that appears in these outlets is potentially valuable.

The question of peer review

In a recent analysis of 516 articles containing the term ‘predatory publishers’ and its variations published between 2010 and 2019, 48% of the articles (n = 248) could be broadly characterized as primarily informing or alerting researchers to the problem of deceptive publishing. Articles highlighting the potential research waste and scientific harm these publications represent cite the fact that the articles appearing in them have not been adequately peer-reviewed. A lack of peer review is a major recurring theme in the significant amount of work that has been done to characterize deceptive publishers. A quick search online will reveal hundreds of checklists, notices from scholarly publishing associations, commentaries and research guides that generally conclude that ‘such journals do not provide the peer review that is the hallmark of […] scholarly publishing’. One can also refer to stings and pranks performed by researchers and journalists that expose the lack of peer review in these outlets as further support for the contention that these outlets do not provide peer review. Despite criticism of the methodologies or ethics of these hoaxes, they do seem to paint a portrait of journals that are not overly concerned with scientific integrity.

Reports of these stings and pranks generally do not appear in peer-reviewed journals, but rather as new items, editorials, blog posts and social media conversations. In fact, it seems that there has been limited research done on peer review in questionable journals with only a few peer-reviewed studies to date on the topic. In four articles that touch on the subject, those by Cobey et al., Shaghaei et al. and Cohen et al. looked at author perception of peer review in questionable journals post-publication, while McCutcheon et al. also looked to validate the quality of peer review by performing a post-publication peer review exercise.

Cobey et al., who were specifically interested in the question of peer review, found that the majority of participants surveyed (83.3%) believed their article had undergone peer review and among those, most (79.7%) felt ‘the peer review was substantial and helpful’. Shaghei et al., who investigated author motivations for submitting to potentially dubious venues, reported that their interviewees had varying experiences with the submission and review processes, with some noting they had to ‘change their articles and provide additional information’ and others noting a ‘lack of feedback or only limited corrections of their articles’. Cohen et al., seeking to understand both author and editor motivations and perspectives on experiences publishing in deceptive outlets, found that 78.2% of authors recalled a peer review and that 68.0% had to submit revisions. Finally, McCutcheon et al. performed a structured post-publication peer review on a sample of articles from suspected ‘predatory’ journals and from relatively good journals and asked authors to rate their experiences with the reviewers and editors of these journals. Their results confirmed their predictions that raters would find that there would be more spelling, grammatical, statistical and methodological errors in the suspected ‘predatory’ journals and that raters would score the article’s literature review and ‘overall contribution to science’ as better in ‘non-predatory’ journals. With regard to author perception of the review experience, though, they found that authors did not give consistently poorer ratings to ‘predatory’ journals except for one criteria that asked them to consider whether the editors/reviewers were well-qualified.

These limited pieces of data on author perception of peer review in questionable outlets are pulled from larger studies covering broader questions and cannot be generalized, nor should they be. They are limited for a variety of reasons, mostly noted by the authors of the articles. The response rate to the Cobey et al. survey was quite low (14%) and both the recruitment e-mail and the survey instrument stated that the respondent was being contacted because the research team believed that the author had published in a possibly ‘predatory’ journal, thereby introducing a possible source of bias in the results. Shaghei et al.’s limited sample size of six interviewees, and the semi-structured nature of their interview script that did not specifically ask about the peer review process, limits the applicability of this study to the question of peer review processes in questionable outlets. Cohen et al. also noted that low response rates, among other factors, limit the generalizability of their results. Additionally, even for those authors that reported experiencing peer review, the rigour and appropriateness of the review cannot be known. McCutcheon et al. give particular attention in their discussion to the potential for cognitive dissonance on the part of authors when evaluating the quality of peer review and the overall speed and their satisfaction with the publication process.

Recently, Grudniewicz, Moher, Cobey et al. described a consensus statement on ‘predatory journals’ that was reached as part of the work of a research programme based out of the Ottawa Hospital Research Institute’s Journalology group. In this article they state that they chose to omit the question of peer review from their definition because it was too subjective to include. It is unfortunate the group left this element out of the definition given the limited generalizability of the studies that have looked at this question. When considering the question of peer review, one should weigh the amount of available work that cites a lack of peer review as a defining characteristic of these publications against the limited studies to the contrary. After all, this research is not labelled ‘a significant threat to science’ only because a researcher was misled about made-up impact factors or oblique pricing. The threat is implied because the research has not been vetted.

Reframing

It is important that we reconsider the narrative around the research that appears in these deceptive outlets. There is a considerable amount of stigma surrounding this topic and this appears to have at least some impact on the research in the field. Cohen et al. note that ‘from provided comments, there is clearly a stigma around being associated with a predatory journal for both authors and editors’. Recall too that Cobey et al. noted the presumption of predation in their survey instrument as a limitation in their research.

Let us shift our thinking to remove the stigma around the research that appears in these outlets and proceed with the assumption that at least some of this research has not been peer reviewed. Let us also take into account reports in the literature of authors who see their work ‘published’ without consent, notice, fee payment, nor copyright agreement. Without these hall-marks of legitimate publishing, must we consider all this research as formally published? Or can at least some of it be reframed as not yet validated by the scholarly research community?

The San Francisco Declaration on Research Assessment (DORA) stresses ‘the need to assess research on its own merits rather than on the basis of the journal in which the research is published’. While McCutcheon et al. found in their study that overall the quality of articles in ‘predatory’ outlets was of lower quality, they were struck by the variability in quality of the articles in these outlets. While the majority were of poorer quality, two received high marks and were misidentified as coming from a ‘predatory’ outlet. In another post-publication peer review study in nursing, Oermann et al. rated the overall quality of 358 articles in presumed ‘predatory’ outlets and found that while 47.7% were considered of poor quality, 47.2% were rated as average, and 3.7% were considered excellent. Should we continue then to discount this research completely despite the fact that there is at least some evidence that quality research does appear in these outlets? If we reframe our thinking about this research and consider that it has not yet been validated, and is therefore potentially valuable, should we continue as scholarly communities to ignore or ‘write off’ this research? Or, should we have a serious conversation about what we do with the research that appears in these outlets?

Options

As a scholarly communication librarian, I advise researchers often on the topic of deceptive publications and, for those unfortunate few who find their work ‘published’ by a deceptive publisher, the question of ‘what now?’ inevitably arises. For those authors who inadvertently find their work in deceptive publication outlets, there exists scant advice on the question of what to do next. Out of the 516 articles referred to previously on ‘predatory publishing’, only 8% (n = 40) deal with the question from a post-publication standpoint. Of these, 16 look specifically at the question of what to do next regarding the article itself, while 24 examine other post-publication issues such as the impact on career progression, considerations for future knowledge synthesis, or examine ethical or legal perspectives.

In this literature, articles by Balehegen, Dadkhah, and Memon deal with the question of what to do next most directly. They all suggest that once an author becomes aware of the nature of the publication outlet, they should withdraw their paper. There appears to be consensus for this course of action, with several other articles and publication guidelines recommending it. This recommendation comes despite noting that this may involve complicated and lengthy correspondence or that the journal may demand retraction fees (which they stress should not be paid). Dadkhah and Harris propose that after the article is retracted, the author could submit it to a new journal with a supporting explanation to the editor of the new journal. A case report posted to the Committee on Publication Ethics (COPE) supports this, though under very specific circumstances. The World Association of Medical Ethics (WAME) guidance on the topic suggests that ‘authors whose legitimate research was published in predatory journals should have a mechanism for submitting […] to a legitimate peer-reviewed journal’. However, this solution is not universally accepted, and is even rejected in COPE’s discussion document on predatory publishing. One suspects that the lack of discussion on what to do with research that appears in deceptive outlets and the stigma surrounding this topic generally makes finding a consensus difficult, leaving authors without much guidance for this potentially valuable and not yet validated research. In fact, it seems that researchers cannot even agree on whether or how to list these publications on an academic CV.

The solution

Given that some studies into author motivations to submit to these journals seem to suggest that authors are frustrated by the current academic publishing system, a solution that proposes authors submit to a new journal may not find much traction. The current approach to ignore or write off this research also falls short of a satisfactory solution. These authors appear already committed to OA, so there is an opportunity to explore other advances in scholarly communication to provide them a solution to the question of what to do next when they discover the deceptive nature of the publication outlet. I therefore propose that an author could retract or withdraw the article, acknowledge its ‘prior publication’ and submit it to a preprint server to make it available for open peer review. The feasibility of this solution will be discussed in the following sections.

Open peer review and preprint servers

Open peer review, like predatory publishing, does not have a standardized definition. However, in a systematic review, Ross-Hellauer proposes that open peer review can be seen as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of open science. They propose seven main traits which may or may not be present in a particular system of open peer review and that can be combined in any number of ways. Similarly, the preprint and preprint server landscape is equally complex and evolving. In their recent review of this landscape, Ernesto Galbán Rodríguez identifies 20 prominent preprint servers and proposes five non-exclusive categories to define them. Given the complexities of both open peer review and preprint servers, this paper does not purport to provide a single specific recommendation for authors, but rather imagines that an individual solution will depend on the author’s experience with the original publication venue and their requirements for validation and preservation.

If an author contends that inadequate or no peer review was performed and therefore their article is analogous to a preprint, they could post their retracted article to a preprint server in their discipline. This solution is dependent on the community reframing these articles as non-peer-reviewed preprints. Currently, most preprint servers, such as BioRxiv, ChemRxiv and Preprints.org, state that the material posted should not have been previously published. However, these policies are set by advisory boards and groups who manage and set policies for these platforms and they could be revised, and guidelines established. Most of these platforms perform a cursory review of any submitted articles, therefore the author could submit their work with an acknowledgement of its appearance elsewhere and an explanation or supporting evidence of the lack of peer review. For example, they could provide their submitted manuscript and the published version to demonstrate that there is no discernible difference between the two. If the advisory boards of preprint servers were to accept the proposed reframing of these articles and shift their current policies to allow for these postings, the risk of losing this science would be mitigated since most preprint servers have robust preservation policies. For example, the Centre for Open Science, which runs the infrastructure for OSF Preprints, SocAxiv and PhysArxiv, etc., has a policy and funding for back-up and preservation and BioRxiv archives its papers at Portico.

For the validation function, many preprint servers such as those listed above also allow for open commenting or annotations, so they could support a form of open peer review. For those that do not support open peer review, another feature could be employed that Ross-Tellauer characterizes as ‘open platform’ or ‘decoupled peer review’ whereby a different organizational entity is employed for the peer review function. Two platforms, Pubpeer and ScienceOpen, use a DOI look-up tool in order to use their commenting features, so an author could actively solicit reviews from individuals in their fields using the DOI provided by their preprint server.

Copyright and ethics

In order to fully consider the viability of this proposal, one must also consider questions of both copyright and publication ethics. In the case report on the COPE Forum alluded to in the previous section, the analysis of the appropriateness of retracting and republishing hinges on whether or not a copyright transfer agreement has been signed. This is a correct analysis, and authors should examine any publishing contract for copyright transfer, particularly given the dubious nature of these outlets. However, one should not place so much importance on this particular aspect that it becomes a barrier to considering this solution. It is important to note that with most OA publications, the copyright remains with the author. But even if the journal holds the copyright, there is usually a Creative Commons licence attached to it and this is often true for even deceptive publishers. For example, OMICS, the dubious publisher at the centre of a US$50 million lawsuit brought by the US Federal Trade Commission for deceptive practices, states that all articles are available under a Creative Commons Attribution licence. As long as the article is under a Creative Commons licence that would permit the reposting of the article, and the place of first ‘publication’ is cited and acknowledged, there is no reason under a copyright framework that this would not be permissible.

When it comes to publication ethics, WAME, COPE and the International Committee of Medical Journal Editors (ICMJE) all address the question of duplicate publication in their various guidelines and forums. In these discussions, duplicate publication under an ethical framework is addressed from the standpoint of deception on the part of the author. If there is no deception on the part of the author and they are transparent about the initial ‘publication’, I would propose then that there is no ethical breach. As noted above, WAME suggests that there should be a mechanism for authors to submit to a legitimate journal, but it does not provide substantial guidance to journal editors on how to handle such requests. In the COPE discussion document, there are contradictory stances on the republishing of these articles, with one section stating ‘an article appearing in a fake journal cannot be submitted for publication in a legitimate journal, as that counts as attempted redundant/duplicate publication’ and the following section stating ‘occasionally, upon author explanation an editor will consider an article previously accepted by a fake journal. Generally, however, this is not the case’. Given their importance in establishing and maintaining ethical standards for publication, it is unfortunate that there is a lack of guidance and agreement on this topic. If the scholarly community were to accept that there needs to be a shift in the conversation surrounding articles appearing in deceptive outlets, WAME, COPE and ICJME should in response consider developing clear guidelines for authors and editors regarding the republishing of retracted articles and consider the inclusion of posting to preprint servers as part of this guidance.

Conclusion

As it stands currently, there is no way for authors to retract an article, acknowledge its prior ‘publication’ and submit it for open peer review on a preprint server. However, the limitation is social and political, not technical, and so I urge disciplinary communities and publication stakeholders to think about the question of what to do with the at-risk research in their field. Rather than dismissing it or ignoring it, researchers must reframe how they think of this potentially valuable and at-risk research and consider it as an issue the disciplinary community must try to solve.

Abbreviations and Acronyms

A list of the abbreviations and acronyms used in this and other Insights articles can be accessed here – click on the URL below and then select the ‘full list of industry A&As’ link: http://www.uksg.org/publications#aa

Competing interests

The author declares that they have no competing interests.

References

Stefan Eriksson and Gert Helgesson, “Time to stop talking about ‘predatory journals’,” Learned Publishing 31 (2018): 181–183, DOI: https://doi.org/10.1002/leap.1135 (accessed 29 April 2020).
Bo-Christer Björk, Sari Kanto-Karvonen and J. Tuomas Harviainen, “How Frequently are Articles in Predatory Open Access Journals Cited,” Preprint, submitted December 20, 2019, https://arxiv.org/pdf/1912.10228.pdf (accessed 29 April 2020).
Cenyu Shen and Bo-Christer Björk, “‘Predatory’ Open Access: A Longitudinal Study of Article Volumes and Market Characteristics,” BMC Medicine 13 (2015): 230, DOI: https://doi.org/10.1186/s12916-015-0469-2 (accessed 29 April 2020).
Walt Crawford, “PPPPredatory Article Counts: An Investigation,” Cites & Insights 16, no. 1 (2016): 2–10, http://citesandinsights.info/civ16i1.pdf (accessed 29 April 2020).
Shen and Björk, “‘Predatory’ Open Access”, 6.
Mark Ware and Michael Mabe, “The STM Report: An overview of scientific and scholarly journal publishing,” 4th ed., (2015): 27, http://www.stm-assoc.org/2015_02_20_STM_Report_2015.pdf (accessed 29 April 2020).
Walt Crawford, “Gray OA 2012–2016: Open Access Journals Beyond DOAJ,” Cites & Insights 17, no. 1 (2017): 30, https://cical.info/civ17i1.pdf (accessed 29 April 2020).
Najmeh Shaghaei et al., “Being a Deliberate Prey of a Predator: Researchers’ Thoughts after having Published in a Predatory Journal,” Liber Quarterly 28, no. 1 (2018): 8, DOI: https://doi.org/10.18352/lq.10259 (accessed 29 April 2020).
Rob Johnson, Anthony Watkinson and Michael Mabe, “The STM Report: An overview of scientific and scholarly journal publishing,” 5th ed. (2018): 25, https://www.stm-assoc.org/2018_10_04_STM_Report_2018.pdf (accessed 29 April 2020).
Heather Morrison, “Dramatic Growth of Open Access December 2015,” The Imaginary Journal of Poetic Economics (blog), December 31, 2015, https://poeticeconomics.blogspot.com/2015/12/dramatic-growth-of-open-access-december.html (accessed 29 April 2020).
Heather Morrison, “Dramatic Growth of Open Access Dec. 31, 2019,” Dataset available on Scholars Portal Dataverse, V1. (2020), https://doi.org/10.5683/SP2/CHLOKU (accessed 29 April 2020).
Jocalyn Clark and Richard Smith, “Firm action needed on predatory journals,” BMJ 350 (2015): h210, DOI: https://doi.org/10.1136/bmj.h210 (accessed 29 April 2020).
Andrew J Cohen et al., “Perspectives From Authors and Editors in the Biomedical Disciplines on Predatory Journals: Survey Study,” Journal of Medical Internet Research 21, no. 8 (2019): e13769, DOI: https://doi.org/10.2196/13769 (accessed 29 April 2020); Mehdi Dadkhah, “What can authors do for the papers they published in predatory journals?,” Polish Archives of Internal Medicine, 126, no. 7–8 (2016): 7–8, DOI: https://doi.org/10.20452/pamw.3485 (accessed 29 April 2020); Dalmeet Singh Chawla, “Predatory-journal papers have little scientific impact,” Nature News, (2020), https://www.nature.com/articles/d41586-020-00031-6 (accessed 29 April 2020).
Jeanette Hatherill, “At-Risk Articles: Brief Literature Analysis,” Report (2020) http://hdl.handle.net/10393/40276 (accessed 29 April 2020).
Christine Laine and Margaret A. Winker, “Identifying Predatory or Pseudo-Journals,” World Association of Medical Editors. February 15, 2017, https://www.wame.org/identifying-predatory-or-pseudo-journals (accessed 29 April 2020).
Joseph Stromberg, “A reporter published a fake study to expose how terrible some scientific journals are,” Vox (2014), https://www.vox.com/2014/4/24/5647106/a-reporter-published-a-fake-study-to-expose-how-terrible-some (accessed 29 April 2020).
Mike Taylor, “Anti-tutorial: how to design and execute a really bad study,” Sauropod Vertebra Picture of the Week (blog), October 7, 2013, https://svpow.com/2013/10/07/anti-tutorial-how-to-design-and-execute-a-really-bad-study/ (accessed 29 April 2020); Gunther Eysenbach, “Unscientific spoof paper accepted by 157 “black sheep” open access journals – but the Bohannon study has severe flaws itself,” Gunther Eysenbach’s random research rants (blog), October 5, 2013, http://gunther-eysenbach.blogspot.com/2013/10/unscientific-spoof-paper-accepted-by.html (accessed 29 April 2020).
Kelly D Cobey et al., “Knowledge and motivations of researchers publishing in presumed predatory journals: a survey,” BMJ Open 9 (2019): e026516, DOI: https://doi.org/10.1136/bmjopen-2018-026516 (accessed 29 April 2020); Shaghaei et al., “Being a Deliberate Prey of a Predator,” p1–17; Cohen et al., “Perspectives From Authors and Editors”, e13769; Lynn E. McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” North American Journal of Psychology 18, No. 3 (2016): 427–440.
Cobey et al. “Knowledge and motivations of researchers,” 4.
Shaghei et al. “Being a Deliberate Prey of a Predator,” 10.
Cohen et al., “Perspectives From Authors and Editors,” e13769.
McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 434.
McCutcheon et al., 435.
Cobey et al. “Knowledge and motivations of researchers,” 8
Cohen et al., “Perspectives From Authors and Editors,” e13769
McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 435–436.
Agnes Grudniewicz et al., “Predatory journals: no definition, no defence,” Nature 576 (2019): 210–212, DOI: https://doi.org/10.1038/d41586-019-03759-y (accessed 29 April 2020).
Clare Fiala, Bryant Lim and Eleftherios P. Diamandis, “The growing problem of predatory publishing: a case report,” Clinical Chemistry and Laboratory Medicine 58, no. 2 (2019): e51–e53, DOI: https://doi.org/10.1515/cclm-2019-0798 (accessed 29 April 2020).
Cohen et al., “Perspectives From Authors and Editors,” 8.
Cobey et al., “Knowledge and motivations,” 8.
Marjorie A. Bowman, John W. Saultz, and William R. Phillips, “Beware of Predatory Journals: A Caution from Editors of Three Family Medicine Journals,” The Journal of the American Board of Family Medicine 31, no. 5 (2018): 671–676, DOI: https://doi.org/10.3122/jabfm.2018.05.180197 (accessed 29 April 2020); Alan H. Chambers, “How I became easy prey,” Science 364, no. 6440 (2019): 602, DOI: https://doi.org/10.1126/science.364.6440.602 (accessed 29 April 2020); John P. Harris, “Hazards of predatory publication,” ANZ Journal of Surgery 88, no. 1–2 (2018): 9, DOI: https://doi.org/10.1111/ans.14255 (accessed 29 April 2020); Aamir Raoof Memon, “How to respond to and what to do for papers published in predatory journals?,” Science Editing 5, no. 2 (2018): 146–149 DOI: https://doi.org/10.6087/kcse.140 (accessed 29 April 2020); MD Witham and H Runcie, “Turning predator into prey – the problem of predatory journals,” Journal of the Royal College of Physicians of Edinburgh 47, no. 1 (2017): 3–4, DOI: https://doi.org/10.4997/JRCPE.2017.101 (accessed 29 April 2020).
“San Francisco Declaration on Research Assessment,” https://sfdora.org/read/ (accessed 29 April 2020).
McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 434.
Marilyn H. Oermann et al., “Quality of articles published in predatory nursing journals,” Nursing Outlook 66, no. 1 (2018): 4–10, DOI: https://doi.org/10.1016/j.outlook.2017.05.005 (accessed 29 April 2020).
COPE Forum, Case 16–22 “Withdrawal of accepted manuscript from predatory journal,” https://publicationethics.org/case/withdrawal-accepted-manuscript-predatory-journal (accessed 29 April 2020).
Hatherill, “At-Risk Articles,” 5.
Balehegn, “Increased Publication in Predatory Journals,” 97–100.
Dadkhah, “What can authors do…?,” 7–8.
Memon, “How to respond to and what to do,” 146–149.
Fiala, Lim and Diamandis, “The growing problem of predatory publishing,” e51–e53; H. Benjamin Harvey and Debra F. Weinstein, “Predatory Publishing: An Emerging Threat to the Medical Literature,” Academic Medicine 92, no. 2 (2017): 150–151, DOI: https://doi.org/10.1097/ACM.0000000000001521 (accessed 29 April 2020).
COPE Council, “COPE Discussion Document: Predatory Publishing,” (2019): 1–14, DOI: https://doi.org/10.24318/cope.2019.3.6 (accessed 29 April 2020).
Dadkhah, “What can authors do…?”, 7–8.
Harris, “Hazards of predatory publication,” 9.
COPE Forum, Case 16–22 “Withdrawal.”
Laine and Winker, “Identifying Predatory or Pseudo-Journals.”
Bowman, Saultz and Phillips, “Beware of Predatory Journals,” 673.
COPE Council, “COPE Discussion Document,” 9.
Jaime A. Teixeira da Silva and Panagiotis Tsigaris, “Academics must list all publications on their CV,” KOME – An International Journal of Pure Communication Inquiry 6, no. 1 (2018): 94–99, DOI:https://doi.org/10.17646/KOME.2018.16 (accessed 29 April 2020).
Cobey et al., “Knowledge and motivations,” 4–5; McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 436.
Tony Ross-Hellauer, “What is open peer review? A systematic review [version 2; peer review: 4 approved],” F1000Research 6 (2017) 588, DOI: https://doi.org/10.12688/f1000research.11369.2 (accessed 29 April 2020).
Ernesto Galbán Rodríguez, “Preprints and preprint servers as academic communication tools,” Revista Cubana de Información en Ciencias de la Salud 30, no. 1 (2019): 1–27, http://rcics.sld.cu/index.php/acimed/article/view/1324/802 (accessed 29 April 2020).
Open Science Framework FAQs, https://help.osf.io/hc/en-us/articles/360019737894-FAQs#Backup-Preservation-Policy (accessed 29 April 2020); https://help.osf.io/hc/en-us/articles/360019737894-FAQs#what-if-you-run-out-of-funding-what-happens-to-my-data (accessed 29 April 2020).
BioRxiv FAQs, https://www.biorxiv.org/about/FAQ (accessed 29 April 2020).
COPE Forum, Case 16–22 “Withdrawal.”
FTC Press Release, “FTC Charges Academic Journal Publisher OMICS Group Deceived Researchers,” August 16, 2016, https://www.ftc.gov/news-events/press-releases/2016/08/ftc-charges-academic-journal-publisher-omics-group-deceived (accessed 29 April 2020).
COPE Council, “COPE Discussion Document,” 8–9.

[B1] Stefan Eriksson and Gert Helgesson, “Time to stop talking about ‘predatory journals’,” Learned Publishing 31 (2018): 181–183, DOI: https://doi.org/10.1002/leap.1135 (accessed 29 April 2020).

[B2] Bo-Christer Björk, Sari Kanto-Karvonen and J. Tuomas Harviainen, “How Frequently are Articles in Predatory Open Access Journals Cited,” Preprint, submitted December 20, 2019, https://arxiv.org/pdf/1912.10228.pdf (accessed 29 April 2020).

[B3] Cenyu Shen and Bo-Christer Björk, “‘Predatory’ Open Access: A Longitudinal Study of Article Volumes and Market Characteristics,” BMC Medicine 13 (2015): 230, DOI: https://doi.org/10.1186/s12916-015-0469-2 (accessed 29 April 2020).

[B4] Walt Crawford, “PPPPredatory Article Counts: An Investigation,” Cites & Insights 16, no. 1 (2016): 2–10, http://citesandinsights.info/civ16i1.pdf (accessed 29 April 2020).

[B5] Shen and Björk, “‘Predatory’ Open Access”, 6.

[B6] Mark Ware and Michael Mabe, “The STM Report: An overview of scientific and scholarly journal publishing,” 4th ed., (2015): 27, http://www.stm-assoc.org/2015_02_20_STM_Report_2015.pdf (accessed 29 April 2020).

[B7] Walt Crawford, “Gray OA 2012–2016: Open Access Journals Beyond DOAJ,” Cites & Insights 17, no. 1 (2017): 30, https://cical.info/civ17i1.pdf (accessed 29 April 2020).

[B8] Najmeh Shaghaei et al., “Being a Deliberate Prey of a Predator: Researchers’ Thoughts after having Published in a Predatory Journal,” Liber Quarterly 28, no. 1 (2018): 8, DOI: https://doi.org/10.18352/lq.10259 (accessed 29 April 2020).

[B9] Rob Johnson, Anthony Watkinson and Michael Mabe, “The STM Report: An overview of scientific and scholarly journal publishing,” 5th ed. (2018): 25, https://www.stm-assoc.org/2018_10_04_STM_Report_2018.pdf (accessed 29 April 2020).

[B10] Heather Morrison, “Dramatic Growth of Open Access December 2015,” The Imaginary Journal of Poetic Economics (blog), December 31, 2015, https://poeticeconomics.blogspot.com/2015/12/dramatic-growth-of-open-access-december.html (accessed 29 April 2020).

[B11] Heather Morrison, “Dramatic Growth of Open Access Dec. 31, 2019,” Dataset available on Scholars Portal Dataverse, V1. (2020), https://doi.org/10.5683/SP2/CHLOKU (accessed 29 April 2020).

[B12] Jocalyn Clark and Richard Smith, “Firm action needed on predatory journals,” BMJ 350 (2015): h210, DOI: https://doi.org/10.1136/bmj.h210 (accessed 29 April 2020).

[B13] Andrew J Cohen et al., “Perspectives From Authors and Editors in the Biomedical Disciplines on Predatory Journals: Survey Study,” Journal of Medical Internet Research 21, no. 8 (2019): e13769, DOI: https://doi.org/10.2196/13769 (accessed 29 April 2020); Mehdi Dadkhah, “What can authors do for the papers they published in predatory journals?,” Polish Archives of Internal Medicine, 126, no. 7–8 (2016): 7–8, DOI: https://doi.org/10.20452/pamw.3485 (accessed 29 April 2020); Dalmeet Singh Chawla, “Predatory-journal papers have little scientific impact,” Nature News, (2020), https://www.nature.com/articles/d41586-020-00031-6 (accessed 29 April 2020).

[B14] Jeanette Hatherill, “At-Risk Articles: Brief Literature Analysis,” Report (2020) http://hdl.handle.net/10393/40276 (accessed 29 April 2020).

[B15] Christine Laine and Margaret A. Winker, “Identifying Predatory or Pseudo-Journals,” World Association of Medical Editors. February 15, 2017, https://www.wame.org/identifying-predatory-or-pseudo-journals (accessed 29 April 2020).

[B16] Joseph Stromberg, “A reporter published a fake study to expose how terrible some scientific journals are,” Vox (2014), https://www.vox.com/2014/4/24/5647106/a-reporter-published-a-fake-study-to-expose-how-terrible-some (accessed 29 April 2020).

[B17] Mike Taylor, “Anti-tutorial: how to design and execute a really bad study,” Sauropod Vertebra Picture of the Week (blog), October 7, 2013, https://svpow.com/2013/10/07/anti-tutorial-how-to-design-and-execute-a-really-bad-study/ (accessed 29 April 2020); Gunther Eysenbach, “Unscientific spoof paper accepted by 157 “black sheep” open access journals – but the Bohannon study has severe flaws itself,” Gunther Eysenbach’s random research rants (blog), October 5, 2013, http://gunther-eysenbach.blogspot.com/2013/10/unscientific-spoof-paper-accepted-by.html (accessed 29 April 2020).

[B18] Kelly D Cobey et al., “Knowledge and motivations of researchers publishing in presumed predatory journals: a survey,” BMJ Open 9 (2019): e026516, DOI: https://doi.org/10.1136/bmjopen-2018-026516 (accessed 29 April 2020); Shaghaei et al., “Being a Deliberate Prey of a Predator,” p1–17; Cohen et al., “Perspectives From Authors and Editors”, e13769; Lynn E. McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” North American Journal of Psychology 18, No. 3 (2016): 427–440.

[B19] Cobey et al. “Knowledge and motivations of researchers,” 4.

[B20] Shaghei et al. “Being a Deliberate Prey of a Predator,” 10.

[B21] Cohen et al., “Perspectives From Authors and Editors,” e13769.

[B22] McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 434.

[B23] McCutcheon et al., 435.

[B24] Cobey et al. “Knowledge and motivations of researchers,” 8

[B25] Cohen et al., “Perspectives From Authors and Editors,” e13769

[B26] McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 435–436.

[B27] Agnes Grudniewicz et al., “Predatory journals: no definition, no defence,” Nature 576 (2019): 210–212, DOI: https://doi.org/10.1038/d41586-019-03759-y (accessed 29 April 2020).

[B28] Clare Fiala, Bryant Lim and Eleftherios P. Diamandis, “The growing problem of predatory publishing: a case report,” Clinical Chemistry and Laboratory Medicine 58, no. 2 (2019): e51–e53, DOI: https://doi.org/10.1515/cclm-2019-0798 (accessed 29 April 2020).

[B29] Cohen et al., “Perspectives From Authors and Editors,” 8.

[B30] Cobey et al., “Knowledge and motivations,” 8.

[B31] Marjorie A. Bowman, John W. Saultz, and William R. Phillips, “Beware of Predatory Journals: A Caution from Editors of Three Family Medicine Journals,” The Journal of the American Board of Family Medicine 31, no. 5 (2018): 671–676, DOI: https://doi.org/10.3122/jabfm.2018.05.180197 (accessed 29 April 2020); Alan H. Chambers, “How I became easy prey,” Science 364, no. 6440 (2019): 602, DOI: https://doi.org/10.1126/science.364.6440.602 (accessed 29 April 2020); John P. Harris, “Hazards of predatory publication,” ANZ Journal of Surgery 88, no. 1–2 (2018): 9, DOI: https://doi.org/10.1111/ans.14255 (accessed 29 April 2020); Aamir Raoof Memon, “How to respond to and what to do for papers published in predatory journals?,” Science Editing 5, no. 2 (2018): 146–149 DOI: https://doi.org/10.6087/kcse.140 (accessed 29 April 2020); MD Witham and H Runcie, “Turning predator into prey – the problem of predatory journals,” Journal of the Royal College of Physicians of Edinburgh 47, no. 1 (2017): 3–4, DOI: https://doi.org/10.4997/JRCPE.2017.101 (accessed 29 April 2020).

[B32] “San Francisco Declaration on Research Assessment,” https://sfdora.org/read/ (accessed 29 April 2020).

[B33] McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 434.

[B34] Marilyn H. Oermann et al., “Quality of articles published in predatory nursing journals,” Nursing Outlook 66, no. 1 (2018): 4–10, DOI: https://doi.org/10.1016/j.outlook.2017.05.005 (accessed 29 April 2020).

[B35] COPE Forum, Case 16–22 “Withdrawal of accepted manuscript from predatory journal,” https://publicationethics.org/case/withdrawal-accepted-manuscript-predatory-journal (accessed 29 April 2020).

[B36] Hatherill, “At-Risk Articles,” 5.

[B37] Balehegn, “Increased Publication in Predatory Journals,” 97–100.

[B38] Dadkhah, “What can authors do…?,” 7–8.

[B39] Memon, “How to respond to and what to do,” 146–149.

[B40] Fiala, Lim and Diamandis, “The growing problem of predatory publishing,” e51–e53; H. Benjamin Harvey and Debra F. Weinstein, “Predatory Publishing: An Emerging Threat to the Medical Literature,” Academic Medicine 92, no. 2 (2017): 150–151, DOI: https://doi.org/10.1097/ACM.0000000000001521 (accessed 29 April 2020).

[B41] COPE Council, “COPE Discussion Document: Predatory Publishing,” (2019): 1–14, DOI: https://doi.org/10.24318/cope.2019.3.6 (accessed 29 April 2020).

[B42] Dadkhah, “What can authors do…?”, 7–8.

[B43] Harris, “Hazards of predatory publication,” 9.

[B44] COPE Forum, Case 16–22 “Withdrawal.”

[B45] Laine and Winker, “Identifying Predatory or Pseudo-Journals.”

[B46] Bowman, Saultz and Phillips, “Beware of Predatory Journals,” 673.

[B47] COPE Council, “COPE Discussion Document,” 9.

[B48] Jaime A. Teixeira da Silva and Panagiotis Tsigaris, “Academics must list all publications on their CV,” KOME – An International Journal of Pure Communication Inquiry 6, no. 1 (2018): 94–99, DOI:https://doi.org/10.17646/KOME.2018.16 (accessed 29 April 2020).

[B49] Cobey et al., “Knowledge and motivations,” 4–5; McCutcheon et al., “How Questionable Are Predatory Social Science Journals?,” 436.

[B50] Tony Ross-Hellauer, “What is open peer review? A systematic review [version 2; peer review: 4 approved],” F1000Research 6 (2017) 588, DOI: https://doi.org/10.12688/f1000research.11369.2 (accessed 29 April 2020).

[B51] Ernesto Galbán Rodríguez, “Preprints and preprint servers as academic communication tools,” Revista Cubana de Información en Ciencias de la Salud 30, no. 1 (2019): 1–27, http://rcics.sld.cu/index.php/acimed/article/view/1324/802 (accessed 29 April 2020).

[B52] Open Science Framework FAQs, https://help.osf.io/hc/en-us/articles/360019737894-FAQs#Backup-Preservation-Policy (accessed 29 April 2020); https://help.osf.io/hc/en-us/articles/360019737894-FAQs#what-if-you-run-out-of-funding-what-happens-to-my-data (accessed 29 April 2020).

[B53] BioRxiv FAQs, https://www.biorxiv.org/about/FAQ (accessed 29 April 2020).

[B54] COPE Forum, Case 16–22 “Withdrawal.”

[B55] FTC Press Release, “FTC Charges Academic Journal Publisher OMICS Group Deceived Researchers,” August 16, 2016, https://www.ftc.gov/news-events/press-releases/2016/08/ftc-charges-academic-journal-publisher-omics-group-deceived (accessed 29 April 2020).

[B56] COPE Council, “COPE Discussion Document,” 8–9.