KEY TAKEAWAYS

• A year after the launch of their ‘reviewed preprint’ model, the journal eLife has released their key findings.
• eLife report over 6,200 submissions, 2.5× faster time to publication, and no significant change in quality.

In January 2023, eLife made the radical decision to end the process of accepting or rejecting papers after peer review, in favour of publishing ‘reviewed preprints’. A year on, they have released their key findings.

What is the ‘reviewed preprint’ model?

In this model, all articles selected for peer review are published on the eLife website as a reviewed preprint alongside an eLife assessment, public reviews, and a response from the authors (if provided).

What are the key results?

In the first year, eLife report:

• over 6,200 submissions received and more than 1,300 reviewed preprints published
• over 2.5× faster time from submission to publication than the legacy model
• no significant change in the quality of submissions (based on ratings for significance and strength of evidence)
• quality of eLife assessments and public reviews rated highly by authors.

When the new model was launched, eLife reported that views across academic publishing were mixed, with concerns that:

• authors would not submit their work
• editors and reviewers would not want to be involved
• articles would be of low quality or only from researchers with the most confidence in their work.

However, a year on, eLife consider the reality to be much more encouraging, highlighting how:

• editors and reviewers have been able to focus on summarising the strengths and weaknesses of an article, with their views open for debate
• authors and reviewers have been able to provide exchange without fear of articles being rejected
• the majority of authors have revised their articles in response to reviewer comments, resulting in what eLife believe to be ‘better science all around’.

The majority of authors have revised their articles in response to reviewer comments, resulting in what eLife believe to be ‘better science all around’.

What’s next?

Going forward, eLife commit to continued evolution and adaptation. One proposal is to extend this approach to articles that may not typically be published by broad-interest journals, such as important negative or preliminary findings.

eLife welcome ideas to help them achieve these aims. They also encourage other publishers to adopt some aspects of their approach by making their software infrastructure freely available.

————————————————–

KEY TAKEAWAYS

• Single blinding in peer review may be subject to unconscious bias, putting authors from wealthier, English-speaking countries at an advantage.
• Double blinding can improve equity in peer review and even increase reviewer numbers, thus creating a more time-efficient process.

The shortfalls of the current peer review process have long been debated. Now, a real-life experiment conducted by a journal’s own editorial team leads them to call time on the traditional single-blind peer review system, in favour of a more equitable double-blind approach.

How does double-blind review improve equity?

Writing in a post on the LSE Impact Blog, Professor Charles Fox (Executive Editor of the journal in question, Functional Ecology, at the start of the experiment), explained his team’s conclusions that:

Authors from wealthy countries and those with higher levels of English language proficiency receive an advantage under current peer review processes.

In the study, around 3,700 papers submitted to the journal over a period of 3 years were randomly assigned to either single-blind (ie, only the reviewer was anonymous) or double-blind (ie, reviewers and authors were anonymous) peer review. The group found that:

• single blinding can be subject to positive biases relating to the country of origin of authors
• authors from wealthier, English-speaking countries received higher scores and were more likely to be invited to proceed to the revisions stage, when their identities were known
• no biases were identified in relation to gender.

As a result of the experiment, all peer review at Functional Ecology is now double blind.

What are the challenges with double-blind review?

Prof. Fox acknowledges that concerns exist around potential costs and limitations associated with double blinding, and that there is a generally held assumption that individuals would be less keen to review under such a system. The group, however, found the reverse to be true, with double blinding leading to increased reviewer numbers and a more time efficient process.

Issues can also arise in maintaining the anonymity of authors, for a number of reasons:

• Individuals can be recognisable to their peers because of their specialism, research methods, etc.
• Manuscripts may have been previously submitted to a preprint server or earlier data published as part of a longer-term trial.

Sixty percent of reviewers in the study stated that they knew, or suspected, the identity of authors despite anonymisation, and in 90% of these cases they were correct.

What is the best way forward?

While the study’s findings reinforce existing evidence that unconscious bias exists within peer review, single blinding is still standard practice for most journals. Some offer optional anonymisation (ie, authors can choose whether to be identified), but Prof. Fox argues that this does not go far enough. In an ‘opt in’ system, authors from more affluent countries – who are more likely to benefit from positive bias – would perhaps be unlikely to hide their identity.

Overall, Prof. Fox maintains that any potential challenges associated with double blinding do not outweigh the benefits of improved objectivity. He calls on journals to follow Functional Ecology’s example and make the switch to mandatory double-blind peer review.

————————————————–

KEY TAKEAWAYS

• Nature is giving authors the option to publish anonymised peer review reports alongside their article.
• Opening up the peer review process promotes transparency and could benefit the research community and general public.

Although an important tool for scientific progress and upholding rigorous standards of proof, peer review reports have typically remained confidential from the wider research community. In a bid to promote transparency in publishing, Nature is piloting a trial to give authors the opportunity to publish their discussions with referees.

In 2016, Nature Communications started offering authors the option to publish peer reviewers’ comments and rebuttal letters alongside their articles. Following a positive uptake in 2021 (approximately 70%), the journal has now gone a step further, announcing that peer review files will be published for all accepted research articles that were submitted after 1 November 2022. For Nature, who began piloting this option in February 2020, almost half (46%) of authors agreed to publish anonymised peer review reports in 2021, and indications in early 2022 suggest that this number is rising.

Almost half (46%) of authors agreed to publish anonymised peer review reports in Nature in 2021.

Nature strongly encourages researchers to consider publishing their exchanges with reviewers, citing the following benefits to the scientific research community and general public:

• promoting transparency
• preserving valuable scholarship
• providing insight into the peer review process to both early-career researchers and those who study peer review systems
• recognising the contributions of peer reviewers
• highlighting discussion on valid potential caveats or limitations of studies
• allowing readers to better critically assess the robustness of the study conclusions
• enabling authors to raise supportive arguments not suitable for integration within the article itself.

Given the promising results from the Nature’s pilot, the journal hopes that more authors will opt to publish their referee reports with a view to improve how science is represented.

—————————————————–

KEY TAKEAWAYS

• As preprint publications increase in popularity, mechanisms to encourage transparent public review are needed.
• The FAST principles provide a framework for use by authors, reviewers, and the wider community to foster engagement in preprint discussions.

The publication of preprint articles has gathered pace in recent years, accelerating rapidly during the COVID-19 pandemic. A key advantage of preprints is that they can be scrutinised by a diverse audience ahead of submission to traditional scholarly journals. Despite the benefits of public feedback, just 5–10% of preprint articles on bioRxiv and medRxiv receive publicly accessible comments, with many reviewers preferring to provide feedback privately.

To help facilitate rapid and constructive preprint feedback, ASAPbio established a Working Group to develop best-practice guidance for public commentary and engagement with open preprint discussions. In their guest post on The Scholarly Kitchen, Sandra Franco Iborra, Jessica Polka, and Iratxe Puebla of ASAPbio summarised the FAST principles for preprint feedback that were developed.

The 14 FAST principles are grouped into 4 central themes:

• Focussed: comments and feedback should focus on the scientific content and not the suitability for potential target journals.
• Appropriate: reviewers should reflect on their potential biases and engage in scientific discourse respectfully and with integrity.
• Specific: feedback should be candid, assess a study’s claims against the data presented, and be clear on whether issues identified are major or minor.
• Transparent: reviews should be as open and transparent as possible and credit any co-reviewers. Those not comfortable signing their review can disclose their background or expertise alongside their comments.

The authors highlight that the unique features of the FAST principles mean that they are relevant to all involved in feedback, including journals, authors, and the wider community. Importantly, they are not intended to replace the reviewer guidance already provided by traditional scholarly journals, but rather complement it, to facilitate communication between authors and peer-reviewers and help promote positive behaviours for peer-reviewers.

The authors hope that the FAST principles will contribute to a broader conversation on the review process, helping produce a more positive and diverse culture.

The authors propose that encouraging public review of preprint articles could help journals expand and diversify their reviewer pool by identifying junior researchers and those located across broader geographical regions.

There has already been a move by some journals to incorporate preprint reviews into their editorial processes. Both Review Commons and Peer Community In Registered Reports (PCI RR) provide journal-independent preprint review, which is accepted by several affiliated journals. The FAST principles could be used to support this process, defining the expectations for preprint reviews that will ultimately be acceptable to scholarly journals.

The authors hope that the FAST principles will contribute to a broader conversation on the review process, helping produce a more positive and diverse culture.

—————————————————–

KEY TAKEAWAYS

• Disclosing reviewer identities can improve peer review transparency, but its risks and benefits remain unclear.
• Signed peer review offers scholars an opportunity to claim credit for their work, but will not in itself remove bias from the process.

Despite the growing support for improving transparency in the peer review process, attitudes towards revealing reviewer identities remain mixed. In their article for the Scholarly Kitchen, Véronique Kiermer and Alison Mudditt explore the arguments for and against disclosing reviewer identities alongside published articles, and discuss learnings from publishers who have implemented different approaches to open review.

The benefits of open-identities peer review have been described in terms of:

• accountability: increasing the transparency of any competing interests
• credit: recognising reviewers’ contribution to this important academic activity
• quality: encouraging more thorough reviews.

However, Kiermer and Mudditt cautioned that the following risks should be considered:

• Bias: masking author and reviewer identities has traditionally been used to counteract biases in peer review.
• Impact on vulnerable researchers: early-stage researchers may be more susceptible to retaliatory actions such as negative subsequent peer reviews.
• Rigor and candour: researchers may be less willing to provide critical assessments or may trade positive reviews with other academics.
• Halo/horn effect: readers might overlay their perceptions of the credibility of the reviewer on the research article, instead of judging the research on its merit.

Various journals have adopted versions of open review. The BMJ identified a small improvement in constructive feedback following the release of reviewer identities; however, this was coupled with decreased willingness to review manuscripts. In contrast to the BMJ data, PeerJ reported that reviewers choosing to disclose their identity gave more subjective and positive reviews, which could suggest that candour is influenced by social pressures.

Nature journals offer their referees the option to be named on published papers (with authors’ permission), but restrict access to the peer review reports. Overall, 80% of Nature articles name at least one reviewer, which likely reflects a desire amongst reviewers to receive credit for their efforts.

Since 2019, authors publishing in PLOS journals have had the option to release their article’s peer review history, including peer reviewers’ names. Subsequently, in 22% of PLOS journal articles at least one reviewer has chosen to reveal their identity. PLOS also observe a strong desire for credit and have enabled reviewers to add certified proof of review to their ORCID records.

Kiermer and Mudditt believe that all stakeholders can benefit from greater transparency in peer review, although there is some ambiguity as to the value of naming individual reviewers, and a limited understanding of its impact. They stress that on its own, disclosure of reviewer identities is unlikely to overcome the bias inherent in the peer review process, and that greater diversity and representation among reviewers and editors is needed. Further exploring the impact of social identities could help reduce bias and increase trust and transparency in peer review.

—————————————————–

Women are systematically under-represented in academic publishing as authors, referees and editors. But does the peer review system play a role in the publication gender gap? According to a recent study published in Science Advances, manuscripts submitted or co-authored by women are generally not penalised in the peer review process, and those with all-women or cross-gender author lists actually have a higher probability of being published.

Professor Flaminio Squazzoni and colleagues analysed data from almost 350,000 manuscript submissions to 145 journals across biomedicine and health, physical sciences, life sciences, social sciences and humanities. Most of the submitting authors (75%) and referees (79%) were men, supporting previous research on the gender disparity in publications.

Three possible sources of bias were investigated:

• Editorial selection of referees: manuscripts with a higher proportion of women among the authors were more likely to be reviewed by women. The authors note that this may be an intentional journal preference, or could reflect gender disparity in expert referee roles.
• Referee recommendations: manuscripts authored by women received more positive reviews in some fields of research. This could reflect manuscript quality: the authors highlight previous studies suggesting that women invest more in their manuscripts to overcome expected editorial bias. For all but one of the research fields, women referees provided more positive recommendations than men.
• Editorial decisions: in some fields, manuscripts with a higher proportion of woman authors were more likely to be accepted, but there was no systematic bias against manuscripts submitted by women across the journals and disciplines analysed.

Overall, Professor Squazzoni and colleagues found that manuscripts submitted or co-authored by women were generally not penalised during the peer review process.

Manuscripts submitted or co-authored by women were generally not penalised during the peer review process.

However, these results do not mean that peer review and editorial processes are free from bias. Factors such as age, ethnicity, or institutional prestige could influence editorial processes and have gender implications. Moving forward, the authors suggest that greater gender diversity in editorial teams and referee pools could help promote inclusion and participation of women.

——————————————————–

——————————————————–

History has shaped the format of scientific publishing, with the process of manuscript submission to the journal of choice, peer review and acceptance (or rejection, and a repeat of the cycle) being the norm. Unfortunately, this process can result in significant delays to publication. In an article for Times Higher Education, Professor Hilal Lashuel and Benjamin Stecher propose a system that would turn this publishing model on its head.

Lashuel and Stecher suggest a role reversal, with authors sharing their research publicly and journal editors competing to publish it.

Under their proposal, the article would remain open post-publication, allowing authors to update their work as the research continues. Lashuel and Stecher acknowledge that the increasing popularity of preprints signals that a shift in this direction is starting to happen, but feel that there is more to do. They highlight the following:

• Articles should be indexed and retrievable via a single open database to allow all within the scientific community to provide feedback – although incentives and cultural change may be needed to encourage full engagement with this process.
• A dynamic publication system would enable authors to easily correct or improve their data, while tracking contributions could facilitate authors being rewarded for this typically unpublished work.
• Investment is required to deliver long-term sustainability of repositories.

Meanwhile, some journals have adopted processes which synergise with existing preprint activities. Some have looked to preprint servers to source content, while eLife is among those who have taken this relationship a step further. The journal has been trialling ‘Preprint Review’, where preprints posted on bioRxiv are reviewed and considered for publication in eLife in parallel. Under the initiative, review reports tailored to the interests of readers, rather than authors, are posted on bioRxiv via their Transparent Review in Preprints service, aiming to make the peer review process more valuable across the scientific community. With such initiatives and proposals being introduced, it is clear that the medical publishing landscape is changing – we look forward to seeing how it continues to evolve.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

——————————————————–

Summary by Jo Chapman PhD from Aspire Scientific

——————————————————–

With thanks to our sponsors, Aspire Scientific Ltd and NetworkPharma Ltd

In a recent Nature Index article, Dalmeet Singh Chawla highlights that between 2012 and 2016, more than 500 papers were retracted for compromised, rigged, or faked peer review, with some authors going as far as to completely fabricate their suggested experts. With such a risk for bias and misconduct why do many journals still accept, and often request, suggestions for peer reviewers?

Essentially, requesting suggested peer reviewers is a quick way to find experts who are best fitted to review the work, which is important as the delays and potential for editorial bias from peer review can be a source of frustration for researchers. Also, it is becoming increasingly difficult to recruit peer reviewers, at least partly due to a phenomenon known as ‘reviewer fatigue’.

As noted by Singh Chawla, journals must determine whether the benefits of allowing suggested peer reviewers outweigh the potential risks. Even top publishers have different lines in the sand when it comes to accepting suggestions for reviewers, with some not accepting them at all, others discriminating based on previous co-authorship or other criteria, and others relying on the Editor’s discretion.

The issue extends from publishing to grant proposals, with research from the Swiss National Science Foundation (SNSF) showing that reviewers recommended by grant applicants are four times more likely to give favourable feedback than those who are chosen at random. This finding led the SNSF to change their policy in 2016, to prevent applicants from recommending individuals to assess their funding applications.

At a time when the current process of peer review is being debated, publishers and funders alike need to weigh up the benefits and risks of accepting suggested reviewers to maintain their value and reputation in the scientific community.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

——————————————————–

Summary by Robyn Foster PhD from Aspire Scientific

——————————————————–

With thanks to our sponsors, Aspire Scientific Ltd and NetworkPharma Ltd

Firstly, the authors identify 7 key forms of open review:

1. Open identities: identities of authors and reviewers are known to one another;
2. Open reports: peer review comments are published alongside an article;
3. Open participation: involvement of the wider community;
4. Open interaction: discussions between authors and reviewers;
5. Open pre-review manuscripts: manuscripts are available prior to peer review;
6. Open final version commenting: the final version can still be reviewed and commented on;
7. Open platforms: the review is conducted by an organisation other than the publisher.

They go on to describe how each form of review is associated with both advantages and disadvantages. For example, ‘open identities’ may increase accountability, lead to a more thorough review and allow potential conflicts of interest to be identified, but could also discourage reviewers from voicing strong criticisms.

Regarding specific guidance, which was created in close consultation with a group of experts, Ross-Hellauer and Görögh advise editors to identify which elements of open review they wish to utilise and recommend that editors set clear goals outlining what they hope to achieve by implementing such systems. The authors highlight the need to consider the technological feasibility and cost implications of open peer review and suggest that new processes are made optional initially or introduced in a phased manner. Communication and engagement with the community, and identification of academics to advocate open peer review are also recommended. In addition, publishers are advised to evaluate performance, by measuring review quality and acceptance rates.

The authors note that open peer review processes are becoming adopted by a growing number of journals and it is their hope that these guidelines will ‘prove useful in setting expectations and guiding best-practice’.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

—————————————————–

Summary by Debbie Sherwood BSc from Aspire Scientific

——————————————————–

With thanks to our sponsors, Aspire Scientific Ltd and NetworkPharma Ltd

In the trial, as in the journal’s existing editorial process, papers are first assessed at the submission stage by a senior editor. If the work is considered to be of a high scientific standard or significance, authors will then be invited to submit the paper for external review. However, while only half of papers that are peer reviewed are accepted for publication in the journal’s current editorial system, in the trial all papers sent for external review will essentially be accepted for publication. Following peer review it will be the author’s decision as to how they respond to the editorial and reviewer feedback. This may mean performing additional work, editing the manuscript, responding to specific points in the rebuttal letter, or withdrawing the paper if serious flaws are identified. The editors will assess whether the authors have fully addressed the peer review issues or whether minor or major issues remain unresolved. Regardless of the outcome of this assessment, providing the author is happy to proceed, the article will be published alongside the reviewer reports, decision letter and author response. The editor’s assessment will also be published and will appear within the manuscript at the end of the abstract.

eLife hope this new approach will allow the journal to become a venue for critical and transparent research evaluation, as well as encouraging open and constructive dialogue between reviewers and authors and increasing the efficiency of peer review. They anticipate unpredictable reactions from the authors, editors and reviewers of the 300 submissions that will form the trial. eLife will publish a report of the trial outcomes – watch this space for an update!

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

——————————————————–

Summary by Alice Wareham PhD from Aspire Scientific

——————————————————–

With thanks to our sponsors, Aspire Scientific Ltd and NetworkPharma Ltd