Stanford researchers discuss the benefits – and perils – of science without peer review

Usually, if biomedical researchers want to publish in a high-quality journal, their research must first withstand review by several outside experts. These “peer reviewers” judge the quality of the research, ask questions and offer critiques.

When it comes to COVID-19 research, that conventional model has been upended.

The urgent need for immediate solutions to the COVID-19 pandemic has led biomedical researchers to favor a different form of publishing, called preprints – full articles made publicly available before passing the gauntlet of peer review.

“It’s a great way to get preliminary results out and shared with the wider community, which can encourage collaboration and speed up the science,” said Russ Altman, the Kenneth Fong Professor and professor of bioengineering, of genetics, of medicine and of biomedical data science at Stanford University. “Of course, the negative is that it’s not peer reviewed, so people have to remember that what they’re reading might actually be slightly – or totally – wrong.”

The main advantages of preprints are speed and open access. Whereas traditional publishing can take many months, posting a preprint is an instant way to share findings with colleagues and stake a claim on new insights. Preprints are also freely available to the public – in contrast to journal articles, which are usually hidden behind paywalls. Used responsibly, preprints have the potential to accelerate and improve research, inspiring collaborations and sharing failures or negative results that might never make it to the pages of a journal.

“Everyone is realizing how this ability that we’ve developed really only in the past few years is an amazing tool for disseminating knowledge,” said Tim Stearns, the Frank Lee and Carol Hall Professor in the School of Humanities and Sciences and chair of the biology department.

On the other hand, as many labs pivot to preprinting COVID-19 research in hopes of fast-tracking potentially life-saving science, there is also potential for wider dissemination of poor-quality work.

“Sometimes one piece of wrong information or misleading information can damage ten good papers. Negative things and scary things – those can spread faster than positive things,” said Stanley Qi, assistant professor of bioengineering and of chemical and systems biology at Stanford.

#OpenScience

Preprints are part of a broader and accelerating movement called open science that aims to make scientific research – not only the final publications but also the data, samples and software – public, transparent and accessible. For Stanford postdoctoral researcher Morgan Kain, open science presents both an opportunity and a risk.

“I want people to be able to access my work as soon as possible but I understand I often need to keep my work under wraps before sharing it with the world in the interest of my own career,” Kain said. “I think this is a conflict many scientists face regularly. Given this, I think it is really nice to see so many people being extremely open and collaborative in order to understand what’s going on in the interest of helping people.”

Preprints exist within an internet ecosystem that includes other open-access websites. So, an eye-catching preprint can launch real-time discussion among colleagues, and, oftentimes, the public is privy to the conversation.

“It’s been cool to see how much of the COVID-19 virus response work is being done online, in real time, openly available to everybody,” said Erin Mordecai, assistant professor of biology in the School of Humanities and Sciences. “Many of the top scientists are doing analyses and posting them on Twitter first, and then posting them on places such as GitHub and preprint servers like bioRxiv. I think Twitter is one of the best places to follow what’s going on.”

This online dialogue can resemble a more formal peer-review process and lead to improvements in the paper and launch collaborations. However, non-experts are just as able to see, comment on and disperse preprints – with mixed consequences.

“The discussion between experts and semi-experts is valuable for scientists working on COVID-19 but it can be confusing for the public,” said Mordecai. “Without some background knowledge, it’s hard to know who the credible experts are.”

An excess of good intentions

Altman estimates that there have been thousands of preprints published about the novel coronavirus and COVID-19 in the past few months.

“Let’s assume everybody has the best motives, they’re just being a little big eager. But there’s too much stuff now,” said Altman, who is also associate director of the Stanford Institute for Human-Centered Artificial Intelligence. “We have some ridiculous number of brand-new papers on COVID and nobody can read all that.”

As a result of this inundation, a server Altman has published on many times recently refused a preprint paper from his team. The paper was about potentially repurposing drugs to address the novel coronavirus, and Altman was publishing it with the goal of attracting collaborators. The server administrators told him that they are changing how they screen COVID-19 and novel coronavirus papers, and are unable to assess his computational biology research because it’s outside the bounds of their expertise. Soon after the refusal, Altman successfully published the paper on a different preprint server.

“I understand the arguments. They worry that uninformed or overly zealous patients will self-medicate in dangerous ways based on preliminary evidence and that there will be a proliferation of misinformation,” said Altman. “But, in the specific case of COVID-19, the benefits of rapid dissemination of new ideas to scientific colleagues far outweigh the risks of misinterpretation by the lay public.”

After COVID-19

Right now, everyone’s attention is on COVID-19, but there will be a time when that’s no longer the case. Once that happens, will science return to more formal means of communication, collaboration and publishing?

“There are many other unsolved diseases besides COVID-19 and a lot of these problems require coordination, timely dissemination of information and public attention,” said Qi. “They will require discussion within the scientific community but also discussion with the public, doctors, policymakers. So, I think preprints will continue to play a big role beyond the current pandemic.”

But in order for preprints to earn a secure place among traditional routes of science dissemination, other aspects of professional and academic science may need to change first, said Judith Frydman, the Donald Kennedy Chair in the School of Humanities and Sciences and professor of genetics.

“The publication requirements for faculty positions and grants will need to balance the open and fast access provided by preprints with the requirement for peer-review,” Frydman said. “As things stand, people still have to publish their work in journals where the quality and rigor of the work is vetted by experts in their field.”