Science speeds up during coronavirus pandemic — but at what cost?

Every day we learn something new about Covid-19 — like how the virus can damage organs outside the lungs, or how some kids can become severely ill.

One day, there’s excitement over a drug like hydroxychloroquine. Seemingly the next, several studies find it has no benefit.

Adding to the deluge of information are numerous studies, many of which have not gone through the standard scientific vetting process from experts in the same field who evaluate the quality of a study before it can publish.

This process, called peer review, helps weed out results that are misleading — or wrong.

“I have colleagues that have patients come into their office waving articles that they printed from a preprint server promising … miracle cures for Covid, which have never been studied, could be dangerous, and that’s a big problem,” said Dr. Rita Redberg, a professor of medicine at the University of California, San Francisco and the editor-in-chief of the journal JAMA Internal Medicine.

“You have preprints suggesting that things worked, but they had no control group,” she said. “There’s a lot of value to peer review. And that doesn’t happen on a preprint server.”

As the outbreak spreads at breakneck speed, so does information about the virus. But experts say there’s a balancing act between sharing findings quickly and taking the time to ensure they’re scientifically sound.

“It’s new and it’s great that so many people are doing research,” Redberg said. “You want to get information out fast, but you want information out that’s reliable.”

Surges in preprint publications

During the pandemic, there’s been a flood of studies posted to preprint servers.

Two of the most prominent — bioRxiv and medRxiv — have more than 3,300 papers related to Covid-19. Since January, up to 100 papers about the virus have been added each day.

The preprint server focused on medicine, medRxiv –pronounced “med archive” — launched nearly a year ago.But its growth was slow and steady during its first eight months, according to John Inglis, cofounder of bioRxiv and medRxiv. In January, there were 224 submissions, but that number has doubled each month. In May, it is expected to exceed 2,200.

Right now, scientists have an “intense need to share the latest information as fast as possible,” Inglis said. “The preprint servers were there to perform that function for them.”

But not every paper makes it to the preprint servers. All submissions go through a screening process, where in-house staff and working scientists check to see that the paper meets certain requirements. This is to ensure that the content isn’t plagiarized, offensive, non-scientific or pose a health or safety risk, according to the servers’ websites.

“These platforms are open but operate with responsibility,” Inglis said.

Unlike peer review, this screening process doesn’t look at the quality of the work or how easy the findings are to reproduce, so anything that’s posted is considered a preliminary report.

“It’s not conclusive. It shouldn’t guide clinical practice. And it shouldn’t be reported in the media as established information,” he said.

Redberg cautioned similarly, “I think the problem is — particularly if they’re picked up in the media or on social media people post — that it’s very hard for the public to separate an article that has undergone a rigorous peer review process from one that has not.”

Still, some say there’s value in preprint publications in helping inform policymakers or setting the tone for future research. Some of these papers eventually turn into journal papers, but only after peer review.

“People want to see data sooner rather than later,” said Dr. Jeffrey Flier, a professor of medicine and the former dean of Harvard Medical School. “The traditional journals just weren’t set up to make that happen.”

Journals expedite peer review

Even traditional journals have found ways to speed up their review process.

In an open letter, nine publishers put the call out for volunteers who have relevant expertise to rapidly review Covid-19 papers. The “rapid reviewers” are also asked to identify “important and crucial Covid-19 preprints” to save time if those preprints end up getting submitted to their peer-reviewed journals.

“Scholarly publishers are working together to maximize the efficiency of peer review, ensuring that key work related to Covid-19 is reviewed and published as quickly and openly as possible,” according to the open letter posted on April 27 to the Open Access Scholarly Publishers Association website. The journals in this association make content available online and free to the public.

The expedited process for Redberg’s journal, JAMA Internal Medicine, relies on speed and fewer reviewers. Covid-19 manuscripts start getting reviewed the day they come in, she said.

“They’re not going through our usual review process because that takes two or three weeks at best,” she said. “So we’re either asking reviewers to review them really quickly, or just sending them ourselves to our senior editorial team and our colleagues and asking for quick reviews.”

This cuts down the review process from several weeks to several days, if the paper is accepted, she said.

Many more journals are in the same position. A research paper in April analyzed 14 journals and found that turnaround times for Covid-19 articles were dramatically faster than those published before the pandemic. On average, turnaround time fell from 117 to 60 days.

“While the acceleration of journals’ publication process is laudable from the perspective of quick information dissemination, it also raises concerns relating to the quality of the peer review process and the quality of resulting publications,” the authors said in their study, which was itself a preprint posted on bioRxiv without undergoing peer review.

From preprint to peer-review

That bioRxiv preprint also analyzed other preprints: In total, 2,012 preprints about the outbreak were published through April 4 across seven preprint servers. Of these, 129 ended up in a peer-reviewed journal.

Since 2013, when bioRxiv started, about 70% of its preprints have become journal articles, according to co-founder Inglis. For medRxiv, it’s about 20% — but that may not give the full picture, since the archive is less than a year old, and peer review can take longer than that.

It’s hard to say what the conversion rate for Covid-19-related preprints will be.

“People are so desperate to share their latest insights. I think the question of eventual publication is taking a back seat to the need to share,” Inglis said. “It’s quite likely that a much greater percentage of the papers that have been posted during the pandemic will actually not turn into journal papers.”

One reason: Information about the virus is constantly evolving, he said. So what someone finds in February could be old news a few months later.

And even if a preprint makes it to peer review, that doesn’t make it foolproof.

“Having peer review doesn’t ensure that something’s true,” said Flier, the doctor from Harvard. Even with more quality checks, there could be mistakes — some so big that journals might have to retract studies.

Scientific research has exploded during the pandemic. But the publishing landscape was changing before Covid-19, according to Flier.

“There’s going to be much more open access, much more early sharing of information. There are going to be new peer review mechanisms,” he said.

An enduring challenge might be figuring out how to deal with information overload.

“Is it solid data? Is it badly carried out?” said Flier. “Everyone is trying to process the new information.”