Peer Review Matters: Research Quality and the Public Trust

Peer review has been the foundation of scholarly publishing and scientific communication since the 1665 publication of the Philosophical Transactions of the Royal Society. The benefits and advantages of peer review in scientific research, and particularly medical research, are manifold and manifest.¹  Journals, editors, and peer reviewers hold serious responsibility as stewards of valid information, with accountability to the scientific community and an obligation to maintain the public trust. Anesthesiology states its aspiration and its responsibility on the cover of every issue: Trusted Evidence. Quality peer review (more specifically, closed or single-blind peer review, in which the identity of reviewers is confidential) is a foundational tenet of Anesthesiology.

Peer review grounds the public trust in the scientific and medical research enterprise, as well as the substantial public investment in scientific research. Peer review affords patients some degree of comfort in placing their trust in practitioners, knowing that they should be informed by the best possible, vetted evidence.

Quality peer review enriches and safeguards the scientific content, transparency, comprehensibility, and scientific integrity of published articles. It can enhance published research importance, originality, authenticity, scientific validity, adherence to experimental rigor, and correctness of results and interpretations and can identify errors in research execution. Peer review can help authors improve reporting quality, presentation clarity, and transparency, thereby enhancing comprehension and potential use by clinicians and scientists. Careful scrutiny can identify whether research has appropriate ethical principles, regulatory approvals, compliance, and equitable inclusion of both sexes. Peer review should consider the appropriateness of authorship and can detect duplicate publication, fabrication, falsification, plagiarism, and other misconduct.

Peer review should serve as a tempering factor on overenthusiastic authors and overstated conclusions, unwarranted extrapolations, conflation of association with causality, unsupported clinical recommendations, and spin. Spin is a well known, unfortunately common, and often insidious bias in the presentation and interpretation of results that seeks to convince readers that the beneficial effect of an experimental treatment exceeds what has actually been found or that minimizes untoward effects.^2–4 

Manuscripts often change substantially between the initial submission and the revised and improved published version. Improvement during the peer review process is not apparent to readers, who only see the final, published article, but is well known to authors, reviewers, and editors. Peer review is a defining difference in an era of proliferating predatory journals and other forms of research dissemination. Anesthesiology reviewers and editors devote considerable effort in service to helping authors improve their scientific communications, whether published in this journal or if ultimately elsewhere.

In the domain of clinical research, peer review does not change the scientific premise of an investigation, the hypothesis, or the study design, although it frequently improves their communication. Peer review does not change clinical research data, although it often corrects, enhances, or strengthens the statistical analysis of those data and can markedly improve their presentation and clarity. More importantly, peer review can assess, correct, and improve the interpretation, meaning, importance, and communication of research results—and importantly, confirm that conclusions emanate strictly from those results. Peer review may occasionally fundamentally revise or even reverse clinical research interpretations and recommendations. Each of these many functions enhances reader understanding and should ultimately improve patient care.

Peer review is not a guarantee of truth, and it can be imperfect. Medical history provides many examples of peer-reviewed research that was later found to be incorrect, typically through error or occasionally from misconduct. However, peer review certainly was and remains an essential initial check and quality control that has weeded out, or corrected before publication, innumerable reports of research of insufficient quality or veracity that otherwise would have been published and thereby become publicly accessible. Additionally, science should be “self-correcting,” and peer review is one of the most important factors responsible for such correction. Peer review remains an element by which medical science achieves the “self-correction” that drives progress.

Quality peer review does take time. So also do the initial preparation of manuscripts and the modifications made by authors in response to peer review. Anesthesiology endeavors to provide both quality and timely peer review. Our time to first decision averages only 16 days.

The increasing emphasis on fast research dissemination, often absent quality peer review, comes mostly but not exclusively because of the immediacy of the internet and broader media and societal trends. In an era in which the companies whose major product is the immediacy of information are the economic leaders (Facebook, Twitter, Google, and Apple), it is unsurprising that the immediacy of information is challenging that of quality as the value proposition in the research marketplace. Nevertheless, fast is not synonymous with good. We believe that sacrificing quality on the altar of speed is unwise, benefits no one (except perhaps authors), and may ultimately diminish trust in medical research and possibly even worsen clinical care.

Another recent societal problem is the growing spillover of political and media communication trends into scientific communication. Almost half of Americans believe that science researchers overstate the implications of their research, and three in four think “the biggest problem with news about scientific research findings is the way news reporters cover it.”⁵  Scientific conclusions may be perverted through internet-based campaigns of disinformation and misinformation and dissemination of misleading and biased information.⁶  This threatens the public trust in the scientific enterprise and scientific knowledge.⁷  Social media has made science and health vulnerable to strategic manipulation.^7,8  It is also “leaving peer-reviewed communication behind as some scientists begin to worry less about their citation index (which takes years to develop) and more about their Twitter response (measurable in hours).”⁸  Peer-reviewed journals cannot reverse these trends, but they can at least ensure that scientific conclusions when presented are correct and clearly stated.

In addition to the premium on dissemination speed versus peer review quality, a new variant of rapid clinical research dissemination has emerged that abrogates peer review entirely: preprints. Preprints are research reports that are posted by authors in a publicly accessible online repository in place of or before publication in a peer-reviewed scholarly journal. The preprint concept is decades old, rooted in physics and mathematics, in which authors traditionally sent their hand- or typewritten manuscript draft to a few colleagues for feedback before submitting it to a journal for publication. With the advent of the internet, this process was replaced by preprint servers and public posting. With the creation of a preprint server for biology and the life sciences (bioRxiv.org), the posting of unreviewed manuscripts by basic biomedical scientists has exploded in popularity and practice. Next came the creation of medRxiv.org, a publicly accessible preprint server for disseminating unpublished and unreviewed clinical research results in their “preliminary form”⁹  and more so a call for research funders to require mandatory posting of their grantees’ research reports first on preprint servers before peer-reviewed publication.¹⁰ Lack of peer review is the hallmark of preprints.

The main arguments offered by proponents of preprints are the free and near-immediate access to research results, claimed acceleration of the progress of research by immediate dissemination without peer review, and the assumption that articles will be improved by feedback from a wider group of readers alongside formal review by a few experts. Specifically claimed advantages of preprints are that they bypass the peer review process that adversely delays the dissemination of research results and “lifesaving cures” and “the months-long turnaround time of the publishing process and share findings with the community more quickly.”¹¹  In addition it is claimed that preprints address “researchers recently becoming vocally frustrated about the lengthy process of distributing research through the conventional pipelines, numerous laments decrying increasingly impractical demands of journals and reviewers, complicated dynamics at play from both authors and publishers that can affect time to press” and enable “sharing papers online before (or instead of) publication in peer-reviewed journals.”¹¹ 

Preprints for clinical research have been justifiably criticized.^2,12–15  Most importantly, medical preprints lack safeguards afforded by peer review and increase the possibility of disseminating wrong or incorrectly interpreted results. Related concerns are that preprints are unnecessary for and potentially harmful to scientific progress and a significant threat with potential consequence to patient health and safety. Preprint server proponents “assume that most preprints would subsequently be peer reviewed,”¹⁰  possibly before or after formal publication (if published), thus enabling correction or improvement (before or after publication). However, it is estimated that careful peer review of a manuscript takes 5 to 6 h.^1,16  It seems highly unlikely that busy scientists will surf the web in search of preprints on which to spend half a day providing concerted informative peer review.

Preprint enthusiasts claim that peer review after posting will provide scholarly input, facilitate preprint improvement, and enhance research quality. In fact, such peer review has been scant with biologic preprints, and it seems naïve to expect it with medical preprints. In reality, most preprints receive few comments, even fewer formal reviews, and many comments that are “counted” to support the notion that preprints do undergo peer review actually come through social media; a tweet is hardly a substantive review. The idea that comments on servers will replace quality peer review is not happening now and seems unlikely to transpire. Moreover, a survey found that the lack of peer review was an important reason why authors deliberately choose to post via preprint.¹⁷  Additionally, postdissemination peer review takes longer than traditional prepublication peer review, and there remains concern by authors who do value peer review about the quality of the post-preprint peer review process and the quality of posted preprints.¹⁷ 

Preprint server proponents state “the work in question would be available to interested readers while these processes (peer review) take place, which is more or less what happens in physics today.”¹⁰  The lives of patients are different than the lives of subatomic particles. Preprints deliberately “decouples the dissemination of manuscripts from the much slower process of evaluation and certification.”¹⁰  However, it is exactly that coupling that validates clinical research, benefits patients, improves health, and engenders public trust.

The potential for free and unfettered distribution of raw, unvetted, and potentially incorrect information to be consumed by clinicians and patients cannot be called a medical advance. Use of such information by news outlets and online web services to promote “new” and “latest” research further misinforms the public and patients and is a disservice.

Relegating peer review to the realm of option and afterthought is not in the interest of research quality and integrity or of patients and public health. There is no apparent value in abrogating peer review of clinical research and all its many attendant benefits in ensuring the quality of clinical research available to practitioners and patients. Practitioners and patients have historically not seen the unreviewed manuscript submissions that eventually become revised peer-reviewed publications. Doing so now, given the sizable fraction of clinical research manuscripts that are rejected for publication and the substantial changes in most that are published, by providing the public with unreviewed preprints seems to carry considerable risk.

An additional problem is that the same research report can be posted on several preprint servers or websites or multiple versions may exist on the same preprint site. Various versions may be the same or different, and the final peer-reviewed published article (if it ever exists) may bear little semblance to the various posted versions, which remain freely available. Which version is correct? Availability of various differing reports of the same research risks competing or incorrect information and can only generate confusion. Scientific publishing decades ago banned publication of the same research in multiple journals owing to concerns about data integrity and inappropriate reuse. Restarting this now, via preprints, seems unwise—especially in medicine.

The public cannot and should not be expected to differentiate between posting and peer-reviewed publication. Unfortunately, and worse, even some practitioners do not understand the difference. Posting is often referred to erroneously as publication. Indeed, even the world’s most prestigious scientific journals refer to posting as publication.¹⁸  Such conflation blurs the validity of information. That peer-reviewed publications and preprints both receive digital object identifiers further blurs their distinction and may give the latter more apparent credibility in the eyes of the lay public. The preprint community (servers and scientists) continues to claim simultaneously that preprints are and are not publications, depending on how such claims meet their proclivities. Although the bioRxiv server contains the disclaimer “readers should be aware that articles on bioRxiv have not been finalized by authors, might contain errors, and report information that has not yet been accepted or endorsed in any way by the scientific or medical community” on a web page,¹⁹  it is not on the preprint itself for readers to see (perhaps this disclaimer, and the one below, should appear on the cover page of every preprint and as a footnote on every page). Fortunately, the medRxiv home page (http://www.medrxiv.org) states the following disclaimer: “Preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.” Then why bother?

The popularity of preprints in the basic science world has exploded in the last 5 yr, with the number of documents posted to preprint servers increasing exponentially.²⁰  While acknowledging the noble reasons given by preprint servers and authors for the dissemination of research by posting, three other apparent reasons are less noble. The first is competition for research funding. Major research funders (e.g., the National Institutes of Health) do not allow citation of unpublished manuscripts in grant applications but do allow citation of preprints.^21,22  The second is the preoccupation of authors with the speed of availability. There is a growing (and disappointing) trend of authors perceiving a need to claim priority (“we are the first to report…”), grounded perhaps on fear of being “scooped.” The third is the pursuit of academic promotion, which is based largely on the number of peer-reviewed publications listed on a curriculum vitae. We now see faculty listing preprints in the peer-reviewed research publications section of their curriculum vitae. All these drivers (priority, science advancement, reputational reward, and financial return)⁷  are investigator centric. They are neither quality-centric nor patient-centric.

Who benefits if clinical research quality is sacrificed at the altar of speed? Certainly, it is not patients, public health, or the public trust in science, medicine, and the research enterprise. Enthusiasm for preprints seems to be emanating mostly from investigators, presumably because of academic or other incentives,²³  including the desire for prominence and further funding. Is this why we do medical research? Should we be investigator- or patient-centric?

Little in the argumentation espoused by proponents of clinical preprints attends to their benefit to patients. Indeed, posted preprints without all the scrutiny and benefits of peer review may lack quality and validity and may report flawed data and conclusions, which may hurt patients.^17,23  As stated previously, “clinical studies of poor quality can harm patients who might start or stop therapy in response to faulty data, whereas little short-term harm would be expected from an unreviewed astronomy study.”¹² 

The importance of peer review in clinical research and the downside of its absence in posted preprints is illuminated by the COVID-19 pandemic. As of this date (October 1, 2020), there are 9,222 unreviewed COVID-19 SARS–CoV-2 preprints posted: 7,257 on medRxiv and 1,965 on bioRxiv.²⁴  To date, 33 COVID-19 articles have been retracted (0.37%), and 5 others have been temporarily retracted or have expressions of concern.²⁵  Of the 33 retractions, 11 (33%) were posted on an Rxiv server. The overall retraction rate in the general peer-reviewed literature is 0.04%.²⁶ 

Based upon one of the unreviewed COVID-19 medical preprints,²⁷  the Commissioner of the U.S. Food and Drug Administration (the government agency entrusted more than any other to protect public health) and the President of the United States announced that convalescent plasma from COVID-19 survivors was “safe and very effective” and had been “proven to reduce mortality by 35%.”²⁸  Although the Commissioner later, after scientific uproar over that misinformation, “corrected” his comment in a tweet (a back page retraction to a front page headline),²⁹  the preprint was used to justify a Food and Drug Administration decision to issue an emergency use authorization for convalescent plasma to treat severe COVID-19. Would these errors have been prevented by peer review? We will never know.

Even if priority in clinical (and basic) research is valued, compared to the unquestionable value of quality, clinical preprints have questionable necessity in establishing precedence in contemporary times. Clinical trials registration, which makes fully public the existence of all such research, establishes both who is doing what and when. Some investigators may even publish their entire clinical protocol, to further make their studies known and by whom and when.

For hundreds of years, patent medicines (exotic concoctions of substances, often addicting and sometimes toxic) were claimed to prevent or cure a panoply of illnesses, without any evidence of effectiveness or safety or warning of potential harm. These medical elixirs, the magic potions of snake oil salesmen and charlatans, were heavily advertised and promoted to ailing, sometimes desperate, and thoroughly unsuspecting citizens—all without any oversight, regulation, quality control, or peer review. It was not until the 20th century that medical peer review and the requirement for evidence of effectiveness and safety reigned in the “Wild West” and launched the modern era of medicine, yielding the scientific discovery, progress, and improvement in human health seen today. This era rests on the bedrock of peer review, the quality ideal, and the evidence that constitutes the foundation for evidence-based medicine.

Will clinical preprints become the patent medicines of the new millennium? Do they portend the unrestricted and unregulated spillage of anything claimed as research, by anyone, and absent the quality control afforded by peer review? Like the patent medicines of a bygone era, which were heavily promoted by the newly developed advertising industry, will “posted” clinical research become fodder for the medical advertising industry and media at large, pushing who knows what information and claims on practitioners and a public already deluged with endless promotions and claims with which they cannot keep up or verify? An unsuspecting public is incapable of differentiating between the “posting” of any research observation by anyone with access to a computer and proper scholarly “publication” of peer-reviewed results and conclusions. This is particularly true of vulnerable patients with severe and/or incurable diseases, who may grasp at anything. Moreover, continuous claims of “breakthroughs” and “proven treatments” based on preprints, followed by backpedaling after challenges and outcries, further reduces public confidence in the scientific endeavor as a whole. This can create the perception that clinical science is unreliable and might be a matter of turf wars and politics instead of reliable valid evidence.

Over the past century and throughout the world, legislation has been passed and government agencies have been created to protect the public and maintain their trust in the medicines they take. Few would advocate dismantling the protections against patent medicines. Why now consider dismantling the peer review process in clinical research?

In 2019, the editors of several journals expressed a well articulated principle that they will not accept clinical research manuscripts that had been previously posted to a preprint server.³⁰  Their rationale was that the benefit of preprint servers in clinical research did not outweigh the potential harm to patients and scientific integrity. Major specific concerns included: “1) Preprints may be perceived by some (and used by less scrupulous investigators) as evidence even though the studies have not gone through peer review and the public may not be able to discern an unreviewed preprint from a seminal article in a leading journal; 2) It seems unlikely that the kind of prepublication dialogue that has taken place in other academic disciplines (e.g. mathematics and physics) will take place in medicine or surgery because the incentives are very different; 3) Preprints may lead to multiple competing, and perhaps even conflicting, versions of the ‘same’ content being available online at the same time, which can cause (at least) confusion and (at most) grave harm; and 4) For the vast majority of medical diagnoses, a few months of review of a study’s findings do not make a difference; the pace of discovery and dissemination generally is adequate.” These editors’ concerns and approach merit consideration if not more widespread adoption.

The potential for practitioner and public confusion regarding the difference between unregulated preprints and peer-reviewed publication is substantial. Indeed, the posting of preprints is often incorrectly termed “publication.” Peer-reviewed publications versus posted “publications” will soon become a difference without a distinction. Moreover, authors cannot have it both ways. They cannot claim a preprint as a publication for purposes of a grant (and now in some universities potentially for purposes of a degree, appointment, and/or promotion), yet claim it is not a publication for the purposes of submission to a peer-reviewed journal that does not allow prior publication. More importantly, the peer review imperative in clinical research and the role it plays in research quality, the evidence base, and patient care, constitutes an obligation to patient safety that cannot and should not be abrogated.

Peer review, clinical research quality, and the public trust in clinical research all now face an unprecedented assault. Quality peer review is a foundational tenet of Anesthesiology and underlies the Trusted Evidence we publish. Quality, timely, and unpressured peer review will continue to be a hallmark of Anesthesiology, in service to readers, patients, and the public trust.

We thank Ryan Walther, Managing Editor, and Vicki Tedeschi, Director of Digital Communications, for their valuable insights.

Dr. Clark has a consulting agreement with Teikoku Pharma USA (San Jose, California). Dr. Levy reports being on Advisory and Steering Committees for Instrumentation Laboratory (Bedford, Massachusetts), Merck & Co. (Kenilworth, New Jersey), and Octapharma (Lachen, Switzerland). Dr. London reports financial relationships with Wolters Kluwer UptoDate (Philadelphia, Pennsylvania) and Springer (journal honorarium; New York, New York). The remaining authors declare no competing interests.