Reporting of Preclinical Research in Anesthesiology: Transparency and Enforcement

Imagine reading a clinical study where investigators gave patients a drug thought to speed recovery from sedation after anesthesia or a placebo. The description and results of the study in the article include the following statements:

What the investigators actually did was the following:

Had the investigators completely reported their actual methods, it’s unlikely that a journal would accept such an article, or that a reader would put much stock in its results. To ensure adequate reporting for clinical trials, Anesthesiology requires submitted research to conform to the CONsolidated Standards of Reporting Trials (CONSORT) guidelines. Among many reporting elements, CONSORT requires the following: (1) an adequate description of the experiments to allow other researchers to replicate them, (2) report of the measures used to reduce bias, including whether and how random allocation and blinding methods were used, (3) how the sample size was determined, and (4) the data analysis plan. These are the same reporting elements that we will now require in all preclinical studies.

It took decades for clinical investigators to embrace these elements as critical to interpretable, reproducible, and actionable science. Given the extent that modern preclinical research lacks rigor regarding these elements, the reporting quality in such studies is “reminiscent of the situation in clinical research approximately 50 yr ago.”²  Reporting these elements is only present a minority of the time (or not at all for sample size calculations), even in journals that strongly endorse the ARRIVE statement.^3,4  The poor reporting quality in preclinical research may reflect many causes, but the consequences of poor reporting can be readily observed. The lack of reporting rigor may underlie the inability of independent industry laboratories to replicate a majority of landmark studies from academic laboratories performing cancer, cardiovascular, and stroke research.^2,5,6  Failure of clinical translation and of replication of preclinical research was cited by leaders of the National Institute of Neurologic Diseases and Stroke⁷  and the National Institutes of Health⁸  when they called on journals, investigators, and funders to improve education in good scientific design and in transparent reporting of essential research design elements.

Authors are encouraged to review the full ARRIVE guidelines¹  (in addition to the citation, they can be directly accessed at www.nc3rs.org.uk/arrive-guidelines) before submission of preclinical studies to Anesthesiology. However, the following items will be particularly scrutinized in research submissions.

• 1. Describe the experiments adequately to allow other researchers to replicate them

This is unchanged from our current requirement, and investigators are encouraged to report the key aspects of the experiments that would allow an experienced investigator outside of their laboratory to attempt replication of the study. All studies that were performed should be reported, not just those that support the hypothesis, including the number of animals in these studies and the statistical analysis. Pilot studies used to define conditions should be described only to the extent that they would aid in replication.

• 2. Report whether measures to reduce bias were used, including random allocation and blinding, and how they were performed

Some investigators argue that random allocation is not necessary because they are studying inbred or highly homogeneous animal populations and that blinding is not necessary because the animal is effectively blinded to treatment. However, the need for these procedures is underscored by changes in animal behavior due to seasonal changes in the source of the protein in commercial animal chow⁹  and large interindividual animal variability in behaviors before and after surgery.¹⁰  In addition, we now know that environmental influences can alter subsequent biology and physiology via epigenetic and other mechanisms, despite presumed identical genomes. Similarly, experimenter blinding is essential whenever possible given that unintentional experimenter bias can influence measurements as evidenced by the fact that effect sizes of interventions are lower in studies when blinding is performed.¹¹ 

• 3. Report how the sample size was determined

Although many preclinical articles include multiple experiments, it should be reported for each experiment whether there was an a priori defined primary outcome measure and sample size based on estimates of variance and minimum biologically meaningful effect sizes. We recognize the need for exploratory science, and it is quite likely that unblinded, nonrandomized experiments might be included in an article as preliminary observations. Very small sample sizes in preclinical research may result in a high likelihood of false results and in misestimation of the true effect size, and the ethics of such unreliable research has been questioned.¹²  Concerns over the unreliability of small sample size have led at least one journal to only accept studies with a minimum sample size of five.¹³  Thus, in addition to a power calculation, at very small sample sizes, the reliability of the observation should be considered.

• 4. Report the data analysis plan

Prospective definition of primary outcome(s) and an analysis plan are needed to design a high-quality study that has a good chance of being replicated in future studies. In clinical research, prospective documentation of these design aspects is required through trial registration. Although trial registration is not required for preclinical research, the authors should state whether primary outcomes and an analysis plan were established before the study started and to declare what elements of the analysis were derived after examination of the data (i.e., post hoc). Clinical research investigators report the number of subjects recruited into the trial, randomized into conditions, and the number excluded from the analysis, as well as the reasons for exclusion. This same practice should be reported for each experiment involving animals. Although there may be cases where a majority of animals are excluded from data analysis due to technical failures, providing this information is extremely valuable to other investigators who wish to replicate the experiment or method. Whether any data were excluded as outliers should also be reported, including how outliers were defined and whether this was done prospectively and before unblinding. Often, it is advisable to report the analysis with and without outliers to allow a reader to evaluate the data in both contexts.

As noted, despite journal endorsement of these and other elements of the ARRIVE guidelines for reporting preclinical research, articles in these journals report the elements only a small minority of the time. Furthermore, there has been little improvement in reporting practices over the past 3 yr and little difference between journals with high- or low-impact factors.^3,4  For the past several years, Anesthesiology has scanned all clinical trials with custom-designed software to identify elements of CONSORT that are not included, and we will do the same for preclinical research for these elements of ARRIVE. The goal of these efforts is not to reduce the amount of preclinical research we publish or to discourage authors from considering Anesthesiology for publication of their preclinical research. Rather, the goal of these efforts is to enhance trust by our readers in the quality of the science we publish and to enhance trust by investigators that this published work is more likely to replicated and perhaps translated into improved care of patients.

Supported, in part, by grant R37-GM48085 from the National Institutes of Health, Bethesda, Maryland.

Dr. Eisenach is the Editor-in-Chief of Anesthesiology, and his institution receives salary support from the American Society of Anesthesiologists (ASA), Schaumburg, Illinois, for this position. Dr. Houle is the statistical Editor of Anesthesiology, and his institution receives salary support from the ASA for this position. Dr. Warner declares no competing interests.