Inappropriate Citation of Retracted Articles in Anesthesiology and Intensive Care Medicine Publications

The Retraction Watch database (http://retractiondatabase.org/RetractionSearch.aspx?),⁶  maintained by Retraction Watch (New York, New York), was used to identify the retracted articles on anesthesiology and critical care.⁷  Retraction Watch is a nonprofit news outlet, founded in 2010, that covers retractions and related issues in scholarly literature.

We collected a dataset of retracted articles published up to August 16, 2021. The database categorizes retracted articles by subject matter. We extracted the subgroup of interest labeled in the dataset as “(HSC) Medicine - Anesthesia/Anesthesia,” comprising both intensive care– and anesthesia-related articles. Each identified article was manually checked to confirm that the articles had actually been retracted.

The following data were extracted from the retrieved articles: title, authors, publication date, retraction date, and journal name.

For each retracted article, a complete list of the citing articles published since the year of retraction was retrieved from Scopus. After the initial screening, two members of the research team (A.D.C. and S.D.P.) independently performed a second screening, using the full publication date as a reference, to identify erroneously included articles. Our analysis included both articles and book chapters, and we excluded scientometric articles discussing retraction practices.

We collected the following information for each retrieved paper: corresponding authors’ mailing details and country, number of authors, article type, and publication date.

For both the retracted articles and citing articles, the Scimago Journal Rank was retrieved using the publication year of the citing article as a reference. Scimago is a publicly available portal that includes the journal and country scientific indicators developed from the information contained in the Scopus database and utilizes the Scimago Journal Rank as a metric to evaluate journal impact based on the citations, references, and number of articles in the journals indexed in Scopus.⁸  Scimago assigns journals to one or more subject categories, of which “Anesthesiology and Pain Medicine” and “Critical Care” are two separate categories of 313 categories. For each category, the journals are divided into quartiles (1, 2, 3, and 4) based on the Scimago Journal Rank, with quartile 1 comprising journals with the top 25% highest Scimago Journal Rank and quartile 4 comprising those with the lowest 25% Scimago Journal Rank, without a prespecified minimum or maximum Scimago Journal Rank for each quartile.

An online survey was also conducted. The survey protocol was examined by the Institutional Review Board of the University Hospital of Padua, Italy, and a waiver for formal approval was granted considering the nature of the study (protocol reference: 65681).

We followed the existing recommendations for designing high-quality surveys.^9,10  After a preliminary bibliographic search in order to identify existing and validated surveys, the survey items were designed following Peterson’s brief, relevant, unambiguous, specific, and objective (BRUSO) model,¹⁰  followed by a two-phase process of pretesting. We referred to the checklist for reporting results of Internet E-Surveys (CHERRIES)¹²  to enhance the quality of our study.

The bibliographic search of PubMed, Scopus, Embase, and World of Science was conducted by a member of the team (A.D.C.) using a combination of the following terms: (“scientometric” OR “retraction” OR “retracted” OR “citation”) AND (“survey” OR “cross-sectional”).

Since no existing surveys or survey questions related to this topic were found, we developed a survey comprising seven single-choice and multiple-choice questions to investigate the awareness and motivations for citing a retracted article. The number of questions was intentionally low to decrease the rate of both nonresponders and partial responders, but we did not randomize the order of the questions. The complete list of survey questions is available in Supplemental Digital Content 1 (http://links.lww.com/ALN/C873).

To enhance the effectiveness of the survey, a pretest was conducted in two phases.

In the first phase, the survey team reviewed the whole survey with particular attention to the order of the questions to avoid “skip” or “branch” logic mistakes in the survey flow. “Skip” and “branch” are logic tools used in survey design. The “skip” mechanism permits respondents to skip questions based on users’ responses, while the “branch” mechanism creates two or more survey branches based on respondents’ answers. In the second phase, the survey was sent to 20 residents of the University Hospital of Padua, Padua, Italy, to identify any problems in the order or clarity of the questions. Moreover, testers were asked to record the time required to complete the survey, which was confirmed to be less than 5 min.

The population of interest in this survey was identified as the authors of articles citing retracted literature. The corresponding author of each article was contacted by email and asked to respond by email. For articles citing more than one retracted article, the corresponding authors were contacted via a single email, requesting a response for all citations. To minimize the rate of nonresponders, five reminders were sent since the initial contact, with at least 7 days between each reminder. No incentives were offered to participate in this survey.

If the retrieved email address was not functional or there was no email address mentioned in the article, further corresponding author contacts were searched in more recently published articles and institutional websites.

The corresponding authors were asked to respond to the email by attaching the completed electronic version of the survey, and the responses were collected for 3 months from October 11, 2021, to January 13, 2022.

Each of the corresponding authors of articles citing retracted articles was assigned a consecutive integer number to store the information in an Excel (Microsoft Corporation, USA) file protected by a password known to only one of the authors (A.D.C.). The same author stored the file in a system other than where the database was stored, and without access to a third party. A consecutive integer was used to assign survey responses to the corresponding database data. After this, emails containing the survey response were deleted. Moreover, at the end of the survey, the file storing the emails of the corresponding authors was deleted. The collected data regarding retracted articles will be available upon reasonable request from the Retraction Watch and the Center for Scientific Integrity (New York, New York) to researchers, subject to the Retraction Watch data use agreement, while the deidentified survey data generated and analyzed in this study are available upon reasonable request.

No a priori power calculation was used to guide the sample size. The normality of the distribution of quantitative characteristics was verified using the Shapiro–Wilk test. Nonnormally distributed variables are presented as median (first and third quartiles), while categorical variables are presented as numbers and percentages.

Nonnormally distributed variables were compared using the Mann–Whitney U test. Categorical variables were compared between the two groups using the chi-square test or Fisher exact test when appropriate. All tests were two-tailed; P values less than 0.05 were considered statistically significant.

Nonresponse bias was evaluated by comparing the following variables, using the appropriate tests described above, between the respondent and nonrespondent groups: country of the corresponding authors, number of authors per article, publication year, type of journal publishing the article, and article type. Nonresponse bias occurs when the characteristics of nonresponders from a sample differ significantly from those of responders; its assessment is vital in cross-sectional studies, as this type of bias could limit the generalizability of the results. All statistical analyses were conducted using R version 3.4.0 (2017-04-21).

The flowchart of the study is shown in figure 1A. From the Retraction Watch database, we identified 478 retracted articles on anesthesiology and intensive care medicine. Among these, 220 retracted articles (46.0%) were cited at least once.

The initial citation search yielded 2,277 potential articles citing a retracted article. Considering that 554 articles were published before retraction occurred and 155 were scientometric articles discussing article retraction, 1,568 articles were included in the qualitative analysis.

Of these 1,568 included articles, 166 were considered duplicates, as some cited more than one retracted article; thus, 1,402 articles were included in the survey analysis.

Of the included articles, there were 415 (30%) review articles (91 of 415 meta-analyses), 397 (28%) randomized controlled trials, 258 (18%) cohort studies, 132 (9%) animal or laboratory research, 96 (7%) book chapters, 42 (3%) case report/case series, 26 (2%) editorial/comments, 14 (1%) guidelines/expert opinions, 2 (<1%) surveys, and 20 (1%) articles classified as “others.”

The median number of authors of the articles was 5 (3 to 6), and the elapsed time from retraction to citation was 3 (1 to 5) yr, with a maximum of 28 yr.

According to the Scimago classification of journals, 394 (28%) articles were published in anesthesiology or critical care journals, 862 (61%) were published in journals of other fields, 50 (4%) were not categorized, and the remaining 96 (7%) were book chapters. Anesthesia and intensive care journals with the corresponding number of articles with retracted citations and the number of published articles in the time period studied are available as Supplementary Digital Content 2 (http://links.lww.com/ALN/C874).

Most of the cited retracted articles were published in anesthesiology or intensive care journals (1,069, 76%), with the majority of articles published in high-ranking journals (993 in quartile 1 journals, 58 in quartile 2, 13 in quartile 3, and 5 in quartile 4). Regarding the citing articles, only 394 (28%) were published in anesthesiology or critical care journals; there were 134 articles published in quartile 1 journals, 140 in quartile 2, 77 in quartile 3, and 41 in quartile 4, while the remaining two were not ranked.

A flowchart of the survey is shown in figure 1B, with 1,402 potential corresponding authors contacted for inclusion in the survey analysis. The corresponding author’s email address was not reported or recoverable elsewhere in 105 articles. In total, 1,297 emails were sent, 91 (7.0%) of which were not delivered because the mailboxes were full or no longer existed.

After the last reminder, we had 480 (34%) responses to our survey, with 63 (4%) respondents refusing to participate. Finally, 417 (30%) respondents were included in the analysis.

Most respondents were unaware of having cited a retracted article (n = 372, 89%). This occurred despite a strategy to check for the presence of retracted articles (n = 182, 49%) and at least one author checking the reference before submission (n = 262, 70%). The main reasons for this were missed identification of the retraction status in the journals and/or search database (n = 229, 61.6%) and the use of a stored copy of the article (n = 42, 11%). Notably, the majority of respondents, 249 (67%), did not believe that the retracted article played a role in the study design or discussion of the results.

Some respondents (n = 45, 11%) were aware of the retraction status. Most researchers knowingly citing retracted articles used them as an example of a retracted article (n = 35, 78%).

Responses to the survey are summarized in table 1.

The analysis of the nonresponders is shown in table 2. Responders’ and nonresponders’ characteristics did not differ based on the country (fig. 2), number of authors per article, publication year, and type of journal publishing the article. Nevertheless, there may have been bias regarding the type of articles published (P = 0.009).

The main result of our study was that most authors citing a retracted article in their research were unaware of the retracted status, mainly because of inadequate notification of the retraction status in journals and/or databases and the use of stored copies.

Although several analyses have been conducted on the topic of retracted articles in anesthesiology and critical care,^1,13  no study has evaluated the retraction issue using a combination of comprehensive analysis and a survey-based approach. This phenomenon includes many problems that should be addressed through multicomponent research strategies. For example, an important aspect to be evaluated is the citation of articles many years after retraction. In their longitudinal analysis (from 1960 to 2020), Hsiao and Schneider¹⁴  highlighted that retracted papers continue to be cited, but their citation trend progressively decreases. Notably, in our study, we found that citations often occurred within 10 yr after the publication of the retracted article; however, there were sporadic cases of citations of articles occurring up to three decades after their retraction. In this period, the inclusion of data from retracted studies in evidence-based medicine analyses can occur easily¹⁵  and potentially affect the reliability of scientific evidence.¹⁶ 

The link between journal ranking (quartile 1 to quartile 4), retracted articles, and their citations is another aspect that requires careful evaluation. Notably, although Fiore et al.¹  demonstrated that there is no difference in the quartile 1 to quartile 4 distribution of retracted articles, the retracted articles published in high-ranking journals (quartile 1) were cited most frequently. This is not surprising, as authors tend to cite articles from prestigious and high-impact journals. In an investigation of approximately 4,000 articles, the average impact factor of the cited journals at the time of publication was 3.262.¹⁷  Furthermore, in our investigation, most citations were found in journals of different scientific areas, and only approximately one fourth of these were in anesthesia- and critical care–related journals. Thus, these findings suggest that approaches useful for abating retracted investigations, such as replication strategies involving reproduction of the same study project in another setting,¹⁸  are needed. These approaches should be combined with more rigorous publication processes and substantial mentorship programs for the authors, editors, and reviewers. According to Nato et al.,¹⁹  multicomponent approaches should be encouraged to strengthen the “pact of trust” between authors, journals, and readers. Interestingly, the Reducing the Inadvertent Spread of Retracted Science project is an ambitious program aimed at reducing the unintended spread of retracted literature. It provides information on the research institutions, scientific societies, government agencies, funding companies, publishers, and other stakeholders involved. The research agenda includes the development of guidelines to recognize and prevent the dissemination of retracted manuscripts.²⁰ 

Careful analysis of the causes of retraction is crucial for promoting required improvements. Therefore, in the second step of this study, the mechanisms underlying citations of retracted articles were investigated using a survey.

Identification of retracted articles is indispensable. In their editorial, Pfeil and Goldhammer highlighted that many manuscripts had been retracted on the publisher’s website but not on PubMed or other directories. Furthermore, a lack of retraction notification is common. They suggested sending quarterly or annual reports containing lists of retracted articles to authors of scientific articles included in journal databases (e.g., corresponding authors), perhaps divided by specialty.²¹  In our investigation, most authors (89%) were unaware of the retraction status of the article they cited, and there are probably multiple reasons for this lack of knowledge. For example, some authors commonly use printed or stored copies, and they may fail to verify whether the article they are citing has been retracted. Nonetheless, several software programs (e.g., EndNote 20, Clarivate, London, United Kingdom) have been recently developed and implemented to detect retractions. Their routine use could be advantageous in the resolution of this potentially serious problem.

Authors must exclude retraction when checking references, and it is important to follow precise strategies. Interestingly, in the survey responses, approximately 70% of the respondents reported that citations were checked by one or more authors. This finding was comparable with that of the authors who were aware of having cited a retracted article (71%) and those who were not (70%; table 1). Nonetheless, in the latter group, approximately half of the respondents failed to adopt a retraction-checking strategy.

Almost all respondents (94%) declared that editors or reviewers did not raise issues regarding retracted references. Retraction-checking strategies vary among the journals. These approaches should be a routine part of the reporting guidelines, especially for systematic reviews and meta-analyses. For example, some journals recommend using an ad hoc string that includes the term “retraction” and the author’s name. Nevertheless, according to the International Committee of Medical Journal Editors, authors are responsible for checking that none of the references cite retracted articles.²² 

The identification of a retracted article is a key step in avoiding citation. In our investigation, the responders of the group of authors who were not aware of the article being retracted affirmed that neither was the retraction note adequately highlighted by the journal (15%) nor was it clearly indicated in the search database (39%). Although most retracted articles contain a retraction note, according to Nair et al.,¹³  the quality of these watermarks varies among journals. Evidently, the absence of pre-established labels on retracted articles complicates the recognition process. Hence, these labels should be clearly applied to library databases. Moreover, as suggested by Rong et al.,²³  standardizing the processes after retraction is warranted to draw attention to controversial articles.

Remarkably, up to one in five of those who recognized that the citation was retracted affirmed that they quoted it because of its relevant findings (16%) or assumed that the retraction did not influence research validity (7%). However, most authors (78%) used citations as examples of retracted literature. The latter seems to be a legitimate use of retracted citations, but it is recommended that authors include the retraction notice.

Finally, we evaluated whether citing retracted articles affects the quality of the articles that cited them. Two thirds of the authors stated that the retracted reference was not substantially important for their paper. Consequently, the reasons for citation of these articles remain unclear.

First, a single-source–based search strategy, the Retraction Watch database, may not have allowed the identification of all retracted articles. Nevertheless, we have previously reported that Retraction Watch summarizes elements from different datasets, even those not commonly used for systematic research.¹ 

Second, the survey results do not allow for generalization given the presence of a possible nonrespondent bias for the article type characteristics (P = 0.009). Despite this limitation, the bias analysis of the respondents was satisfactory for all the other variables assessed (table 2).

Third, we included all papers citing retracted literature, regardless of the elapsed time since retraction. We recognize that papers published shortly after the retraction may not be able to identify the retraction status regardless of the efforts of the authors, reviewers, and editors.

Although citation of retracted literature in the anesthesiology and intensive care medicine fields occurred in approximately 1,000 articles, which could be considered uncommon, it is still a significant phenomenon. Moreover, citations occurred up to three decades after retraction. Articles published in high-impact journals accounted for most of the citations. Most of the corresponding authors (89%) were unaware of the retraction of the cited article, mainly because of inadequate notification of the retraction status in the journals and/or web databases and the use of stored copies.

Support was provided solely from institutional and/or departmental sources.

Dr. Navalesi’s research laboratory received grants/research equipment from Draeger (Lubeck‚ Germany), Intersurgical SPA (Mirandola, Italy), and Gilead (Foster City‚ California). Dr. Navalesi receives royalties from Intersurgical SPA for the Helmet Next invention. He also received speaking fees from Getinge (Gothenburg‚ Sweden), Intersurgical SPA, Gilead, MSD (Rahway‚ New Jersey), Draeger, and Medicair (Vigonza, Italy). Dr. Navalesi has no conflict of interest to declare in relation to this manuscript. The other authors declare no competing interests.

Survey, http://links.lww.com/ALN/C873

Anesthesia and Intensive Care Journals, http://links.lww.com/ALN/C874