Abstract
Acute respiratory distress syndrome (ARDS) remains a serious postoperative complication. Although ARDS prevention is a priority, the inability to identify patients at risk for ARDS remains a barrier to progress. The authors tested and refined the previously reported surgical lung injury prediction (SLIP) model in a multicenter cohort of at-risk surgical patients.
This is a secondary analysis of a multicenter, prospective cohort investigation evaluating high-risk patients undergoing surgery. Preoperative ARDS risk factors and risk modifiers were evaluated for inclusion in a parsimonious risk-prediction model. Multiple imputation and domain analysis were used to facilitate development of a refined model, designated SLIP-2. Area under the receiver operating characteristic curve and the Hosmer–Lemeshow goodness-of-fit test were used to assess model performance.
Among 1,562 at-risk patients, ARDS developed in 117 (7.5%). Nine independent predictors of ARDS were identified: sepsis, high-risk aortic vascular surgery, high-risk cardiac surgery, emergency surgery, cirrhosis, admission location other than home, increased respiratory rate (20 to 29 and ≥30 breaths/min), Fio2 greater than 35%, and Spo2 less than 95%. The original SLIP score performed poorly in this heterogeneous cohort with baseline risk factors for ARDS (area under the receiver operating characteristic curve [95% CI], 0.56 [0.50 to 0.62]). In contrast, SLIP-2 score performed well (area under the receiver operating characteristic curve [95% CI], 0.84 [0.81 to 0.88]). Internal validation indicated similar discrimination, with an area under the receiver operating characteristic curve of 0.84.
In this multicenter cohort of patients at risk for ARDS, the SLIP-2 score outperformed the original SLIP score. If validated in an independent sample, this tool may help identify surgical patients at high risk for ARDS.
Acute respiratory distress syndrome is a serious postoperative complication that remains difficult to predict in individual patients
The investigators tested and refined the previously reported surgical lung injury prediction model in a multicenter cohort of 1,562 at-risk surgical patients
Sepsis, high-risk aortic vascular surgery, high-risk cardiac surgery, emergency surgery, cirrhosis, admission location other than home, increased respiratory rate (20 to 29 and ≥30 breaths/min), Fio2 greater than 35%, and Spo2 less than 95% were all significant predictors of acute respiratory distress syndrome
ACUTE respiratory distress syndrome (ARDS) is a leading cause of postoperative hypoxemic respiratory failure.1 Postoperative ARDS substantially affects patient-important outcomes, with associated mortality rates between 15 and 45%.1–4 Beyond mortality, long-term health status can be compromised, with functional limitations sometimes persisting 5 yr after an ARDS episode.5 Postoperative respiratory failure also increases healthcare resource utilization, with substantial increases in lengths of stay and hospital costs.6
Besides best-practice supportive therapies such as lung-protective ventilation,7 conservative fluid management,8 and early neuromuscular blockade,9 no proven treatment exists for established ARDS. Thus, interest in ARDS prevention is increasing. Indeed, prevention of ARDS is a key priority for the National Heart, Lung, and Blood Institute. This newfound emphasis has manifested in recent ARDS working group publications, including the recent “Notice of Intent to Publish a Funding Opportunity Announcement for Clinical Centers for the National Heart, Lung, and Blood Institute Prevention and Early Treatment of Acute Lung Injury Clinical Trials Network.”*10 Importantly, recent investigations suggest the potential to prevent postoperative complications by modifying perioperative care. Futier et al.11 noted fewer postoperative complications and improved healthcare resource utilization in a multicenter clinical trial evaluating the effect of intraoperative lung-protective ventilation strategies for patients undergoing major abdominal surgery. Indeed, the efficacy of such interventions may be particularly notable in surgical patients at high risk for complications such as ARDS.
To make progress in preventing postoperative ARDS, we need an improved ability to identify patients at high risk for this life-threatening complication. To this end, recent publications have outlined risk-prediction algorithms that may be used to identify patients at high risk for ARDS.3,12–14 These algorithms have specifically targeted mixed medical and surgical populations with acute illnesses (e.g., Lung Injury Prediction Score [LIPS]),12 elective surgical populations (e.g., surgical lung injury prediction [SLIP] score),3,14 and those who have experienced major trauma.13 In contrast, the risk of postoperative ARDS among heterogeneous, high-risk surgical populations, including patients seen in the emergency department with acute illnesses, has not been adequately addressed. In addition, previous prediction algorithms specifically evaluating risk of ARDS in surgical populations (e.g., the SLIP score) have been limited to single-center studies, thereby limiting their generalizability.3,13,14
To address these important knowledge gaps, we leveraged the infrastructure of the U.S. Critical Illness and Injury Trials Group and the LIPS study cohort.12 The LIPS study12 was a multicenter, prospective, observational cohort study to determine the frequency and outcome of acute lung injury (ALI)/ARDS in patients at risk (those with at least one major predisposing condition for ALI/ARDS). That primary study also aimed to validate the previously derived LIPS.15 The study population for the LIPS investigation, involving 20 hospitals in the United States and 2 in Turkey, included 5,584 medical patients and surgical patients undergoing a wide range of elective and emergent surgical procedures. In the current study, we aimed to test and refine the previously reported SLIP model in a multicenter cohort of surgical patients at risk for postoperative ARDS.
Materials and Methods
Study Design
This is a secondary analysis of a previously reported multicenter, prospective cohort investigation.12,16 The prospective study was approved by the institutional review board at each participating institution. Approval was also granted for ancillary studies such as the current investigation. The Strengthening the Reporting of Observational Studies in Epidemiology guidelines were followed in the design and reporting of this observational study.16,17
Study Population
Details of the study population have been previously described.12,16 In brief, consecutive adult patients were enrolled prospectively in 19 hospitals and retrospectively (after hospital discharge) in 3 hospitals over a 6-month period, beginning in March 2009. Participating institutions included both community and academic medical centers. Twenty of the included hospitals were located in the United States, with 2 additional institutions located in Turkey. Inclusion criteria consisted of admission to the hospital with the presence of at least one major risk factor for ARDS and age older than 18 yr. Variables considered major risk factors for ARDS included aspiration, pneumonia, sepsis, shock, pancreatitis, high-risk trauma, or high-risk surgery. Standardized definitions were used to identify these risk factors (high-risk trauma18–20 , high-risk surgery,1,21–23 aspiration,18,20,22,24 sepsis,19,20,22,25 shock,25–27 pneumonia,18,22,25,28 and pancreatitis25,29–32 ).
For the current investigation, the study population described above was restricted to patients who underwent a surgical procedure. Patients were excluded if they came to the hospital with prevalent ARDS, were transferred from an outside hospital, died in the emergency department, were admitted for comfort or hospice care, or had been previously enrolled in the study (i.e., for patients with multiple hospital admissions, only the first hospital admission was included in the database).16
Predictor Variables
In the initial prospective cohort study,12 demographic information and additional clinical characteristics were collected during the first 6 h after initial emergency department evaluation or preoperatively at the time of hospital admission for those undergoing high-risk elective surgery by review of the study participant’s medical record. ARDS risk factors (high-risk trauma, high-risk surgery, aspiration, pneumonia, sepsis, shock, and pancreatitis) and potential risk modifiers, including alcohol abuse,1,33–35 obesity (body mass index >30 kg/m2),36 hypoalbuminemia (albumin <3.4 g/dl),18,37 use of chemotherapy currently or within 6 months of the hospital admission,34,38 fraction of inspired oxygen (Fio2) greater than 35%,39 tachypnea (respiratory rate ≥30 breaths/min),18,34 oxygen saturation (Spo2) less than 95%,12 acidosis (pH <7.35),12,18 and diabetes mellitus,18,40 were also extracted. Standardized definitions were used to characterize these variables, as previously described.12
Details regarding the nature and urgency of the surgical procedures performed were also recorded. Procedural categories characterized as high risk for ARDS in the initial prospective cohort study included cardiac surgery, aortic surgery, thoracic surgery, spine surgery, and acute abdominal surgery.12 On the basis of recent data suggesting differential risks for ARDS among specific procedures within the high-risk procedural categories of cardiac, aortic, and thoracic (noncardiac) surgery, these procedural categories were further characterized into low or high risk, as previously described.3 Investigators and study coordinators at each site reviewed the study’s standard operating procedures and received structured online training for definitions of each risk factor before study initiation. In the three hospitals that collected data retrospectively, the investigators followed the same protocols and definitions, but data were collected after hospital discharge.
Outcome Variables
The primary outcome was development of ALI or ARDS during the hospitalization. Of note, new consensus definitions for adjudicating diagnoses of ARDS have recently been endorsed.41 An important change in these new recommendations is the removal of the term acute lung injury or ALI, which previously characterized less severe forms of ARDS.42 However, because this investigation was multicenter and because both enrollment and ALI/ARDS adjudication were performed before this new ARDS definition was endorsed, we have maintained the ALI and ARDS definitions proposed by the 1994 American-European consensus conference.42 Specifically, to be allocated a diagnosis of ALI or ARDS, the following elements were required: development of acute, bilateral pulmonary infiltrates on chest radiography, a Pao2/Fio2 ratio less than 300 (<200 for ARDS), and absence of signs of left atrial hypertension as the main explanation for pulmonary edema. All these diagnostic criteria needed to be present during the same 24-h interval to meet criteria for ALI/ARDS. The adjudication of ALI/ARDS was made locally at each participating center by a member of the study team. Investigators and study coordinators at each site underwent a structured online tutorial for the assessment of ALI/ARDS before study initiation.
Statistical Analysis
Sample Size Estimation
This study constituted a secondary data analysis of an existing database. As such, no formal power analyses were conducted. The available sample size, however, was evaluated to ensure there would be an adequate number of ALI/ARDS cases to support the logistic regression modeling. Preliminary examination of the database suggested that the ALI/ARDS incidence in this cohort would be 8% (approximately 125 ALI/ARDS events). Using the standard rule of thumb of having at least 10 events for each covariate, the database was believed to be sufficiently large, provided that model-building strategies were used to minimize the number of independent variables used simultaneously. The following sections detail this process.
Analysis Plan Overview
Dichotomous variables are presented as number (percentage); continuous data are presented as median (interquartile range [IQR]). For initial descriptive analyses, comparisons between the patients with and without ALI/ARDS during their hospitalization were performed with Pearson chi-square tests or Fisher exact tests, as appropriate, for categorical variables. Continuous variables were tested with the Wilcoxon rank sum test (Mann–Whitney U statistic).
The primary analysis consisted of evaluating the performance of the previously reported SLIP model (table 1)3 compared with the performance of a revised predictive algorithm (termed SLIP-2) that included additional patient characteristics which may be predictive of ALI/ARDS in this population. This revised algorithm analysis was planned a priori, recognizing the potential for unique ALI/ARDS predictive factors in this more heterogeneous surgical population, which included patients with major risk factors for ALI/ARDS as well as patients undergoing emergency surgery.
Modeling Overview
For validation of the SLIP prediction algorithm, SLIP points were used to calculate the risk score for each study participant, as described previously.3 The association between the SLIP score and development of ALI/ARDS was evaluated using single-variable logistic regression, with the cumulative SLIP score as the independent variable. Model discrimination was assessed by calculating the area under the receiver operating characteristic curve (AUC), otherwise known as the concordance statistic (or C statistic). Model calibration was assessed using the Hosmer–Lemeshow goodness-of-fit test statistic.
The overall goal of the revised SLIP-2 predictive algorithm analysis was to develop a parsimonious collection of variables that would improve prediction of ALI/ARDS in a heterogeneous high-risk surgical population. To accomplish this objective, model building proceeded in two stages (see figure, Supplemental Digital Content 1, https://links.lww.com/ALN/B40). In the first stage, biologically plausible predictor variables were categorized into specific domains representing mechanistic pathways believed to affect the development of ALI/ARDS. Of note, many of the ARDS risk factors were evaluated in more than one domain because of their potential role in more than one pathway that may be involved in ARDS pathogenesis. Multiple imputation was used for missing values.43 The second stage of the analysis (the domain analysis) was accomplished in three steps.44
In step 1 of the domain analysis, a “fully conditional specification” multiple imputation method was used to impute 10 different datasets for the missing variables based on the remaining variables in each specific domain. Univariate logistic regression was then performed for each variable in the 10 imputed datasets. The results across the imputation dataset were synthesized (with PROC MIanalyze in SAS; SAS Institute, Inc., Cary, NC) to obtain overall parameter estimates and P values for each variable considered in the domain. Variables that passed the initial screening evaluation (P < 0.25) were then assessed for multicollinearity before variable reduction in each domain. When multicollinearity was detected, the variable with the largest likelihood ratio test value was chosen for further modeling.
In the second step, a maximal model consisting of all screened-in variables within a domain from step 1 was specified, and backward elimination was used to reduce the number of independent variables in the domain. Modeling was again conducted on the imputed dataset, and a more stringent criterion of P value less than 0.05 was used to retain the variable at the end of step 2. Once variables were selected within a domain, all selected variables were pooled together to develop the overall predictive model in step 3 of the analysis. Because the missing data pattern was altered when multiple domains were combined, 10 new imputed datasets were generated using all the variables considered in the third step. Multicollinearity and screening were conducted as in step 1 of the model-building process. Backward elimination was used with a P value less than 0.05 requirement for retention in the final model.
To facilitate the development of the revised SLIP-2 score, continuous variables from the domain analyses were reclassified as categorical variables in the final models using previously defined and clinically relevant cut points: Fio2 (>35 vs.≤35%), respiratory rate (20 to 29 and ≥30 vs. 0 to 19 breaths/min), and Spo2 (<95 vs. ≥95%).12 SLIP-2 points were then assigned to each predictor in the final model by multiplying the predictor’s parameter estimate (β coefficient) by 10 and rounding down to the nearest integer.
For validation of the SLIP-2 prediction algorithm, a two-step process was used. First, SLIP-2 points were assigned and summed for each participant. The association between the SLIP-2 score and development of ALI/ARDS for the final model was then evaluated in the original dataset (without imputation) via single-variable logistic regression, with the cumulative SLIP-2 score as the independent variable. Model discrimination and calibration were evaluated on the basis of the AUC and the Hosmer–Lemeshow goodness-of-fit test statistic, respectively. Second, model optimism was estimated using Harrell’s RMS package for R.† This model validation approach uses the bootstrap method to resample the data to account for sampling variation that may contribute to overfitting and overestimation of model fit, or optimism. Optimism in model performance, as measured by the change (decrease) in the C statistic and the le Cessie and van Houwelingen test,45 was assessed for each of the 10 multiple imputation datasets. All the model building and analysis steps were carried out using SAS software v9.3. Model validation using the RMS package was conducted using R v.3.0.0 (R Foundation for Statistical Computing, Vienna, Austria).
Results
A study participant flow diagram is presented in figure 1. After exclusions, 1,562 patients met all inclusion criteria and were included in this investigation. Postoperative ALI/ARDS developed in 117 study participants (7.5%; 95% CI, 6.3 to 8.9%). The mortality rate was significantly higher among those with ALI/ARDS than that among those without (10.3 vs. 1.6%; P < 0.001), as was the median (IQR) hospital length of stay (14 [7 to 25] days vs. 6 [4 to 10] days; P < 0.001). Baseline demographics and clinical characteristics are shown in table 2.
Study participant flow diagram. ALI = acute lung injury; ARDS = acute respiratory distress syndrome; ED = emergency department; LIPS = Lung Injury Prediction Score.
Study participant flow diagram. ALI = acute lung injury; ARDS = acute respiratory distress syndrome; ED = emergency department; LIPS = Lung Injury Prediction Score.
Of the 1,562 study participants, 1,433 had all variables needed to calculate the original SLIP score. All 129 patients for whom a SLIP score could not be calculated were missing information related to alcohol use. SLIP scores for these patients (n = 1,433) ranged from 0 to 75, with a median (IQR) of 11 (0 to 21). Median (IQR) SLIP score was 17 (0 to 25) in those with ALI/ARDS (n = 108) and was 11 (0 to 19) in those without (P = 0.06). The original SLIP predictive algorithm was a poor discriminator between patients who did and did not have ALI/ARDS, with an AUC (95% CI) of 0.56 (0.50 to 0.62).
As expected, in the initial descriptive analysis, numerous baseline demographics and clinical characteristics, as well as known ALI/ARDS risk factors and risk modifiers, differed significantly between those with and without ALI/ARDS (table 2). The first stage of the domain analysis (table 3) identified 29 unique variables from seven domains to be significantly associated with ALI/ARDS (P < 0.25). Traumatic brain injury, chronic obstructive pulmonary disease, and respiratory rate, in domains 1, 4, and 7, respectively, were excluded because of multicollinearity. The rest of the significant variables were carried forward to the second stage of domain analysis, in which only 15 variables retained significance (P < 0.05). None of these 15 variables showed multicollinearity. Of these, 14 variables were included in the final SLIP-2 multivariable model (table 4). Aspiration was excluded because of the limited number of cases (n = 4). A total of nine variables remained as significant predictors of postoperative ALI/ARDS in the final model: admission source other than home, sepsis, cirrhosis, high-risk cardiac surgery, high-risk aortic vascular surgery, emergency surgery, increased respiratory rate (with two defined cutoffs: 20 to 29 and ≥30 breaths/min), Fio2 higher than 35%, and oxygen saturation less than 95%. The final SLIP-2 scoring algorithm is presented in table 5.
Surgical lung injury prediction-2 scores for all the study participants (N = 1,562) ranged from 0 to 57, with a median (IQR) of 9.5 (0 to 20). The median (IQR) SLIP-2 scores were 27 (20 to 37) for those with ALI/ARDS versus 7 (0 to 18) for those without ALI/ARDS (P < 0.001). The SLIP-2 model effectively discriminated between those who did and did not have development of ALI/ARDS (C statistic, 0.84; 95% CI, 0.81 to 0.88). Figure 2 shows the discrimination comparison between the SLIP and SLIP-2 scores for this sample (P < 0.001). Furthermore, model validation using the 10 imputed datasets showed similar discrimination (mean C statistic, 0.84). A good fit (P > 0.05 on the Hosmer–Lemeshow goodness-of-fit test) was observed in all 10 imputed datasets. Use of Harrell’s RMS package with 200 bootstrap replicates per imputed dataset showed that the SLIP-2 score remained a valid predictor for the development of ALI/ARDS. The mean optimism-corrected C statistic over the multiple imputation datasets was 0.837, which is highly consistent with the estimate obtained without bootstrap resampling. Similarly, all of the le Cessie and van Houwelingen tests,45 which are similar in interpretation to the Hosmer–Lemeshow goodness-of-fit test, indicated good calibration (P > 0.05) for all 10 datasets.
Receiver operating characteristic curves comparing the original surgical lung injury prediction (SLIP) model with the revised SLIP-2 predictive algorithm. AUC = area under the receiver operating characteristic curve.
Receiver operating characteristic curves comparing the original surgical lung injury prediction (SLIP) model with the revised SLIP-2 predictive algorithm. AUC = area under the receiver operating characteristic curve.
By AUC analysis, the optimal cutoff for the SLIP-2 score to maximize the Youden index was 19. Using this cutoff, we found a positive likelihood ratio (95% CI) of 3.31 (2.89 to 3.79), a negative likelihood ratio of 0.24 (0.16 to 0.36), a sensitivity of 82% (74 to 88%), and a specificity of 75% (73 to 78%). Table 6 shows the sensitivity analysis using the cutoff of 19 and 2 additional SLIP-2 cut points. Three groups of patients were defined on the basis of the SLIP-2 cut points identified in the sensitivity analysis: persons at low risk (<10 points on SLIP-2), moderate risk (10 to 30 points on SLIP-2), and high risk (>30 points on SLIP-2) for postoperative ALI/ARDS. Using all available data (without multiple imputation for missing data), SLIP-2 score could be calculated for 1,246 study patients (316 [20%] had missing observations for SLIP-2 scoring criteria variables). By this scale, 50% of study patients (n = 623) were assigned to the low-risk group, 40.4% (n = 503) were in the moderate-risk group, and 9.6% (n = 120) were in the high-risk group. The associated frequency of ALI/ARDS in each of these groups is shown in figure 3.
Incidence of acute respiratory distress syndrome (ARDS) by risk group. Frequency of ARDS increased with increasing associated risk, as determined by the surgical lung injury prediction model 2 (SLIP-2) score: low-risk group (n = 623), moderate-risk group (n = 503), and high-risk group (n = 120).
Incidence of acute respiratory distress syndrome (ARDS) by risk group. Frequency of ARDS increased with increasing associated risk, as determined by the surgical lung injury prediction model 2 (SLIP-2) score: low-risk group (n = 623), moderate-risk group (n = 503), and high-risk group (n = 120).
Discussion
In this multicenter investigation evaluating surgical patients at high risk for ALI/ARDS, we tested the predictive accuracy of the recently described SLIP model.3 Recognizing the potential for differing risk profiles in this more heterogeneous and high-risk cohort, we aimed to further refine the SLIP algorithm, thereby permitting its use in more diverse surgical populations, such as those with coexisting ARDS risk factors or those undergoing emergency surgery. Our results suggest that the original SLIP score performs poorly in this more diverse and acutely ill surgical population. In contrast, the refined algorithm (SLIP-2) accurately estimated risk of postoperative ALI/ARDS in this high-risk cohort.
The inability to identify patients who are at greatest risk for ARDS is a major barrier to progress in ARDS prevention, and recent reports from content experts have specifically called for studies aiming to facilitate the identification of high-risk populations.10,30 In a workshop convened by the National Heart, Lung, and Blood Institute evaluating the current state of ARDS research, the development of strategies to perform ARDS prevention trials was identified as a key strategic priority.10 As this working group noted, “Identification of patients at risk early in the course of their illness (e.g., in the emergency department or operating room) may allow earlier implementation of ALI prevention strategies.”10 In addition, targeted enrollment of study participants using innovative risk-assessment tools can greatly enhance study efficiency while decreasing study-related costs. The potential for such gains has recently been described for the multicenter Lung Injury Prevention Study with Aspirin clinical trial.12,46
Recently, several studies have begun to address this topic. Gajic et al.12 developed the LIPS model for mixed medical and surgical populations; similarly, the SLIP algorithm was developed for use in elective surgical populations.3 Watkins et al.13 have also developed an ARDS prediction model in a trauma population. Although these investigations provide an excellent starting point, their findings are unlikely to be generalizable to more diverse and high-risk surgical populations. Indeed, we have confirmed that the SLIP score3,12 performs poorly in this context. Moreover, previous investigations targeting surgical populations have been restricted to a single center, thereby limiting their generalizability. The current study evaluated a multicenter surgical cohort with major risk factors for ARDS, as well as those undergoing emergency surgery. This population is of particular interest because the risk of ARDS is high and the associated effects on patient-important outcomes are significant.
It may not be surprising that the SLIP model did not perform well in this study given the stark differences in the target populations. Specifically, the SLIP model was designed to detect the risk of ALI/ARDS in patients undergoing elective surgery and in those with no other ARDS risk factors. In contrast, inclusion into the current study required that patients have at least one major ARDS risk factor. Furthermore, 272 of our patients (17.4%) underwent an emergency surgical procedure. All such patients were excluded from the SLIP model development cohort.
In contrast, the refined SLIP-2 prediction model performed well in this high-risk cohort. Nine ALI/ARDS risk factors were identified from four major ARDS-related domains: oxidative stress or reperfusion injury (sepsis), type of surgical procedure (high-risk cardiac surgery, aortic vascular surgery, and emergency surgery), baseline health status (cirrhosis, admitted from a location other than home), and physiologic markers of acute illness (tachypnea, Fio2 >35%, and Spo2 <95%). This parsimonious collection of routinely available clinical variables predicted the risk of postoperative ALI/ARDS with very good accuracy (AUC [95% CI], 0.84 [0.81 to 0.88]). Furthermore, additional validation of the model using Harrell’s optimism approach suggested the current model to be less optimistic.
Importantly, all but one of the identified ALI/ARDS predictor variables appear robust, with previous investigations corroborating their association with ALI/ARDS (admission source,18 sepsis,19,20,22,25 high-risk cardiac surgery,1,3,12,47–50 high-risk aortic surgery,1,3,12 emergency surgery,12,48 tachypnea,18,34 oxygen supplementation,12,39 and hypoxemia12 ). Although the association between cirrhosis and ALI/ARDS has been less consistently documented,3,18,51,52 there is clear evidence for biologic plausibility because liver injury can affect the host inflammatory response and increase lung microvascular permeability.52 Moreover, surgical patients with advanced liver disease are at risk for blood product administration, which is another risk factor for ALI/ARDS.4,18,50
Of note, some statistical factors may partly explain why SLIP-2 outperformed the original SLIP score. First, the SLIP-2 model included more variables to estimate the risk of ALI/ARDS. Second, the SLIP score was developed in a different dataset, and its application to this study represents a validation study. As a result, it is not surprising that the performance was lower in this sample. Third, external validation of SLIP-2 was not possible at this time for lack of a unique validation dataset, so the parameter estimates and variables selected for inclusion in the model may be “overfit” to the available data. We attempted to mitigate this concern through bootstrap resampling, but further research using independent data is needed to fully understand how robust SLIP-2 will be outside the confines of the data used to develop the model.
This investigation has several strengths. Unlike the previous single-center studies,3,13,14 this study includes a large, heterogeneous, multicenter, surgical population. This target cohort is of particular interest given these patients’ risk of postoperative ALI/ARDS. This study also included clinical data that are readily available in the medical record. Thus, we believe that we have meaningfully enhanced the usability of the model. The vast majority of the data were also collected prospectively with explicit standard operating procedures, and the statistical methodology was deliberate and detailed. We believe that these factors help to ensure both the accuracy of the data collected and the validity of the study findings.
In addition to the strengths identified above, several limitations deserve note. First, this study was a secondary analysis of a previous prospective cohort investigation, and potentially relevant ARDS risk predictors may not have been included in the original data extraction. Recognizing the importance of the surgical procedure in determining the risk of postoperative ARDS, we supplemented the original list of variables with additional details regarding the surgical procedure. However, the multicenter nature of the initial investigation precluded our ability to include additional details relating to intraoperative exposures and patient responses. Therefore, we were unable to evaluate intraoperative and postoperative risk predictors, despite many such factors having been previously associated with postoperative ALI/ARDS (e.g., blood transfusion, ventilator management, longer surgical duration, fluid resuscitation, and disturbed hemodynamics).1,2,4,14,49,50,53–56 Nonetheless, we emphasize that the expected use of this ARDS prediction algorithm is to determine risk of ARDS as early as possible, preferably before the surgical procedure.
Our investigation also shares limitations with other clinical ARDS studies relating to the reproducibility of the ARDS diagnosis. Reproducibility may be particularly pertinent when studying postoperative ARDS because atelectasis and ventilatory support are not uncommon after surgery and may confound the ARDS diagnosis. To mitigate this concern, structured training in ALI/ARDS assessment was required for all primary investigators at each site before study onset. In addition, the vast majority of data were collected prospectively, ensuring close follow-up for ALI/ARDS development. The total number of ALI/ARDS cases was also somewhat limited (n = 117). This limited number of outcomes, as well as the low frequency of some potentially important risk factors (e.g., aspiration, pneumonia, amiodarone use), may have masked important associations with ALI/ARDS. Additional concerns include the potential effects of missing data, as well as the lack of an independent validation cohort. To address the issue of missing data, a “fully conditional specification” multiple imputation strategy was used. Also, although an independent validation cohort is not available, the new SLIP-2 algorithm was internally validated by evaluating the final SLIP-2 model in the original dataset (without imputation) via single-variable logistic regression.
In addition to the noted limitations, we also acknowledge the modest overall performance of the SLIP-2 algorithm. Specifically, the sensitivity and specificity at the optimal cut point were 82% (95% CI, 74 to 88%) and 75% (95% CI, 73 to 78%), respectively. The substantial heterogeneity of the study population most likely affected model performance. Although we might expect improved accuracy if the model was developed in a more restrictive patient population, the goal of this work was to create a model that can predict risk of ALI/ARDS in a heterogeneous high-risk surgical population. Moreover, we note that our predictive accuracy is similar3,12 or favorable13 to the previously reported prediction algorithms that have focused on more selective surgical populations. We further recognize that model performance might be improved with additional variables and more complex modeling strategies. However, our primary aim was to develop and validate an accurate, efficient, and scalable model that could be used for the preoperative identification of high-risk participants for future ARDS prevention trials. Time-sensitive studies on ARDS prevention require very efficient risk-prediction strategies. Although increasing complexity of a model may provide additional predictive accuracy, this would come at the cost of usability and scalability. In addition, alternate cut points may be selected if the intended use of the model requires improved sensitivity or specificity (table 6).
In conclusion, we have refined and internally validated a parsimonious postoperative ARDS risk–prediction model in a multicenter cohort of patients at high risk for postoperative ARDS. The SLIP-2 model effectively discriminates risk of postoperative ALI/ARDS. This tool may improve clinicians’ ability to estimate risk of postoperative ARDS. By improving our ability to identify patients at higher risk for ARDS, the SLIP-2 score may also enhance the feasibility of mechanistic studies and ARDS prevention trials.
Acknowledgments
The authors thank the U.S. Critical Illness and Injury Trials Group: Lung Injury Prevention Study for collaborating with them on this investigation (see appendix for a listing of collaborating investigators and clinical centers).
This study is supported by a Career Development Grant provided by the Foundation for Anesthesia Education and Research, Rochester, Minnesota; grant numbers U01-HL108712-01, UL1 TR000135, and KL2 TR000136, National Institutes of Health, Bethesda, Maryland; and the Mayo Clinic Critical Care Research Committee, Rochester, Minnesota.
Competing Interests
The authors declare no competing interests.
Available at: http://grants.nih.gov/grants/guide/notice-files/NOT-HL-13-168.html. Accessed March 15, 2014.
Available at: http://cran.r-project.org/web/packages/rms/index.html. Accessed October 21, 2013.
References
U.S. Critical Illness and Injury Trials Group: Lung Injury Prevention Study Participating Centers and Investigators
Mayo Clinic, Rochester, Minnesota: Adil Ahmed, M.D.; Ognjen Gajic, M.D.; Michael Malinchoc, M.S.; Daryl J. Kor, M.D.; Bekele Afessa, M.D.; Rodrigo Cartin-Ceba, M.D.; Rickey E. Carter, Ph.D.; Departments of Internal Medicine, Health Sciences Research and Anesthesiology
University of Missouri, Missouri, Columbia: University of Missouri-Columbia: Ousama Dabbagh, M.D., M.S.P.H., Associate Professor of Clinical Medicine; Nivedita Nagam, M.D.; Shilpa Patel, M.D.; Ammar Kar; and Brian Hess
University of Michigan, Ann Arbor, Michigan: Pauline K. Park, M.D., F.A.C.S., F.C.C.S., Co-Director, Surgical Intensive Care Unit, Associate Professor, Surgery; Julie Harris, Clinical Research Coordinator; Lena Napolitano, M.D.; Krishnan Raghavendran, M.B.B.S.; Robert C. Hyzy, M.D.; James Blum, M.D.; Christy Dean
University of Texas Southwestern Medical Center in Dallas, Dallas, Texas: Adebola Adesanya, M.D.; Srikanth Hosur, M.D.; Victor Enoh, M.D.; Department of Anesthesiology, Division of Critical Care Medicine
University of Medicine and Dentistry of New Jersey, New Jersey: Steven Y. Chang, Ph.D., M.D., Assistant Professor, MICU Director, Pulmonary and Critical Care Medicine; Amee Patrawalla, M.D., M.P.H.; Marie Elie, M.D.
Brigham and Women’s Hospital, Boston, Massachusetts: Peter C. Hou, M.D.; Jonathan M. Barry, B.A.; Ian Shempp, B.S.; Atul Malhotra, M.D.; Gyorgy Frendl, M.D., Ph.D.; Departments of Emergency Medicine, Surgery, Internal Medicine and Anesthesiology, Perioperative and Pain Medicine, Division of Burn, Trauma, and Surgical Critical Care
Wright State University Boonshoft School of Medicine and Miami Valley Hospital, Dayton, Ohio: Harry Anderson III, M.D., Professor of Surgery; Kathryn Tchorz, M.D., Associate Professor of Surgery; Mary C. McCarthy, M.D., Professor of Surgery; David Uddin, Ph.D., D.A.B.C.C., C.I.P., Director of Research
Wake Forest University Health Sciences, Winston-Salem, North Carolina: J. Jason Hoth, M.D., Assistant Professor of Surgery; Barbara Yoza, Ph.D., Study Coordinator
University of Pennsylvania, Philadelphia, Pennsylvania: Mark Mikkelsen, M.D., M.S.C.E., Assistant Professor of Medicine, Pulmonary, Allergy, and Critical Care Division; Jason D. Christie, M.D.; David F. Gaieski, M.D.; Paul Lanken, M.D.; Nuala Meyer, M.D.; Chirag Shah, M.D.
Temple University School of Medicine, Philadelphia, Pennsylvania: Nina T. Gentile, M.D., Associate Professor and Director, Clinical Research, Department of Emergency Medicine, Temple University School of Medicine; Karen Stevenson, M.D., Resident, Department of Emergency Medicine; Brent Freeman, B.S., Research Coordinator; Sujatha Srinivasan, M.D., Resident, Department of Emergency Medicine
Mount Sinai School of Medicine, New York, New York: Michelle Ng Gong, M.D., M.S., Assistant Professor, Pulmonary, Critical Care, and Sleep Medicine, Department of Medicine
Beth Israel Deaconess Medical Center, Boston, Massachusetts: Daniel Talmor, M.D., Director of Anesthesia and Critical Care, Associate Professor of Anaesthesia, Harvard Medical School, Boston, Massachusetts; S. Patrick Bender, M.D.; Mauricio Garcia, M.D.
Massachusetts General Hospital Harvard Medical School, Boston, Massachusetts: Ednan Bajwa, M.D., M.P.H., Instructor in Medicine; Atul Malhotra, M.D., Assistant Professor; B. Taylor Thompson, Associate Professor; David C. Christiani, M.D., M.P.H., Professor
University of Washington, Harborview, Seattle, Washington: Timothy R. Watkins, M.D., Acting Instructor, Department of Medicine, Division of Pulmonary and Critical Care Medicine; Steven Deem, M.D.; Miriam Treggiari, M.D., M.P.H.
Mayo Clinic Jacksonville: Emir Festic, M.D.; Augustine Lee, M.D.; John Daniels, M.D.
Akdeniz University, Antalyia, Turkey: Melike Cengiz, M.D., Ph.D.; Murat Yilmaz, M.D.
Uludag University, Bursa, Turkey: Remzi Iscimen, M.D.
Bridgeport Hospital, Yale New Haven Health, New Haven, Connecticut: David Kaufman, M.D., Section Chief, Pulmonary, Critical Care, and Sleep Medicine, Medical Director, Respiratory Therapy
Emory University, Atlanta, Georgia: Annette Esper, M.D.; Greg Martin, M.D.
University of Illinois at Chicago, Chicago, Illinois: Ruxana Sadikot, M.D., M.R.C.P.
University of Colorado, Boulder, Colorado: Ivor Douglas, M.D.
Johns Hopkins University: Jonathan Sevransky, M.D., M.H.S., Assistant Professor of Medicine, Medical Director, JHBMC MICU.