Development and Validation of a Score for Prediction of Postoperative Respiratory Complications

PULMONARY complications such as pneumonia, failure to wean, and postextubation respiratory failure represent the second most frequent type of postoperative complication after wound infection,^1–3  with an incidence estimated to range from 2.0 to 5.6% following surgery.^2,4–6  Postextubation respiratory failure has been shown to be one of the most significant factors associated with poor patient outcomes, leading to a longer hospital stay,^2,3,7  higher financial cost,^1,2  and increases in 30-day mortality to as high as 18-fold.^7,8  Predicting which patients are at highest risk for postextubation respiratory failure is clinically important as it allows for triage to care settings in the postoperative period, which allows for more intensive monitoring.

Because postoperative complications better predict short- and long-term mortality than preoperative and intraoperative factors,²  recent research has focused on identifying preoperative factors that predispose patients to postextubation respiratory failure.^7–9  These studies have identified a range of factors that are independently associated with postoperative respiratory failure including age, chronic obstructive pulmonary disease history, American Society of Anesthesiologists (ASA) score, surgery type, and functional status, to name a few. Despite the great number of identified predictors, there is no widely accepted simple evaluation tool to quickly assess the likelihood of postextubation respiratory failure at the bedside. Moreover, available scores do not predict early postoperative respiratory outcome variables, such as deoxygenation in the early postoperative period because most of them are based on a surgical database that do not include anesthesia record derived information,^4,5,8–10  Furthermore, available studies focus on respiratory failure occurring within 30 days after surgery. We were interested to focus on early postoperative respiratory failure, which we believe can be attributed more likely to the potentially preventable effects of anesthesia.⁸  Our aim was to develop and validate a simple score that can be used by clinicians in the triage of surgical patients at the time of initial preoperative evaluation, to estimate their risk of postoperative respiratory failure.

This study is a cohort study of all surgical cases undergoing general anesthesia at Massachusetts General Hospital, for whom inpatient admission was planned. Massachusetts General Hospital is a multidisciplinary, tertiary care facility and teaching hospital in Boston, Massachusetts. On an average, 40,000 surgical procedures are performed each year. Data from three databases were retrieved and combined to provide deidentified pre-, intra-, and postoperative information: the Anesthesia Information Management System, the respiratory therapy billing database, and the hospital billing database. This study was approved by the Institutional Review Board of the Massachusetts General Hospital, Boston, Massachusetts.

Cases identified were those that had a surgical procedure between January 1, 2007 and April 30, 2010 within the main operative suites. Surgical cases were only included if the patient was intubated at the beginning and extubated at the end of the procedure.

Cases were excluded if patients were younger than 18 yr of age, if they received their care predominantly outside the main campus, if information regarding main demographics (e.g., age, sex, ASA score) were missing, or the surgery was planned as an ambulatory procedure. Further, cases were excluded if the patient had subsequent surgical procedures that were separated by less than 10 days, to minimize the correlated nature of the data.

Patient characteristics considered included those routinely collected for the preanesthesia evaluation including sex, age, body mass index, ASA score,^11,12  and information on specific illnesses, which were assembled into the Charlson Comorbidity Index.¹³  The latter was calculated using codes of the 9th version of the International Statistical Classification of Diseases and Related Health Problems.¹⁴  This information had to be documented before surgery in the online accessible records. Further information was retrieved regarding the intended surgery type (burn, gynecology, neurosurgery, urology, cardiac, general, oral/maxillofacial, orthopedic, plastic, pediatric, thoracic, transplant, trauma and vascular surgery, and surgical oncology) and whether the procedure was designated as emergent. An emergency procedure was defined as a surgical procedure for which the patient is supposed to arrive in the operating room no later than 30 min after the patient has been booked as an emergent case, accompanied by the person who requested the case to be conducted.

The primary endpoint for this study was reintubation, defined as any intubation after primary extubation in the operating room within the first 3 postoperative days requiring unplanned mechanical ventilation either in the operating room, the postanesthesia care unit, or the intensive care unit. The study endpoints were either retrieved based on online available information, or chart review. Reintubations for a surgical complication, requiring an intubation for a subsequent surgical procedure were not part of the primary endpoint.

Medical records for patients who were identified as potentially having developed an endpoint (extubation billing code followed by evidence of respiratory service afterward) were reviewed and adjudicated based on consensus by an independent committee, who were blinded to the preoperative patients’ characteristics.

Further, the time between primary extubation and reintubation was determined.

We also tested the predictive value of the derived model against further endpoints. The first, reintubation after primary extubation based solely on information available online, was chosen to provide comparability to external databases, which are based on billing data. It was defined as a combination of an existing extubation code at the end of the surgery in the Anesthesia Information Management System database and an entry in the respiratory therapy billing database indicating postoperative mechanical ventilation. The interval between those two codes had to be less than or equal to 3 days, as determined on the timestamps. Given the importance of postextubation respiratory failure as a source of postoperative mortality, we also tested the model against the endpoint of all-cause mortality within index hospitalization, and due to the potential for survivor bias, also against a composite endpoint of death and adjudicated reintubation.

Finally, another anesthesia-related endpoint was chosen, combining deoxygenation below 80% or reintubation, both within the first 10 min after extubation. A hemoglobin oxygen saturation below 80% with a decrease of at least 3% within the first 10 min after extubation was defined as a desaturation. Data from pulse oximeters are sampled every 20 s by the Anesthesia Information Management System, and the nadir value, taken during 10 epochs of 1-min duration each before and after extubation, were used for statistical analysis.

Continuous, normal distributed variables were expressed as mean ± SD, ordinal as median (interquartile range [IQR]), and categorical variables as percentages (frequency), if not otherwise specified. The entire cohort was randomly divided equally into a training and validation dataset. For this, a probability-based approach with simple random sampling was chosen.¹⁵  Differences between the training and validation dataset regarding the demographics and outcomes were examined, using unpaired t test for continuous, unpaired Mann–Whitney U test for ordinal, and chi-square test for categorical variables. On the basis of the training dataset, referred surgical services were assigned to be “high-risk” if the reintubation rate was above the reintubation rate in the training dataset (0.38%)

Univariable analysis was performed in the training dataset to examine differences between reintubated and nonreintubated patients, as defined in the primary endpoint, using unpaired t test, Mann–Whitney U test, and chi-square test. On the basis of the literature review, we identified important clinical predictors of postoperative respiratory complications, which were available to the anesthesiologist in the preadmission area. Using multivariable logistic regression analysis, the initial model was fit with all relevant covariates, which were significant in univariate analysis, such as age, high-risk service, emergency procedure, ASA score, Charlson Comorbidity score, history of congestive heart failure, cerebrovascular, peripheral artery, chronic pulmonary, and renal disease. From this initial model, predictors were removed in a backward stepwise fashion if they had a P value of 0.05 or less and the model was refit. The final model included only predictors with a P value less than 0.05. If continuous or ordinal variables remained in the model, they were stratified based on the optimal cutpoint on the receiver-operating characteristics curve. The Hosmer–Lemeshow test was performed to determine the goodness of fit of the model, whereas a P value of 0.05 or less indicates that there is no significant difference between observed and expected count of outcomes across all risk levels. Given the fact that patients had multiple procedures within the enrollment period (separated by at least 10 days), the potential effect from the clustered data structure was determined. The estimates were recalculated using a nonlinear mixed-effects models (SAS proc NLmixed; SAS Institute, Cary, NC) accounting for clustering on a patient level and compared with the results from the logistic regression model.

A point value was assigned to each predictor proportional to the estimates from the logistic regression. For this, we divided the β estimate of each predictor by the smallest estimate. These results were rounded to the nearest whole number to define the score point values. The predictive value of the score for reintubation was assessed using c-statistics, which is equivalent to the area under the receiver-operating characteristics curve (AUC).¹⁶  We further calculated positive and negative likelihood ratios for strata of the score. Also, probabilities for getting reintubated depending on the score values were determined.

For the validation cohort, we calculated the derived score for each surgical case and compared the distribution of the score values with the one from the training cohort, using Mann–Whitney U test. The predictive value of the score for reintubation was evaluated in the validation dataset using a logistic regression model. The calculated c-statistic and the estimated probabilities for reintubation based on the score were compared with those derived from the training dataset. Similarly, the predictive value of the score was determined for the secondary endpoints. Finally, the score was correlated to the time to event using Spearman correlation coefficient.

All statistical tests were performed by using the software SAS (Version 9.2; SAS Institute Inc.) and a two-sided P value of less than 0.05 was considered statistically significant.

A total of 57,100 surgical cases were identified. Of these, a total of 23,331 cases were excluded because they had missing values for covariates, received their care predominantly outside the main campus, the age was less than 18 yr at the time of surgery, or one patient had multiple procedures within a time period of less than 10 days. In this case, only the first procedure remained in the cohort. In addition, all cases that were planned as an ambulatory surgery were excluded. The final cohort consisted of 33,769 surgical cases within 29,924 patients (fig. 1). Of patients with multiple surgeries (n =3,088), the majority (82%) had two procedures with a median time difference in between of 133 (IQR: 47–330) days.

Within the entire cohort, patients undergoing surgical procedures were on average 55±17 yr old and about half of them were men (47%). The most frequent referring surgical service was orthopedic surgery (22.5%). Only 5.4% (N = 1,825) of the cases were performed as emergency procedures (table 1).

A total of 60% of the cases (N = 20,274) had at least one documented history of the comorbidities, which are included in the Charlson Comorbidity score. Most frequent were myocardial infarctions, malignancies, and chronic pulmonary disease. All other disease classes were observed with a prevalence of less than 10% in the entire cohort. One third (30%) had a Charlson Comorbidity score of 3 or more. No differences were observed between the training and validation dataset regarding patients’ demographics (table 1), except for body mass index (P = 0.02). However, the magnitude of this imbalance was minor.

On the basis of the adjudication of the medical records, in 137 cases, we were able to identify clear evidence of timing and cause of tracheal reintubation within the first 3 postoperative days after primary extubation in the operation room (primary endpoint, 0.41%). The median time between primary extubation and reintubation was 6.4h (IQR, 0.5–47.1). Figure 2 shows the reasons for postoperative respiratory failure and reintubation based on chart review. The five most common reasons were pulmonary edema, atelectasis, pneumonia, impaired brain function, and aspiration. Multiple reasons for reintubation were reported, and each patient had on average 2.3 reasons for postextubation respiratory failure. With a definition of reintubation solely based on information in the electronic records and respiratory care database, an additional 89 cases were identified (secondary endpoint, total N = 226, 0.67%). A total of 111 (0.33%) patients died after surgery within index hospitalization. Reintubation based on medical record adjudication or on billing codes was associated with an increased crude odds for death (odds ratio [OR]: 72 [95% CI, 44–119] and 51 [95% CI, 32–81], respectively).

On the basis of the training dataset (table 2), patients of reintubated cases (N = 65) were on an average older (64.4±16.4 vs. 54.7±16 yr; P < 0.0001) and had a higher ASA and Charlson Comorbidity score. These cases were more frequently emergency procedures as compared with nonreintubated cases (table 2). The following referred surgical services were identified to be associated with a reintubation rate above average in the training database, vascular surgery, transplant surgery, neurosurgery, thoracic surgery, general surgery, and the burn center with 1.01, 0.85, 0.68, 0.63, 0.62, 0.51, and 0.47%, reintubation rates, respectively, and combined to so called “high-risk services”. In comparison, the reintubation rates for the remaining referring services were 0.24% for orthopedic surgery, 0.13% for urology, and 0% for all other referred services combined. Also, all assessed comorbidities were more frequent in reintubated cases with an exception of malignancies (P = 0.63), dementia (P = 0.76), and myocardial infarctions (P = 0.11).

On the basis of the training dataset, the following preoperative patient characteristics were the strongest independent predictors for postoperative reintubation in multivariable analysis: ASA score of 3 or more, emergency procedures, referred high-risk service, a history of congestive heart failure, and chronic pulmonary disease (table 3). The final model yielded a c-statistic of 0.81. The Hosmer–Lemeshow test demonstrated a well calibrated model (P = 0.32). Interestingly, the histories of chronic pulmonary disease and congestive heart failure as individual predictors were superior to the Charlson score, which combines 17 disease classes.

On the basis of the β coefficients for the final model, a point value of 3, 3, 2, 2, and 1 were assigned to the preoperative predictors: ASA score 3 or more, emergency procedure, if referred from a high-risk service, history of congestive heart failure, and chronic pulmonary disease, respectively (table 3). If accounting for the clustered data structure, similar estimates would have been derived, although the clustered data structure reached significance (P = 0.006; table 3).

The summed point values of the developed score ranged from 0 to 11 (median 0 [IQR, 0–3]) points (the maximum point value achieved in this dataset was 9) and were on an average higher in patients who did versus those who did not get reintubated within the first 3 postoperative days (median score value: 4 [IQR, 3–6] vs. 0 [IQR, 0–3] points; P < 0.0001). The odds of getting reintubated increased by 1.7-fold per one-point increase in the score (OR, 1.72 [95% CI, 1.55–1.91]; P < 0.0001). The score led to an AUC under the receiver-operating characteristics curve of 0.81, and a point value of 0 was associated with an estimated probability of 0.097% for reintubation, whereas a score value of 7 points or more led to a probability of 6.37% (table 4). The positive likelihood ratios for a score value of 4 or more and 7 or more points were 4.5 and 16.2, respectively. In contrast the negative likelihood ratio for a score value of 0 points was 0.2

The validation dataset (N = 16,884) with 72 reintubated patients, based on the medical record adjudication and 102 positive billing codes, was used to confirm the predictive value of the developed score. The median score value was 0 (IQR, 0–3), which did not differ from the training dataset (P= 0.32). The score led to an AUC of 0.796 and the associated probabilities were similar as derived from the training dataset (table 4). A weak correlation between higher score values and shorter time-to-events was observed, however it did not reach statistical significance (r = −0.18; P = 0.12). Extending this analysis to the entire database, the correlation reached significance (r = −0.22; P = 0.008).

Furthermore, the developed score also demonstrated a good predictive value for reintubation, which was defined on the basis of electronic records and billing data (AUC, 0.81), indicating applicability to external databases. Due to the high association between reintubation and death (OR between 51 and 72, see above), the score was also predictive for mortality within index hospitalization. For each point value of the score, the odds for death increased by 1.8-fold (OR, 1.78 [95% CI, 1.60–1.98]; P < 0.0001; AUC, 0.81). Similarly, for the composite endpoint of death and reintubation, the AUC was 0.79 (95% CI, 0.75–0.84).

Within the first 10 min after postoperative extubation, 172 cases from the validation database were reintubated (N= 4), deoxygenated below 80% (N = 165), or both (N=3). For each one-point increment of the score, the odds increased by 1.2-fold for this combined endpoint (OR, 1.18 [95% CI, 1.10–1.27]; P < 0.0001).

On the basis of preoperatively available characteristics, we developed a simple score (score for prediction of postoperative respiratory complications [SPORC]), which was highly predictive of postextubation respiratory failure leading to reintubation (fig. 3). Using a separate dataset, the SPORC score demonstrates high validity and allows the risk stratification of patients undergoing surgery. Given an estimated probability of 0.1% for reintubation in patients with a SPORC of 0, a score value of 7 or more points leads to a probability of 6.4%, which represents clinical meaningful information.

Unplanned reintubation, as used in our investigation, is one of the most rigorous markers for postextubation respiratory failure, which is known as a significant source of morbidity and mortality for surgical patients.^7,8,17  Also, in our study, reintubation within the first 3 postoperative days was highly associated with an increased risk (72-fold) for in-hospital death. Given the potential for survivor bias for the endpoint reintubation, the SPORC also demonstrated good discriminative capacity for a combined endpoint of reintubation and death.

A few previous studies have developed multifactorial models for predicting overall patient risk of developing postextubation respiratory failure.^4,5,8–10  These studies and the current investigation all share the objective of developing clinically applicable risk scores for postoperative respiratory complications in surgical patients. However, there are some significant differences in the selected cohorts and applied methods. We believe that sufficient preoperative risk stratification—well before the patient enters the operation room—may improve postsurgical triage and maximize the time available to gather necessary resources (e.g., allocate respiratory therapists with the potential for continuous positive airway pressure treatment in the recovery room). With a positive likelihood ratio of 16.2 for point values of 7 or more, and a negative likelihood ratio of 0.2 for a point value of 0, SPORC has the potential for significant risk restratification. These likelihood ratios are comparable with those derived from other scores. In a study of Lee et al.¹⁸  developing a score predicting postoperative cardiac events, the lowest score value was associated with a negative likelihood ratio of 0.24, whereas the highest score value was associated with a positive likelihood ratio of 4.6, given an overall event rate of 2% in their cohort. Also, similar posttest probabilities were achieved between both risk prediction tools.

Shifting regulations, shrinking budgets, and the imperative to provide the highest quality of patient care means that clinicians will need to define the value of new screening tools, which are supposed to predict low-frequency events. What is the value of predicting a ‘true’ postoperative respiratory complication? A tailored perioperative intervention may improve outcome and reduce costs—but from a economical healthcare point of view, it is important to show that such a benefit outweighs the consequences of missed events, and false-positive predictions associated with the use of a new prediction instrument. Similar to the revised cardiac risk index, we believe, that the SPORC adds significant value to future clinical trials in perioperative respiratory medicine. Some data suggest that patients at a high risk of postoperative respiratory failure may require additional perioperative treatment, such as noninvasive ventilation,^19,20  early mobilization,²¹  and/or pharmacological treatment,²²  but the evidence level of these trials is low at this point. The SPORC will help us define target populations for time and resource consuming intervention trials, which we believe are needed to improve perioperative respiratory outcome of future patients.

To allow possible interventions well before surgery, we focused exclusively on factors available during the patient’s preoperative care by the anesthesiologist. Intraoperative and postoperative factors, such as operation time,¹⁰  and laboratory values,^4,5  although important predictors of morbidity, were thus, not included in our score.

An additional objective in developing our score was to maintain its generalizability for use in a wide spectrum of cases. To this end, we did not exclude patients based on procedure type. This is a major difference from previous published scores, which often have excluded complex interventions, such as transplants^4,5,10  or low acuity.^4,5,8  Accordingly, differences have been observed regarding reintubation rates. Although the previous literature reported rates of postextubation respiratory failure from 0.83 to 3.3%,^7–10  we only observed a rate of 0.41%. A common source of variability is the definition of the primary endpoint postextubation respiratory failure. Previous studies have defined postextubation respiratory failure as assisted ventilation lasting more than 24,²³  48,^4,5,9  or 72h^6,8  cumulatively after surgery; some studies explicitly define reintubation as being sufficient for classification of postextubation respiratory failure,^4,5,23  whereas others do not. Some studies captured reintubation within the first 72h,⁸  whereas others focused on 30 days^4,5,7,9,10  after surgery. We defined postextubation respiratory failure as reintubation within the first 3 postoperative days, leading to unplanned mechanical ventilation after primary extubation in the operating room. We and others have selected this definition because it is the most specific for serious respiratory failure in the period where complications can be more confidently attributed to anesthesia and the effects of the surgical procedure.⁸  Finally, poor documentation of mechanism and timing of reintubation needs to be considered in this context. In our study, the incidence of reintubation based on pure billing-code derived data was more than 50% higher compared with a chart review derived approach, where existing information on timing and mechanism of reintubation were part of the definition of postextubation respiratory failure. It is important to state that the SPORC presented here demonstrates also a good predictive value for reintubation defined on the basis of electronic records and billing data (AUC, 0.81), indicating applicability to external databases.

Many variables (more than 30) are associated with postextubation respiratory failure. Common predictors are ASA score, surgical service, and whether the procedure is an emergency.^{3–8,10,24,25}  These factors were also included in our SPORC score. The ASA score is an assessment of the patient’s overall health status before surgery that has been consistently shown to be a predictor for postextubation respiratory failure and mortality.^{2,4,7,9,10,23,24}  Also, the operating service and type of surgery have been previously shown to be associated with postextubation respiratory failure.^{4,5,7–10,25}  In these studies, cardiothoracic, vascular, abdominal, and neurological surgeries have traditionally been defined as the procedures with the highest risk for complications and mortality. For these services, we also showed an increased risk in our cohort and extended this definition of high-risk service to transplant surgery, which in our data was associated with a reintubation rate of 0.85%. An additional operative feature, frequently found to be a valuable predictor for postextubation respiratory failure in the literature, is the emergency status of the procedure.^{2,4–7,9,10,24,25}  The assignment of emergency status is traditionally left to the discretion of the surgery and anesthesia team; in our hospital it is defined as “a surgical procedure where the patient is supposed to arrive in the operating room no later than 30 min after the patient has been booked as an emergent case, accompanied by the person who requested the case to be conducted emergently”. In addition to these factors, we have identified two previous existing conditions as being independent predictors for postextubation respiratory failure: history of chronic pulmonary disease and congestive heart failure.

Including the past medical history of the patient is a critical point in the risk prediction for postextubation respiratory failure. Preexisting pulmonary disease, particularly chronic obstructive pulmonary disease, is commonly shown as an independent predictor for postextubation respiratory failure.^4,5,8  Further, history of congestive heart failure has also been described to be associated with an increased risk of reintubation.^4,8  Instead of single diseases, the Charlson Comorbidity Index is an instrument to quantify a patient’s health status, that predicts short-term mortality.¹³  We showed that the two existing conditions mentioned above were at least equal to the Charlson Comorbidity score (capturing 17 distinct medical conditions) in their association to postextubation respiratory failure. Due to the complexity of calculating the Charlson Comorbidity score, our approach adds simplicity to the score.

Most of the previous published studies are based on a surgical database and therefore, do not include endpoints derived from the anesthesia record.^4,5,8–10  Because our database contains information from the Anesthesia Information Management System, the performance of the score was also tested against a combined endpoint of deoxygenation below 80% or reintubation, both within the first 10 min after extubation. Each one-point increment of the score was associated with a 1.2-fold increased risk for desaturation or reintubation shortly after extubation. Being able to identify patients who are at highest risk for complications during this vulnerable time adds additional information for the correct allocation of resources.

Many patients undergoing surgical procedures in our study were reintubated for pulmonary edema, pneumonia, atelectasis, and aspiration. On the basis of this information, we speculate that it is possible that residual paralysis on the first day after surgery may be a contributing factor to reintubation on a subsequent day. Partial paralysis affects the ability to protect the airway during swallowing and would thereby increase the propensity to aspirate or develop pneumonia. The proposed mechanism of pulmonary edema may be negative pressure pulmonary edema; paralytics can induce an upper airway obstruction in the absence of respiratory pump muscle dysfunction.²⁶ 

Our results must be interpreted under several limitations. Like any investigation of data on file, our study’s findings are dependent on the quality of the database, which is open to a variety of measurement biases. We aimed to create a highly specific database by gathering data from a variety of sources and verifying all billing code positive postextubation respiratory failure-positive cases with electronic or/and paper record adjudication. With these measures in place, we are confident our initial database, which has been utilized in previous respiratory studies,^27,28  has a high sensitivity for likely postextubation respiratory failure cases—but our final database included in this trial adds a high degree of specificity. Despite these efforts, it is not possible to guarantee that no information was left out of the patient charts and thus, our databases. Furthermore, we cannot exclude that we missed patients after being discharged who experienced postextubation respiratory failure and were admitted to a different hospital. Due to the study design, our score is not applicable to pediatric patients. Finally, as this is a single-site study at a tertiary medical center, we acknowledge the question of applicability to the general population, which would be addressed by a validation in a nationwide patient population. In addition, it is important to emphasize that the SPORC, by definition, can only explain a fraction of variables that affect the risk of postextubation respiratory failure. Intraoperative management as well as elements of postoperative care affect the incidence of postoperative complications but cannot be captured by the SPORC.^28–31 

Despite these limitations, the SPORC represents a simple approach that can be used by clinicians to assess the risk of postextubation respiratory failure as soon as patients are scheduled for surgery. Although the current study does not address strategies to prevent the incidence of postextubation respiratory failure, the implementation of such a score would optimize the postoperative care of patients by improving triage and alerting care providers to those who might require closer monitoring.

In summary, we have developed, on the basis of seven preoperative available patients’ characteristics, a simple score for the prediction of postoperative respiratory complications, called SPORC. The SPORC score can be used preoperatively by anesthesiologists to reliably predict severe postoperative respiratory complications (fig. 3).