“Understanding the weight of these events will enable us to confirm or correct predictions made during preoperative assessment, so that we can make faster decisions about complicated surgical patients.”
CRITICAL care planning for surgical patients is not an easy task because it involves anticipating the risk of adverse effects in most of the vulnerable patients we have treated. The difficulty arises from the unpredictability of complications that emerge both during surgery and afterwards in patients admitted to critical care units for periods that may extend over many days and weeks. Wherever resources are already stretched to the limit, decision-making becomes more burdensome because of the high cost of critical care.
In this issue of Anesthesiology, Wanderer et al.1 propose an intraoperative prediction model for unplanned admission to the critical care unit. The authors examined more than 70,000 anesthetic records with more than 4,500 events (unplanned admissions), giving them sufficient leeway to test a large number of hypothetical predictors. Through bootstrapping, they provided assurance against overfitting of the model and strengthened the finding of internal validity. The performance of the model was compared with that of the Surgical Apgar Score2 using an interesting series of time-dependent receiver-operating characteristic curves with an acceptable discrimination value at 1 h before case end. Sensitivity, specificity, and predictive values of the model were calculated for a threshold of 5% of predicted probability of unplanned admission. The optimal cut points based on the areas under the curves can be disputed, as the authors do not report the criteria for their choice. Alternative cutoffs might well be considered in the interest of avoiding excessive dichotomization and establishing an intermediate gray zone, as suggested by Ray et al.3 Furthermore, because predictive values can be severely affected by prevalence, likelihood ratios would have shed further light.3 In this study, a positive and negative likelihood ratio of approximately 4 and 0.17, respectively, have a moderate degree of predictive usefulness.
The nine predictors found to be associated with hemodynamic events are consistent with what others have emphasized, that hemodynamic instability appears to play a key role in postoperative outcome.4–9 Arterial oxygen saturation as measured by pulse oximetry (specifically, a decrease in the ratio of arterial oxygen saturation to the fraction of inspired oxygen) was among the predictors, consistent with its recently identified implication in respiratory complications.10 Three predictors are factors well known before surgery (American Society of Anesthesiologists class, age, and high-risk surgery); but although these factors can be considered in preoperative risk prediction and critical care planning, they may also bear a relation to intraoperative complications. Emergency operation proved to be another predictor of unplanned admission, unsurprisingly, because it is recognized that emergency status involves higher risk. But the fact that Wanderer et al.1 classified admissions as unplanned for all patients who underwent an emergency operation deserves a comment: it is anomalous to define a candidate predictor in terms of the dependent variable, the outcome of interest. When a factor is defined in this way, its involvement in the model may be overestimated. Although that categorical decision may be logical from an administrative standpoint within a hospital, from a medical point of view the destination of many emergency patients could be planned before surgery, and if these cases had been defined differently in this study, the model may well have been different.
The longer we watch a patient and weigh the information at hand, the more accurately we can predict. But a prediction that comes late, too close to the event we anticipate, will be less useful than one that comes earlier. Once we are certain, and the need for an immediate response is at hand, prediction is no longer an issue. Under normal conditions, the best time for risk prediction is during the preoperative visit. There, the patient can be duly informed and the anesthesiologist can prescribe preventive measures and foresee postoperative care requirements. Accumulated experience tells us that some intraoperative factors are known or can be anticipated in this visit. Examples are type of intervention, surgical incision, aggressiveness, or expected duration of surgery. Clearly, unusual intraoperative events (complications) cannot be taken into account when planning: such events are weighed in the operating room with a view to change the care plan as needed. When unplanned admission to a critical care unit becomes necessary, events have unfolded in a way that unfortunately leaves little room for choice. The outcome specified by Wanderer et al.1 —unplanned admission—is a truly complex real-life event because such admission involves a complication followed by a clinical decision. It seems logical to assume that decisions may be tightly constrained by critical care admission criteria, which may be more or less restrictive in different settings. The criteria operating in this study are difficult to determine, however, given that data were gathered retrospectively and it may be that future external validation of the model in different hospital systems could provide different results.
Whenever a multivariable analysis of risk factors is attempted, we begin to debate whether we have an explanatory or a predictive model. An ideal model would be one that gives us as much explanatory information as possible but that also predicts reasonably well. Because it is too bold to infer causal relationships from associations in observational studies, models are usually assumed to be predictive and their explanatory value depends on the biological plausibility of the predictors. Paradoxically, although we may manage to foresee events based on modeled associations, prediction in the narrow range of 1 or 2 h will probably be of limited practical use for hospitals with poor resources, where more time for maneuvering is a key to success.
After a predictive model has been built by multivariable analysis and internally validated by assessment of its ability to identify the individuals who will develop the outcome as well as the congruence between the predicted and observed frequencies, we must next ask about the future use of this model. Predictive models make sense only if they are useful in upcoming situations. If a model has been ideally based on a heterogeneous sample aspiring to represent a whole population, we can hope that it will be externally validated more easily. In contrast, a model based on a highly specific sample is less likely to be transportable. In any case, the proof of the pudding is in the eating as the old saying goes. Therefore, we should always plan as robust as possible an external validation of any model to verify the limits of its usefulness and establish a hierarchy of models based on their performance.11 What is needed is validation in different geographical settings, by independent research groups and even at different historical moments. Furthermore, even in a well-constructed predictive model, recalibration may be necessary to obtain an acceptable performance if the model is applied in highly selective contexts or ones that are different in any way from those of the development sample (e.g., pediatric medicine, tertiary hospitals, particular ethnic groups and so on).12,13 Nowadays, despite advances in computational and statistical methods in the health sciences, we are cautious about searching for a predictive score that will be useful everywhere and forever. Continuous adjustments are necessary to optimize these tools.
We conclude by emphasizing that the model developed by Wanderer et al.1 shows the importance of intraoperative events as predictors of patients’ admission to critical care units. Understanding the weight of these events will enable us to confirm or correct predictions made during preoperative assessment, so that we can make faster decisions about complicated surgical patients. However, we encourage other researchers and clinicians to study this tool further to substantiate its transportability. The more evidence we have of the utility of any model in different settings, the more our theories will begin to meet the needs of practice.