In Reply

We thank Dr. Hyder for his interest in our recent article published in Anesthesiology, “Preoperative Surgical Risk Predictions Are Not Meaningfully Improved by Including the Surgical Apgar Score: An Analysis of the Risk Quantification Index and Present-On-Admission Risk Models.”¹ 

As suggested by Dr. Hyder, we performed additional analyses using an alternative sampling interval for vital signs and added a calculation of risk reclassification to better test the clinical utility of the Surgical Apgar Score (SAS) when combined with preoperative risk stratification models.

A sampling method for slowest heart rate (HR) and lowest mean arterial pressure (MAP) was established before initiating data analyses. The method was based on “windows” or intervals of data and was established as follows: 10-min nonoverlapping windows, with windows beginning at the time of incision (0 to 10 min, 11 to 20 min, 21 to 30 min, etc.). Within each window, a median value was determined. Median values for HR and MAP were the basis for the original SAS investigations, and median values were chosen for this investigation. Estimated blood loss as recorded by the in-room anesthesia provider was calculated for the entire case.² 

We also added a calculation of risk reclassification to better test the clinical utility of the SAS. The use of a reclassification measure may be applied to provide a more clinically meaningful assessment of change in risk prediction. A concept of categorizing patients into high- and low-risk groups is clinically intuitive and actionable, as we treat high-risk patients differently, such as with admission to the intensive care unit. Traditionally, risk prediction models have been evaluated using the area under the receiver operating characteristic curve, along with model calibration, Brier score, information criteria, etc., but this can be an insensitive measure for model comparison in a healthcare setting, providing little direct clinical relevance. Since its description in 2006, much interest has been generated in reclassification, which assesses the ability of new models to more accurately classify individuals into higher or lower risk strata. This has led to new methods of evaluating and comparing risk prediction models, including the reclassification calibration test and the net reclassification index (NRI). Pencina et al.³  developed the NRI and the integrated discrimination improvement (fig. 1).

After performing analyses using alternative sampling for vital signs and calculating risk reclassification, the Risk Quantification Index and present-on-admission preoperative risk models were not meaningfully improved by adding intraoperative risk using the SAS, as determined by the NRI value of 0.02 (P = 0.10). These analyses supported the original findings: adding the SAS did not substantively improve predictions. In addition to the estimated blood loss, lowest HR, and lowest MAP, other dynamic clinical parameters from the patient’s intraoperative course may need to be combined with procedural risk estimate models to improve risk stratification.