Prediction Model for In-hospital Mortality Should Accurately Predict the Risks of Patients Who Are Truly at Risk

With great interest, we read the article by Le Manach et al.¹  The article presents a prediction model for postoperative in-hospital mortality with very good discriminative abilities (C statistic of 0.93 in a validation cohort). However, we contend the conclusion that the predictive model is well calibrated.

A prediction model should first and foremost provide accurate predicted probabilities. When validating a prediction model, it is essential to answer the question whether predicted probabilities correspond to observed probabilities, especially for patients who may have a clinically relevant risk of the predicted outcome.

The reported calibration plot (fig. 2 in the article¹ ) seems to show a well-calibrated model. However, the calibration plot is truncated at a predicted probability of 0.10, and the plot shows only 9 out of 10 deciles. Patients with the highest risks seem to have been omitted from the reported calibration plot. Figure 3 of the article¹  shows the observed mortality in the validation cohort for a wider range of risk scores. Supplemental Digital Content 3 reports the predicted probabilities for all Preoperative Score to Predict Postoperative Mortality (POSPOM) scores. If we overlay the predicted probabilities for all POSPOM scores onto figure 3 of the article,¹  we observe that the prediction model greatly overestimates the in-hospital mortality risk in the high-risk patients (fig. 1). Although the high-risk patients form only a small group, they are in fact the patients for whom the prediction model is most clinically relevant. We would not want to be the physician who communicates a 62% risk of death to a patient (POSPOM value of 40) while the actual risk is 23%.

From the results presented in the article, it is impossible to deduce what the reason is for the discrepancy in the results. There are several inconsistencies in the reporting of the predicted probabilities. The text of the article mentions a predicted probability of 5.65% for a POSPOM value of 30, whereas the Supplemental Digital Content reports a predicted probability 7.403%. There is no way to reconstruct the predicted probability as the model’s intercept is not reported. It is possible that the predicted probabilities in Supplemental Digital Content 3 are wrong, which then could explain the overestimation of the in-hospital mortality risk. Even then, it would still be a necessity to report a continuous calibration plot of the model for the entire range of predicted probabilities, with a histogram within that calibration plot.

We believe that the prediction model may be of great value to both physicians and patients, but only after the overestimation of the high-risk patients has been addressed.

The authors declare no competing interests.