The authors developed a score for predicting the risk of postoperative complications
The score was developed from 1,094 patients and validated in 830 patients from six French hospitals
Severe complications occurred in about 11% of each cohort
The positive predictive value was poor, but the negative prediction value was excellent and might be used to identify patients who do not need critical care
Craniotomy for brain tumor displays significant morbidity and mortality, and no score is available to discriminate high-risk patients. Our objective was to validate a prediction score for postoperative neurosurgical complications in this setting.
Creation of a score in a learning cohort from a prospective specific database of 1,094 patients undergoing elective brain tumor craniotomy in one center from 2008 to 2012. The validation cohort was validated in a prospective multicenter independent cohort of 830 patients from 2013 to 2015 in six university hospitals in France. The primary outcome variable was postoperative neurologic complications requiring in–intensive care unit management (intracranial hypertension, intracranial bleeding, status epilepticus, respiratory failure, impaired consciousness, unexpected motor deficit). The least absolute shrinkage and selection operator method was used for potential risk factor selection with logistic regression.
Severe complications occurred in 125 (11.4%) and 90 (10.8%) patients in the learning and validation cohorts, respectively. The independent risk factors for severe complications were related to the patient (Glasgow Coma Score before surgery at or below 14, history of brain tumor surgery), tumor characteristics (greatest diameter, cerebral midline shift at least 3 mm), and perioperative management (transfusion of blood products, maximum and minimal systolic arterial pressure, duration of surgery). The positive predictive value of the score at or below 3% was 12.1%, and the negative predictive value was 100% in the learning cohort. In–intensive care unit mortality was observed in eight (0.7%) and six (0.7%) patients in the learning and validation cohorts, respectively.
The validation of prediction scores is the first step toward on-demand intensive care unit admission. Further research is needed to improve the score’s performance before routine use.
NEUROSURGERY remains the cornerstone of curative treatment in brain tumor but is associated with high perioperative morbidity and mortality.1 The risk of perioperative mortality is more than twofold compared with the average mortality risk when adjusted to a patient’s baseline severity.1 This can be explained by the life-threatening complications that may occur during the perioperative period: intracranial bleeding, intracranial hypertension, and status epilepticus, among others.2–5 It has therefore been suggested that overnight postoperative monitoring in an intensive care unit (ICU) be mandatory for all patients undergoing elective craniotomy.2,3,6 However, systematic ICU admission uses medical resources, increases costs, reduces the number of ICU beds for emergencies, and is not associated with improved postoperative outcome.7 Although some risk factors have been identified,8,9 no validated prediction score exists to differentiate patients with high perioperative risk of complications from those who do not require ICU admission. From a medicoeconomic point of view, the use of such scores could help reduce healthcare costs by providing adequate care to high-risk patients only.
Our primary objective was to develop and validate a score that could help physicians decide which patients require overnight ICU admission after elective intracranial neurosurgery to avoid the unnecessary admission of low-risk patients. The secondary objectives of the study were to assess perioperative morbidity and mortality in patients undergoing elective brain tumor craniotomy.
Materials and Methods
This was a multicenter observational study (ClinicalTrials.gov identifier NCT 01801813). The protocol was approved by the Institutional Review Board (Groupe d’Ethique dans le Domaine de la Santé) of the University Hospital of Nantes (Nantes, France). The learning cohort comprised patients undergoing craniotomy for a brain tumor in one center (Nantes) from January 1, 2008, to December 31, 2012. This cohort was developed from a retrospective analysis by screening two prospective databases: Clinicom (Siemens, Germany) for clinical and biologic data (radiologic findings, tumor histology, or other demographic variables) and Pégase (Thélème, France) for perioperative data (such as during surgery, during ICU stay). The data provided in the software enabled us to gather data on primary outcome, baseline demographics, tumor, and perioperative management. We therefore avoided selection bias and included all patients who were operated for brain tumor according to histology. For the validation cohort, a prospective analysis was performed in patients undergoing cerebral craniotomy for brain tumor from January 1, 2013, to December 1, 2015, in six French university hospitals (the University Hospitals of Beaujon, Clichy, Assistance Publique des Hôpitaux de Paris; La Timone, Assistance Publique des Hôpitaux de Marseille; Nantes; Rennes; Strasbourg; and Toulouse). Because our study was purely observational, consent was waived. Oral and written information was provided to patients in the validation cohort. Our Institutional Review Board waived the requirement to provide information for the retrospective analysis. The study was prepared in accordance with transparent reporting of a multivariable prediction model for individual prognosis or diagnosis guidelines.
Adult patients (older than 18 yr) undergoing elective neurosurgery with craniotomy for a brain tumor confirmed after histologic analysis were eligible for this study.
Patients with stereotactic biopsy for brain tumor were not included. Patients undergoing craniotomy for simple biopsy, aneurysm clipping, arteriovenous malformation, cerebral cavernoma, or central nervous system infections and urgent craniotomy were not eligible for this study.
We collected demographic data such as age, sex, American Society of Anesthesiologists class, history of epilepsy, use of preoperative medications such as antiepileptic drugs, β-blockers, previous history of brain tumor surgery,10 tumor histology, location and intracerebral radiologic severity criteria such as mass effect on median structures, peritumoral edema, size of the tumor,11 perioperative management such as duration of anesthesia, duration of surgery,12 surgical position,13 operative administration of mannitol, fluid administration, blood loss, and highest and lowest arterial blood pressures.8 Postoperative data such as extubation time, use of mechanical ventilation, intracranial hypertension, intracranial bleeding, urgent neurosurgery, and seizures were recorded. The list of data recorded during the study is provided in Supplemental Digital Content 1 (http://links.lww.com/ALN/B775).
The primary objective was to develop and validate a score that could predict early severe postoperative neurosurgical complications in the first 24 h in the ICU after elective brain tumor neurosurgery to improve ICU triage and safely discharge patients to wards.
Definition of the Primary Outcome
To define the primary outcome variable, we developed a list of complications that could lead to severe postoperative neurosurgical complications that require at least 24 h of ICU monitoring14 : moderate to severe intracerebral bleeding confirmed on brain computed tomography scan possibly requiring neurosurgical evacuation, intracranial hypertension confirmed on brain computed tomography scan or with intracranial probe or external ventricular drainage (defined as intracranial pressure at or above 20 mmHg), status epilepticus or seizures (clinical or confirmed by electroencephalogram), need for tracheal intubation and mechanical ventilation after the neurosurgical procedure, impaired consciousness (Glasgow Coma Score at or below 13), unmanageable agitation requiring restraint or sedation, severe swallowing disorders leading to aspiration and respiratory failure (oxygen saturation measured by pulse oximetry at or below 90% or requiring oxygen therapy), unexpected severe motor deficit (motor score at or above 3), and finally death in the perioperative period. In case of minor postoperative intracranial bleeding on brain computed tomography scan but without significant symptoms, a patient could be discharged from the ICU depending on each center’s protocol.
The aim of this study was to develop and validate a score specific for neurologic complications. Patients with postoperative complications unrelated to the neurosurgical procedure were therefore not considered for the primary outcome variable (e.g., allergy, iatrogenic complications such as pneumothorax after central venous catheter insertion, pacemaker dysfunction).
The secondary objectives of our study were the description of perioperative management and perioperative morbidity and mortality.
Perioperative patient morbidity was defined as follows: patient readmission to the ICU during hospitalization and length of hospital stay. We also recorded in-ICU and in-hospital mortality.
Continuous variables were expressed as median (interquartile range) for nonparametric data or mean ± SD for parametric data. Qualitative variables were expressed as N (%).
To construct the risk model for primary outcome, the least absolute shrinkage and selection operator was used for potential risk-factor selection with logistic regression. Conventional selection methods based on P values failed to obtain an adequate multivariable model. Indeed, the number of events in our population was small compared with the number of risk factors tested (125 early postoperative neurosurgical complications and 35 potential risk factors). With the least absolute shrinkage and selection operator method,15 the usual P < 0.05 is not considered for variable selection. This method penalizes the sum of the absolute values of the regression coefficients leading some coefficients to shrink to 0 and thus simultaneously perform variable selection.15 The shrinkage parameter, called λ, is generally selected through the cross-validation method. This method was first performed on the learning cohort and resulted in the selection of 26 potential risk factors. Then, to reduce the number of risk factors, the optimal λ was determined using graphical consideration with an area under the receiver operating characteristic curve of 73% and eight nonzero coefficients in the model.
To validate this predictive model, (1) apparent performance was estimated in the learning cohort and (2) reproducibility was estimated in the validation cohort. Discrimination was evaluated using the area under the receiver operating characteristic curve and its 95% CI, and calibration was assessed using the Hosmer–Lemeshow test. The Brier score was calculated to measure the accuracy of probabilistic predictions. The Brier score provides the probability of a model (i.e., the CranioScore) to predict the occurrence of an outcome (i.e., postoperative complications). The value of the Brier score is between 0 and 1; the closer to 0 the Brier score, the better the model to predict the outcome. Finally, the CranioScore score was constructed with the regression coefficients identified in the multivariable model. We calculated the predicted probabilities of complications based on this score. We classified high-risk or low-risk postoperative complications according to various CranioScore thresholds. The aim of our score is to improve ICU triage and safely discharge patients to wards. This strategy implies favoring a score with the best negative predictive values to avoid false negatives. We therefore tested various CranioScore values in the learning cohort and chose which threshold was the best to discriminate patients with high or low risk of postoperative complications that require overnight ICU monitoring.
The discriminative ability of this dichotomy (sensitivity, specificity, positive predictive value, and negative predictive value) was estimated in both cohorts. We retained the threshold with the best negative predictive value and which could be helpful in the ICU triage process.
An a priori sample size calculation was not conducted, and all available data were used to include a minimum of 100 events.16 We therefore decided to include at least 100 patients with postoperative complications in the learning cohort to obtain an adequate sample size. Regarding the validation cohort, we included a cohort with at least a 40% of the size of the learning cohort to provide robust external validity.16,17
All analyses were performed on complete cases. Model selection with the least absolute shrinkage and selection operator method was performed using the penalized package in R, and SAS statistical software version 9.3 (SAS Institute, USA) was used for other analyses.
Description of the Learning Cohort
We included 1,094 patients in the first cohort from January 1, 2008, to December 31, 2012. A flowchart of the learning cohort is available in Supplemental Digital Content 2 (http://links.lww.com/ALN/B776). Mean age was 57 yr (±15), and the sex ratio of male:female was 521(47.6%):573(52.4%). Demographics and comorbidities for both cohorts are presented in table 1. Brain tumors were meningioma in 355 (32.4%) patients, glioma-glioblastoma in 252 (23%) patients, and metastasis in 238 (21.8%) patients (table 2). Intraoperative data in the learning and validation cohorts are available in table 3. Respectively, 125 (11.4%) patients presented early postoperative neurosurgical complications in the learning cohort, and 114 (16.6%) patients presented complications in the validation cohort, in accordance with the primary outcome variable. Complications are provided in Supplemental Digital Content 3 (http://links.lww.com/ALN/B777).
Multivariable Analysis: Independent Risk Factors for Neurologic Life-threatening Complications
To adequately select the variables associated with early postoperative complications, a least absolute shrinkage and selection operator procedure was performed (Supplemental Digital Content 4, http://links.lww.com/ALN/B778), and eight factors were selected for the multivariable model. The optimal λ (shrinkage parameter) was graphically determined so that the number of risk factors was the lowest (considering the number of events) and the highest area under the receiver operating characteristic curve criterion. These factors were used to develop the score (table 4): Glasgow Coma Score before surgery at or below 14 (odds ratio [OR], 4.55; 95% CI, 1.88 to 11.03; P = 0.0008), history of brain tumor surgery (OR, 2.87; 95% CI, 1.74 to 4.72; P < 0.0001), greatest brain tumor diameter (mm; OR, 1.01; 95% CI, 1.00 to 1.02; P = 0.1), midline shift at or above 3 mm11 (OR, 1.67; 95% CI, 1.04 to 2.69; P = 0.03), transfusion of packed erythrocytes or plasma or platelets (OR, 1.69; 95% CI, 1.01 to 2.83; P = 0.04), maximum systolic arterial pressure during surgery (mmHg; OR, 1.01; 95% CI, 1.01 to 1.02; P = 0.0002), minimal systolic arterial pressure (mmHg) during surgery (OR, 0.99; 95% CI, 0.97 to 1.00; P = 0.08), and duration of surgery (OR, 1.37; 95% CI, 1.19 to 1.58; P < 0.0001). There were no missing data for all patients included in the learning cohort (N = 1,094). The model showed an area under the receiver operating characteristic curve of 0.73 IC95% (0.68 to 0.77), Hosmer–Lemeshow test, P = 0.2 (fig. 1).
External Validation of the Score in an Independent Cohort
We tested the robustness of our model in an independent multicenter (six ICUs) prospective cohort of 830 patients undergoing scheduled neurosurgery included from January 1, 2013, to December 31, 2015, in six centers. A flowchart of the validation cohort is available in Supplemental Digital Content 2 (http://links.lww.com/ALN/B776). Ninety (10.8%) patients presented early postoperative neurologic complications. Demographics, the type of brain tumor, and perioperative data in this cohort are available in tables 1–3, respectively. The score was applied in 748 patients (82 patients had missing data for at least one of the selected variables). The area under the receiver operating characteristic curve of the score in this cohort was 0.70 IC95% (0.64 to 0.76), and the Hosmer–Lemeshow test P value was 0.1 (fig. 1). The Brier score in this cohort was 0.13.
Definition and Usefulness of a Predictive Score for Complications
The robustness of the multivariable analysis prompted us to validate a score with the selected risk factors. The CranioScore based on these factors (table 4) provides a calculated probability of postoperative neurosurgical complications for each patient and is therefore expressed as a percentage. The CranioScore is only applicable in patients with all available data (no missing data in any of the selected variables). Several probability cutoffs for predicted complications were tested to delineate sensitivity, specificity, and positive and negative predictive values in the learning cohort (table 5). In the learning cohort, a 3% threshold had a sensitivity of 100%, a specificity of 6.2%, a positive predictive value of 12.1%, and a negative predictive value of 100%. With a threshold greater than 3%, 1,034 patients in the learning cohort and 660 patients in the validation cohort presented such values and would have been classified as high-risk patients requiring overnight ICU monitoring. Among these patients, 125 (11.4%) patients with complications would have been accurately classified as “high risk” in the learning cohort. No patients would have been misclassified in the learning cohort. In the validation cohort, 85 (10.2%) patients with complications would have been accurately classified as “high risk.” Only one (0.1%) patient would have been misclassified as low risk in the validation cohort. On the other hand, 60 (5.4%) patients in the learning cohort and 88 (10.6%) patients in the validation cohort had a CranioScore at or below 3% and could have been discharged directly to a ward. Table 6 provides the classification of patients according to various values of their CranioScore (1%, 2%, 3%, and others) in the two cohorts. Based on these data (table 6) and using the CranioScore formula (Supplemental Digital Content 5, http://links.lww.com/ALN/B779), a predicted percentage of complications greater than 3% could be proposed to limit the risk of false negatives (patients classified as low risk of complications but who will develop a complication). Two examples of calculations of the CranioScore are provided (Supplemental Digital Content 5, http://links.lww.com/ALN/B779).
The perioperative management of patients in the validation and learning cohorts is displayed in table 3. In the learning cohort, 8 (0.7%) patients died in the ICU, and 16 (1.5%) died in the hospital. In the validation cohort, 6 (0.7%) patients died in the ICU, and 9 (1.1%) died in the hospital. Secondary outcome data can be found in table 1.
We validated a score predicting early severe postoperative neurosurgical complications within the setting of elective craniotomy for brain tumor. This score could provide substantial help in discriminating patients requiring mandatory overnight ICU monitoring to screen and treat major complications.
In an international observational study,18 in-hospital mortality for patients requiring elective noncardiac surgery was higher than expected (up to 4%) with wide variation between countries. Moreover, indirect patient ICU admission after surgery was associated with higher mortality than patients with direct admission, meaning that complications after elective surgery should have been better anticipated.19 These data favor systematic postoperative ICU admission. However, in a recent multicenter international study, direct ICU admission after surgery did not appear to improve hospital mortality.7 The benefit of systematic ICU admission is therefore questionable. Up to now, monitoring patients after elective intracranial surgery in an acute care setting has been recommended (ICU, Neuro-ICU) because elective neurosurgery involves substantial morbidity and mortality compared with other types of surgery. An on-demand rather than a routine ICU admission policy could be profitable to high-risk patients and institutions.
In a nationwide multicenter database,1 patients undergoing neurosurgery had more than a twofold risk of perioperative mortality compared with average mortality. Moreover, mortality after neurosurgery has not decreased over the last few decades.1,20 This high mortality rate can be explained by life-threatening complications that can occur, such as cerebral hematoma,9 status epilepticus,13 and difficulty in weaning the patient from mechanical ventilation.21 The incidence of complications after a neurosurgical procedure can be as high as 14.3%, especially after craniotomy.21 These data urgently call for modification of postoperative ICU admission protocols and enhanced screening of patients. These scores could play a role in deciding ICU admission. In neurosurgery after elective craniotomy, selective rather than routine ICU admission could be both safe and cost-effective,6 but the medical data that could improve this process are currently lacking. With a CranioScore value at or below 3%, patients could be safely discharged from the recovery room to a surgical ward. The percentage of patients with such CranioScore values could seem low. However, given the current policy of systematic ICU admission after neurosurgery, this would be the first step toward safe, medically justified, and cost-effective on-demand ICU admission after intracranial neurosurgery. Such a strategy would be highly innovative because on-demand ICU admission has not been tested in other contexts of high-risk surgery such as cardiac surgery.
Unlike cardiac surgery, there are very few scores to predict outcome in the setting of neurosurgery with craniotomy.22,23 To the best of our knowledge, only 25 studies on preoperative risk assessment are available in this setting.24 The scores currently available, such as American Society of Anesthesiologists or Karnofski Performance Status, are not specific for neurosurgery patients and usually have small samples.24 The specific preoperative sex, Karnofsky, American Society of Anesthesiology, location, edema grading system associating sex, Karnofski Performance Status, American Society of Anesthesiologists class, location of the brain tumor, and edema was built to predict 1-yr outcome after meningioma surgery in the elderly.25 The sex, Karnofsky, American Society of Anesthesiology, location, edema score is useful in a selected population of patients undergoing elective craniotomy but is not helpful in the prediction of early postoperative complications with all types of brain tumors. The CranioScore is the first designed nationwide cohort with unselected brain tumor types in patients undergoing neurosurgery. Because we also provide a validation cohort, our results should be applicable to other settings. We have also identified some previously described risk factors such as the duration of surgery,13 which will strengthen applicability.
Our study has limitations. First, in spite of this large cohort, the sample size of patients with a postoperative complication is rather low, and the selection of the adequate variable to uphold in a traditional regression model could be inadequate, owing to the parsimonious rule. Least absolute shrinkage and selection operator models are not subjected to this limitation even with a high number of variables. Second, our findings suggest associations and not causation, although we provided a large independent validation cohort for external validation of the CranioScore. It is therefore possible that specific therapeutic targets based on the risk factors would improve outcomes. Third, the decision to switch hospital policies from systematic to on-demand ICU admission could involve some major logistics changes. It should enhance ICU bed availability, but it should be accompanied by increased nurse staff training, education, and monitoring in surgical wards. Fourth, a CranioScore cannot be easily calculated. However, this should not be a major drawback with the widespread use of online free calculators and medical apps on smartphones. Fifth, our cohort had a low incidence of awake craniotomy or rare surgical procedures that may have a higher risk of complications such as craniopharyngioma resection. Our score may not apply in such a setting, as well as in patients undergoing craniotomy for other procedures (aneurysm clipping and arteriovenous malformation, among others). In addition, the CranioScore is not applicable in case a patient has missing data in one of the selected variables. However, these variables are routinely monitored. Last, patient ICU admission could rely on the comorbidities for a given patient18 and not only on the potential occurrence of postoperative complications. In cases of patients with severe comorbidities, other specific scores can be used26 to evaluate overall perioperative risk.
The CranioScore is a validated score predicting the risk of severe postoperative neurosurgical complications in elective craniotomy for brain neoplasms. It should be of interest to help an attending physician in the on-demand ICU admission process after craniotomy. Given the potential consequences of misclassification, further research should focus on improving the specificity of prediction scores. The addition of biomarkers could be a promising tool in the prognostication of outcome after neurosurgery27 and could be tested to enhance the validity of our score.
The authors thank Anne-Sophie Crouzet and Laurence Pacaud, both research nurses in the Anesthesia and Critical Care Department, Hôpital Laennec, University Hospital of Nantes, Saint-Herblain, France; and Delphine Flattres-Duchaussoy and Cécilia LeBel, both assistant researchers in the Anesthesia and Critical Care Department, Hôtel Dieu, University Hospital of Nantes, Nantes, France, for their precious help in the logistics of this study.
The authors also appreciate the reactivity, methodologic help, and logistics of the Société Française d’Anesthésie-Réanimation Research Network and all of the doctors and nurses involved in data collection at all sites.
Support was provided solely from institutional and/or departmental sources.
Dr. Asehnoune received payments from FRESENIUS (Sèvres, France), Laboratoire français du fractionnement et des biotechnologies (Courtaboeuf, France), and BAXTER (Guyancourt, France).
Appendix: Collaborators of the Société Française d’Anesthésie-Réanimation (SFAR) Research Network
Marie-Pierre Bonnet, M.D., Ph.D., Anesthesia and Critical Care Department, Hôpital Cochin, Paris, France; Morgan Le Guen, M.D., Ph.D., Anesthesia and Critical Care Department, Hôpital Foch, Suresnes, France; Valeria Martinez, M.D., Anesthesia and Critical Care Department, Hôpital Raymond Poincaré, Garches, France; and Romain Pirracchio, M.D., Ph.D. and Amélie Yavchitz, M.D., Anesthesia and Critical Care Department, Hôpital Européen Georges Pompidou, Paris, France.