Perioperative Mortality, 2010 to 2014: A Retrospective Cohort Study Using the National Anesthesia Clinical Outcomes Registry

PERIOPERATIVE mortality has decreased by an order of magnitude in the past 5 decades in high-income countries, despite a population increasingly burdened with severe comorbidities.¹  Anesthesiology as a specialty was identified in the 1999 Institute of Medicine publication To Err is Human: Building a Safer Health Care System as a model for creation and dissemination of patient safety improvements.²  Process and patient care improvement efforts to mitigate the causes of preventable perioperative mortality and morbidity, such as malignant hyperthermia, airway or intubation misadventures, and undetected hypoxia, have succeeded in making these relatively rare causes of harm.

The development of large epidemiologic data sets has facilitated the investigation of such low-incidence events. Recognizing the need for a database focused on anesthesia-related outcomes, the American Society of Anesthesiologists created the Anesthesia Quality Institute (AQI) in 2009. Among its other patient safety initiatives, the AQI oversees the National Anesthesia Clinical Outcomes Registry (NACOR). NACOR is a nonprofit data repository created to improve the quality of anesthesia care, and has been collecting deidentified billing and electronic health record data from anesthesia practices in the United States since 2010. Participation in NACOR is voluntary, although participants are required to contribute a minimum administrative data set for each case. At the time of writing this article, NACOR contained almost 30 million anesthesia cases from nearly 500 distinct practices at more than 2,500 facilities. A subset of practices contributes data on perioperative outcomes; at the time of writing this article, 16% of the cases in the NACOR data set contained outcomes data.

The capabilities and limitations of NACOR’s outcomes data set have only begun to be described in the peer-reviewed literature.³  Herein, we perform an exploratory retrospective cohort study of perioperative mortality events, defined as death within 48 h of induction of anesthesia, and other perioperative complications documented in the first 4 yr of NACOR case collection. Our intent was to identify factors associated with perioperative mortality available in the minimum data set collected by NACOR.

The structure and data management and validation of NACOR are briefly described in the appendix and have been described by Liau et al.³ 

This study is a retrospective analysis of a cohort of cases in a large quality-focused data set. This article has been structured according to the STrengthening the Reporting of OBservational studies in Epidemology (STROBE) recommendations for reporting of cohort studies.⁴ 

The NACOR data set is itself deidentified. The human studies committee of the University of California, San Francisco (San Francisco, California) approved this study and waived the requirement for informed consent.

NACOR is a national repository for administrative, clinical, and quality-focused data from practices in the United States that provide anesthesia care. Data from practices’ billing, clinical, or quality data documentation systems are imported directly via customized programming routines. Practices are expected to contribute at least a minimum data set of approximately 20 elements per case. More detailed clinical or quality data are contributed by a subset of practices. All U.S. anesthesia practices are eligible to participate. Case collection began on January 1, 2010; the data set used here includes the convenience sample of all cases containing outcomes data uploaded to NACOR between inception of the database and May 31, 2014.

Cases from practices not reporting outcomes data were not eligible for this study. Further, we excluded cases where the procedure code and anesthesia codes were not reported and there was no documented anesthesia type or it was documented as “other,” because there was not enough information to make conclusions about contributing risk factors in those cases. Because of the essential differences in which analgesia provided by an anesthesia provider for vaginal delivery is billed, we excluded those cases.

Because of varying degrees of practice participation, data completeness varied among practices. Therefore, we focused on analyses using the variables typically provided to NACOR as a part of the minimum data set. These include practice/facility factors (U.S. region, Medicare-defined facility type), patient factors (age, sex, American Society of Anesthesiologists physical status [ASAPS], emergency, or elective case), and procedural factors (surgical and anesthesia Current Procedural Terminology (CPT) codes, principal anesthesia type, U.S. Agency for Healthcare Research and Quality’s Clinical Classifications Software code, case start time, case duration). The full list of variables in the minimum data set available in the NACOR participant user file is available through the AQI.⁵ 

Patient ages were grouped into categories according to the predetermined age ranges defined by AQI (see table 2). Because some practices did not differentiate between ASAPS 1 and ASAPS 2 patients, they were grouped together for analysis. Surgery type was determined from anesthesia CPT code, which was the most complete variable that provided information about the surgical procedure; if this was missing, the American Society of Anesthesiologists crosswalk code (derived from surgical CPT) was used. Procedure start time was grouped according to the definitions suggested by Kelz et al.⁶  Emergency case status was derived from provider reporting. Cases were also considered emergent if any of the four surgical CPT codes potentially listed for an individual case indicated an emergent surgical procedure (CPT code 99140). Because very few cases were documented as “elective,” we assumed a case was nonemergent if an ASA physical status had been documented without emergency case (“E”) modifier or surgical CPT code indicating emergency.

Perioperative outcomes are listed in NACOR’s document Outcomes of Anesthesia: Core Measures.⁷  Briefly, these outcome definitions represent consensus recommendations formalized on May 11, 2013 and developed jointly between the AQI and the Multicenter Perioperative Outcomes Group. Reporting of individual events is typically the responsibility of the anesthesia provider. A significant (i.e., reportable) event was generally considered to be one that was unanticipated and required active intervention.

Mortality was defined as death within 48 h of induction and was typically reported by the anesthesia provider. NACOR does not collect time of death; mortality is reported as a Boolean variable (i.e., present or not present) without further details on temporal relationship to surgery. Other outcomes are further defined in a document available through the AQI.⁸ 

Basic descriptive statistics were calculated for the data set, and univariate comparisons were performed with chi-square test for categorical values and a t test for age as a continuous measure (i.e., when comparing excluded and included cases).

The primary analytic strategy was a hierarchical mixed-effects model, with adjustment for random effects by anesthesia practice, and by facility (within anesthesia practice). All predictors from univariate analyses were entered into a logistic regression model, with the exception of facility type and region, which were not included because of varying penetrance of NACOR participation and consequent unequal representation of certain regions and/or facility types. The primary analysis did not adjust for missing data because of the paucity of complete variables that could be used to impute missing values; cases with missing data for one or more variables of interest were excluded from the regression models.

To further investigate the association between time of day and perioperative mortality, we examined the association in a post hoc sensitivity analysis intended to reduce the impact of emergency procedures that were not coded as such. Given the substantially smaller sample size, hierarchical mixed-effects modeling failed to converge, and instead a logistic regression with variances adjusted for clustering by practice was used. After excluding known emergency procedures and ASAPS 5 patients, we limited the population to adult patients with a Clinical Classifications Software code suggesting an elective procedure, following the method suggested by Sessler et al.⁹  We also excluded cases with a start time before 6:00 am or after 11:00 pm, similar to the study by Kelz et al.⁶  This limited the population to 308,047 cases with 38 deaths. Because of this substantially smaller population, we removed surgery type from the model. The variables entered into the logistic regression model were otherwise the same as for the primary model, with variance adjusted for clustering by practice.

A second post hoc sensitivity analysis was performed to adjust, with better granularity, for mortality risk using the Risk Quantification Index (RQI) published by Dalton et al.¹⁰  The RQI uses a population-based measurement of procedure-specific mortality risk associated with the particular surgical CPT code, in combination with patient’s age and ASAPS, and has shown excellent discrimination.^10,11  It can be calculated only for those CPT codes that are matched to a mortality risk estimate; thus, RQI was added into the logistic regression model for patients in whom the RQI could be calculated. Most notably, this excluded obstetric and ocular cases from this subgroup analysis. Patients younger than 18 yr were also excluded, because this index was derived in adults. As with the elective subgroup, the relatively smaller sample size resulted in nonconvergence of attempted hierarchical modeling, so a logistic regression with variance adjusted for clustering by practice was used. There was no significant collinearity with age or ASAPS, thus those variables were retained.

A third post hoc sensitivity analysis adjusted for Medicare physician work relative value units (RVUs), a standardized measure of the time, expertise, and skill required for a physician to perform a given procedure. Surgical CPT codes were linked to RVUs using a cross-talk from the Centers for Medicare & Medicaid Services Physician Fee Schedule (January 2014 release). Covariates entered into the hierarchical mixed-effects model were the same as for the primary statistical model, with the addition of work RVUs as a continuous covariate.

The final post hoc sensitivity analysis used propensity scores to estimate the impact of late case starts. “Late” cases were defined as those beginning between 4:00 pm and 10:59 pm, and “day” cases began between 7:00 am and 3:59 pm. Cases occurring outside those hours, patients younger than 18 yr, ASAPS 5 patients, and emergency cases were excluded. A propensity score to predict likelihood of being scheduled as a “late” case was created using logistic regression with variance adjusted for clustering within practices, adjusting for age, sex, ASAPS, facility type, and location (to account for differing practice patterns). Although the logistic model predicted “day” or “late” assignment significantly better than chance, the area under the receiver operator characteristic curve was 0.625, consistent with relatively random assignment to “day” or “late” as would be expected if assignment was a result of scheduling convenience. We then performed 1:1 matching of “day” cases to “late” cases, with exact matching for ASA crosswalk code and within a caliper distance of 0.01 for the logistic probability model. In this way, we rigorously controlled for a case mixture that potentially varies with time. Because there were far more “day” cases than “late” cases, the sort order of the data markedly affected propensity matching and the rate of mortality events in the matched “day” patients. Accordingly, a modified bootstrapping approach was taken, in which the “day” patients were randomly sorted and propensity matching was run; this was performed 500 times, and the mean and SD for the number of deaths in the “day” cases after 500 replications was reported.

A P value less than 0.05 was considered to indicate statistical significance. All statistical tests were two tailed. Statistical analyses were performed using Stata/MP 13.1 (StataCorp, USA). The hierarchical mixed-effects logistic regression was performed with the Stata meqrlogit command, and logistic regression with variance adjusted for clustering within practices was accomplished with the Stata logit command. Propensity matching used the Stata psmatch2 command.

After exclusions for cases with missing data for both surgical and anesthesia procedure types and vaginal deliveries, the final data set included 2,866,141 cases and 944 deaths within 48 h of induction of anesthesia (fig. 1).

Included records are compared with excluded records in table 1. Excluded records were more likely to have missing data for ASAPS. Excluded patients were younger, represented more females than males, and had a higher proportion of neuraxial anesthesia, reflecting the 65,318 episodes of analgesia for vaginal delivery that were excluded. Excluded cases were also less likely to have been coded as emergent and were less likely to have had a case start time between 7:00 am and 3:59 pm. There were also significant differences in region and facility type among included versus excluded records; these differences were attributable to large practices, which dominated the region and/or facility type and had missing data effectively requiring elimination of the practice or facility from the final data set. The included records were from 60 practices, which provided anesthesia care at 197 facilities.

Missing data for the six primary variables of interest (patient age, sex, ASA physical status and emergency case status, primary anesthesia type, case start time, and surgery type) varied substantially by practice. The distribution of missing data by practice is shown in figure 2. The median percent of cases with missing data, by practice, was 6.0% (interquartile range, 0.7 to 73.0%). A single practice that did not report operative start times contributed 472,353 cases, comprising 62.1% of the cases in this study with incomplete data.

The unadjusted rate of perioperative mortality within 48 h of induction of anesthesia remained reasonably constant over the studied period (fig. 3). Raw rates of perioperative death and unadjusted odds ratios, by variables of interest, are given in table 2. The highest raw death rates were seen in ASAPS 4 and 5 patients, emergent surgeries, and patients younger than 1 yr.

The distribution of cases and deaths according to patient age is shown in figure 4A; patients younger than 1 yr were disproportionately represented among the deaths. Figure 4B depicts the distribution of cases and deaths according to the operative start time and demonstrates an increased raw mortality rate for case start times before 7:00 am or after 3:59 pm.

Hierarchical mixed-effects logistic regression analysis was performed for cases for which the data were complete. Age less than 1 or more than or equal to 65 yr; ASAPS 3, 4, or 5; and case start time after 4:00 pm were significantly and independently associated with perioperative mortality (table 2).

Several post hoc sensitivity analyses were performed to further explore the association between case start time and perioperative mortality. For the first sensitivity analysis, we further limited the cohort to patients undergoing 1 of 24 elective case types starting between 6:00 am and 10:59 pm and excluded emergent cases, ASAPS 5 patients, and children. By using logistic regression with clustering by practice and adjusting for sex, age, ASAPS, and surgery type, a significant independent association between mortality and a start time after 6:00 pm was again demonstrated with odds ratio of 3.98 (95% CI, 1.54 to 10.3, P = 0.004). There was no significant increase in mortality in this subgroup for cases starting between 4:00 pm and 5:59 pm (table 3), although the point estimate was consistent with findings from the whole data set–adjusted model.

The second post hoc sensitivity analysis incorporated the RQI as a highly discriminative measure of adult surgical mortality risk. RQI could be calculated for only 867,186 patients. After casewise deletion for other missing variables in the clustered logistic regression, this subgroup analysis was performed on 715,540 patients. Point estimates for mortality risk associated with after-hours operative start times were substantially similar to the whole cohort, non–risk-adjusted model, although the CIs were wider (table 3).

The final post hoc sensitivity analysis linked 2,144,320 patients with a value for work RVUs, in an attempt to adjust for surgical complexity with greater granularity. Of these patients, 1,738,246 had complete data for the other variables of interest and were included in the hierarchical mixed-effects model, which confirmed the association between perioperative mortality and after-hours start times (table 3).

This post hoc subgroup analysis excluded emergency cases, ASAPS 5 patients, and children and used propensity matching to approximate “random” assignment to day–start (7:00 am to 3:59 pm) versus late to start (4:00 pm to 10:59 pm) case times. Propensity matching was performed within a cohort of 1,500,479 cases, 7.2% of which began between 4:00 pm and 10:59 pm. Because the number of deaths in the matched day–start cases markedly varied depending on sort order, the propensity match was run 500 times. There were 37 deaths in the cohort of cases starting after 4:00 pm and 25.5 ± 3.9 deaths in the matched cases (95% CI, 17.9 to 33.1) starting between 7:00 am and 3:59 pm.

The most common outcomes concurrently documented in patients who died within 48 h of induction of anesthesia were major or minor hemodynamic instability (35.0%), major or minor respiratory complications (8.1%) including airway/intubation complications (1.7%), upgrade of care (4.2%), and resuscitation (2.8%).

This is an early report of outcomes data available in the NACOR data set, which was created by the ASA for the reporting of quality metrics and anesthesia-related perioperative outcomes. We confirmed several known associations between perioperative mortality and patient age, ASA physical status, and emergency case status. We also demonstrate that, in this data set, operative start time is associated with perioperative 48-h mortality after a limited degree of adjustment for comorbidity, with higher risk of death in surgical cases starting after 6:00 pm. This association was confirmed in several post hoc sensitivity analyses and a post hoc propensity-matched cohort and offers a potentially modifiable target for perioperative risk reduction.

The overall 48-h mortality rate of 33 per 100,000 anesthetics is lower than the unadjusted perioperative mortality rate of 89 per 100,000 for high–human development index countries derived from an impressive recent meta-analysis.¹  This likely reflects differences in reporting, as practices in NACOR may be reporting deaths ranging from intraoperative only to up to 48 h postoperatively, depending on data collection methods at individual participating practices. Some of the studies included in the Lancet report included deaths up to 30 days postoperatively (although the majority of studies focused on the first 7 days after surgery). The lower mortality rate in NACOR may also reflect underreporting or other limitations of the data set itself.

The association between mortality and case start time has been debated in the literature, with most large studies of unselected surgical populations reporting no association between time and 30-day operative mortality.^6,9  We are not aware of any previous studies evaluating the impact of operative start time on 48-h mortality. There is a known association between later operative start time and perioperative complications^6,12  and, specifically, anesthetic-related adverse events,¹³  perhaps as a result of provider fatigue (reviewed by Warltier et al.¹⁴ ), care transitions, or other factors.

In our study, the most common complications reported concurrently in patients who died included airway complications (1.7%), resuscitation (2.8%), respiratory complications (8.1%), and hemodynamic instability (35.0%). There is evidence that “failure to rescue,” providers’ failure to detect and/or prevent a clinically significant deterioration, occurs more commonly in situations of reduced staff availability (e.g., in hospitals with lower amounts of nursing care per patient¹⁵  or higher patient-to-nurse ratios¹⁶ ). Because the majority of anesthesia care in the United States is provided between 7:30 am and 3:30 pm,¹⁷  we would expect fewer staff to be available to respond to emergencies after hours, in parallel with the decreased workload. It is plausible that in a large data set such as this, the subtle impact of perioperative adverse events (which occur with greater frequency outside elective case hours),^6,12  and “failure to rescue” from those adverse events, on 48-h mortality rate could be detected, whereas smaller studies have been unable to find an association.

Our findings must be considered in the light of several additional limitations in this data set. Data on concurrent adverse events in patients who died may be incomplete, if providers preferentially document the “worst” complication (i.e., mortality) and not other adverse events that occurred with, or perhaps contributed to, the death. An association between time of day and outcome may be because of important confounding variables, such as emergency surgery. We have attempted to exclude emergency cases not coded as such using a post hoc elective subgroup analysis and propensity score analysis, which largely confirmed the findings from the whole data set model. Sensitivity analyses using two methods of controlling for procedure-associated risk (the RQI and work RVUs) also produced substantially similar results. NACOR does not uniquely identify patients in the data set; there is no way to eliminate patients who had multiple separate anesthetics. Definition of ASAPS, entry of emergency case status, and entry of outcomes were typically left to the individual anesthesia provider’s discretion; thus, these metrics likely differ among providers. The accuracy of the other variables was dependent on coding by practice; because NACOR deidentifies its data, there was no way to spot check this data entry for accuracy. In addition, we have no information on the cause of death for any patient who died. Although retrospective cohort studies in general cannot determine causality, we additionally emphasize that we cannot speculate on the relative contributions of anesthetic, versus surgical, versus patient-related or systems components to the reported deaths. Furthermore, although the NACOR data set undergoes continual refinement and error checking, we found a large amount of missing data even within the administrative variable set. Table 1 documents the missing data rates, which ranged from 0.9 (patient sex) to 18.0% (case start time) in the overall data set for the limited list of variables that required to address our research question.

In summary, cases from the first 4 yr of NACOR data collection indicate a perioperative 48-h mortality rate of 33 per 100,000 cases. Significant associations with perioperative mortality include age less than 1 yr or more than or equal to 65 yr, ASAPS, emergency case status, and operative start time after 6:00 pm. The association with later operative start time was robust to post hoc subgroup analyses and a propensity score analysis. Although causal inference cannot be drawn from a retrospective cohort study, our findings suggest a clinically plausible, potentially modifiable risk factor for perioperative mortality that merits further investigation. Because of concerns surrounding missing data, possible residual confounding, and nonstandardized mortality reporting, these findings are preliminary. Improvements in practice reporting will maximize data completeness and generalizability and will allow further investigation of regional and national contributors to perioperative morbidity and mortality.

The authors thank the Anesthesia Quality Institute and National Anesthesia Clinical Outcomes Registry for collecting and sharing the data that made the preparation of this manuscript possible. They also thank Richard P. Dutton, M.D., M.B.A. (Anesthesia Quality Institute, Schaumburg, Illinois), for his thoughtful comments on the draft manuscript; I. Elaine Allen, Ph.D. (University of California, San Francisco, San Francisco, California), for her assistance with statistical analysis; and Jonathan Wanderer, M.D. (Vanderbilt University, Nashville, Tennessee), for statistical code used to calculate the Risk Quantification Index.

Dr. Whitlock is a research scholar in the Department of Anesthesiology and Perioperative Care and receives departmental funding.

The authors declare no competing interests.

The National Anesthesia Clinical Outcomes Registry (NACOR) electronically and automatically collects deidentified data from all cases performed in participating practices. Incoming data undergo several automated validity checks, which are further described under Data Validation. The data set itself is maintained with Microsoft SQL Server 2008 R2 (Microsoft Corporation, USA).

Through programming customized for each practice and/or facility, standard administrative data (i.e., year and month, duration of anesthesia billing time, patient age, American Society of Anesthesiologists physical status, procedure billing codes, etc.) are mapped to standardized fields within the NACOR database. Depending on the field, these data elements may be entered by the anesthesia provider, registration staff, or billing specialists.

Beyond the standard administrative dataset, NACOR also invites the sharing of data regarding certain patient-focused outcomes and data from anesthesia information management systems, which may include drug administration and physiometric data, among other items. Patient-focused outcomes are typically entered by the anesthesia provider at the time of the postoperative assessment. Data from an electronic anesthesia information management system may be entered automatically (i.e., from automated noninvasive blood pressure cuff monitoring, pulse oximetry, end-tidal carbon dioxide monitoring, etc.) or manually (drug names and dosages) by anesthesia providers. Once programming and mapping of standardized fields is complete and verified, the transfer of data from practices to NACOR is fully automated and requires no manual abstraction.

Incoming data from practices is mapped into staging tables, where a series of checks are performed to ensure source-level and case-level data integrity.

First, the structure and format of the new data are checked to ensure that input formats (i.e., string, integer, etc.) are appropriate to the target fields. The data then undergo some cleaning at the individual record level, where records missing essential data elements are eliminated and a series of between-field logic checks ensure that complementary fields (e.g., procedure start time and procedure end time) do not contain mutually exclusive values. Individual values at the case level are checked against a predetermined range of valid values. The value distribution from the set of new cases is also compared against previous data from that same practice to ensure that the new data are consistent with the distribution of that variable from the practice’s previous records. Inconsistencies at any step are flagged for manual review by NACOR staff.

These cleaned and checked data are then inserted into the main NACOR data set. Automated whole data set record validation provides reports comparing historically provided cases with newer data submissions. This includes trend analysis by date of import into NACOR and by date of case to interrogate gradual drift or rapid shifts in data value ranges.

The NACOR participant user file, which is made available to researchers, is updated on a quarterly basis, both retrospectively (as practices contribute historical cases) and contemporaneously. Data fields with low reliability or high heterogeneity are not released to users in the participant user file; because those fields undergo further refinement at the NACOR or practice level, they may become available in future.