This study sought to evaluate the effectiveness of an active survey method for detecting anesthesia nonroutine events (NREs). An NRE is any aspect of clinical care perceived by clinicians or observers as a deviation from optimal care based on the context of the clinical situation.


A Comprehensive Open-ended Nonroutine Event Survey (CONES) was developed to elicit NREs. CONES, which consisted of multiple brief open-ended questions, was administered to anesthesia providers in the postanesthesia care unit. CONES data were compared with those from the same hospital's anesthesia quality assurance (QA) process, which relied on self-reporting of predefined adverse events.


CONES interviews were conducted after 183 cases of varying patient, anesthesia, and surgical complexity. Fifty-five cases had at least one NRE (30.4% incidence). During the same 30-month period, the QA process captured 159 cases with 96.8% containing at least one NRE among the 8,303 anesthetic procedures conducted (1.9% overall incidence). The CONES data were more representative of the overall surgical population. There were significant differences in NRE incidence (P < 0.001), patient impact (74.5% vs. 96.2%; P < 0.001), and injury (23.6% vs. 60.3%) between CONES and QA data. Outcomes were more severe in the QA group (P < 0.001). Extrapolation of the CONES data suggested a significantly higher overall incidence of anesthesia-related patient injury (7.7% vs. only 1.0% with the QA method).


An active surveillance tool using the NRE construct identified a large number of clinical cases with potential patient safety concerns. This approach may be a useful complement to more traditional QA methods of self-reporting.

AN increasingly well-accepted model of patient safety1,2distinguishes between active errors, which are most commonly linked with frontline providers’ actions or inactions, and latent conditions, which are systemic issues that predispose to subsequent active errors. Common examples of latent conditions are dysfunctional organizational structure and policy, inadequate training, faulty communication, and poorly designed medical devices.3–6An adverse outcome is believed to result primarily from the rare coincidence of multiple events evolving from latent conditions, triggered by more readily apparent active errors. Identification and correction of latent conditions is crucial to improving safety, but it is unclear how to best identify these potential pathways to patient harm.

The most popular method currently used in hospital quality assurance (QA) programs (table 1) is to examine adverse events that have already occurred and “trace back” the problem attempting to pinpoint a “root cause.” Although this approach has a role, especially as a hypothesis-generating activity, it is limited for several reasons. Retrospective analysis of adverse events is contaminated by cognitive (especially hindsight, outcome, and attribution) bias, making an accurate assessment of causality often impossible.1,7–9Typical QA event analyses yield specific recommendations to prevent the same event from reoccurring. However, the cause of adverse events is often multifactorial and complex, and the “same event” (i.e. , identical sequence of subevents and failures) is unlikely to reoccur. As well, reporting and analysis of adverse events are problematic because of clinicians’ reluctance to report mistakes due to concerns of social, legal, and regulatory retribution.10Finally, traditional QA systems may bias clinicians toward reporting only specific types of events (e.g. , sentinel or predefined event categories) and thus do not capture many aspects of suboptimal care that might facilitate understanding of deficiencies in healthcare system performance and opportunities for improving patient safety.10 

Table 1. Definition of Terms and Abbreviations 

Table 1. Definition of Terms and Abbreviations 
Table 1. Definition of Terms and Abbreviations 

Therefore, although traditional QA is an important component of institutional patient safety efforts, it is a cumbersome process with many limitations. We report the results of a pilot study that examined an event detection and analysis technique that could serve to supplement traditional QA efforts based on clinician self-reporting.

An event discovery tool is needed that is not hampered by precategorized events, is nonjudgmental, has high compliance, facilitates discovery of latent conditions, and provides ample data to inform intervention strategies. One such approach may be the collection of nonroutine events (NREs).11,12An NRE is any aspect of clinical care perceived by clinicians or observers as a deviation from optimal care based on the context of the clinical situation. Unlike approaches that focus on specific “adverse events,” NRE identification is open-ended. Observers or clinicians identify all events that are unusual, are unexpected, or deviate from optimal care. Therefore, the NRE construct extends the definition of noteworthy information beyond the occurrence or near occurrence of patient injury.12While NREs include “near misses”10and “critical incidents,”13,14they also encompass events that may not have an immediately obvious link to adverse outcomes but still could provide early clues to important latent conditions in healthcare systems. For example, the failure of the results of routine preoperative laboratory test values to be available just before surgery would not generally be considered by most anesthesia providers to be a “near miss.” Yet, capture and analysis of several of these NRE could bring to light process problems in specimen delivery, clinical laboratory operations, or results reporting that might otherwise remain unappreciated until a patient injury event occurred as a consequence.

We developed the Comprehensive Open-Ended Nonroutine Event Survey (CONES) to evaluate an alternative method of reliably collecting clinical event data. CONES data were compared with data from the same institution’s existing anesthesiology QA program. This study was based on three hypotheses: (1) active surveillance using CONES would yield more events than would be collected using traditional self-reporting QA methods; (2) the CONES approach would capture a wider variety of events than those identified by the QA process, thereby providing more information on which to base system understanding and improvement; and (3) the CONES sample would be more representative of the overall surgical population than would the QA sample and, thus, would permit a more accurate picture of the true incidence of different types of events and their contributory factors.

Materials and Methods

After obtaining institutional review board approval, a trained nonclinician research assistant interviewed anesthesia providers using the CONES structured survey tool on 87 days, selected at convenience for the researchers, over a 30-month period at the Veterans Affairs San Diego Healthcare System, San Diego, California. Because the survey was specifically targeted to capture NREs occurring in the operating rooms, the assistant interviewed anesthesia providers after they had handed off their patients to the postanesthesia care unit nurse or, if necessary, during slow periods of the next case. A standardized interview form was used to document NREs based on the clinicians’ response to nine open-ended questions ( appendix 1). After identifying an NRE, additional questions sought data about its etiology and potential contributor factors.12Only the data from the CONES interview were available for subsequent analysis and event coding.

Data collection rarely took more than 5 min. No identifiable patient or provider data were obtained. The NRE interviewers were independent of the department, and the process was strictly confidential and anonymous. Interviewers were college educated but had no clinical experience before their comparable training in preparation for this study. The training began with a month of basic reading about anesthesia and observation of providers in the operating room, similar to a medical student rotation. Trainees observed both real and videotaped cases under close supervision of more experienced observers. Interviewers were subsequently trained in the CONES interview process by shadowing an experienced interviewer during postanesthesia care unit interviews over multiple days of data collection. Expert reviewers were not actively providing patient care at this institution during this study (except for the principal investigator, who only served as a final arbiter). The two primary reviewers did not collect data, interact with the data collectors, or attend departmental QA conferences.

The intent of the CONES tool was to capture all events that were thought to deviate from optimal care. The CONES was designed to first ask general questions in an unbiased manner about anything that might have deviated from routine or optimal for that provider ( appendix 1). Therefore, the initial nine questions were used to identify NREs. The cause of an NRE is usually multifactorial, and therefore, the questions were intended to identify a variety of these factors. The initial question sequence was the same every time, and an affirmative response to any of the first nine questions led to the second set of questions, several of which were open-ended and required more detailed responses, designed to ascertain the nature and etiology of the putative NRE. The CONES questions were refined through an iterative process based on pilot postanesthesia care unit surveys as well as feedback from clinician participants, interviewers, and reviewers. All of the items in the final CONES version were found to elicit meaningful responses.

Traditional Quality Assurance

Over the same period, the Veterans Affairs San Diego Healthcare System’s QA system collected data regarding specific anesthesia care events. After every anesthetic, the anesthesia service expected its providers to fill out a form indicating whether one or more categories of adverse events had occurred during the case ( appendix 2). If a QA form (which allowed the option of checking a “no event” box) was not completed for a specific anesthetic, the department administrator contacted the responsible certified registered nurse anesthetist or anesthesia resident and asked him or her to complete the form for that case. QA data forms and associated peer-review analyses were deidentified and transferred to the study database. Only data about the patient and situation that appeared in the final QA report were included in the analysis.

For surgical population demographics, data sets of all procedures performed in the operating room during the study period were acquired from both the anesthesia and surgical services. The databases were compared and found to be largely congruent. The anesthesia database was thought to contain more accurate data about patient age, anesthetic type, and American Society of Anesthesiologists (ASA) physical status. The surgery database was used to provide the type of surgery performed.

Data Management

All identifiable patient and provider information was removed from all data sets. Each case was assigned a unique randomly generated case number, as was each NRE identified. Data obtained from the two methods, along with any demographic, medical, and peer-review (for QA data only) summaries were entered into a custom password-protected Filemaker Pro® (Santa Clara, CA) database. NRE data were entered into the database in a uniform manner, regardless of the source (i.e. , whether CONES or QA derived). Because provider identity was excised, each case was treated independently even though the same anesthesia provider or surgeon might have been involved in several cases. Because it was possible for the two methods to both capture the same operative event, it was prospectively decided that, to address this potential statistical confound, all duplicate cases would be excluded from data analysis (table 2).

Table 2. Two Cases Obtained by Both Event Detection Methods during the Study Period 

Table 2. Two Cases Obtained by Both Event Detection Methods during the Study Period 
Table 2. Two Cases Obtained by Both Event Detection Methods during the Study Period 

Event Analysis

Two clinical experts independently reviewed each event in both data sets and determined whether reported events met the definition of an NRE. For all of the analyses described below, if there was disagreement between expert reviewers, a consensus was reached after discussion of each disputed case. If agreement was not obtained, a third expert reviewed the case, and the majority opinion was selected. To reduce hindsight bias, reviewers were shielded from knowledge of NRE outcome or clinical course until the end of their review. Reviewer-completed ratings and annotations were stored in the database. Before any statistical analyses, all events were reviewed one last time to ensure maximal consistency of classification and coding of all cases.

Each NRE identified from QA- and CONES-derived data were categorized according to its patient impact (PIE), patient injury type, outcome severity, and putative contributing factors (see definitions in table 1). Patient injury was defined as any unanticipated side effect or complication of anesthesia that affected the patient’s postoperative course or quality of life. Therefore, with this patient-centric definition, “injuries” included events that required the need for a higher level or prolonged postoperative care (e.g. , surgical intensive care unit admission instead of discharge home) or emotional distress (e.g. , case cancellation, intraoperative recall).

A structured taxonomy of 337 event descriptors, organized hierarchically by clinical manifestation, was applied to each NRE. This comprehensive taxonomy described both patient impact and injury events within major categories as well as subcodes for specific events. For example, under the major category of Airway Events, one would find codes for difficult ventilation, difficult intubation, esophageal intubation, inadvertent endotracheal extubation, and so forth.

Over a 1-yr period before this study, multiple anesthesia experts in our department refined this taxonomy iteratively. In collaboration, Dr. Ping-Wing Lui et al.  15at the Chang-Gung Memorial Hospital, Taoyuan, Taiwan, successfully used a version of this taxonomy in a quality improvement study involving almost 200,000 anesthetized patients. The event classification scheme was iteratively developed and refined before being applied in this study.12 

To simplify analysis and interpretation, the discrete PIE categories were condensed into eight composite categories (table 3) based on common organ systems affected (e.g. , all primary pulmonary events were grouped together) or presumed mechanisms of action (e.g. , the Anesthesia category included needle-related complications of regional anesthesia and of central venous catheterization).

Table 3. Composite Patient Impact Events 

Table 3. Composite Patient Impact Events 
Table 3. Composite Patient Impact Events 

For each patient injury, the reviewer used the ASA Closed Claims Study Injury Severity Scale to code the magnitude of injury.16Injuries were rated from 0 (no injury) to 9 (death) depending on the severity of patient harm. Emotional injury was rated as 1. Insignificant, minor, and major temporary injuries were rated as 2, 3, and 4, respectively. Minor, significant, major, and grave permanent injuries were rated as 5, 6, 7, and 8, respectively. Because it was possible for each case to have several NREs and for each NRE to have multiple outcomes, the most severe outcome for all NREs in a case was assigned to that case.

Finally, each reviewer assessed whether any of 21 discrete contributory factors (see table 4footnote) seemed to play a role in the occurrence of each NRE. This postevent analysis was used to delineate differences between types of NREs and potential common sources or mechanisms of event etiology deserving of intervention. The contributory factor classification scheme was established at the beginning of the project based on the literature and 20 yr of human factors experience. For analysis, the contributory factors were condensed into 11 composite contributory factors (table 4).

Table 4. Contributory Factors 

Table 4. Contributory Factors 
Table 4. Contributory Factors 

Statistical Analysis

The principal objective of this study was to evaluate differences between CONES and traditional QA data in terms of case variables (i.e. , patient, anesthesia, and surgical factors), NRE incidence, presence of patient impact or injury, outcome severity, and putative contributory factors. Case variables, NRE incidence, and presence of patient impact or injury were evaluated by case using the Pearson chi-square test with the Yates continuity or the Fisher exact test. Patient outcome severity score data were categorically grouped (0, 1–3, 4, 5, >6) and then analyzed using the Fisher exact test.

The 11 composite contributory factor categories were considered statistically independent, because dependencies were accounted for when collapsing the original discrete categories into the composite categories. A logistic regression model was used to test for differences between CONES and QA with respect to the presence of individual contributory factors. Individual factors were then tested using the Pearson chi-square test with the Bonferroni correction applied to adjust for testing multiple comparisons.

Similarly, PIEs were collapsed into eight categories that were considered independently. A logistic regression model was used to test for differences between CONES and QA with respect to the count distribution of individual PIEs (0, 1, or > 1). The Pearson chi-square test was used to compare the distribution of PIEs between CONES and QA. Individual categories were then tested using the Pearson chi-square test with the Bonferroni correction applied to adjust for testing multiple comparisons.

All statistical analyses were conducted using the R statistical software package, which is an on-line open-source software application.††Differences were considered statistically significant if P  was less than 0.05. P  values less than 0.001 were reported simply as less than 0.001. Samples with counts of 3 or fewer did not have P  values computed. Multiple comparison adjustment via  Bonferroni correction was applied when required.


The active survey method using the nonroutine event construct identified multiple events that may have otherwise gone undetected and therefore unexamined with traditional QA reporting. During the 30-month study period, a total of 8,303 anesthetic associated procedures were performed at this hospital. For at least several hours on 87 weekdays over a period of approximately 630 regular workdays, 183 CONES surveys were performed, representing 2.2% (183 of 8,303) of cases. Of this sample, 31.3% of the cases contained an NRE. During the same period, a total of 1.9% (159 of 8,303) of all the cases were reported and evaluated per traditional QA analysis according to departmental policy. Two cases captured by both methods were excluded from subsequent analysis (table 2), and all data reported hereafter reflect this.

Patient and provider variables in the CONES data were highly reflective of the surgical population (table 5). By contrast, patients in the QA data set were slightly older (P < 0.05). A greater proportion of QA cases were performed by residents (P < 0.001). General anesthesia was more common, and monitored anesthetic care was much less frequent in the QA data set compared with the surgical population as a whole (P < 0.001). The QA data set contained patients with a higher ASA classification (i.e. , sicker patients; P < 0.001), and emergent cases were more common (P < 0.05).

Table 5. Case Demographics 

Table 5. Case Demographics 
Table 5. Case Demographics 

Compared with the population, cardiothoracic (P < 0.05) and ophthalmologic (P < 0.01) surgical procedures were underrepresented in the CONES data, whereas plastic surgical cases were overrepresented (P < 0.001). Neurosurgical (P < 0.001), cardiothoracic (P < 0.001), and vascular (P < 0.01) surgery cases were overrepresented in the QA data. The QA process did not detect any ophthalmology case events.

Nonroutine events occurred in 55 (30.4%) of 181 CONES cases and 156 (98.1%) of 159 QA cases (P < 0.001). Among cases with NREs, there was patient impact in 74.5% versus  96.2% of CONES and QA cases, respectively (P < 0.001). Similarly, patient injury among NRE cases occurred in 23.6% of CONES and 60.3% of QA cases (P < 0.001). Severity of injury was significantly worse in the QA data (P < 0.001; fig. 1). However, if one linearly extrapolates the incidence determined from the CONES data sample to all surgical cases during the study period, the CONES method detected a sevenfold greater incidence of patient injury events (7.7% for CONES vs.  1.1% for QA). The greater frequency of event identification using the CONES process suggests that many more safety-critical events, including near misses, could be captured in a robust QA system that incorporated additional methods of event identification.

Fig. 1. Histogram of American Society of Anesthesiologists Closed Claims Injury Severity Scores. CONES = Comprehensive Open-ended Nonroutine Event Survey; QA = quality assurance. 

Fig. 1. Histogram of American Society of Anesthesiologists Closed Claims Injury Severity Scores. CONES = Comprehensive Open-ended Nonroutine Event Survey; QA = quality assurance. 

Among CONES cases, the most common PIEs were airway, cardiovascular, other (miscellaneous), and surgical events (table 3). Among QA cases, cardiovascular, pulmonary, airway, and patient disposition issues were most frequent. In the logistic regression model, PIEs related to the neurologic, cardiovascular, and pulmonary systems were less frequent in the CONES data set compared with the QA data (all P < 0.001). A lower proportion of CONES NRE-affected patients were sent to higher levels of postoperative care (e.g. , fewer required care in the intensive care unit). However, this difference did not attain significance in the overall logistic regression.

Among CONES cases, the most common factors seeming to contribute to NRE were Patient, Provider,‡‡Equipment, and Preoperative preparation factors. For QA-derived NRE, Patient, Provider,‡‡Surgical, and Drug-related factors were most frequent (table 4). Patient-related factors (i.e. , preexisting disease or unexpected response) were less  commonly cited in CONES NREs (52.7%) than those detected with the QA process (84.6%; P < 0.001). In contrast, equipment factors (especially poor usability) (27.3% vs.  4.5%; P < 0.001) and environmental factors (10.9% vs.  0.6%; P = 0.001) were more commonly cited in CONES compared with QA NREs. In the logistic regression model, only surgical (P < 0.05) and patient (P < 0.005) factors proved significantly different between the two data sets.

There were patterns to CONES-detected NREs that identified patient safety issues that might not have emerged through traditional QA reporting. For example, a number of cases were reported in which significant desaturation occurred after extubation. This finding led to further study, analysis, and implementation of simulation-based training and more explicit department-wide appreciation of extubation risks.


This study examined the potential value of the NRE-based survey technique as a complement to traditional QA. Applying CONES to a representative sample of surgical patients revealed an incidence of patient injury seven times greater than that suggested by data captured using a traditional QA process. The CONES approach revealed patient impact events (i.e. , events associated with changes in patient physiologic or psychological status) 12 times more frequently than self-reported QA data. The QA process generally captured the more serious patient injury events, but it did not detect many mild to moderate injury events and most near misses. CONES captured a broader array of the range and magnitude of patient safety issues in perioperative care than the traditional method of event detection. Its greater sensitivity, provider compliance, and ability to use nonclinician interviewers suggest that the CONES technique may be an effective complementary strategy for collecting data about perioperative anesthesia events. Perhaps ultimately a modified combination of the two will be shown to improve event detection and analysis, thereby enhancing continual quality improvement processes.

Previous research in this area has examined both intraoperative as well as recovery room occurrences that fell on a continuum of adverse anesthetic-related events. In the published work perhaps most similar to ours, Cooper et al.  17asked providers to self-report events on patient admission to the postanesthesia care unit on a form listing approximately 90 possible events (as opposed to being collected via  facilitated CONES interview). Like CONES, the authors used a wide definition of “recovery room impact events,” and these were collected anonymously and contemporaneously. The incidence of recovery room impact events in their study was 18%, approximately half of what we detected. Similar to our findings, recovery room impact events with more than minor sequelae were infrequent (2.5–3%). In contrast, in an even earlier unpublished 1983 study by R. W. Vaughan, M.D., et al.  at the University of Arizona, a trained investigator observed 451 adult cases and reported that 46.6% of these cases contained at least one “anesthesia-related consequence” (with a definition similar to the “critical incidents” of Cooper et al.  11,12). Therefore, differences in anesthesia event reporting rates include differing definitions of an event as well as methods of identifying those events. An important distinction in our approach from many previous event detection methods is our use of trained examiners actively interviewing clinicians using a progressive cascade of questions and a strict methodology.

Clinician Compliance

The CONES was well received by anesthesia providers at the study hospital. The interview process was not perceived as a threat to autonomy or a hindrance to efficiency. The anonymous process of reporting NRE was appreciated by anesthesia providers to be stress free and had the potential to identify clinical system or process problems. The active surveillance technique using trained interviewers and a structured survey instrument facilitated data collection and hence improved compliance and clinician acceptance. The addition of CONES may help to create a less threatening local event-reporting environment, analogous to the successful model used by the aviation industry to improve flight safety by anonymous submissions to the Aviation Safety Reporting System.§§

Sampling Issues

The CONES was administered in close proximity to the care provided, and the relevant data were collected actively (through interviewer-facilitated structured surveys). Therefore, CONES data collection can be distinguished from traditional QA self-reporting in that (1) CONES is open-ended (i.e. , any type of event can be reported), whereas QA typically uses prescribed categories; (2) CONES data are collected actively, whereas traditional QA systems rely on voluntary self-reporting and/or chart review; and (3) CONES contemporaneously collects data about event etiology, detection, management, and possible contributory factors, whereas in traditional QA systems, such data are only obtained days or weeks after the event is reported. Although both techniques are retrospective, the contemporaneous nature of CONES review may be less prone to “cognitive contamination or bias” that tends to develop with the passage of time and knowledge of the outcome. In addition, the CONES data may be more accurate because it comes directly from the frontline provider immediately after the case as opposed to later via  chart review. The literature suggests serious shortcomings of delayed data collection, with postevent recall and event documentation being significantly affected by memory shaping, hindsight bias, and outcome bias.9,18,19 

CONES data were statistically indistinguishable from the surgical population as a whole in regard to age, sex, ASA physical status, type of anesthesia used, and type of anesthesia provider. In contrast, the QA cases involved sicker patients (higher ASA physical status), were more often emergencies (twice as common as seen in the population), and were reported more often by anesthesia residents than by certified registered nurse anesthetists. As a result, cardiothoracic, vascular, and neurosurgical service patients were represented disproportionately among QA cases.

Events captured by traditional QA self-reporting systems (whether voluntary or “mandatory”) tend to emphasize sicker patients undergoing high-risk surgeries and thus may not provide a reliable picture of the risks to anesthesia safety. The care of very ill patients is complex, and when adverse events occur, numerous factors can obscure the identification of remediable causal variables. Correcting process deficiencies discovered from analyzing events in healthier patients undergoing elective procedures may be more productive and nonetheless benefit the sicker high-risk patients who have a lower tolerance for complications or system failures.

Primarily because of the way CONES data were collected, emergency cases and three types of surgery (cardiothoracic, ophthalmologic, and plastic surgery) were underrepresented. The CONES surveyors rarely encountered emergent patients because most of these cases were completed after the daytime work shift. Similarly, cardiothoracic cases were usually long procedures ending late in the day and postoperatively were transferred directly to the surgical intensive care unit. Almost all ophthalmology patients also bypassed the postanesthesia care unit, being recovered in the same-day surgery unit. Why plastic surgery cases were overrepresented in the CONES sample is unclear. Some of these sampling issues could be remedied by altering the location and work schedule of the CONES surveyor.

Patient Injury

Self-reported QA data has been consistently shown to underestimate the occurrence of patient injury.20–22The CONES method confirms this shortcoming of traditional event reporting. Extrapolating the CONES findings suggested that 7.7% of all patients undergoing anesthesia may be injured, albeit mostly of nominal severity. Half of QA injuries reported were also minimal in severity (1–2 on the Outcome Severity Score). Although the injury severity of QA data was significantly greater than that of CONES data, the small number of CONES cases with injury (n = 13) may be insufficient to assure statistically stable results. Notably, two cases with injury in the CONES database were excluded from this analysis because they were also captured in the QA data. Recalculated injury rates with those cases included suggest an incidence of 8.3% for CONES and 1.2% for self-reported QA data.

The QA process tended to capture the more serious patient injury events, whereas the CONES method captured a larger number and wider variety of events. Although the CONES events tended to be mild to moderate in terms of actual patient injury, they may nonetheless be significant harbingers of patient care quality and satisfaction. Traditional event self-reporting has not typically been a reliable method of capturing near misses. Often, noninjurious events are underreported because it is a common perception that such events do not significantly contribute to patient outcome and are therefore not worthy of reporting. However, near-miss events can provide valuable insights regarding potential sources of system failure and risks of future injury events.10 

Patient Impact Events

The CONES results demonstrate that many patients undergoing anesthesia experience significant physiologic perturbations, most of which do not lead to overt perioperative injury. Almost one quarter of CONES patients experienced such an impact event. Of note, among cases with NREs, QA cases averaged two patient impact events, whereas CONES cases were more commonly singular events. Some might dismiss such impact events as unimportant because they are not “outcomes.” However, they may be harbingers of later postoperative adverse events. Moreover, we suggest that even if transient myocardial ischemia, oxygen desaturation, or profound hypotension, for example, do not cause obvious tissue injury, they represent “at-risk” situations, which are undesirable.23–25The increased frequency of impact events among more severe QA cases gives some credibility to the assertion that these patient impact factors play a role in adverse outcomes.

Contributory Factors

We examined which putative contributory factors were present in cases with NREs (table 4). Factors such as preexisting patient disease weighed more heavily in QA data, suggesting that self-reported QA data and its peer-review analyses tend to emphasize patient phenomena rather than clinical or systems processes. Equipment and environmental factors, which were much more frequent in CONES cases, are potentially important remediable latent conditions. The relative absence of such factors in the QA data may reflect either a failure to capture information about latent conditions in traditional QA processes or reluctance of anesthesia providers to report them. Training of clinicians to recognize and report mutable contributory factors may be beneficial.

Methodologic Limitations

To some extent, CONES, like traditional event self-reporting, is vulnerable to individual practice patterns, behaviors, and cognitive biases. It can be difficult for clinicians who are not trained in event recognition to identify and analyze evolving event situations. Individual clinicians show tremendous variability in reporting styles.21,26Explicit training of clinicians and support staff in NRE recognition and description may be useful. The clinical reviewers in our study were authors of this article and therefore represent a potential analytical bias.

Because both approaches are retrospective in nature, they can be influenced by both outcome bias—“the influence of outcome knowledge on evaluations of decision quality”27—and hindsight bias—“the tendency for people with outcome knowledge to exaggerate the extent to which they would have predicted the event beforehand.”9,27,28QA data, which tend to emphasize more serious adverse events, are particularly prone to these biases. In addition, QA event submission often involves many days of follow-up, analysis, and discussion with peers and superiors, during which time conclusions may be drawn prematurely or shaped by “conventional wisdom” about what happened and why. The assessment of causality and culpability is strongly influenced by knowledge of the outcome.29,30Events with bad outcomes are more commonly judged “serious,” whereas events with minor outcomes are viewed as unimportant. However, these “minor” events represent an opportunity to identify the latent conditions that pose a threat to future patients. The nature of CONES focuses attention on process, thus providing insight on mechanisms of failure rather than the failure itself. Therefore, the effects of outcome and hindsight bias may be minimized.

The reduced effort required for clinicians to report events with CONES, and the fact that data collection occurs immediately after the event, may similarly reduce recall bias (the human tendency to only remember unique or novel events) and reporting bias (the reporting or documentation of such events).31CONES, however, can be affected by postevent cognitive shaping (e.g. , rationalization, justification).18,32For example, the number and type of NREs reported could be decreased if providers rationalized some types of events as “routine.” An NRE for one clinician may be more routine for another. For example, a provider who had not been properly trained in the use of a medical device may frequently have trouble using that device but might be less likely to report events related to this misuse as an NRE.

These potential biases could still be reduced further. If data were collected prospectively (e.g. , through videotaping or direct observation of clinical care11,33,34), there would be an independent mechanism to detect and validate NREs.12Second, the CONES surveyor could use specific probes and questions to elucidate events or practices that deviate from standards of care. Third, clinicians could be given training and reminders to increase their awareness of and willingness to report NREs. In contrast, self-reporting systems are prone to other problems, including providers failing or forgetting to report significant events or details, filling out forms prematurely (e.g. , before the end of the case), or completing forms hastily or incompletely because of production pressure. The face-to-face interactive nature of CONES has the potential to yield a larger amount of high-quality event data.

Our laboratory has videotaped hundreds of anesthesia cases,35as have Mackenzie et al.  36,37in the trauma suite, and others.38Using prospective video analysis of cases at the same institution, the incidence of NREs (approximately 35%) was remarkably similar to that found with the CONES method reported here,34,35thereby providing additional validation of the CONES approach as a reliable method of capturing clinical events. The collection, archiving, processing, and analyses of video records is complicated and resource intensive,35perhaps relegating these methods for the present to research applications. In contrast, CONES may be a practical method of everyday clinical quality improvement.

Traditional hospital QA systems tend to focus on sharp-end issues and do not adequately evaluate the potential impact of latent failure modes, including equipment, environmental, and system factors. The structure of the CONES data collection and analysis methods were designed to more readily capture these types of contributory factors. It would be possible to redesign QA systems to emphasize such factors to a greater degree. This would be a valuable direction and is supported by some early results by others with human factors- or systems-oriented morbidity and mortality conferences, for example.

A disadvantage of the CONES method is that the limited amount of data that can be collected about the nature of the event because of production pressure (i.e. , getting the next case started) precludes exhaustive analysis of individual cases. This could be obviated by follow-up, perhaps using detailed structured interviews,39but these would require substantially more resources. In addition, the CONES method does not capture any patient injuries that manifest after handover of the patient in the postanesthesia care unit. This is a serious limitation of the injury component of the current study and must be addressed in future studies (e.g. , through 30-day patient follow-up).


The purpose of this study was to establish feasibility and potential value. Although future studies will be necessary to evaluate formally the cost-effectiveness of CONES as an adjunct to traditional QA methods, several observations are warranted. The CONES surveyor in this study did not have formal medical training. If sufficient safety information is generated, this additional cost of personnel may prove to be justified. In many hospitals, existing QA personnel could be used to collect and analyze data using the CONES approach. Intermittent or random sampling seems effective and would reduce the overall cost of the technique. Although the cost of collecting the events was nominal, the cost of analysis of the captured events was significant because we had to create the infrastructure and compensate the clinician reviewers.

Unbiased analysis of clinical events by domain experts requires appreciable time. CONES data can theoretically be analyzed more quickly than traditional QA reviews, because CONES data are more circumscribed and can be more readily collated in a standardized structured format. On the other hand, in its current format, CONES data lack the rich detail of chart-review based QA narratives. However, a CONES database is potentially more amenable to quantitative analyses, and the descriptive statistics could be automated.


We describe a method of collecting data about undesirable clinical events that seems to yield new information compared with traditional QA methods. The CONES approach of active surveillance is analogous to strategies already in the public health sector, where passive self-reporting systems are widely appreciated to be unreliable and therefore active surveillance methods are commonly used to augment established processes.40 

CONES introduces a systems approach to anesthesiology quality improvement advocated by patient safety experts.29,41–43A larger volume of potentially useful data are generated, and reporting compliance is improved. The method prompts provider recall, minimizes the burden of interrupted workflow, and creates a nonjudgmental climate within which to report events by respecting provider autonomy, confidentiality, and workload. CONES emphasizes a systems perspective in data collection, analysis, and process improvement.

The typically less severe events captured with the CONES method represent valuable QA information because they (1) are much more frequent and represent a “signal” that can be detected and followed over time in response to QA interventions; (2) may be precursors to or illuminate underlying causes of/contributors to more serious future events; and (3) may have a greater overall impact on patient satisfaction and other “pay-for-performance” anesthesia indicators. In summary, CONES may be an effective complement to existing anesthesiology quality improvement programs.

The authors thank Oralia Loza, M.S. (Biostatistician, University of California, San Diego), for statistical assistance; Samuel Hughes, M.D., Ph.D. (Technician, University of California, San Diego), Amanda Lamond, B.S. (Technician, University of California, San Diego), Amy Groom, B.S. (Technician, University of California, San Diego), and Zobeida Torres, B.S. (Technician, University of California, San Diego), for data collection; Karen Ruberg (Administrative Assistant, University of California, San Diego) and Kelly Vo, B.S. (Administrative Assistant, University of California, San Diego), for technical assistance; Susana Hazelden, B.A. (Editorial Assistant, Clinical Investigation Department, Naval Medical Center San Diego, California), for editorial assistance; and all of the clinicians who participated in the research (San Diego, California).


Reason J: Human Error. Cambridge, Cambridge University Press, 1990
Cambridge University Press
Reason J: Managing the Risks of Organizational Accidents. Aldershot, United Kingdom, Ashgate, 1997
Aldershot, United Kingdom
Kohn LT, Corrigan JM, Donaldson MS: To Err is Human: Building a Safer Health System. Washington, DC, National Academy Press, 1999
Washington, DC
National Academy Press
Weinger MB, Pantiskas C, Wiklund ME, Carstensen P: Incorporating human factors in the design of medical devices. JAMA 1998; 280:1484
The Human Factors Design Process for Medical Devices. Arlington, Virginia, Association for the Advancement of Medical Instrumentation, 2001, p 38
Arlington, Virginia
Association for the Advancement of Medical Instrumentation
Thomas MP: Medication errors. Clin Pediatr 2003; 42:287–94
Rasmussen J, Pejtersen AM, Goodstein LP: Cognitive Systems Engineering. New York, John Wiley, 1994
New York
John Wiley
Woods DD, Cook RI: Perspectives on human error: Hindsight biases and local rationality, Handbook of Applied Cognition. Edited by Durso FT, Nickerson RS, Schvaneveldt RW, Dumais ST, Lindsay DS, Chi MTH. New York, John Wiley and Sons, 1999, pp 141–72Durso FT, Nickerson RS, Schvaneveldt RW, Dumais ST, Lindsay DS, Chi MTH
New York
John Wiley and Sons
Tversky A, Kahneman D: Judgment under uncertainty: Heuristics and biases. Science 1974; 185:1124–31
Barach P, Small SD: Reporting and preventing medical mishaps: Lessons from non-medical near miss reporting systems. BMJ 2000; 320:759–63
Weinger MB, Slagle J: Human factors research in anesthesia patient safety: Techniques to elucidate factors affecting clinical task performance and decision making. J Am Med Inform Assoc 2002; 9:S58–63
Weinger MB, Slagle J, Jain S, Ordonez N: Retrospective data collection and analytical techniques for patient safety studies. J Biomed Inform 2003; 36:106–19
Cooper JB, Newbower RS, Long CD, McPeek B: Preventable anesthesia mishaps: A study of human factors. Anesthesiology 1978; 49:399–406
Cooper J, Newbower R, Kitz R: An analysis of major errors and equipment failures in anesthesia management: Considerations for prevention and detection. Anesthesiology 1984; 60:34–42
Lui P, Weinger MB, Liu F, Tso A, Lin K: A continuous quality improvement program reduced perioperative adverse outcomes in a large hospital in Taiwan (abstract). Anesthesiology 2004; 101:A1278
Brunner EA: Analysis of anesthetic mishaps: The National Association of Insurance Commissioners’ closed claim study. Int Anesthesiol Clin 1984; 22:17–30
Cooper JB, Cullen DJ, Nemeskal R, Hoaglin DC, Gevirtz CC, Csete M, Venable C: Effects of information feedback and pulse oximetry on the incidence of anesthesia complications. Anesthesiology 1987; 67:686–94
Buckhout R: Eyewitness testimony. Scientific American 1974, pp 23–32
Scientific American
Bazerman MH, Neal MA: Negotiating Rationally. Chapters 2–8. New York, The Free Press, 1992
New York
The Free Press
Lawton R, Parker D: Barriers to incident reporting in a healthcare system. Qual Saf Health Care 2002; 11:15–8
Cullen DJ, Bates DW, Small SD, Cooper JB, Nemeskal AR, Leape LL: The incident reporting system does not detect adverse drug events: A problem for quality improvement. Jt Comm J Qual Improv 1995; 21:541–8
Firth-Cozens J: Barriers to incident reporting. Qual Saf Health Care 2002; 11:7
Buist M, Bernard S, Nguyen TV, Moore G, Anderson J: Association between clinically abnormal observations and subsequent in-hospital mortality: A prospective study. Resuscitation 2004; 62:137–41
Madjlessi-Simon T, Mary-Krause M, Fillette F, Lechat P, Jaillon P: Persistent transient myocardial ischemia despite beta-adrenergic blockade predicts a higher risk of adverse cardiac events in patients with coronary artery disease. J Am Coll Cardiol 1996; 27:1586–91
Pelter MM, Adams MG, Drew BJ: Transient myocardial ischemia is an independent predictor of adverse in-hospital outcomes in patients with acute coronary syndromes treated in the telemetry unit. Heart Lung 2003; 32:71–8
Elder NC, Vonder Meulen M, Cassedy A: The identification of medical errors by family physicians during outpatient visits. Ann Fam Med 2004; 2:125–9
Henriksen K, Kaplan H: Hindsight bias, outcome knowledge and adaptive learning. Qual Saf Health Care 2003; 12 (suppl 2):ii46–50
Bradfield A, Wells GL: Not the same old hindsight bias: Outcome information distorts a broad range of retrospective judgments. Mem Cogn 2005; 33:120–30
Dekker S: The Field Guide to Human Error Investigations. Aldershot, England, Ashgate Publishing, 2002
Aldershot, England
Ashgate Publishing
Caplan RA, Posner K, Cheney FW: Effect of outcome on physician judgments of appropriateness of care. JAMA 1991; 265:1957–60
Gorin AA, Stone AA: Recall biases and cognitive errors in retrospective self-reports: A call for momentary assessments, Handbook of Health Psychology. Edited by Baum A, Revenson T, Singer J. Mahwah, New Jersey, Lawrence Erlbaum Associates, 2001, pp 405–13Baum A, Revenson T, Singer J
Mahwah, New Jersey
Lawrence Erlbaum Associates
Hastorf A, Cantril H: They saw a game: A case study. J Abnorm Soc Psychol 1954; 49:129–34
Fraind DB, Slagle JM, Tubbesing V, Hughes S, Weinger MB: Reengineering intravenous drug and fluid administration processes in the operating room: Step one: Task analysis of existing processes. Anesthesiology 2002; 97:139–47
Weinger MB, Slagle J, Reddy S: Multiple measures of anesthesia workload during teaching and nonteaching cases. Anesth Analg 2004; 98:1419–25
Weinger MB, Gonzales DC, Slagle J, Syeed M: Video capture of clinical care to enhance patient safety: The nuts and bolts. Qual Safe Health Care 2004; 13:136–44
Mackenzie C, Xiao Y, Hu PF, Seagull FJ, Hammond C, Bochicchio G, Chiu W, O’Connor J, Gerber-Smith L, Dutton R: Video clips as a data source for safety performance. Proc Hum Factors Ergo Soc 2002; 46:1414–7
Mackenzie CF, Jefferies NJ, Hunter WA, Bernhard WN, Xiao Y: Comparison of self-reporting of deficiencies in airway management with video analyses of actual performance. LOTAS Group. Hum Factors 1996; 38:623–35
Michaelson M, Levi L: Videotaping in the admitting area: A most useful tool for quality improvement of the trauma care. Eur J Emerg Med 1997; 4:94–6
Klein GA, Calderwood R, MacGregor D: Critical decision method for eliciting knowledge. IEEE Trans Sys Man Cyber 1989; 19:462–76
Teutsch S, Churchill R: Principles and Practice of Public Health Surveillance, 2nd edition. Oxford, England, Oxford University Press, 2000
Oxford, England
Oxford University Press
Rasmussen J: The concept of human error: Is it useful for the design of safe systems in healthcare? Safety in Medicine. Edited by Vincent C, De Mol B. Amsterdam, Permagon, 2000, pp 31–47Vincent C, De Mol B
Woods DD, Cook RI: Nine steps to move forward from error. Cognition Technol Work 2002; 4:137–44
Cook RI, Render M, Woods DD: Gaps in the continuity of care and progress on patient safety. BMJ 2000; 320:791–4
Rasmussen J: Skills, rules, and knowledge: Signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Sys Man Cyber 1983; SMC-13:257–66
Weinger MB, Vora S, Herndon CN, Howard SK, Smith BE, Mazzei WJ, Rosekind MR, Gaba DM: Evaluation of the effects of fatigue and sleepiness on clinical performance in on-call anesthesia residents during actual night time cases and in simulated cases, Proceedings of Enhancing Patient Safety and Reducing Errors in Health Care. Chicago, National Patient Safety Foundation, 1999, pp 306–10
National Patient Safety Foundation
Weinger MB: Human-use medical device interactions in the anesthesia work environment, Human Automation, Interaction: Research and Practice. Edited by Mouloua MKJ. Mahwah, New Jersey, Lawrence Erlbaum, 1997, pp 233–48Mouloua MKJ
Mahwah, New Jersey
Lawrence Erlbaum
Cook RI, Woods DD: Adapting to new technology in the operating room. Hum Factors 1996; 38:593–613
Vicente KJ: Less is (sometimes) more in cognitive engineering: The role of automation technology in improving patient safety. Qual Saf Health Care 2003; 12:291–4
Woods DD, Cook RI, Billings CE: The impact of technology on physician cognition and performance. J Clin Monit 1995; 11:5–8
Weinger M, Englund C: Ergonomic and human factors affecting anesthetic vigilance and monitoring performance in the operating room environment. Anesthesiology 1990; 73:995–1021
Lundstrom T, Pugliese G, Bartley J, Cox J, Guither C: Organizational and environmental factors that affect worker health and safety and patient outcomes. Am J Infect Control 2002; 30:93–106
Perry SJ: Profiles in patient safety: Organizational barriers to patient safety. Acad Emerg Med 2002; 9:848–50

Appendix 1: Nonroutine Event Questionnaire

  1. If you were to imagine the perfect case involving this particular patient, were there any deviations from ideal?

  2. Did you make any unanticipated or unplanned actions or interventions?

  3. Were any unusual requests or demands made on you?

  4. Was there anything distinctive about other peoples’ actions or interactions with you?

  5. Did anything occur that affected your workload or vigilance?

  6. Did anything occur that was annoying or distracting?

  7. Did your tools and equipment perform as expected?

  8. Did the patient do anything that was out of the ordinary or notable?

  9. Did the patient respond as expected to your treatment and interventions?

If any nonroutine events occurred during the case, the following additional questions were asked:

  • 10. Please briefly describe the event.

  • 11. What were your initial clues that something was atypical?

  • 12. Why did it occur?

  • 13. How did you respond (if you did)?

  • 14. What factors, issues, or other events influenced your response?

  • 15. What was the result of your intervention?

  • 16. Was your attending present when the event happened? YES NO

  • 17. If not, was your attending made aware that the event occurred? YES NO

  • 18. Was there any patient injury? YES NO

  • 19. Was the injury (injuries) preventable? YES NO

  • 20. What could have been done to prevent injury?

  • 21. If there was no patient injury, estimate the percent likelihood that the event, if not managed correctly, could have led to patient injury (0–100%).

  • 22. Did the nonroutine event affect the patient’s perioperative course in any significant way? That is, did it result in, for example, case cancellation, or a change in postoperative management or patient disposition (intensive care unit instead of ward admission)?

Appendix 2: Documentation of Quality Assurance Reviews 

Appendix 2: Documentation of Quality Assurance Reviews 

Appendix 2: Documentation of Quality Assurance Reviews (continued) 

ARDS = acute respiratory distress syndrome; ASA = American Society of Anesthesiologists (physical status); CNS = central nervous system; DOU = Direct Observation Unit; ENT = ear, nose, and throat; GI = gastrointestinal; ICU = intensive care unit; MAC = monitored anesthetic care; OR = operating room; PACU = postanesthesia care unit; PostOp = postoperative; PreOp = preoperative; Spo2= oxygen saturation measured by pulse oximetry; VA = Veterans Affairs. Reprinted with permission of Anesthesia Service, VA San Diego Healthcare System, Mark Mitchell, M.D., Service Chief. 

Appendix 2: Documentation of Quality Assurance Reviews (continued) 

ARDS = acute respiratory distress syndrome; ASA = American Society of Anesthesiologists (physical status); CNS = central nervous system; DOU = Direct Observation Unit; ENT = ear, nose, and throat; GI = gastrointestinal; ICU = intensive care unit; MAC = monitored anesthetic care; OR = operating room; PACU = postanesthesia care unit; PostOp = postoperative; PreOp = preoperative; Spo2= oxygen saturation measured by pulse oximetry; VA = Veterans Affairs. Reprinted with permission of Anesthesia Service, VA San Diego Healthcare System, Mark Mitchell, M.D., Service Chief.