Performance Assessment in Airway Management Training for Nonanesthesiology Trainees: An Analysis of 4,282 Airway Procedures Performed at a Level-1 Trauma Center

ALTHOUGH the ability to evaluate and effectively manage a wide-range of airway conditions is the defining skill of an anesthesiologist, other physicians with acute care responsibilities are expected to be competent in airway management. The question then arises as to the best means of training the nonanesthesiologist to manage the airway in both the stable and unstable patient during their residency training program. The means for providing airway management training to nonanesthesiologists has taken multiple approaches. Many residencies have established formal training programs utilizing simulation-, cadaveric-, and/or didactic-based formats combined with “on the job” clinical experience.^1–4  Unfortunately, there are still barriers to training and it is often difficult to achieve a sufficient volume of clinical cases to establish competency during rotations in the primary specialty.⁵  Many of these programs have reached out beyond their specialty to associated anesthesiology departments for additional training and clinical experience. The operating room (OR) has always been considered an ideal location for clinical training in airway management. Given the large volume of procedures, varied patient population, stable conditions for teaching and availability of highly trained staff, many residency training programs have established rotations in the OR for training in airway management. Residencies utilizing this approach include programs in emergency medicine (EM), general surgery, and critical care medicine.

Although the use of an anesthesiology “airway” rotation to expand the nonanesthesiologist’s experience is commonly employed, very little data exist on the utility, clinical exposure, and outcomes derived from these programs. With an increasing emphasis being placed on outcome-based evaluations, there is a need to obtain a greater understanding of how airway skills are obtained and measured in our current training paradigm. In a joint initiative, the Accreditation Council for Graduate Medical Education and the American Board of Emergency Medicine have established the Emergency Medicine Milestone Project.⁶  In this project, airway management is recognized as one of the essential skills for which specific milestones have been created. Although the initial milestones recognize general concepts associated with safe and effective airway management, the only specific outcome target for residents in training is a requirement to accomplish a minimum of 35 intubations during their residency. As the Accreditation Council for Graduate Medical Education Milestone Project continues to evolve, the focus on developing competency-based outcome expectations will require an increased emphasis on creating monitoring programs that can demonstrate skill and knowledge obtainment beyond completion of a specified number of procedures.

To date, there is almost no information on the effectiveness or outcomes associated with anesthesiology-based, clinical airway management rotations for the nonanesthesiology trainee. The current study was undertaken to evaluate the effectiveness of an anesthesiology run airway management rotation in an academic setting that blends exposure to elective, urgent, and emergent cases. We sought to evaluate the effectiveness of the program and to assess for factors that influenced skill attainment across the range of participants. The results of this study may assist in the further refinement of our ability to assess for skill or milestone attainment in the area of airway management.

The trauma anesthesiology division at the R Adams Cowley Shock Trauma Center (STC), Baltimore, Maryland, has an established airway management rotation supporting EM residency and critical care medicine fellowship training requirements. During the period of data collection, all trainees completed either a 4-week or calendar month rotation at STC. The number of patients evaluated in the trauma resuscitation unit (TRU) during this period was approximately 7,800 per year with approximately 5,280 surgical procedures per year being performed in the ORs at STC. Clinical shifts were 12-h in duration, and each trainee completed 12 to 15 shifts during their rotation with a mix of day and night shifts. After orientation on the first day of the rotation, all subsequent training took place in the clinical environment including the TRU, ORs, and intensive care units at STC. Airway procedures taught in the program include endotracheal intubation with both direct and video laryngoscopy, placement of supraglottic airways during operative cases, performance of mask ventilation, and use of airway adjuncts. The airway algorithm and clinical outcomes covering the first 24 h of admission for the institution have been previously described by Stephens et al.⁷ 

After obtaining approval from the Institutional Review Board (University of Maryland-Baltimore, Baltimore, Maryland), prospective, deidentified data collection began in July 2010 and continued through September 2012. All trainees participating in an airway management rotation at STC were asked to self-report all airway management procedures by using a secure, password-protected, online data collection tool, the Shock Trauma Airway Registry (fig. 1). Data to be collected included date and time of procedure, patient characteristics (sex, height, and weight), indication for procedure, location, and attempt-specific details (primary performer of procedure, technique, device, use of manual in-line stabilization/cricoid pressure/mask ventilation before procedure attempt, and success for each attempt). The categories used to describe technique included oral rapid sequence induction, oral with sedative (i.e., standard induction), and oral with no medication before attempts. These categories were included to allow for comparisons with another large self-reported airway management database.⁸  During the period of data collection, all attempts using video laryngoscopy were accomplished using the GlideScope^® (Verathon Medical, Bothell, WA), and the term “glidescope” is used to indicate its use in the following sections. An attempt was defined as any interruption of a procedure required to resume bag-mask-valve ventilation, a change in devices (i.e., change from direct laryngoscopy to videolaryngoscopy), or a change in primary airway manager (i.e., from trainee to attending). For the purposes of this study, adjustment in technique such as reinserting the blade during initial positioning—even when removing it to try another angle—was not considered a new attempt. Similarly, inserting an intubating guide such as a bougie after already beginning the initial attempt without one was not considered a new attempt. For placement of a laryngeal mask airway (LMA), an attempt was defined as insertion into the oral cavity and advancement of the device. In addition, the trainee was asked to provide the best laryngoscopic grade of view they observed during for both direct and video laryngoscopy by using the Cormack–Lehane classification⁹  as well as stating whether or not they personally felt it was a “challenging” intubation.

All trainees were also asked to complete a prerotation survey to include year of training, specialty, prior number of intubations, and previous airway training. In addition, they were asked to rate their comfort level with a number of airway procedures on a 5-point Likert scale. A 3-point scale was used to evaluate self-perceived confidence (1-do not feel confident, 2-confident, and 3-very confident) and competence (1-need to improve, 2-competent, and 3 expert) with airway management. These elements were reassessed in a postrotation survey.

Descriptive statistics were used to describe the cohort. Appropriate parametric and nonparametric tests were applied to describe differences between groups for continuous data; chi-square or Fisher exact tests were used to describe differences between groups of categorical data such as laryngoscopic grade of view. A multivariable logistic regression model, with robust standard error estimates, was constructed to model the outcome of first-attempt intubation success. First, variable inflation factors were calculated to assess multicollinearity among predictors. Predictors with a variable inflation factors greater than 10 were removed from the model. A data-based model was then derived, selecting the remaining predictors based upon stepwise selection by Akaike’s Information Criteria. Two new models were then constructed based on variables thought to be theoretically associated with first-pass endotracheal intubation success. These two multivariate logistic regression models employed general estimating equations and a clustered analysis to control for the effect of clustering by individual trainee. All three models were compared, and results were found to be similar. The final model, which was a logistic regression model controlling for clustering using the cluster-analysis routine available in Stata, included the following covariates: total airway procedures, rotation week, patient age and weight, year of training (i.e., postgraduate training year), whether or not a previous airway course was attended, and indication for intubation. The final model was chosen based upon the ability to perform reliable regression diagnostics and the close similarity in results when compared with both the general estimating equations model derived by Akaike’s Information Criteria. Effect modification was assessed for the categorical variables of total intubations and rotation week; no statistically significant evidence for effect modification was found. The regression model was checked with a wide variety of diagnostics (e.g., hat matrixes for leverage, Pearson residual plots, box plots for influence) to confirm that the assumptions for the regression model were satisfied. The model was found to fit the data adequately (Pearson goodness of fit test; P = 0.39). A bootstrapped logistic regression was performed and compared with the logistic regression model to examine the robustness of the CIs and standard errors. A P value of less than 0.05 was considered statistically significant. All analyses were performed by using Stata Version 12.1 (Stata Corp, College Station, TX).

A total of 4,571 airway procedures by 96 airway trainees were abstracted from the Shock Trauma Airway Registry over a 27-month period (fig. 2). Ten trainees failed to complete a prerotation survey and an additional trainee failed to log at least 20 procedures. These trainees and their procedures (n = 222, 4.9% of total procedures) were excluded from analysis. Incomplete data entry, including failure to identify a successful attempt, resulted in the loss of 54 procedures (1.1%). An additional 13 procedures (0.3%) were excluded due to the failure to identify the trainee as making the initial airway attempt.

A total of 4,282 airway procedures recorded by 86 trainees were included in the analysis. The median number of procedures performed was 50.4 ± 13.2 (range, 20 to 93; 25th quartile = 41; 75th quartile = 57). The majority of procedures occurred in the OR (n = 3,057; 71%); 1,096 procedures took place in the TRU (26%) and 113 in the intensive care unit (3%). Sixteen procedure locations were not logged and were recorded as unknown (0.4%). The distribution of endotracheal intubation attempts, until successful intubation, is shown in figure 3. A maximum of five intubation attempts occurred on three occasions. There were two reported surgical airways (0.05%). One was after a successful intubation with a GlideScope (Verathon Medical, Burnaby, British Columbia, Canada) video laryngoscope due to an anterior neck defect related to the initial trauma. The second was associated with the presence of a previously undiagnosed tracheal stenosis from a previous tracheostomy even though an adequate laryngeal view was obtained. In this case, one intubation attempt was made by the airway rotator, followed by one attempt by the attending before a bedside tracheostomy was performed.

Table 1 represents demographics and first-pass success rates for trainees. Year of training, prior attendance at an airway course, and trainee type were not found to be statistically significant in terms of first-attempt success by the end of the rotation. EM residents had a statistically significant higher rate of success from week 4 compared with week 1, whereas pulmonary critical care fellows, who had a lower first-attempt success rate in week 1 (81.2%), did not. Trainees with a total of 1 to 25 previous intubation attempts had a statistically significant higher week 4 versus week 1 success rate (82.5 vs. 88.8%; P = 0.005).

The distribution of techniques, devices, and indications for airway procedures is described in table 2. The most common indication for an airway procedure was the need for surgery (n = 3,128; 71.5%), followed by altered mental status (n = 196; 4.6%) and combativeness (n = 162; 3.8%). The overall first-attempt success rate for all airway procedures was 89.3%. Oral intubation with rapid sequence intubation had a higher first-attempt success rate versus oral intubation without medications (P = 0.009). There were no significant differences in success rates between other techniques versus oral intubation with no medications (P = 0.95). Oral intubation with a sedative was associated with a higher first-attempt success rate (89.7%) compared with other techniques (P = 0.004). In terms of airway devices, no differences were found when direct laryngoscopy was compared with video laryngoscopy (P = 0.96) or when direct laryngoscopy was compared with use of an LMA (P = 0.14). Video laryngoscopy was associated with a higher first-attempt success rate compared with direct laryngoscopy with a bougie (P = 0.03). There were no differences between direct laryngoscopy versus direct laryngoscopy with a bougie, and no differences between direct laryngoscopy versus LMA. There were too few numbers for the other techniques (e.g., intubating LMA, other supraglottic devices, fiberoptic intubation) to properly identify statistically significant differences and they are aggregately reported as “Other” in table 2. 

First and subsequent attempt pass success rates were analyzed by the number of airway attempts per rotation week (fig. 4). As trainees progressed through the rotation, by week 4, success rates on the first attempt increased significantly from 85% in week 1 to 94% by week 4 (P < 0.001). Subsequently, the number of second airway attempts decreased over the course of the rotation from 13% in the first week to 5% by the last week of the rotation (P < 0.001). When stratified by procedure location, trainees had a higher first-attempt intubation success rate in the OR compared with the TRU (P = 0.001; fig. 5). There were no differences between first-attempt successes in the intensive care unit versus the TRU (P = 0.10). When first-attempt success rate was examined by both location and week of rotation in the OR, first-attempt success rate increased statistically significantly over the course of the rotation (86 vs. 93%; P = 0.001). In the TRU, intubation success also significantly improved over the course of the rotation from 81% in week 1 to 95% by the end of the rotation (P = 0.006). There were an insufficient number of airway interventions to draw any conclusions in the intensive care unit.

Laryngoscopic grade of view for all attempts by using direct laryngoscopy as the initial attempt improved significantly over the course of the rotation, with an increase in grade 1 views from 61 to 74% (P = 0.015), a decrease in grade 2 views from 27 to 17% (P = 0.001), and a decrease in grade 4 views from 2.5 to 0.5% (P = 0.004; fig. 6). The proportion of grade 3 views did not change over the course of the rotation.

The results of the univariate and multivariate analyses are presented in table 3. Trainees who had performed more airway procedures in the past had statistically significant greater adjusted odds of a higher first-attempt success rate compared with those who had lower numbers of procedures. The adjusted odds of first-attempt successes increased over the course of the rotation (odds ratio, 1.23; 95% CI, 1.32 to 1.61; P < 0.0001). For every increase in weight by 1 kg or every increase in age by 1 yr, the odds of first-attempt success rate statistically decreased, when adjusting for the other variables included in the model. A statistically significant lower odds of successful first-attempt success were found for patients classified as requiring intubation for combativeness, face/neck injuries, or traumatic arrest. Respondents reported a lower self-perception of “challenging” intubations by the end of the rotation (32.8% in week 1 vs. 24.1% in week 4; P < 0.001; fig. 7).

Postrotation survey data were available for 56 trainees (66%). Both self-perceived confidence (P = 0.002) and competence (P = 0.0025) increased significantly as assessed by a 3-point Likert scale. Statistically significant (P < 0.01) changes were observed in self-perceived ability to perform direct laryngoscopy, video laryngoscopy, intubation through an LMA, placement of an LMA, and use of an intubating bougie (fig. 8).

By using a prospectively collected training dataset of 4,282 self-reported, airway procedures at an academic, level-1 trauma center during a 27-month period, we observed an improvement in the self-reported first-attempt success rate of nonanesthesiology trainees over the course of a 4-week airway management rotation. In addition, the percentage of cases with a self-reported laryngoscopic grade 1 view increased significantly from 62 to 74% during that same time period—most likely as a result of skill refinement. Logistic regression analysis modeling of first-attempt success rate identified three independent predictors of success: rotation week, number of previous intubation attempts reported before the rotation, and nonemergent indication for airway management (scheduled OR case). In the same analysis, a more emergent indication for intubation (trauma arrest, face/neck injury, overdose, and combative), older age, and obesity were found to be predictors of first-attempt failure by the trainee. Year of training, completion of a formal airway training course, and core specialty did not influence first-attempt success.

Our observed first-attempt and overall success rates of 89.3 and 93.1%, respectively, for nonanesthesiology trainees across the entire rotation are comparable with that reported in other studies of EM resident performance in the emergency department.^8,10–14  When comparing the first-week rate to the that observed during the final week, a significant increase was noted from 85 (1,124 of 1,328) to 94% (618 of 659). First-attempt success rates in previous studies ranged from 74 to 86% which is significantly below the 94% first-attempt success rate observed during the final week of the rotation. Even when looking at only those intubations performed in the TRU, the final week first-attempt success rates of 95% reflect a high degree of success which is similar to the 93% observed in the OR cases for the same time period. Overall, trainees supervised in an anesthesiology-based airway management training program incorporating both emergency and elective cases demonstrated excellent self-reported performance outcomes with significant improvement over a 4-week period.

To our knowledge, this is the first performance assessment of training outcomes from an anesthesiology-based airway management rotation for nonanesthesiologists. With a shifting focus in graduate medical education to measurement and reporting of outcomes through educational milestones,¹⁵  residency training programs need to create systems that are capable of monitoring progress and attainment of these skills. The Accreditation Council for Graduate Medical Education and the American Board of Emergency Medicine have set the minimum number of intubations at 35 without specific outcome criteria suggesting that competency can be achieved at this level. A comprehensive review of competency, learning curves, and assessment methodology is beyond the scope of this discussion, but has been addressed elsewhere.¹⁶  Although a specific number of intubations has not been clearly established, at least one study using cumulative summation analysis for first-pass success rate determined that individual competency was difficult to establish without significantly more than 35 intubations.¹⁷ 

Resources and time constraints are commonly cited as limitations to obtaining analysis of procedural details and most programs already rely on self-reporting to obtain information regarding the quantity and type of procedures performed by their trainees through the use of case logs. Use of an observational database within a training program such as the Shock Trauma Airway Registry can be employed to track outcome-based performance measures. Observational databases have been used previously to evaluate airway training measures outside the OR in a retrospective manner. The National Emergency Airway Registry II was used to produce the largest published series of intubation data by trainees in the United States with the most recent analysis looking at 5,768 intubations from 31 sites over a 57-month reporting period (1997–2002).⁸  More recently, the National Emergency Airway Registry for Children reported on 1,218 intubations by trainees collected from 15 pediatric intensive care units over a 17-month period.¹⁸  In both of these large series, the authors report on aggregate outcomes for first-attempt success rate and overall success rate for different levels of trainees. Unfortunately, they neither did assess longitudinal outcomes for assessment of skill attainment over time nor did look at trainee-specific factors such as cumulative airway management experience and case mix. Nonetheless, observations from these datasets suggest that there is still room for improvement in both groups. Sagarin et al.⁸  found that first and overall success rate increased with increasing years of training with a first-attempt success rate between 82 and 88% at the third postgraduate year or higher. Overall success with multiple attempts was very good at 93 to 94%. Analysis of the National Emergency Airway Registry for Children data was not as encouraging for trainee performance with only a 37% first-attempt success rate for pediatric residents and 70% for pediatric fellows suggesting need for additional training.¹⁹ 

What factors may contribute to the observed success rate and ability to improve performance over the course of our program? An experienced trauma anesthesiologist is present at all airway procedures performed by the trainees in our program, and their presence may be beneficial due to their cumulative experience with this patient population. Many EM training programs use a more senior trainee as the immediate back-up for airway management by a junior trainee. In the National Emergency Airway Registry II dataset, 50% of rescue attempts (change in provider) were done by another trainee compared with 16% (32 of 196) in our study with the remaining rescue attempts taken by the attending anesthesiologist or a certified registered nurse anesthetist. In the National Emergency Airway Registry II series of intubations by EM residents, a failed first attempt followed by a second attempt by the same resident had a success rate of 67%⁸  compared with our observed rate of 92% (152 of 165). The factors leading to our trainees’ high success rate could include: recognition of a difficult airway or low likelihood of success during the first attempt leading to the attending taking over earlier, effective coaching of the trainee, and immediate change in device based on attending observation of first attempt.

An additional factor leading to the higher success rates in our study could be the large number of procedures performed in the OR. Intubations in the emergency department or TRU are more likely to be emergent in nature with less time for preoxygenation and optimization of intubating conditions. Although the initial first-attempt success rate was lowest in the TRU at the end of the first week, there was no discernible difference in success rate by the end of the rotation. Finally, allowing attempts at inserting the airway device to be classified as a first attempt may have contributed to the higher reported success rates.

Techniques for the accurate assessment of proficiency remain a priority for assessing the core clinical competencies.²⁰  In future studies examining trainee proficiency, advanced methods for assessing technical proficiency will be required. One of these methods is cumulative summation analysis.^21–23  Although not reported in this work, efforts to use cumulative summation analysis are underway at our institution, as this may prove to be a valuable method for assessing clinical competency by the end of our airway rotation. Additional projects include expansion of the Shock Trauma Airway Registry online data collection tool to programs currently sending residents to STC for airway management training. To better assess the effectiveness of the program, it would be desirable to monitor pre- and postrotation performance.

There are inherent limitations to this type of studies. First, as an observational study, there were no mandatory airway management protocols other than the general practice of our department allowing for significant practice variation during nonemergent cases. It is also possible that more experienced trainees were allowed to attempt more difficult procedures resulting in a drop in measured success rate. This may have led to an alteration in the learning curve demonstrated in this study.

In addition, we did not perform an audit of the prospectively collected data, and all data entry is self-reported. This could have led to a reporting bias through a trainee’s desire to report more success in the later weeks of training to demonstrate procedural improvement and benefit from the training. We did not perform a retrospective analysis of patient medical records to confirm the dataset, complete missing data, or detect patients intubated by the trainees but not entered in the system. This retrospective analysis would strengthen the study but is not feasible. In addition, adverse events are not reported in the study. Although immediate events were requested during data entry, the limited ability for follow-up would lead to a significant underestimation. Future studies would be strengthened by the addition of this element.

Finally, the use of a self-reported, laryngoscopic view by using the Cormack–Lehane classification as an outcome marker has several potential confounders. First, there may be a poor baseline understanding of the classification system which has been shown to have a poor intraobserver reliability and only fair interobserver reliability.²⁴  In addition, continued practice and teaching during the rotation may have improved trainee recognition resulting in improved laryngoscopic view recognition not dependent on skill acquisition.

Despite these limitations, our study offers insight into airway management training of nonanesthesiologists. An anesthesiology-based airway management rotation is able to improve self-reported and perceived performance for both first-attempt success and laryngoscopic visualization. Within a 4-week period after a median of 50 airway procedures, the majority of EM residents and critical care medicine fellows reported the attainment of a greater than 90% first-attempt success rate. Prior experience improved their overall performance, but their primary specialty did not influence success. Trainee performance at the beginning of the rotation is close to published first-attempt success rates; however, they consistently exceed these rates by the end of the rotation. Although there are multiple areas of potential bias present in self-reported outcomes similar to those in this study, the results can serve as a starting point for discussing the role of outcome tracking in airway training programs. The need to establish educational milestones to measure the transfer of skills and knowledge to our trainees will continue to challenge all of training programs.