Process improvement in healthcare delivery settings can be difficult, even when there is consensus among clinicians about a clinical practice or desired outcome. Airway management is a medical intervention fundamental to the delivery of anesthesia care. Like other medical interventions, a detailed description of the management methods should be documented. Despite this expectation, airway documentation is often insufficient. The authors hypothesized that formal adoption of process improvement methods could be used to increase the rate of “complete” airway management documentation.
The authors defined a set of criteria as a local practice standard of “complete” airway management documentation. The authors then employed selected process improvement methodologies over 13 months in three iterative and escalating phases to increase the percentage of records with complete documentation. The criteria were applied retrospectively to determine the baseline frequency of complete records, and prospectively to measure the impact of process improvements efforts over the three phases of implementation.
Immediately before the initial intervention, a retrospective review of 23,011 general anesthesia cases over 6 months showed that 13.2% of patient records included complete documentation. At the conclusion of the 13-month improvement effort, documentation improved to a completion rate of 91.6% (P < 0.0001). During the subsequent 21 months, the completion rate was sustained at an average of 90.7% (SD, 0.9%) across 82,571 general anesthetic records.
Systematic application of process improvement methodologies can improve airway documentation and may be similarly effective in improving other areas of anesthesia clinical practice.
Detailed documentation of airway management during anesthesia induction is mandatory but in reality it is insufficient
Process improvement science provides strategies and methods increasingly applied in health care with the aim to effectively decrease “evidence-practice” and “knowing-doing” gaps
At the Massachusetts General Hospital, a combination of iterative improvements through Plan-Do-Study-Act cycles, standardization of work flow, data sharing, and additional change management strategies improved the frequency of “complete” airway management documentation from a baseline of 13.2% to over 90% of records in 13 months
Systematic study and application of formal process improvement strategies may similarly enable improvement in other institutions and in other areas of anesthesia practice
THE field of process improvement helps us close the “evidence-practice gap” where practitioners know what to do, but fail to do it.1 Having a good solution to a problem is necessary but not sufficient in solving that problem. In a study by General Electric Corporation, 100% of practice changes evaluated as “successful” were found to have a good technical solution, although over 98% of “unsuccessful” changes also had a good technical solution.* Process improvement methods help identify solutions to problems and, more importantly, implement them successfully.
Airway management is a medical intervention at the core of anesthesia care.2–9 Like other medical interventions, a detailed description of the management methods should be documented. A previous survey of anesthesia records identified only partial completion of selected airway management data entry items.10 This represents a loss of valuable information when patients present for future anesthetics, as subsequent anesthesia teams are unable to benefit from knowledge of previously successful airway management techniques. This disadvantage is magnified for patients with difficult airways.11–13 These observations create a major impetus for improving documentation practices. Our institution therefore looked to process improvement methods to address the deficiency in airway management documentation and hypothesized that formal implementation of these tools could improve our airway documentation practices.
One of the challenges of conducting process improvement is the lack of information regarding the optimal tool or strategy to use in medical environments. In this report, we describe the implementation of a process improvement program and subsequent increase in the frequency of “complete” airway management documentation for patients receiving general anesthesia. We describe each of our many interventions and the corresponding process improvement theory or tool we used to achieve behavioral change. Our primary goal was to improve airway documentation, especially for patients with difficult airways, so that future caregivers can benefit from the information and be alerted to any special conditions. We also wanted to evaluate which process improvement techniques were effective in our department to learn how to more reliably approach future challenges in the delivery of anesthesia care that might also require systematic practice change.
We tested the hypothesis that a combination of process improvement methodologies, including iterative improvements through Plan-Do-Study-Act (PDSA) cycles,14,15 standardization of work flow, data sharing, and additional change management strategies could improve the frequency of “complete” airway management documentation. These methods increased the frequency of complete documentation from a baseline of 13.2% to over 90% of records.
Materials and Methods
This project was completed under approval of the Institutional Review Board at Massachusetts General Hospital, Boston, Massachusetts, for the purpose of this publication; however, the activities described herein would have otherwise met Institutional Review Board exemption criteria, including requirements regarding written/informed consent, given that this is was a process improvement effort aimed at elevating quality of care rather than traditional clinical research.16 A group of seven senior anesthesiologists used a consensus-driven process to initially define a set of necessary elements for “complete” airway documentation. These criteria were disseminated to all anesthesiologists in our department and further refined with their feedback. The final criteria for complete airway management documentation were incorporated into our electronic anesthesia record in the form of lists and checkboxes. The criteria included (1) specification of airway management methods (e.g., spontaneous ventilation, mask, laryngeal mask airway, or endotracheal tube), (2) a description of ease of bag-mask ventilation, if attempted, and notation of any mask adjuncts used, (3) details of tracheal tube insertion (i.e., method, instrument, or adjunct), if attempted, and (4) if intubation was attempted, notation of whether the intubation was difficult or atraumatic (fig. 1). If “difficult intubation” was checked as “yes,” a text display with “Please confirm with attending” would appear below the field (fig. 2) and an automated e-mail with follow-up instructions would be sent to the supervising anesthesiologist. Clinicians were asked to practice concurrent documentation and complete all record entries before the end of the case, but had up to 2 weeks to edit the record before it became permanently archived. Efforts to increase adoption of these criteria were applied in three phases using PDSA continuous improvement cycles that allowed for testing and refining of improvement strategies (table 1).14,15
Phase I/PDSA Cycle 1
Baseline compliance data were retrospectively captured based on the criteria for “complete” airway documentation. A new airway documentation form with a more intuitive layout that reflected clinician workflow was custom-created for our anesthesia information management system (MetaVision; iMDSoft, Dedham, MA) (fig. 3). One field, a designation of “yes” or “no” for “difficult intubation,” was forced (i.e., users could not proceed without completing it). Attending anesthesiologists were asked to determine the appropriateness of the “difficult intubation” designation based on education provided of existing definitions in the literature11,17–26 and clinical judgment. Department-wide education included in-person training (including lunch talks near the operating rooms to increase participation), creation of a conceptual flow chart (fig. 1), and Web-based video instruction on how to use the new form and successfully document all the new required criteria. Clinician feedback and general attitudes toward the new airway documentation criteria were also collected over the training period both verbally during operating room education sessions and via e-mail in response to written communications. Aggregate completion rates were shared with department clinicians in weekly e-mail updates. Finally, the project was linked to a hospital Quality Incentive Program that would award all attending-level physicians with a small payment if at least 50% of the department’s records of general anesthetics contained complete airway documentation at the end of a 4-week measurement period.
Phase II/PDSA Cycle 2
The 18 weeks after the end of the phase I implementation (Do) were used to review (Study) and respond to (Act) clinician feedback and develop a subsequent plan (Plan) for phase II. Two information technology tools were built into the anesthesia information management system to provide clinicians with real-time airway documentation guides. One tool highlighted empty fields that required data input. The second tool was a real-time user query that reported whether airway documentation was complete. Education efforts continued with updated video and in-person trainings and clinicians were asked for additional feedback. The hospital Quality Improvement Program also offered a second financial incentive if clinicians collectively achieved a new target of 75% of records with complete airway documentation at the end of a 9-week measurement period.
Phase III/PDSA Cycle 3
New clinician feedback from phase II was reviewed and used to “Plan” the next iteration of behavioral improvement over the 19 weeks leading up to the next “Do” step of phase III. Process improvement leaders shared confidential reports on a monthly basis via an electronic dashboard. These reports showed individual performance compared to the department aggregate. Records marked as incomplete specifically told clinicians what needed to be added to be considered complete. The hospital Quality Improvement Program offered a third and final incentive payment if clinicians collectively included complete airway documentation in at least 90% of records. Following phase III, passive interventions (anesthesia information management system documentation decision support and ability to review monthly completion reports) were kept in place whereas active interventions (training, communication, and financial incentives) were discontinued.
Compliance data were summarized by mean on monthly or weekly intervals during each stage. We tracked performance during the project using a P-Statistical Process Control Chart. We set control limits at 3 SDs from the mean assuming a binomial distribution. In addition, a generalized linear model was used to assess the linear trend of compliance rate over the progression of each phase. Statistical significance was declared at P value less than 0.05. Statistical analyses were performed using SAS(R) (version 9.3, Cary, NC), Excel(c) and Access(c) (Microsoft Corporation, Redmond, WA).
In phase I, a retrospective review of baseline airway documentation revealed that 13.2% of 23,011 general anesthetic records met all criteria for the new definition of complete airway documentation (see first paragraph in Materials and Methods). A total of 49.8% of records included partial airway documentation, and 37.0% did not have any documentation of airway management. Expected but missing data elements were counted for each incomplete record. The most frequent reason for incomplete documentation was failure to note whether intubation was difficult (fig. 4).
At the end of phase I, the rate of complete documentation reached 66.3%; more than a fivefold increase over baseline. Completion rates increased to a mean of 84.4% over phase II and 90.0% over phase III (fig. 5). A linear trend test demonstrated improvement in each of the three phases of implementation (P < 0.0001). In the 21 months that followed after the conclusion of phase III, the airway documentation completion rate was sustained at an average of 90.7% (SD, 0.9%) across 82,571 general anesthetic records.
Airway management is fundamental to the delivery of anesthesia care. Despite this widespread appreciation, our baseline airway documentation completion rate of 13.2% indicates that behavioral changes were necessary. The goal of this work was to test the use of process improvement as a means to improve documentation of airway management details. In the 13 months of this project, the percent of records that met our local criteria for “complete” airway documentation increased to a sustained rate of over 90%. This success was achieved with a multipronged approach that included three distinct phases and the application of process improvement principles (tables 1 and 2).
Each phase, or PDSA cycle, required a different set of process improvement and change management strategies tailored to the specific state of the project and shifting perceptions, behaviors, and attitudes of the clinicians. For example, initial feedback indicated that the majority of clinicians supported the idea of better documentation, possibly because baseline documentation completion rates were very low. This allowed us to focus phase I primarily on simplifying work flow with a new documentation form and providing education on the criteria, as opposed to building buy-in, which is often otherwise required in the early phases of project implementation.
Developing a set of criteria before phase I provided an explicit shared goal, which is a critical starting point. In addition, having defined criteria also allowed us to decrease the variability in anesthesiologist reports that occurred due to their individualized practices. It is essential to recognize that this effort did not aim to change airway management practices; the aim was to improve documentation of the procedure to manage the airway.
Standardization and low variability are key to high efficiency and high quality in systems-based practices. The specific elements chosen for the criteria are less important than defining them by consensus and ensuring acceptance and knowledge of the definition. To compare results nationally, however, there should be an effort made to get a consensus regarding these criteria.
In phase I, we began deploying a new documentation form that was more intuitive and followed the workflow completed by the anesthetists. This lowered a barrier to change and helped embed the new desired practice into existing workflow, which is another critical step in successful process improvement.
In addition, we began measuring performance and sharing data with the department. This likely resulted in improvement due to the Hawthorne effect, whereby workers change performance solely due to the fact that they are being observed.27 Finally, in our phase I educational efforts, we focused on the need to document a “yes” or “no” for difficult intubation and atraumatic intubation, the management method and, if used, the quality of the bag-mask ventilation. These four elements (33% of all elements in the criteria) accounted for 79% of missing documentation at baseline. This is an example of the “Pareto Principle,” coined by Joseph M. Juran (Department of Industrial Engineering, New York University, New York, New York, 1904–2008) in the 1940s as “the vital few and trivial many.”28 Also called the “80/20 rule,” it posits that often 80% of the effects, or problems, are caused by only 20% of the causative factors.29–31 As a result of applying these principles in phase I, completion rates rose from 13.2 to 66.3% in just 4 weeks.
In studying phase I, we learned that many clinicians felt overwhelmed by the need to memorize the details of the criteria and wanted real-time feedback when documentation was complete. In response, we built decision support tools into the new documentation form as part of our phase II iterative improvements. The rate of improvement in phase II decreased from the rapid improvement in phase I. This is characteristic of the principle of “diminishing returns” which states that greater effort is required for each degree of improvement as the 100% target approaches.
The diminishing returns phenomenon was even more pronounced in phase III, which had a target completion rate of 90%. Individual performance reports were used to close the final performance gap of less than four percentage points. Sharing individual data to drive improvement has been previously demonstrated in anesthesia care.32 Feedback in phase II indicated that many clinicians incorrectly perceived their individual performance as above the departmental average. The individual reports clearly identified which clinicians needed focused communication and additional education on improving airway documentation and helped us achieve our goal of a sustained 90% completion rate.
Although the 80/20 rule leads to fast gains initially, it can also lead to discouragement as the ratio eventually flips and a large amount of effort and resources are required to close a small final gap in performance. From one perspective, we should never settle for a target that is less than 100%. From another perspective, balancing resource expenditure with modest potential benefit (achieving 100% from a level close to 100%) is a very real decision that many groups need to make in the era of increasingly limited resources and should be carefully considered.
Our combined methods allowed us to create a “high-reliability” practice33 where even as active project management, incentives, education, reminders, and all other activities requiring a workforce were discontinued, the department sustained documentation rates at peak performance.
It is possible that we could have reached our 90% goal more quickly if we had not allowed the anesthesiologist to exit the airway documentation form without completing all the necessary fields. However, with all process improvement initiatives, it is important to balance expected benefit with expected burden of planned interventions. Ultimately, we felt the tactic of forcing fields could become a distraction or prevent clinicians from accessing the stat paging, blood ordering, and lab results screens quickly in the event of an urgent or emergent issue. The concept of forced fields was also disliked by clinicians.
The financial incentive payments for achieving our targets were part of an externally funded pay-for-performance program in which our department was obligated to participate. This aspect of our intervention was relatively de-emphasized compared to the objectives of provider preparation and patient safety, which may in part explain the sustained compliance at peak performance despite the discontinuation of financial incentives after March 2012. At the same time, this project could have potentially been completed over a shorter timeline but we were tied to the hospital pay-for-performance intervals which were administered on 6-month cycles and based on group, rather than individual, performance.
There are limitations to process improvement that should be weighed carefully in each “Plan” step of improvement efforts. The first is the limitation of metric definitions used to track performance. These definitions are typically binary or categorical to enable discrete measurement. In this example of airway documentation, we were not able to include free text comments in the automated data query for complete records. Therefore records that had thorough written descriptions but missing entries in the provided drop down boxes were counted as incomplete by the computer system. Baseline data showed that this occurred in less than 5% of cases—low enough for us to accept the trade-off of not being able to manually count and include these records in our monthly data collection. It is important to know the limitations of the plan, but not allow them to paralyze improvement efforts.
The second limitation is that process improvement focuses on process. Process metrics are strongest when designed in connection with an outcome metric, but outcomes are generally more difficult to measure and require substantially more power to discern statistical effect. Our primary objective in improving airway documentation was to increase the likelihood of having detailed information that would better prepare future anesthesia teams providing care for patients returning for surgery. This information might alter an airway management plan and, as a result, lead to fewer intubation attempts, less trauma to the airway, less desaturation and lower airway management-related mortality. The information could also be useful for emergency department and intensive care unit teams needing to secure an airway quickly or transition a patient to mechanical ventilation. Although these potential downstream benefits of detailed airway management information are a logical extrapolation, the actual impact our documentation practices had on these types of events is difficult to determine. The difficulty of measuring this impact, however, should not negate the importance of improving processes. In addition to potentially driving better outcomes, a secondary practical advantage of process measures is that they often satisfy external requirements for performance tracking, like the Joint Commission’s Ongoing Professional Practice Evaluation program or pay-for-performance initiatives driven by insurance carriers.
A third limitation is best described in a study by Davidoff where it is noted that improvement efforts are “hard to standardise, since they are most effective when adapted to the local circumstances.”34 In the “Plan” step of each phase, the interventions we ultimately chose to trial in the “Do” step were the ones that best accounted for the constraints and options specific to our institution. For example, we did not have control over the amount or manner in which the incentives were paid, but we did have the ability to customize our anesthesia information management system and therefore focused on the latter. The specific interventions chosen to improve airway management at another institution may be different depending on the unique constraints and options of each location. Although tailoring process improvement efforts to local need and culture is critical, case studies of generalizable principles may be helpful to institutions facing similar challenges and seeking similar results.
Our experience suggests that systematic application of formal process improvement strategies improves completion rates for airway management documentation in electronic anesthesia records. Applying such methods may similarly enable improvement in other areas of anesthesia practice, and future study is warranted.
The authors thank the Department Chair, Jeanine Wiener-Kronish, M.D., Department of Anesthesia, Critical Care, and Pain Medicine, Massachusetts General Hospital, Boston, Massachusetts, for supporting this work. The authors also extend their sincere gratitude to Shaji Anupama, B.Tech., Micah Flynn, B.A., Gopi Gogada, M.C.A., Mikaela Hoxhalli-Pine, B.S., Karen Kan, M.H.Sc., Diane Kostka, M.H.Sc., Milcho Nikolov, M.S., Lauren Smith, B.S., Morgan Templeton, B.S., and Kalpan Tolia, B.C.O.M., P.G.P.I.T. (The Clinical IT Systems Team, Department of Anesthesia, Critical Care, and Pain Medicine, Massachusetts General Hospital), for developing the decision support and electronic data reporting that supported this project. The authors also thank Fred Millham, M.D., M.B.A. (Department of Surgery, South Shore Hospital, South Weymouth, Massachusetts), K. Trudy Poon, M.S. (Department of Anesthesia, Critical Care, and Pain Medicine, Massachusetts General Hospital), and Victoria Carballo, M.P.H. (Department of Quality, Safety and Value, Partners Healthcare, Boston, Massachusetts), for assistance with statistical analysis, and Melissa Connors, B.A. (Department of Anesthesia, Critical Care, and Pain Medicine, Massachusetts General Hospital), for assisting with the in-person training sessions.
Support was provided solely from institutional and/or departmental sources.
The authors declare no competing interests.
Available at: http://massleague.org/Calendar/LeagueEvents/ClinicalQualityConference/2013/C2ManagingforChange.pdf. Accessed August 10, 2014.