Building the Evidence on Simulation Validity: Comparison of Anesthesiologists’ Communication Patterns in Real and Simulated Cases

HEALTH care is now acknowledged as being delivered by teams and not by individual health professionals.¹  Effective teamwork is recognized as an important contributor to patient safety,¹  and verbal communication underpins many dimensions of teamwork.^2,3  Simulation-based education is often used to develop both effective teamwork and communication, and in assessment and recertification.^4,5  However, important questions remain about the validity of simulation, and there is limited objective evidence comparing teamwork behaviors, including verbal communication, between clinical and simulated environments.

Assessment of anesthesiologists in the workplace typically takes place in routine situations. It is unclear whether effective patterns of communication displayed in routine situations will be sustained in a crisis. An advantage of simulation is the ability to recreate rare or challenging clinical events, including crises.

Therefore, in this study, we set out to explore communication among anesthesiologists, anesthesia technicians, and other members of the operating room (OR) team during simulations of cases with and without a crisis, and in comparable routine cases in the OR.

As our theoretical framework, we used the Brunswikian approach to ecological validity, where the validity of participant behaviors is a function of the validity of the environmental cues⁶  and, in the context of assessment, that the observed behaviors in the assessment condition (e.g., simulation environment) predict behaviors in the open environment.⁷  We found no previous observational studies comparing communication in the simulated environment with communication in the clinical environment. We designed simulations to provide comparable environmental cues with those seen in ORs in our institution. In an observational study, Manser et al.⁸  compared anesthesiologists’ activity patterns in a simulated and a real OR, using a structured computerized coding system consisting of 43 observable, predefined behaviors, and demonstrated similar patterns, although at higher activity levels in the simulation. We have applied this methodology to verbal communications and have examined the extent to which the simulation environment elicited similar verbal communications as a comparable clinical environment. Further justification for using simulation to teach and assess communication would be provided if the verbal communications elicited by the simulation environment resemble those elicited by a real case.

Our primary hypothesis was that anesthesiologists would exhibit similar verbal communication patterns in a routine OR case and routine simulated case. Our secondary hypothesis was that anesthesiologists would exhibit different communication patterns in routine cases (in the OR or simulated environment) and simulated cases involving a crisis.

Ethics approval was obtained from the Northern Ethics Committee Y, Auckland, New Zealand (approval NTY/081 1S IEXP). All study participants, and all staff members and patients present in the designated OR during filming, gave written informed consent. The study design is summarized in figure 1.

Each participant team comprised an anesthesiologist and an anesthesia technician. In New Zealand, an anesthesia technician is an assistant who has completed 3 yr of vocationally focused training at a tertiary institution. Within the OR team, the anesthesiologist and technician work closely together as a subunit. We included only anesthesiologist consultants or residents in their final year of training.

Manser et al.⁸  demonstrated differences in activity patterns with a sample of six anesthesiologists in two scenarios over five cases. In our previous work, we demonstrated clinically relevant effects with a sample of 20.^9,10  Taking feasibility into account, we aimed to enroll 20 anesthesiologists and anesthesia technicians.

Using unobtrusive audiovisual equipment, we video recorded the anesthesiologists in the OR over the course of a full-day elective general surgical operating list (two to five cases). The automated anesthesia records were collected for each case.

We used a METI HPS full body human patient simulator (Medical Education Technology Inc., Sarasota, FL) in a simulated OR, using the same automated record-keeping system as used in the OR (Safersleep Inc., Nashville, TN)¹¹  to create an accurate record of physiological data and events.

In the simulations, the study participants (the anesthesiologist and anesthesia technician) worked with a full OR team, comprising experienced faculty acting in the roles of surgeon, scrub nurse, and circulating nurse. The circulating nurse also assisted the participants in tasks specific to the simulation environment, e.g., fetching equipment, taking blood samples for analysis.

The routine simulation, modeled on the cases typical of the OR where filming took place, involved a 70-yr-old female undergoing elective laparoscopic surgery. The case proceeded uneventfully apart from a change in blood pressure of approximately 20% from baseline. In a second scenario, participants then managed one of two crisis simulations: profound hypotension with hyperkalemia or an unstable tachyarrhythmia. Although the crisis simulations differed, they were both manifestly more complex than the routine simulation.

We followed a standardized protocol for familiarization and briefing to the simulation environment. Simulations began when the anesthesiologist first entered the simulated OR to begin the anesthesia for the patient and ended with the completion of surgery. The routine case was followed by the case containing the crisis. The day concluded with an educational debrief.

We wished to compare two video segments in each of the three conditions for each anesthesiologist: an “induction” segment and an “event” segment. We recorded several hours of video for each anesthesiologist in the OR consisting of two to five cases. From these, a research assistant (not involved in the simulation component of the study, the coding or analysis) selected video segments on the basis of the criteria listed in table 1.

We used a structured coding framework to code anesthesiologists’ verbal communications, allowing computerized recording and analysis of relevant communication events. The methodology was based on the observation system developed by Manser et al. to observe anesthesiologists’ actions during simulated and real OR cases.^12,13  The starting point for the coding framework was well-established dimensions of teamwork taken from the extensive literature on effective teamwork, crew resource management, and nontechnical skills.^14–16  These were workload management; problem solving and decision making; open communication environment; and sharing mental model. We then developed observable verbal manifestations on each dimension of teamwork. These were identified initially on the basis of published examples,¹⁶  then reviewed by a group of experts, and subsequently piloted. This resulted in a list of eight mutually exclusive items for communication types, within which a subset of four items that related to sharing of situational information.

All relevant communication events in the videos were coded initially on the basis of to whom the anesthesologist was communicating (six mutually exclusive items) and the communication type. A subset of these communication types involved sharing situational information about the patient (four items), and these were then coded against these items. The chosen communication events were not an exhaustive list of all possible communications.

The coding framework was piloted by three researchers viewing six video segments, following which descriptors for the items were refined to address ambiguity.

Two coders, both with clinical experience in the operating theatre, were trained in the use of the observation tool. The order of the videos was randomized for coding. Using Observer XT video-analysis software (Noldus Information Technology, Wageningen, The Netherlands), the two coders independently coded all videos using the coding framework. A 10% sample of these initial independent codings showed 85% agreement on coded verbal communications. The two coders then met and reviewed every communication event where their coding differed and reached consensus on each event and item over the course of a single viewing. The final codings for analysis were thus a consensus of the two coders.

After the simulation and before any discussion, the anesthesiologist and anesthesia technician completed an on-line questionnaire exploring their perceptions of the realism of the simulations.

The percentages of all coded items were then calculated for each anesthesiologist in each condition, and compared using a mixed-model ANOVA that included anesthesiologist as a random factor, and condition as a fixed within-subject effect. When the overall condition effect for a particular item was significant (two-tailed P < 0.05) from the ANOVA, we proceeded to pair-wise comparisons among the three conditions, using Fisher protected least significant difference test, applying the Bonferroni correction for multiple comparisons. The differences between conditions are represented as Cohen D effect sizes, calculated as the difference between the means divided by the SD. All analyses were undertaken using SPSS V19 (IBM SPSS Statistics, Armonk, NY).

We recruited 20 anesthesiologists (17 consultants and 3 senior residents) from the pool of 58 eligible anesthesiologists and 20 anesthesia technicians.

Of the 20 anesthesiologists recorded in the simulated cases, we were unable to record three in the OR as they had moved to other hospitals. This generated a total time of 2,469.2 min of video, with an average per video segment of 21.66 min (median, 20 min; range, 6.6–84.8 min; interquartile range, 16.5–22.5 min).

From this data, we identified 2,502 relevant verbal communication events between the anesthesiologist and other members of the OR and simulated OR teams, 48.7% of which involved sharing of situational information about the patient. The different items, expressed as percentages of all 2,502 coded communication events, are shown in table 2. Items that were coded less often than 5% of the total were excluded from further analysis.

The number of communication events in the crisis simulations was considerably higher than in either of the other two conditions: 16% (400) were in the OR cases; 27% (683) in the routine simulations; and 57% (1,419) in the crisis simulations. The average number of coded communication events per video segment per condition was 12.5 (OR), 17.5 (routine simulation), and 36.5 (crisis simulation).

The mean percentages of all coded items for each anesthesiologist in each condition are shown in table 3.

The only significant differences between the OR and the routine simulation were the percentages of communications with the circulating nurse. The circulating nurse was a faculty member with the specific role of assisting participants with tasks related to working in the simulation environment.

There were, however, significant differences between the crisis simulation and the other two conditions in the items related to sharing of situational information. Significantly higher percentages of verbalizing patient status, assessing patient status, and proposing a plan of action occurred in the crisis simulation than in the real OR. Proposing a plan of action was also significantly different between the routine simulation and the crisis simulation. In addition, there was a trend toward differences in assessing patient status.

The pair-wise comparisons of the routine simulation with the other two study conditions (crisis simulation and OR; table 4) indicate that the effect sizes for the comparisons that were not statistically significant were small to moderate, generally less than 0.50. This would indicate that the sample size of 20 anesthesiologists provided a sufficient power to detect significant and potentially important differences between the conditions.

Participants rated team communication as realistic in 86% of responses (n = 80). Participants considered their communications generally occurred with a similar frequency in the simulations as in comparable cases in the OR (table 5).

Our results support our primary hypothesis that anesthesiologists would exhibit similar verbal communication patterns in a routine OR and a routine simulated case. Anesthesiologists directed their communications to different members of the team in similar percentages in the OR and in the simulations, with the exception of communications with the circulating nurse. The latter was a member of the faculty assigned to assist participants with tasks specific to the simulation environment and, not unexpectedly, significantly more communications were directed to this person in the simulations.

Our secondary hypothesis was that anesthesiologists would exhibit different communication patterns in routine cases (in the OR or simulated environment) and simulated cases involving a crisis. Again, our results support this hypothesis.

This is the first study that has attempted to compare anesthesiologists’ verbal communications in the operating theatre with those in the simulated environment. Manser et al. looked at another aspect of ecological validity of the simulation environment and provided evidence that anesthesiologists undertook similar tasks during simulated and OR cases. They suggested that understanding the effect of the simulation environment on teamwork behaviors was key to create valid learning experiences.⁸  In this study, we explored the effect of simulation and the occurrence of a crisis on key team communications and identified areas of similarity and difference. Although in this study we have not rated quality of communication, the items in our coding tool are closely linked to established dimensions of teamwork. This could conceivably allow more specific research into which types of communication are desirable; for example, does verbalizing more situational assessment information encourage more suggestions from other team members, and subsequently better patient management decisions?

Increased sharing of situational information about the patient in crisis simulations makes intuitive sense in view of the additional challenges involved and supports previous findings.¹⁷  Although the ability of our coding framework to detect this difference supports its construct validity, this finding suggests that communication patterns in routine simulations may not be predictive of those in crisis simulations. This finding may also apply in the clinical environment. Although future research to explore the validity of simulation for assessing crisis management should optimally compare real and simulated crises, this is not likely to be feasible. Our findings of similarity between routine simulated and OR cases, but differences in simulated cases containing a crisis, provide some limited support for the proposition that anesthesiologists’ ability to manage a crises may be better assessed in a simulator than by inference from the management of routine OR cases. Furthermore, far fewer team-related communications occurred in the OR than in the crisis simulations, compounding the difficulty in assessing teamwork communication in the real setting.

Modern approaches to validation consider a unified concept of validity where evidence from different sources is required to build the argument.^18,19  Much of the evidence on simulation validity rests on comparisons of novice and expert performance in the simulator.^20,21  Experts would be expected to perform better than novices in real life and, if this is also the case in a simulation, it would support the validity of the simulation. Although other types of validity evidence exist, in this study we looked for evidence on ecological validity, determining whether the simulation elicited similar or different responses from participants. From this and Manser’s study, we can now say that anesthesiologists’ activity patterns and teamwork-related verbal communications support these important components of ecological validity of the simulation environment.

Selecting a random sample from the limited number of available anesthesiologists was not feasible and may introduce bias. However, the paired study design is a strength because it allowed participants to act as their own controls.

Our study was not designed to demonstrate equivalence between the conditions, but rather to explore similarities and possible differences. In general, demonstration of equivalence requires a larger study than the demonstration of differences. For this reason, we have reported the effect size of comparisons between the conditions. A larger sample size with more statistical power may well identify further more subtle differences between the three conditions; however, the small to moderate effect sizes in our study suggest it was adequately powered to detect any important differences. Our results may be useful to plan future studies of equivalence.

The coding framework that we used is novel. To ensure content validity, we consulted the literature and drew on the experience of senior anesthesiologists and a human factors expert. The ability of the framework to detect different communication patterns with the circulating nurse lends support to its validity in that it has detected an anticipated difference (due to this person’s role in the simulation). We had no previous knowledge of interrater agreement with the coding framework. Similar studies using observation tools for coding observable events, where no judgment on quality or completeness is required, have used a single coder.²²  An interim reliability check before consensus supported high levels of agreement between our two coders, suggesting only one coder would be necessary for future studies.

Although we could manipulate the simulation environment, we were unable to manipulate the clinical environment and this inevitably led to some compromise in comparability of the chosen video segments. We chose to study intubation and a change in blood pressure of 20% from baseline, which could be expected to involve similar tasks and workload demands in the simulated and clinical environment.

In conclusion, in our study, anesthesiologists demonstrated similar communication patterns in routine simulated cases and routine OR cases, but we found differences between routine OR and simulated cases and crisis simulations. The similarity in teamwork-related communications elicited from anesthesiologists in routine simulated cases, and the real setting lends support for the ecological validity of the simulation environment and its value in teamwork training. Different communication patterns and frequencies under the challenge of a crisis support the use of simulation to assess crisis management skills.