Pulse oximetry is considered a standard of care in both the operating room and the postanesthetic care unit, and it is widely used in all critical care settings. Pulse oximeters may fail to provide valid pulse oximetry data in various situations that produce low signal-to-noise ratio. Motion artifact is a common cause of oximeter failure and loss of accuracy. This study compares the accuracy and data dropout rates of three current pulse oximeters during standardized motion in healthy volunteers.
Ten healthy volunteers were monitored by three different pulse oximeters: Nellcor N-200, Nellcor N-3000, and Masimo SET (prototype). Sensors were placed on digits 2, 3, and 4 of the test hand, which was strapped to a mechanical motion table. The opposite hand was used as a stationary control and was monitored with the same pulse oximeters and an arterial cannula. Arterial oxygen saturation rate varied from 100% to 75% by changing the inspired oxygen concentration. While pulse oximetry was both constant and changing, the oximeter sensors were connected before and during motion. Oximeter errors and dropout rates were digitally recorded continuously during each experiment.
If the oximeter was functioning before motion began, the following are the percentages of time when the instrument displayed a pulse oximetry value within 7% of control: N-200 = 76%, N-3000 = 87%, and Masimo = 99%. When the oximeter sensor was connected after the beginning of motion, the values were N-200 = 68%, N-3000 = 47%, and Masimo = 97%. If the alarm threshold was chosen as pulse oximetry less than 90%, then the positive predictive values (true alarms/ total alarms) are N-200 = 73%, N-3000 = 81%, and Masimo = 100%. In general, N-200 had the greatest pulse oximetry errors and N-3000 had the highest dropout rates.
The mechanical motions used in this study significantly affected oximeter function, particularly when the sensors were connected during motion, which requires signal acquisition during motion. The error and dropout rate performance of the Masimo was superior to that of the other two instruments during all test conditions. Masimo uses a new paradigm for oximeter signal processing, which appears to represent a significant advance in low signal-to-noise performance.
CONTINUOUS monitoring of arterial hemoglobin oxygen saturation by pulse oximetry has been a standard of care in the operating room since 1990 and in the recovery room since 1992. *Despite universal agreement on the importance of pulse oximetry monitoring, little progress has been made in reducing the incidence of failure to display valid data. A prospective study in the operating room in 1991 found an overall incidence of oximeter failure near 1%. [1]The term failure was defined in this study as a gap of more than 15 minutes in the anesthesia record without pulse oximetry (SpO2) data. Use of the term failure rate is problematic, because nearly every published study has applied a different definition. A recent intraoperative study defined data failure as the presence of a gap of more than 10 min in continuously recorded SpO2data. [2]To avoid confusion, we use the term data dropout to describe any interruption in continuous SpO2data, and dropout rate as the percentage of time when SpO2data are not provided by the oximeter.
By any definition, dropout rates in the recovery room appear significantly greater than in the operating room, and the most common cause of data dropout and false alarms in awake patients appears to be motion artifact. [3]In fact, a pediatric critical care study found that 68% of all alarms in the unit were false, and 71% of the pulse oximeter alarms were false. [4]In a prospective study of 9,578 patients in the recovery room, patient motion was responsible for 56% of the 106 cases in which pulse oximetry was “completely abandoned,” and for 29% of the 123 cases in which it was “temporarily abandoned.”[5].
A volunteer study in 1990 examined the effects of sinusoidal motion on four pulse oximeters during mild hypoxemia. [6]The authors concluded that all four instruments experienced significant delays in detecting hypoxemia, and that during motion the oximeters “frequently” reported nonexistent hypoxemia. However, this study did not quantify errors caused by motion in terms of bias (mean error) or precision (standard deviation of error). Despite the increased interest in motion-generated errors since 1990, Severinghaus [7]concluded in 1992, “There is relatively little to report as to methodological advance in pulse oximetry since 1988.”
The past 2 years have seen increased activity by pulse oximeter manufacturers in developing new instruments with better artifact rejection. Among others, Nellcor (Pleasanton, CA) has developed the N-3000 “Symphony” Pulse Oximeter as an improvement in low signal performance over their N-200 series. Although the N-200 instrument included a “C-lock” feature that synchronized the absorbance wave form with the electrocardiogram, this feature was used infrequently in clinical practice and is not included in the N-3000. Masimo Corporation (Mission Viejo, CA) has developed a completely new algorithm for processing the light absorbance wave forms produced at the 660- and 940-nanometer wave lengths of the oximeter sensor. Their “Signal Extraction Technology” or Masimo SET uses adaptive digital signal processing to cancel the noise signal that is common to the two wave lengths. Initial volunteer studies with this technique have shown promise. [8].
These recent developments in pulse oximeter technology prompted the current study, which is aimed at determining the impact of the new technologies on resistance to motion artifact. This was a volunteer study comparing the performance during standardized motion of three pulse oximeters: Nellcor N-200, Nellcor N-3000, and Masimo SET. The Masimo and Nellcor N-3000 were chosen because they both represent new technologies aimed at reducing motion artifact. The N-3000 is approved by the Food and Drug Administration and commercially marketed; the Masimo SET is an experimental prototype that is not yet federally approved. The N-200 was selected because it exhibited the best performance of the “conventional” oximeters tested in our previous preliminary experiments. [8]The present study covers the range of hemoglobin oxygen saturations between 100% and 75%, with two different motions occurring at various controlled intervals during the experiment.
Methods
Ten healthy volunteers participated in this study, which was reviewed and approved by the Human Subjects Review Committee. Participants included seven men and three women, whose average age was 26.6 + 6.7 yr (range, 19 to 42 yr); all were nonsmokers with no evidence of vascular or other systemic disease. After giving informed consent, each participant had a 20-gauge, 1.5-inch cannula placed in the radial artery of the nondominant wrist. Three pulse oximeter sensors were then attached to the test hand (the hand opposite the radial cannula): Nellcor N-200, Nellcor N-3000 “Symphony,” and Masimo SET prototype. (The C-lock feature of the N-200 was not used during this study because it actually deteriorated performance in the preliminary motion study. [2]) Additional N-200 and Masimo SET sensors were placed on the cannulated or control hand. All sensors used in the study were disposable, tape-on adult finger sensors; they were firmly attached to the finger and all had the same low mass. The Masimo sensor, provided by the manufacturer, was similar in appearance to the Nellcor disposable. Data obtained from the test hand during various motion protocols were compared with simultaneous data from a similar oximeter whose sensor was located on the stationary control hand. Sensors were located on the index, middle, and ring fingers, and the specific fingers for each instrument were rotated among the volunteers.
Standardized motions called “rubbing” and “tapping” were generated by a motor-driven tilt table to which the test hand was attached. The pivot point of the tilt table was located at the elbow, so that only the forearm and hand moved. In the rubbing motion, the fingertips were positioned against a stationary table such that they slid back and forth across the surface. In tapping motion, the fingertips were positioned to move perpendicular to the table, alternately lifting from and striking the surface. Various motion amplitudes and frequencies were tested. The results depended little on frequency in the range of 1 to 4 Hz, and thus a frequency of 3 Hz was selected for most of the experiments. The motion amplitude of 3 cm was selected from the results of our previous study [7]as a motion that represents a significant challenge to current state-of-the-art pulse oximetry.
All pulse oximeter data were collected electronically using an RS-232 serial communications port on each instrument. Both SpO2and pulse rate from each oximeter were recorded digitally once per second throughout the entire experiment. A 2-ml sample of arterial blood was drawn anaerobically from the radial cannula at each steady-state condition of the study. Each sample was processed using a blood-gas analyzer (Nova Stat-3, Waltham, MA) for pH, PCO2, and POsub 2 and by a co-oximeter (Radiometer OSM-3, Copenhagen, Denmark) for HbO2%, or fractional hemoglobin saturation. Oximeter data from the test hand were compared with simultaneous data from the corresponding control instrument during motion. Data from the control oximeters were compared with co-oximeter results during steady-state conditions. (In the absence of dyshemoglobins, the fractional and functional hemoglobin saturations are equal. Levels of MetHb and COHb were also determined using each co-oximeter measurement.)
The experimental protocol included various combinations of motion starting before or during both slow and rapid changes in arterial hemoglobin oxygen saturation (SaO2). In addition, the oximeter sensors were disconnected and reconnected at various preselected times, forcing the instrument to reacquire the signal during the motion conditions. For the hypoxemia portions, participants breathed through a tight-fitting face mask connected using a circle system to a North American Drager Narkomed 2A Anesthesia Machine. Inspired oxygen fraction (FIO2) was varied by adjusting flows of oxygen and nitrogen through the fresh gas supply from the machine. Inspired and expired oxygen and carbon dioxide concentrations were monitored continuously by a Criticare POET Gas Analyzer.
The protocol for each volunteer was as follows.
1. Normoxemia (room air): After baseline data were recorded for 2 min, the rubbing motion was started and 2 min of data were recorded. (The 2-min interval was selected for all motion conditions because this provided at least 1 min of steady-state effect.) While motion continued, the oximeter sensors were disconnected and reconnected, followed by another 2 min of data recording. Motion was then stopped and 2 minutes of baseline data were again recorded. This sequence was then repeated for the tapping motion.
2. Steady-state hypoxemia: The face-mask was placed on the participant, and the FIO2was adjusted to decrease the control SpO sub 2 values to approximately 75%. An arterial blood sample was aspirated for analysis. The exact protocol described in protocol 1 was repeated (both rubbing and tapping) while maintaining a steady-state control SpO2near 75%. Another arterial blood sample was aspirated before the participant was returned to normoxemia. Pulse oximetry was maintained at 75% for no more than 5 min. Participants were instructed to remove the mask and terminate the test if they experienced any unpleasant symptoms such as nausea, dizziness, palpitations, or chest pain.
3. Rapid changes in SaO2: Each volunteer was stabilized for 1 min at an SpO2of 75%, the rubbing motion was started, and the FIO2simultaneously changed to 100%. Data were recorded during and after this rapid resaturation. The same sequence was repeated for the tapping motion. Finally, after a stable SpO2was obtained during room air breathing, the FIO2was abruptly decreased while the rubbing motion started simultaneously. The control SpO2was stabilized at near 75% for 2 min and then the FIO2was rapidly changed to 100%. The motion was continuous throughout the rapid desaturation and resaturation. The same sequence was repeated for the tapping motion, concluding the experiment for one volunteer. This protocol was followed for all 10 volunteers.
Pulse oximetry data from each oximeter were compared with simultaneous data from the corresponding control instrument to yield an oximeter error, defined as the difference between the two values (test minus control). The bias (mean error) and precision (standard deviation of error) were calculated as described by Altman. [9]Error rate was calculated as the percentage of time when the oximeter error exceeded a specified threshold; that is, 5%(E5), 7%(E7), or 10%(E10). Dropout rate (DR) was defined as the percentage of time when the oximeter provided no SpO2data. Performance index (PI) is the percentage of time when the oximeter provided an SpO2value within 7% of the control SpO2value. By definition, Equation 1.
The statistical significance values of performance differences among the three instruments were evaluated by analysis of variance, with Bonferroni correction used for interval values (bias + precision), and chi-square analysis for outcomes values (E5, E7, E10, DR, PI). A probability value less than 0.05 was considered significant.
Results
No participant terminated a test because of symptoms of hypoxemia. Arterial blood co-oximetry analysis (Radiometer OSM-3) revealed no significant levels of methemoglobin or carboxyhemoglobin in any volunteer. Co-oximetry data also showed that the reference pulse oximeters on the control hand performed within their specified uncertainty limits (+2%) during steady-state conditions.
(Figure 1) shows typical single-subject plots of SpO2versus time for the three pulse oximeters tested: Nellcor N-200, Nellcor N-3000, and Masimo SET. Each plot compares one test oximeter during the two motion conditions with the N-200 control on the stationary hand. Figure 1depicts the normoxemia and gradual desaturation to SaOsub 2 = 75% sections of the protocol (1 and 2). The durations of the two motions are indicated on the bottom of Figure 1, as are the times when the sensors were disconnected and reconnected. For this participant in both normoxemia and hypoxemia, the N-200 underestimated saturation by 5% to 18% during motion. The N-3000 failed to display an SpO2value after a disconnection-reconnection for both motions. The Masimo SET displayed an SpO2value at all times but underestimated saturation by 3% to 6% during hypoxemia and motion. Figure 2shows a similar plot for a rapid desaturation and resaturation that occurred during continuous tapping motion (protocol 3). The Masimo SET tracked the control SpO2well during the desaturation but exhibited some lag during the rapid resaturation. The other two instruments did not track control SpO2in this case.
Figure 1. SpO2versus time for three test pulse oximeters compared with stationary N-200 control (thin line) on opposite hands during normoxemia followed by gradual desaturation. (a) Motion oximeter = N-200;(b) N-3000;(c) Masimo SET. Rubbing and tapping motions begin and stop when indicated by bars. The notch in the motion bar indicates time when oximeter sensors (except control) are disconnected and reconnected during motion.
Figure 1. SpO2versus time for three test pulse oximeters compared with stationary N-200 control (thin line) on opposite hands during normoxemia followed by gradual desaturation. (a) Motion oximeter = N-200;(b) N-3000;(c) Masimo SET. Rubbing and tapping motions begin and stop when indicated by bars. The notch in the motion bar indicates time when oximeter sensors (except control) are disconnected and reconnected during motion.
Figure 2. SpO2versus time showing a rapid desaturation-resaturation occurring during tapping motion. N-200 and N-3000 indicate desaturation before it actually occurs. N-200 reads SpO sub 2 near 80% after the control SpO2value has returned to normal.
Figure 2. SpO2versus time showing a rapid desaturation-resaturation occurring during tapping motion. N-200 and N-3000 indicate desaturation before it actually occurs. N-200 reads SpO sub 2 near 80% after the control SpO2value has returned to normal.
(Table 1) shows the following aggregate parameters for all ten participants: bias + precision; percentage of time when error is equal to or greater than 5%(E5), 7%(E7), and 10%(E10); percentage dropout rate, and performance index; or percentage of time when the instrument provided an SpO2value within 7% of control (Equation 1).
Table 1. Accuracy (Bias +/- Precision, E5, E7, E10), Dropout Rate (DR), and Performance Index (PI) of Three Pulse Oximeters during Motion
(Table 2) shows the effects of these results on the generation of alarms. For this purpose, we selected an alarm threshold of 90% so any oximeter reading of 89% or less would generate an alarm. A true positive, or a correctly detected alarm condition, occurred when the control SpO2was less than 90% and the test oximeter SpO2was also less than 90%. Similarly, control SpO2less than 90% and test SpO2more than 90% is a false negative, or alarm condition missed by the test oximeter. When control SpO2is greater than 90%, then test SpO2greater than 90% is a true negative and test SpO2less than 90% is a false positive. By the accepted definitions, the sensitivity is the fraction of times that an actual alarm condition is correctly detected, or Equation 2.
The specificity is the fraction of times that a nonalarm condition is correctly detected, or Equation 3.
The difficulty with these definitions in this experiment is that they do not account for pulse oximeter dropouts, or interruptions of data, that fall into none of these categories. Thus we have provided two forms of sensitivity and specificity in Table 2: the common definition in which data dropouts are not accounted for, and a modified definition called sensitivity** and specificity** that consider dropouts as either false negatives or false positives, depending on the control SpO2value at the actual time of data interruption.
Discussion
As shown in Figure 1and Figure 2and Table 1and Table 2, the controlled hand motion used in this experiment had significant effects on both accuracy and dropout rate of the pulse oximeters. In addition, the test condition of connecting the oximeters after beginning the motion (Table 1, condition B) proved more strenuous than that of starting the motion after the oximeters were connected and functioning (Table 1, condition A). This is evidenced primarily by the higher dropout rate and lower performance index for condition B compared with condition A in the N-200 and N-3000. The rapid denaturation to 75%(Table 1, condition C) produced similar results, except that the N-3000 exhibited a higher error rate and lower dropout rate when compared with condition B.
The performance of the three instruments during motion differed in several ways. The most obvious is that the Masimo SET exhibited much lower error rates and dropout rates than did the other two oximeters for all three test conditions. The lowest performance index for the Masimo was 95%, compared with 46% for the N-3000 and 68% for the N-200. The N-3000 and N-200 also differed significantly from one another in their motion performance. Although the N-3000 performed better than the N-200 for condition A (PI = 87% compared with 76%), this relationship was reversed for conditions B and C. In fact, the N-3000 exhibited the highest dropout rate of any instrument (DR = 46%) for condition B, motion beginning before oximeter connection. On the other hand, the N-200 demonstrated the greatest SpO2errors of any instrument (E7 = 29%), also during condition B. This suggests that during motion the N-200 has a greater tendency to yield erroneous SpO sub 2 values, whereas the N-3000 is more likely to fail and provide no values. Whether it is preferable to have an erroneous SpO2value or no value at all depends on the magnitude of the error. There is clearly a point when it is better to have no value (with an error message) rather than a very inaccurate value. For this reason we have provided performance data for three error limits: 5%, 7%, and 10%.
(Table 2) interprets the data in the context of alarm performance. We defined the alarm condition as SpO2less than 90%, which is a common alarm threshold. This is also the threshold used in the pediatric intensive care unit study by Lawless, [4]in which a Nellcor pulse oximeter (model unspecified) yielded sensitivity and specificity values of 77% and 92%, respectively. Because no false negatives (displayed SpO2> 90% while control SpO2< 90%) were encountered in our study, Table 2shows sensitivities of 100% for all three instruments. Our reported specificity values of 70% and 80% for N-200 and N-3000 are consistent with the 77% value reported by Lawless, [4]who also reported that the pulse oximeter was responsible for 46% of all false alarms. If oximeter dropouts are considered, most of the sensitivity and specificity values for N-200 and N-3000 decrease. Stated another way, of the 100 actual desaturation “events” recorded, the N-200 detected 98, the N-3000 detected 83, and the Masimo detected 100.
The signal extraction technology developed by Masimo represents a new approach to pulse oximeter signal processing. The traditional paradigm for pulse oximetry assumes that all pulsations in the light absorbance signal are caused by oscillations in the arterial blood volume. This is equivalent to postulating that the blood flow in the region of the sensor passes entirely through the capillary bed rather than through any arteriovenous shunts. The conventional pulse oximeter calculates the ratio of the pulsatile absorbance (AC) to the nonpulsatile absorbance (DC) at each of its two wavelengths, 660 nm and 940 nm:Equation 4.
The instrument then calculates the ratio of these two “pulse-added absorbance” signals:Equation 5.
This value of R is used to find the saturation SpO2in a look-up table built into the oximeter's software. The values in the look-up table are based on experimental data in human volunteers; thus the oximeter's estimate of saturation is empirically based.
If noise is added to either or both of the S(660) and S(940) signals, the value of the ratio R will clearly be affected. In fact, if a great deal of noise is added to both signals, R will be driven toward 1.0, which corresponds to an SpO2value of approximately 85%. This mechanism has been suggested previously to explain the behavior of pulse oximeters in the presence of methemoglobinemia. [10]Motion artifact can add large amounts of noise to both signals, thus overwhelming the oximeter's ability to generate a valid SpO2value.
The Masimo paradigm begins with the assumption that arteriovenous shunting is highly variable, and that fluctuating absorbance by venous blood is the major component of noise during motion. If the venous blood absorbance signal can be isolated and measured, then this component can be removed by adaptive digital filtering. If we decompose S(660) and S(940) into a “true” arterial signal plus a noise component, Equation 6.
Again, R is the ratio of arterial pulse-added absorbances, from which we find the valid value of SpO2as described above. Combining Equation 5(a, b, and c):Equation 7.
N' is effectively a noise reference that varies with the patient's motion. If there is no motion, N' = 0 and S(660)= S(940) X R, which is the relationship for the conventional pulse oximeter given by Equation 4(c).
We now have an equation for N', but unfortunately it depends on R, which is the unknown we actually seek to determine SpO2. To solve this problem, Masimo sweeps through all possible values of R (i.e., SpO2values between 1% and 100%) and generates an N' value for each possible R value. Next the raw signals, S(660) and S(940), are processed with each possible N' noise reference through an adaptive correlation canceler, which yields an output power for each possible value of R (i.e., each possible SpO2from 1% to 100%). The result is a discrete saturation transform plot of the adaptive correlation canceler output power versus possible SpO2value, as shown in Figure 3. (The figure is taken from actual data, provided by Masimo.) This plot has two peaks: The peak at the greater SpO2value (shown at 97%) corresponds to the arterial blood saturation and is selected as the displayed SpO2. The peak at the lesser SpO2value (shown at 80%) may correspond to the venous blood saturation, although this has not yet been confirmed by experiment. The quality of each curve is assessed by the processor and used to assign a confidence level to the resulting SpO2value. The entire sequence described above is repeated once per second on the most recent 6 s of raw data. The Masimo SpO2therefore corresponds to a 6-s running average of arterial hemoglobin saturation, updated every second. This description represents the first public release of the Masimo paradigm for pulse oximeter signal processing.
Figure 3. Discrete saturation transform: A plot of adaptive correlation canceler energy output versus possible SpO2values. Two peaks occur in the energy output. The right-hand peak (ra= 97%) corresponds to the arterial saturation and yields the displayed SpO2value. The left-hand peak (rv= 80%) may correspond to the saturation of the venous blood, although this has not been confirmed experimentally. (Data provided courtesy of Masimo.)
Figure 3. Discrete saturation transform: A plot of adaptive correlation canceler energy output versus possible SpO2values. Two peaks occur in the energy output. The right-hand peak (ra= 97%) corresponds to the arterial saturation and yields the displayed SpO2value. The left-hand peak (rv= 80%) may correspond to the saturation of the venous blood, although this has not been confirmed experimentally. (Data provided courtesy of Masimo.)
These results suggest that the Masimo SET prototype pulse oximeter yields a significant improvement in pulse oximeter performance during patient motion. Results of our earlier study suggest that this technology may also improve performance during low perfusion conditions. [8]Although only two other oximeters were compared with the Masimo in this study (N-200 and N-3000), these two instruments are representative of both second- and third-generation pulse oximeter technology. The N-200 is used widely today, and the N-3000 is being marketed as “the next generation of high performance pulse oximetry…. [with] exceptional performance in high-motion, low-perfusion patient environments.”** Our previous study found grossly similar comparisons of the Masimo with three other current instruments: Criticare 504, Novametrix “Oxipleth,” and Ohmeda 3740. [8]Clinical studies in both the operating room and critical care settings are now needed to evaluate the potential impact of this new technology on patient care and user satisfaction.
*Standards for postanesthesia care. A.S.A. Directory of Members, 56th ed., 1991, p 672.
**Nellcor N-3000 “Symphony” specifications, Nellcor Inc., Pleasanton, California, 1995.
