Failure-to-Rescue, defined as hospital deaths after adverse events, is an established measure of patient safety and hospital quality. Until recently, approaches used to address failure-to-rescue have been focused primarily on improvement of response to a recognized patient crisis, with limited success in terms of patient outcomes. Less attention has been paid to improving the detection of the crisis. A wealth of retrospective data exist to support the observation that adverse events in general ward patients are preceded by a significant period (on the order of hours) of physiologic deterioration. Thus, the lack of early recognition of physiologic decline plays a major role in the failure-to-rescue problem.

AN overview of the failure-to-rescue (FTR) problem and a summary of the efficacy of previous approaches used to address it are presented. Surveillance monitoring, which is based on continuous vital sign monitoring and aims to improve early detection of physiologic deterioration, is described.

Failure-to-Rescue was defined by Silber et al.  in 1992 as hospital deaths after adverse occurrences such as postsurgical complications.1After the publication of the Institute of Medicine report entitled Crossing the Quality Chasm: A New Health System for the 21st Century  2in 2001, FTR was identified as one of the key areas for improvement in patient safety. It was estimated that one method of addressing FTR (Rapid Response Teams[RRTs]) could contribute 66,000 lives saved in the 100,000 Lives Saved campaign.2 

Metrics to estimate FTR rates have been developed and are widely used as indicators of hospital quality. The Agency for Healthcare Research and Quality has developed a measure of FTR intended to address concerns about variation in documentation among reporting institutions and the fact that other metrics of patient safety, such as mortality and complication rates, may be more a measure of patient-related factors than quality of care.1,3FTR metrics are limited to some degree in their usefulness because some patients with advanced illness simply do not want life-prolonging interventions, and some adverse occurrences are not preventable.4Nevertheless, recognition of FTR as a significant issue and an important quality indicator has prompted numerous studies of the underlying causes and the development of systematic approaches to address them.

Ghaferi et al . recently categorized contributors to FTR into two broad classes: timely response (prompt recognition of the complication) and appropriate response (correct management and treatment).5Numerous studies have shown widespread problems in both of these areas,6–10identifying causes ranging from deficiencies in vital sign collections to timely action in response. Silber et al . demonstrated an association between low patient mortality and high nurse-to-patient ratio. This effect may be the result of addressing both identified problems by improving patient monitoring and prompting more timely interventions.1,4,11,12 

Furthermore, Pronovost et al.  showed an association between physician staffing levels in the intensive care unit (ICU) and patients' mortality,13an effect that may be the result of providing the right treatment at the right time. Increasing the number of ICU physicians is an improvement to FTR primarily on the efferent limb of the system; the patient either already had the adverse event that prompted a transfer to the ICU or the patient had been identified to be at high risk and therefore was transferred to the ICU for its improved staffing ratios. ICU physicians also have been used to identify patients on general wards thought to be at risk, as an example of how additional personnel can also be used on the detection side of the problem.

To address the issue of late responses, RRTs were introduced in recognition of the fact that patients show signs of deterioration in the 6-8 h before a cardiac or respiratory arrest.14The results of these previous intervention efforts have been mixed, and only weak evidence indicates that they benefit patients.15,16Early recognition has been identified as the primary determinant of the success of intervention with RRTs.17A recent consensus paper on the afferent limb of Rapid Response Systems (RRS) stated that the focus had been “on the efferent, response component of the system, although evidence suggests that improved vital sign monitoring and recognition of clinical crisis may have outcome benefits.”18 

Attempts to address the afferent limb of FTR systems have led to implementation of Patient Surveillance (PS) systems that use continuous patient vital sign monitoring in the general care setting to facilitate early recognition of physiologic deterioration (fig. 1). PS systems build on previous attempts to improve patient evaluation and deterioration detection and have shown promise in addressing aspects of the FTR problem.19However, PS systems research remains an underexplored area in the field of FTR.

Fig. 1.  Representation of physiologic state deterioration and opportunities for intervention. RRT = Rapid Response Team.

Fig. 1.  Representation of physiologic state deterioration and opportunities for intervention. RRT = Rapid Response Team.

Close modal

In the following discussion we focus on the problem of prompt recognition of and timely response to patients in crisis in the general ward setting, rather than solutions on the response side of the system. We do so because we believe that this domain provides the most opportunity for improvement and research. We present (1) a review of previous work performed to address the problem of FTR, (2) a framework for an emerging approach to FTR based on PS, (3) initial results from a PS system implementation and finally, (4) opportunities for further FTR research.

FTR is a complex, multifaceted issue that has been approached from various directions and with different goals. Although each of these approaches can enhance the safety of our patients, none individually will solve the FTR problem completely. Therefore we must continue to assess the efficacy of current and emerging methods and consider new methods of addressing this problem.

Previous attempts to address unexpected deterioration in hospital inpatients fall into four main groups. First, there have been many retrospective studies of deterioration cases, which often attempt to identify common warning signs. Second, there are some systems that attempt to classify patient risk levels prospectively based on demographic or physiologic indicators. Third, some studies have described or evaluated the use of various technologic monitoring modalities to obtain continuous physiologic state information on hospital inpatients. Fourth, studies evaluated the impact of RRS and its executive arm, Medical Emergency Teams, on patient outcomes.

There has long been interest in using cases of deterioration and death to identify potential risk factors. A 1991 study by Bedell et al.  20examined all cardiopulmonary arrests in which resuscitations were attempted during the year 1981 at Boston's Beth Israel Hospital, a 504-bed university teaching hospital. The stated purpose of the study was to examine specifically those arrests judged to have “resulted from a therapy or procedure or from a clearly identified error of omission.” However, the study presents a thorough review of the demographic and clinical characteristics of all arrests. The mean age of patients who arrested was 70 years, and 43% were female. The most prevalent clinical conditions in patients' medical histories were coronary artery disease (76%) and congestive heart failure (76%). There was a mean of 10 days between hospital admission and arrest, and the arrests were spread relatively evenly over the ward (41%), ICU (37%), and emergency department (18%). Most arrests occurred in patients on the medical service (88%), as opposed to just 12% on the surgical service. Twenty-six percent were classified as respiratory arrests, with the rest having a primarily cardiac mechanism of arrest. Overall, 50% survived the arrest, 33% of those survived to discharge, and 68% of those discharged survived more than 1 yr after arrest.

More recent studies have both pursued more detail on the characteristics of deteriorating patients and widened the scope of the deteriorations analyzed. Bobay et al.  in 2008 examined patient factors in FTR cases at five Midwestern hospitals.21They used a National Quality Forum definition of FTR as death after one of five preventable postsurgical complications: sepsis, gastrointestinal bleeding, deep-venous thrombosis/pulmonary embolism, cardiac arrest/shock, or pneumonia.

A chart review of cases from five metropolitan Midwestern hospitals yielded 376 potential cases of FTR, although only five were ultimately judged to be nurse-sensitive FTR (the remainder had nursing care that “met established guidelines”). Reviewers used patient charts to collect 16 variables, including vital signs and laboratory results from up to 3 days before ICU admission. Heart rate, respiratory rate, temperature, blood sodium concentration, and urine output were shown to be statistically significant indicators of deterioration for this patient group, whereas the Glasgow Coma Scale and oxygen saturation were among the nonsignificant indicators. Age was also an important indicator of deterioration, with most cases occurring in patients older than 70 years, and the study authors suggest that closer monitoring is warranted for this population. The study also emphasizes the difficulty of isolating true FTR cases in large-scale retrospective chart reviews because most of the cases identified as probable by a computer search of complication codes proved to be non-FTR (either operating room deaths or deaths classified as nonpreventable by nurses who followed established guidelines) on closer review.

Buist et al.  in 2004 also investigated the predictive value of several physiologic indicators14using cases of cardiac arrest rather than FTR. Data were collected over 33 weeks in 1999 through daily visits to each ward by the investigators, who reviewed medical records and charts for any adverse events and asked staff members about any undocumented occurrences. Investigators looked specifically for 10 abnormal observations, including Spo2<90%, low or high blood pressure, low or high pulse rate, low or high breathing rate, and loss of or decreased level of consciousness. The 6,303 patients studied (including everyone on the monitored wards, regardless of resuscitation status) represented 38,115 patient days, with a mean stay length of 6 days. Abnormal observations were found in 8.9% of patients, with an overall rate of 4.2 events per 100 bed-days. Low Spo2represented approximately half of all abnormal observations, followed by hypotension at 17.3%, and tachycardia, hypertension, bradycardia, and high respiration rate (each of these at approximately 6%). Most of the abnormal observations were reported to have spontaneously resolved (66.7%), with 21.6% resolved through treatment on the ward.

In the 564 patients who exhibited abnormal physiologic signs, 146 (26%) later died. Univariate linear logistic regression was used to associate abnormal observations and mortality. Independent risk factors were also identified using elimination algorithms, and the odds of mortality associated with each independent factor were then estimated by a multiple linear logistic regression model. The strongest predictor of mortality in univariate analysis was found to be low respiratory rate (less than 6 breaths/min), which was associated with a 13.7-fold increase in the risk of mortality by hospital discharge. Changes in consciousness and high respiratory rate (more than 30 breaths/min) were also found to increase risk of mortality in the univariate model. In the multivariate analysis, changes of consciousness, hypotension, high or low respiratory rate, and low Spo2were all found to be significant independent predictors. High numbers of events also correlated with increased mortality, with a probability of death of 16.2% for patients with only one abnormal observation, compared with 88.2% for four or more. Overall, the study authors observed that changes in neurologic and respiratory status were associated with high risk of mortality, while accounting for less than 5% of all observations.

Although these studies have a diversity of patient populations, aims, and outcomes, they do share some important characteristics. Each relied on chart data and intermittent vital sign data, whether collected prospectively or retrospectively. Each found evidence that at least some unexpected deteriorations can be predicted, but none identified a single overriding factor that would predict all deteriorations. Furthermore, none described attempts to predict deterioration based on their findings. The complexity of data collection and analysis is representative of this class of studies, as are the conclusions presented. More examples are available in the literature.8,22–26 

Risk scoring systems represent an attempt to operationalize the findings of studies such as those presented previously. Various algorithms exist for synthesizing elements of patient medical history, demographic information, and clinical readings into a single score designed to indicate risk of deterioration, and several studies have examined their effectiveness. Whether created with the FTR problem in mind or not, these scoring systems all address the general question of how to evaluate patient severity, an important component of the RRS paradigm.

An early attempt at validation of one scoring system was made in 2001 by Subbe et al .27The study investigated the relationship between a Modified Early Warning Score (MEWS) and increased risk of death or critical care admission. A total of 709 medical emergency admission patients were included over a period of 1 month (although technical restrictions limited data collection for any given patient to 5 days) at a district general hospital; patients who were admitted directly to coronary care or critical care were excluded. Each patient was scored twice daily based on thresholds for nursing measurements of blood pressure, heart rate, respiratory rate, temperature, and the Alert, Voice, Pain, Unresponsiveness measure. For patients with a maximum score over a predetermined threshold of five, the study found increased odds ratios [OR] for death at 6 months (OR, 5.4), ICU admission (OR, 10.9), and high dependency unit admission (OR, 3.3). Critical care transfers were decided by physicians without using the MEWS score. The authors suggest that such a scoring system could be valuable in identifying patients from the general admissions pool who warrant increased clinical attention.

Morrice and Simpson in 2007 compared MEWS with three other algorithms, in a population of 67 adult inpatients.28The Intensive Care Society (ICS) classification divides patients into several levels based on their perceived risk of deterioration as identified by descriptive categories, e.g. , “needs can be met through normal ward care” as opposed to “a patient requiring staff with special expertise.” Although developed for use in ICU settings, ICS has been adapted for general ward use. The MEWS scores patients on physiologic measurements, with measurements outside of normal ranges attracting higher scores and indicating a greater overall risk of deterioration. The Therapeutic Intervention Scoring System (TISS-28) is designed to map the medical interventions required for a particular patient to an estimate of the nursing time taken to provide the care and has been used as a general acuity scoring mechanism for ward patients.

Morrice and Simpson first scored all adult inpatients using the ICS classification, then compared the classification groups using demographic and physiologic variables in addition to the three additional scoring systems listed previously. ANOVA techniques were used to test the hypothesis that the three ICS classification groups did in fact represent distinct patient populations as indicated by the alternative measures. The study concluded that physiologic measurements alone were relatively weak indicators of future deterioration and that although level two ICS patients (requiring a high level of external support) represented a distinct population, it was more difficult to identify significant differences between ICS levels 0 (normal ward care) and 1 (at serious risk of deterioration). In fact, the authors state that there is no single tool currently able to identify at-risk patients.

Kho et al. , also in 2007, explored an alternative approach to risk estimation using automated scoring of electronic medical records of 1,878 patients at a 725-bed academic medical center in Chicago.29This system was based on MEWS but subtracted the mental status element and added more readily available age and body mass index data. Vital signs data were collected from electronic charting performed by ward nurses. Sensitivity and specificity were calculated against the outcome of RRT activation. The study found that the modified MEWS had a receiver operating curve with area similar to the original MEWS, suggesting that an early warning system based only on electronic medical records data could perform as effectively as a more labor-intensive manual scoring technique at indicating RRT activation. The authors suggest that adding such an automated system could provide an additional level of safety in identifying some patients at risk of deterioration.

Finally, a systematic review of 36 papers describing 25 distinct scoring systems was performed by Gao et al.  in 2007.30The systems studied included several variations of Medical Emergency Teams calling criteria (primarily from Australia) and MEWS (primarily from England), together with multiple individual systems from Canada, the United States, and the United Kingdom. The review found that no system met the requirements of a level 1 clinical decision rule (a rule widely validated to change clinical behavior and improve outcomes). The review authors note the wide variety of published systems and a corresponding lack of repeated, statistically sound validation of any particular system. A review of datasets provided by study authors also revealed that many systems have low sensitivity and positive predictive value, and this may be because of high triggering thresholds; sensitivity could potentially be improved by triggering clinical response at lower scoring levels, although this would lead to increased workload.

Despite the preponderance of risk scoring techniques, there is no system that has been shown to perform reliably the difficult task of separating patients who will deteriorate from those of a similar acuity level who will not. All systems discussed here are also limited by their reliance on the medical history, which is relatively static, and on the intermittent vital signs checks performed by nursing staff. Similar results and limitations can be seen in the literature.31–33 

Continuous monitoring systems represent a more proactive approach to identifying patient deterioration, based on the premise that physiologic changes can indicate, and perhaps predict, deterioration episodes. Multiple technologies are in use for both the measuring of physiologic data and its analysis.

Electrocardiograph monitoring has long been used for cardiac patients and has been extended to other patient classes at risk of developing cardiac dysrhythmia. Larson and Brady in 2008 performed a review of the use of inpatient electrocardiogram monitoring to characterize its effect and attempt to identify common criteria for the use of electrocardiogram in the inpatient population.34The study identified numerous shortcomings in the use of electrocardiogram monitoring. Several studies have shown limited utility for electrocardiogram monitoring of patients with low-risk electrocardiographic patterns,35,36and that an electrocardiogram is not necessarily highly reliable in identifying even patients currently undergoing cardiac arrest.37In spite of this finding, physicians were shown to consistently overestimate the usefulness of cardiac monitoring when compared with the actual rate of care decisions made based on electrocardiogram readings. The authors found that unnecessary cardiac monitoring contributed to overcrowding of emergency departments through a shortage of ICU and electrocardiogram-equipped beds, an overburdening of cardiac telemetry wards leading to higher rates of missed cardiac events, and a serious financial burden on both hospitals and patients. The review concluded that correct identification of low-risk patients not in need of cardiac monitoring would offer substantial benefits.

Another approach to continuous monitoring is the multiparameter model advocated by Tarassenko and Oxford BioSignals Ltd. (Oxford, United Kingdom). BioSign brings together five physiologic parameters (heart rate, respiration rate, blood pressure, Spo2, and skin temperature) into a single indicator of patient status.§The scoring system is based on a model of physiologic normality derived from a large training dataset of high-risk adult patient data. Deviations in the measured physiologic values from this learned normal model cause the patient status indicator to rise, and sufficiently large deviations trigger automated alerts.

In a study in which 168 patients in Oxford, England were monitored using BioSign, there were 105 patients with BioSign alerts, with a mean alert rate of one every 7.8 h. Clinicians reviewing the vital signs traces after the fact for the 5 min determined that some were caused by motion artifact. Later work by researchers not associated with the device or manufacturer, however, showed no effect on outcome measures between monitored and nonmonitored patient groups in a randomized control trial.38 

As these studies show, the promise of electronic physiologic monitoring for continuous detection and prediction of deterioration has not been fully realized.

Numerous studies have shown that warning signs of physiologic instability exist before cardiac arrests.8,10,14,16,20,24,39,40Most patients have a period of deterioration that precedes cardiopulmonary events by 6–8 h.7,22Hospitals have not been able to respond to these events early and adequately enough.7,14,22,26,39,40 

The notion of intervention during the time of deterioration, to actually prevent the adverse event, led to the idea of RRS and its executive arm, the RRT. These teams bypass the vertical approach of the medical consultation system (nurse, resident, consultation) to respond in a more timely manner by using horizontal activation systems (the patient's nurse activates the RRT). RRT members are typically drawn from the critical care providers at the institution, and specific activation criteria can vary considerably between hospital systems, although most current systems allow nurses to activate the response at their discretion even for patients who have not met a specific criterion.

The study of the effect of RRTs on outcomes is challenging. Because of the difficulties studying the topic by randomized controlled trials in the same hospital, studies have focused on either hospital randomization (cluster randomization)41–43or single-center before-and-after studies.44–46A recent meta-analysis16found reduced relative risks in hospitals with RRTs for cardiac arrests and hospital mortality but concluded that a general recommendation for RRTs was premature based on the heterogeneity and wide confidence intervals of the effect. Similarly, a Cochrane review performed in 200947identified only two studies42,43that were sufficiently rigorous for inclusion and judged the effects of RRTs to be inconclusive. The success of RRTs depends primarily on early detection of deterioration, and this intuitively makes sense—the RRT can only be as good as the monitoring that activates it. Indeed, studies have shown a circadian activation pattern48–50that correlates with nurse-to-patient ratios. Not surprisingly, Calzavacca et al .17found delayed medical emergency team (MET) activation to be the most important factor effecting outcome. In a recent review of research related to the adoption of medical emergency teams, Tee et al .51stated: “As the MET call criteria depend heavily on physiologic alteration of signs, poor monitoring equipment, methods and recognition by staff may be a major stumbling block in improving outcomes and RRS performance.”

Early detection of deterioration on a broad level requires comprehensive patient surveillance. Patient surveillance in the in-patient general care setting is a new concept in medicine that needs to be differentiated from condition monitoring. Historically, condition monitoring has been applied when physicians have identified patients to be at a risk for problems and then have dedicated resources to these patients (e.g. , cardiotelemetry). The resources may have consisted of closer observation or special monitoring. In contrast, the monitoring used in the operating room by anesthesiologists for low-risk patients is called surveillance monitoring. All patients are monitored per the American Society of Anesthesiologists standards (e.g. , oxygen saturation, blood pressure, electrocardiogram) without exception because the operating room is in itself a factor associated with unexpected events. This monitoring may be combined with that beyond routine or surveillance monitoring that is based on a patient's individual risk profile, known as condition monitoring. Examples of condition monitoring are the use of arterial lines for patients with risk for hemodynamic instability in the ICU or operating room as well as the use of pulmonary artery catheters in patients with risk for low cardiac output failure.

Condition monitoring has been used outside of the ICU and operating room environment in care areas that are usually unmonitored, e.g. , patients with sleep apnea who are using postoperative opioids. Condition monitoring of selected patients in general care settings is similar to monitoring patients in the operating room or the ICU in that alarms have a higher probability of being actionable alarms (alarms that trigger an intervention) than the monitoring of patients not perceived to be at risk using the same static threshold alarms (Andreas H. Taenzer, M.S., M.D., unpublished data, 2011). Data are based on incidence of self-correcting decreases of pulse oximetry oxygen saturation readings less than 93% with patient surveillance in the general care settings versus  actions required by readings of less than 93% in the operating room.

In surveillance monitoring outside the operating room, a large number of patients (e.g. , all patients on one ward, in a section, or an entire hospital) are monitored, making nuisance alarms a predominant problem. In the general ward setting, the nurse-to-patient ratio is lower, and physicians are typically less readily available, making the immediate attention to alarms more difficult. In these environments, the likelihood of alarms being activated is much less than in condition monitoring or surveillance monitoring in high-risk settings such as the operating room. DeVita et al.  stated “there was concern that current technology is clinically inadequate due to a potential for high false positive or negative rates.”18 

Nuisance alarms consist of false-positive and nonactive alarms. The concept of nuisance alarms has been used in the human factor, ergonomic, and medical literature.52–55Frequent nuisance alarms lead to desensitization of personnel and discounting of alarms.52Thus, the implementation of surveillance monitoring requires a different approach to alarm variables than condition monitoring on alarm parameters in terms of static alarm settings as well as notification delay.

Although early warning scoring systems have been able to take data at low temporal resolution and predict deterioration with some success, they are demonstrably unable to catch fast-developing deteriorations that are not preindicated by some element of the patient's medical history or demographic characteristics. Furthermore, vital sign checks are by their nature interruptive; providers checking vital signs will in most cases rouse the patient, intentionally or not. As patients who were asleep or resting become more active to assist in the data collection and to interact with the provider, many of their vital signs change: respiration rate will naturally increase, subsequently increasing Spo2, and heart rate and blood pressure may also rise. This means that assessment of vital signs is not necessarily an accurate measure of patient physiology over time; providers are affecting the readings simply by obtaining them.

Continuous physiologic monitoring systems avoid the limitations of low data rates and misrepresentative readings associated with provider vital sign checks. However, most studies of continuous physiologic monitoring to date have been focused on making the case for feasibility and the value of monitoring systems in real-time detection of adverse events. Those studies that have attempted to predict deterioration in advance of adverse occurrences have not demonstrated significant clinical impact.34,38,56 

Surveillance monitoring of a large patient population must address the problem of human response to frequent alarms. Typically, alarms that are 90% accurate are usually acted upon whereas alarms that are 10% accurate are ignored.57Studies conducted in ICU settings showed that alarms are less than 10% accurate.58–60If surveillance monitoring is to be successful, the problems that cause alarms to be ignored in condition monitoring of ICU patients need to be addressed and overcome. Although training and policy adjustments are important to create an atmosphere where additional monitoring can succeed, appropriate management of alarm rates by adjusting thresholds and notification delays is the single most important factor in effective performance of the human element of surveillance monitoring systems.

Alarm thresholds may be chosen based on the underlying normal distribution of typical physiology of patients in one unit to find the optimal balance between sensitivity and specificity. More recently, an abstract by Taenzer et al.  described that the distribution of heart rates and oxygen saturation varies very little among surgical patients,∥implying that the same static alarm settings may be used for different patient groups. As has been the case with early warning scoring systems, it is likely that thresholds will be developed on an institutional basis to cope with differing patient populations and resource allocation strategies. Surveillance monitoring systems that include the ability to archive physiologic readings can be used to develop these customized thresholds with a sound statistical basis, addressing one of the main objections to widespread validation of existing early warning systems.

Notification delay is the other important concept in alarm frequency management. Appropriate delay eliminates many transient and motion artifact-generated false alarms by stipulating that an alarm condition must persist for a set amount of time before that alarm is annunciated. In a recent publication, Taenzer et al.  instituted a 15-s audio alarm delay at the bedside and an additional 15-s delay for pager annunciation, leading to a 30-s delay before a nurse would be notified by pager of violation of alarm thresholds.61Notification delay has been used as a concept to control nuisance alarms for smoke and fire alarm detectors as well as for network intrusion detection systems. It is increasingly used in ICU settings to help to cope with the problem of alarm fatigue.52In a setting such as inpatient wards where surveillance monitoring is completely new, an alarm delayed by a matter of seconds still represents a major improvement over hourly vital signs checks, and such a delay can have a large effect on nuisance alarm rates.

Filter mechanisms are another method to control alarms. Median filtering has been widely studied, using a variety of different techniques with a wide array of results. Mäkivirta et al.  used preprocessing median filtering in the ICU setting. The filter increased the fraction of true alarms from 12–49%, whereas the false alarm rate remained at 4.5 per hr per patient.62 

Together, alarm thresholds and notification delay represent points of control that an institution can use to bring an appropriate balance of sensitivity and specificity to surveillance monitoring systems. Furthermore, these controls can be adjusted frequently on a unit- or institution-wide basis without the need for staff retraining imposed by changes to manual scoring systems. This fine-grained control is critical to maintaining staff acceptance of the surveillance system while keeping sensitivity at high enough levels to affect patient safety.

Dangers and Downsides of Surveillance

Patient surveillance using biotechnology adds to currently implemented safeguards (direct interaction with patients to assess mental status and sample vital signs) and by no means should be considered to replace other monitoring means at this point. PS should not lure institutions into a false sense of security. Relying entirely on biotechnology and reassigning nurses to different tasks or lowering nurse-to-patient ratios are actions not supported by any data and are considered to be dangerous by the authors at this time.

Premature implementation of systems without fine-grained control of alarms will lead to alarm fatigue by first responders (registered nurses) and lack of acceptance.52In scenarios where the nursing staff is not convinced of the benefits of surveillance, patient acceptance of continuous monitoring usually declines (as experienced by the Dartmouth Patient Surveillance Group). As is commonly the case with introduction of new systems, PS needs to be implemented with support and education for those using and exposed to the system.#

Large gaps still exist with current technology, and we can only improve on the current FTR problem, not solve it. Institutions need to be aware that patients who are being transported through hospitals, waiting for diagnostic tests in hallways or waiting areas, continue to be unmonitored until all surveillance is completely wireless with access everywhere, with complete patient acceptance and fail-free.

Continuous patient monitoring with alarm settings designed for a collective group of patients will benefit a group of patients. This is important to emphasize because it is a departure from what is most common practice in medicine: to tailor and individualize care based on a patient's specific needs. Therefore, PS cannot be assumed to be the sole solution for the detection of physiologic deteriorations of all patients in the general ward setting. It provides a broad underlying safety net for those currently unmonitored, and individual monitoring for patients may be added.

Cost and benefit of surveillance must be carefully considered. Institutions or patient care areas that have no adverse events will likely derive no benefit from PS. Cost effectiveness of surveillance will depend on many factors too complex to discuss here in detail. With cost control in health care of utmost importance, deploying systems with considerable cost needs to be carefully weighed against their benefit. In addition, at times safeguard systems develop autonomy of their own and are not just a subject of cost-benefit considerations, but become ethically mandatory (code teams).

Finally, one has to be aware that continuous monitoring may be prompting unnecessary responses, disturbing patients, distracting and interrupting nurses in their work, and leading to harmful interventions as discussed in other settings of monitoring and screening.63–65Many quality improvement interventions are changes to complex systems and are subject to unintended consequences as a side effect of change as pointed out by Reason.66 

Initial Work on Patient Surveillance Systems

Recently, the effect of implementation of a PS system based on pulse oximetry with wireless nursing notification (Patient SafetyNet™; Masimo, Irvine, CA) on transfers to the ICU and rescue events was described.61Detection of deterioration in this model occurs at an earlier time point than in other described models and goes beyond the optimization of RRTs because the goal is to prompt intervention even before RRT activation becomes necessary. Indeed, a decrease of MET activations from 3.4-1.2 per 1,000 patient discharges (and ICU transfers from 5.2-2.9 per 1,000 patient days) (fig. 2) was shown.61In an accompanying editorial,67Abenstein and Narr remarked that “The importance of the Dartmouth study is introduction of a rudimentary electronic decision support system to a traditionally unmonitored ward and its associated results ... These systems will be able to filter those patients who require intervention from those who do not.”67 

Fig. 2.  Rescues per 1,000 patient days before and after implementation of a patient surveillance system. PSS Unit = surgical unit with Patient Surveillance System (PSS); Comp. Unit 1 = Comparison Unit 1; Comp. Unit 2 = Comparison Unit 2.

Fig. 2.  Rescues per 1,000 patient days before and after implementation of a patient surveillance system. PSS Unit = surgical unit with Patient Surveillance System (PSS); Comp. Unit 1 = Comparison Unit 1; Comp. Unit 2 = Comparison Unit 2.

Close modal

After a decade of research and work on RRS we believe we will enter a decade of research on PS. Much work has been and continues to be done on the efferent limb of RRS; this needs to be complemented by work on the afferent limb. In this review we have argued that continuous patient monitoring should be the next step for early intervention in the FTR field.

Research in PS needs to take into account that this is a multidisciplinary effort that will only work with support from all stakeholders in the process, including patients, nurses, physicians, and administration. Similar to the evaluation of the effect of the response component of RRTs, the success of continuous patient monitoring will be influenced by local hospital culture. We are just beginning to understand why similar process interventions work in some institutions but not others. Therefore, research efforts need to expand beyond the monitoring modalities to the issues of successful deployment: What training should be offered and received by clinicians (nurses and physicians), biotechnologists, and administrators? What structures and processes hinder or facilitate success? Although system configurations will likely continue to be customized to the needs of individual institutions, can a set of best practices for system deployment be described and validated?

Regarding continuous monitoring systems, much work needs to be done in establishing which parameters and what combination of them are the optimal physiologic variables to monitor; if variables should be combined to a summarized score or if they should be considered individually in triggering alarms; if monitoring should be done with static alarm limits, trends, or a combination; if PS should be used universally or only for population groups at risk; if patient monitoring is cost-effective; and if monitoring with automated technology is better or worse than monitoring with additional staff.

As technology or the application of existing technology to new areas progresses, respiratory rate monitoring, transcutaneous carbon dioxide, and hemoglobin are among the variables likely to be added to surveillance based on current hardware development trends. Ongoing research will doubtlessly identify additional physiologic measures. In general, as parameters are added, we need to be aware of the effect of doing so. Adding variables with a specificity of 90% will result in an overall system specificity of 81% for two, 73% for three, and 65% for four variables. Doing the same for a specificity of 95% will yield 90%, 85%, and 81% overall system specificity, respectively. To find an optimized specificity and sensitivity for each parameter as well as for the system as a whole while controlling for nuisance alarms will require much work and likely would be best done in simulation on an available, representative dataset before implementation and testing in clinical settings. To control nuisance alarms, specificity needs to be high, most likely at the cost of sensitivity. As more parameters are added, this balance will become increasingly difficult. The nature of monitoring data also makes data mining and machine learning techniques very attractive. This class of analysis methods can extract time- and frequency-domain relationships between physiologic data and deterioration that are not necessarily evident in statistical approaches nor to standard clinical assessment. A wide range of machine-learning techniques has been applied to clinical data in the past, including fuzzy logic,68neural networks,69,70patient-specific trained classifiers,71wavelets,72syntactic analysis,73Bayesian networks,74and Markovian models.75None of these studies had data on non-ICU general ward patients who underwent continuous monitoring. But the same principles apply and these techniques potentially could introduce smart alarm control.

Monitoring the effect of quality improvement projects in the area of patient safety is challenging. Measurable outcome variables depend on many factors, such as patients, providers, and organizations and their interplay, which remain poorly understood.76,77Unanticipated consequences such as new problems,78,79failure, or uncertainty to reach the desired goal have been common when complex systems have altered with the goal of improving them.80,81 

The outcomes of most interest are rare (mortality, codes, rescue events, and ICU transfers); the criteria for some outcomes are ill-defined, changing, or depend on other factors (when to transfer a patient to the ICU, bed availability); and the denominator, the patient population, is ever changing and so is the associated risk profile. Because of these challenges, the evaluation of new techniques or strategies for patient surveillance is difficult. The measurement of effect requires enormous power and sample sizes because of the rarity of the relevant outcome variables. Similar to other areas of patient safety and quality improvement, it will be easier to study effect with process control charts or before-and-after studies, rather than using methods that establish a very high level of evidence such as cluster randomized controlled trials.

The effect of PS needs to be measured on multiple levels: satisfaction of patients; patient outcomes (as difficult as that may be, as discussed); satisfaction of users; and financial ramifications. Some of theses outcomes can follow examples of related interventions, and others may require new approaches. Patient and family satisfaction likely can be measured with traditional measures such as Likert or Visual Analog Scales. Patient outcomes may be analyzed with similar means as RRT interventions with some caveats. Although rescue calls in traditional evaluations of the efferent limb of RRTs often increase as a sign of a functional and mature system, in PS a reduced number of calls may signify an intervention that occurs so early in a potential deterioration process that a rescue event is not necessary. Transfers to the ICU are considered not very reliable indicators of the success of RRT because the decision to transfer is affected by too many other variables (e.g. , available bed space, changing indications for transfers, varying practices among institutions). In PS, a change in ICU transfers may further indicate a change in culture: physicians may feel comfortable keeping patients on a regular ward because they are now continuously monitored.

The satisfaction of those who use the system is also an important component of assessing PS. Parameters such as nursing satisfaction as well as nursing staff retention rates are easy to track and are meaningful, although isolating the effects of any individual factor on staffing satisfaction is challenging. The addition of PS-specific probes to satisfaction surveys is one potential technique.

Finally, in times of financial constraints in health care, we are obligated to analyze the cost-effectiveness of new systems when we plan to introduce them into medicine (even though this has been rarely done in the past and established systems such as a code team are rarely questioned).

In summary, tremendous opportunities for improvement of patient safety and research exist in the field of PS—a truly new and exciting area in medicine.

The authors thank George T. Blike, M.D., Professor of Anesthesiology, Dartmouth Medical School and Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire.

Silber JH, Williams SV, Krakauer H, Schwartz JS: Hospital and patient characteristics associated with death after surgery. A study of adverse occurrence and failure to rescue. Med Care 1992; 30:615–29
Committee on Quality of Health Care in America, Institute of Medicine, ed. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academy Press, 2001
Committee on Quality of Health Care in America, Institute of Medicine, ed. Crossing the Quality Chasm
Washington, DC
National Academy Press
Silber JH, Rosenbaum PR, Schwartz JS, Ross RN, Williams SV: Evaluation of the complication rate as a measure of quality of care in coronary artery bypass graft surgery. JAMA 1995; 274:317–23
Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K: Nurse-staffing levels and the quality of care in hospitals. N Engl J Med 2002; 346:1715–22
Ghaferi AA, Birkmeyer JD, Dimick JB: Variation in hospital mortality associated with inpatient surgery. N Engl J Med 2009; 361:1368–75
Buist M, Bernard S, Nguyen TV, Moore G, Anderson J: Association between clinically abnormal observations and subsequent in-hospital mortality: A prospective study. Resuscitation 2004; 62:137–41
Schein RM, Hazday N, Pena M, Ruben BH, Sprung CL: Clinical antecedents to in-hospital cardiopulmonary arrest. Chest 1990; 98:1388–92
Hillman KM, Bristow PJ, Chey T, Daffurn K, Jacques T, Norman SL, Bishop GF, Simmons G: Antecedents to hospital deaths. Intern Med J 2001; 31:343–8
Sax FL, Charlson ME: Medical patients at high risk for catastrophic deterioration. Crit Care Med 1987; 15:510–5
McQuillan P, Pilkington S, Allan A, Taylor B, Short A, Morgan G, Nielsen M, Barrett D, Smith G, Collins CH: Confidential inquiry into quality of care before admission to intensive care. BMJ 1998; 316:1853–8
Aiken LH, Clarke SP, Sloane DM, Sochalski J, Silber JH: Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. JAMA 2002; 288:1987–93
Carayon P, Gürses AP: A human factors engineering conceptual framework of nursing workload and patient safety in intensive care units. Intensive Crit Care Nurs 2005; 21:284–301
Pronovost PJ, Angus DC, Dorman T, Robinson KA, Dremsizov TT, Young TL: Physician staffing patterns and clinical outcomes in critically ill patients: A systematic review. JAMA 2002; 288:2151–62
Buist MD, Jarmolowski E, Burton PR, Bernard SA, Waxman BP, Anderson J: Recognising clinical instability in hospital patients before cardiac arrest or unplanned admission to intensive care. A pilot study in a tertiary-care hospital. Med J Aust 1999; 171:22–5
Ranji SR, Auerbach AD, Hurd CJ, O'Rourke K, Shojania KG: Effects of rapid response systems on clinical outcomes: Systematic review and meta-analysis. J Hosp Med 2007; 2:422–32
Winters BD, Pham JC, Hunt EA, Guallar E, Berenholtz S, Pronovost PJ: Rapid response systems: A systematic review. Crit Care Med 2007; 35:1238–43
Calzavacca P, Licari E, Tee A, Egi M, Haase M, Haase-Fielitz A, Bellomo R: A prospective study of factors influencing the outcome of patients after a Medical Emergency Team review. Intensive Care Med 2008; 34:2112–6
DeVita MA, Smith GB, Adam SK, Adams-Pizarro I, Buist M, Bellomo R, Bonello R, Cerchiari E, Farlow B, Goldsmith D, Haskell H, Hillman K, Howell M, Hravnak M, Hunt EA, Hvarfner A, Kellett J, Lighthall GK, Lippert A, Lippert FK, Mahroof R, Myers JS, Rosen M, Reynolds S, Rotondi A, Rubulotta F, Winters B: “Identifying the hospitalised patient in crisis”—a consensus conference on the afferent limb of rapid response systems. Resuscitation 2010; 81:375–82
Litvak E, Pronovost PJ: Rethinking rapid response teams. JAMA 2010; 304:1375–6
Bedell SE, Deitz DC, Leeman D, Delbanco TL: Incidence and characteristics of preventable iatrogenic cardiac arrests. JAMA 1991; 265:2815–20
Bobay KL, Fiorelli KL, Anderson AJ: Failure to rescue: A preliminary study of patient-level factors. J Nurs Care Qual 2008; 23:211–5
Kause J, Smith G, Prytherch D, Parr M, Flabouris A, Hillman K, Intensive Care Society (UK), Australian and New Zealand Intensive Care Society Clinical Trials Group: A comparison of antecedents to cardiac arrests, deaths and emergency intensive care admissions in Australia and New Zealand, and the United Kingdom—the ACADEMIA study. Resuscitation 2004; 62:275–82
Intensive Care Society (UK)
Australian and New Zealand Intensive Care Society Clinical Trials Group
Jacques T, Harrison GA, McLaws ML, Kilborn G: Signs of critical conditions and emergency responses (SOCCER): A model for predicting adverse events in the inpatient setting. Resuscitation 2006; 69:175–83
Franklin C, Mathew J: Developing strategies to prevent inhospital cardiac arrest: Analyzing responses of physicians and nurses in the hours before the event. Crit Care Med 1994; 22:244–47
Galhotra S, DeVita MA, Simmons RL, Dew MA, Members of the Medical Emergency Response Improvement Team (MERIT) Committee: Mature rapid response system and potentially avoidable cardiopulmonary arrests in hospital. Qual Saf Health Care 2007; 16:260–5
Members of the Medical Emergency Response Improvement Team (MERIT) Committee
Goldhill DR, McNarry AF: Physiological abnormalities in early warning scores are related to mortality in adult inpatients. Br J Anaesth 2004; 92:882–4
Subbe CP, Kruger M, Rutherford P, Gemmel L: Validation of a modified Early Warning Score in medical admissions. QJM 2001; 94:521–6
Morrice A, Simpson HJ: Identifying level one patients. A cross-sectional survey on an in-patient hospital population. Intensive Crit Care Nurs 2007; 23:23–32
Kho A, Rotz D, Alrahi K, Cárdenas W, Ramsey K, Liebovitz D, Noskin G, Watts C: Utility of commonly captured data from an EHR to identify hospitalized patients at risk for clinical deterioration. AMIA Annu Symp Proc 2007; 404–8
Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, Esmonde L, Goldhill DR, Parry GJ, Rashidian A, Subbe CP, Harvey S: Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 2007; 33:667–79
Adshead N, Thomson R: Use of a paediatric early warning system in emergency departments. Emerg Nurse 2009; 17:22–5
Garcea G, Gouda M, Hebbes C, Ong SL, Neal CP, Dennison AR, Berry DP: Predictors of severity and survival in acute pancreatitis: Validation of the efficacy of early warning scores. Pancreas 2008; 37:54–61
McCabe A, Duncan H, Heward Y: Paediatric early warning systems: Where do we go from here? Paediatr Nurs 2009; 21:14–7
Larson TS, Brady WJ: Electrocardiographic monitoring in the hospitalized patient: A diagnostic intervention of uncertain clinical impact. Am J Emerg Med 2008; 26:1047–55
Hollander JE, Valentine SM, McCuskey CF, Brogan GX Jr: Are monitored telemetry beds necessary for patients with nontraumatic chest pain and normal or nonspecific electrocardiograms? Am J Cardiol 1997; 79:1110–1
Snider A, Papaleo M, Beldner S, Park C, Katechis D, Galinkin D, Fein A: Is telemetry monitoring necessary in low-risk suspected acute chest pain syndromes? Chest 2002; 122:517–23
Schull MJ, Redelmeier DA: Continuous electrocardiographic monitoring and cardiac arrest outcomes in 8,932 telemetry ward patients. Acad Emerg Med 2000; 7:647–52
Watkinson PJ, Barber VS, Price JD, Hann A, Tarassenko L, Young JD: A randomised controlled trial of the effect of continuous electronic physiological monitoring on the adverse event rate in high risk medical and surgical patients. Anaesthesia 2006; 61:1031–9
Smith AF, Wood J: Can some in-hospital cardio-respiratory arrests be prevented? A prospective survey. Resuscitation 1998; 37:133–7
Goldhill DR, White SA, Sumner A: Physiological values and procedures in the 24 h before ICU admission from the ward. Anaesthesia 1999; 54:529–34
Bristow PJ, Hillman KM, Chey T, Daffurn K, Jacques TC, Norman SL, Bishop GF, Simmons EG: Rates of in-hospital arrests, deaths and intensive care admissions: The effect of a medical emergency team. Med J Aust 2000; 173:236–40
Hillman K, Chen J, Cretikos M, Bellomo R, Brown D, Doig G, Finfer S, Flabouris A, MERIT study investigators: Introduction of the medical emergency team (MET) system: A cluster-randomised controlled trial. Lancet 2005; 365:2091–7
MERIT study investigators
Priestley G, Watson W, Rashidian A, Mozley C, Russell D, Wilson J, Cope J, Hart D, Kay D, Cowley K, Pateraki J: Introducing Critical Care Outreach: A ward-randomised trial of phased introduction in a general hospital. Intensive Care Med 2004; 30:1398–404
Buist MD, Moore GE, Bernard SA, Waxman BP, Anderson JN, Nguyen TV: Effects of a medical emergency team on reduction of incidence of and mortality from unexpected cardiac arrests in hospital: Preliminary study. BMJ 2002; 324:387–90
Bellomo R, Goldsmith D, Uchino S, Buckmaster J, Hart G, Opdam H, Silvester W, Doolan L, Gutteridge G: Prospective controlled trial of effect of medical emergency team on postoperative morbidity and mortality rates. Crit Care Med 2004; 32:916–21
DeVita MA, Braithwaite RS, Mahidhara R, Stuart S, Foraida M, Simmons RL, Medical Emergency Response Improvement Team (MERIT): Use of medical emergency team responses to reduce hospital cardiopulmonary arrests. Qual Saf Health Care 2004; 13:251–4
Medical Emergency Response Improvement Team (MERIT)
McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M: Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev. 2007; CD005529
Galhotra S, DeVita MA, Simmons RL, Schmid A, members of the Medical Emergency Response Improvement Team (MERIT) Committee: Impact of patient monitoring on the diurnal pattern of medical emergency team activation. Crit Care Med 2006; 34:1700–6
members of the Medical Emergency Response Improvement Team (MERIT) Committee
Jones D, Bates S, Warrillow S, Opdam H, Goldsmith D, Gutteridge G, Bellomo R: Circadian pattern of activation of the medical emergency team in a teaching hospital. Crit Care 2005; 9:303–6
Jones D, Bellomo R, Bates S, Warrillow S, Goldsmith D, Hart G, Opdam H: Patient monitoring and the timing of cardiac arrests and medical emergency team calls in a teaching hospital. Intensive Care Med 2006; 32:1352–6
Tee A, Calzavacca P, Licari E, Goldsmith D, Bellomo R: Bench-to-bedside review: The MET syndrome-the challenges of researching and adopting medical emergency teams. Crit Care 2008; 12:205
Graham KC, Cvach M: Monitor alarm fatigue: Standardizing use of physiological monitors and decreasing nuisance alarms. Am J Crit Care 2010; 19:28–34; quiz 35
Woods D: The alarm problem and directed attention in dynamic fault management. Ergonomics 1995; 38:2371–93
Xiao Y, Seagull FJ, Nieves-Khouw F, Barczak N, Perkins S: Organizational-historical analysis of Systems, Man and Cybernetics, Part A: Systems and Humans. IEEE Transactions 2004; 34:772–8
Grundgeiger T, Sanderson P: Interruptions in healthcare: Theoretical views. Int J Med Inform 2009; 78:293–307
Amantini A, Fossi S, Grippo A, Innocenti P, Amadori A, Bucciardini L, Cossu C, Nardini C, Scarpelli S, Roma V, Pinto F: Continuous EEG-SEP monitoring in severe brain injury. Neurophysiol Clin 2009; 39:85–93
Bliss JP, Dunn MC: Behavioural implications of alarm mistrust as a function of task workload. Ergonomics 2000; 43:1283–300
Tsien CL, Fackler JC: Poor prognosis for existing monitors in the intensive care unit. Crit Care Med 1997; 25:614–9
Schoenberg R, Sands DZ, Safran C: Making ICU alarms meaningful: A comparison of traditional versus  trend-based algorithms. Proc AMIA Symp 1999; 379–83
Lawless ST: Crying wolf: False alarms in a pediatric intensive care unit. Crit Care Med 1994; 22:981–5
Taenzer AH, Pyke JB, McGrath SP, Blike GT: Impact of pulse oximetry surveillance on rescue events and intensive care unit transfers: A before-and-after concurrence study. Anesthesiology 2010; 112:282–7
Mäkivirta A, Koski E, Kari A, Sukuvaara T: The median filter as a preprocessor for a patient monitor limit alarm system in intensive care. Comput Methods Programs Biomed 1991; 34:139–44
Welch HG: Overdiagnosis and mammography screening. BMJ 2009; 7:1425
Welch HG, Black WC: Overdiagnosis in cancer. JNCI 2010; 102:605–613
Welch HG, Woloshin S, Schwartz LM: The sea of uncertainty surrounding ductal carcinoma in situ -the price of screening mammography. JNCI 2008; 100:228–9
Reason J. Managing the Risks of Organisational Accidents. Aldershot: Ashgate Publishing, 1997
Ashgate Publishing
Abenstein JP, Narr BJ: An ounce of prevention may equate to a pound of cure: Can early detection and intervention prevent adverse events? Anesthesiology 2010; 112:272–3
Becker K, Thull B, Käsmacher-Leidinger H, Stemmer J, Rau G, Kalff G, Zimmermann HJ: Design and validation of an intelligent patient monitoring and alarm system based on a fuzzy logic process model. Artif Intell Med 1997; 11:33–53
Lewicke AT, Sazonov ES, Corwin MJ, Schuckers SAC: Reliable determination of sleep versus  wake from heart rate variability using neural networks. In: Neural Networks, 2005. IJCNN 2005 Proceedings. 2005 IEEE International Joint Conference, 2394–9
Parmanto B, Deneault LG, Denault AY: Detection of hemodynamic changes in clinical monitoring by time-delay neural networks. Int J Med Inform 2001; 63:91–9
Qu H, Gotman J: A patient-specific algorithm for the detection of seizure onset in long-term EEG monitoring: Possible use as a warning device. IEEE Trans Biomed Eng 1997; 44:115–22
Saeed M, Mark R: A novel method for the efficient retrieval of similar multiparameter physiologic time series using wavelet-based symbolic representations. AMIA Annu Symp Proc 2006; 679–83
Saxena SC, Tandon VK, Gupta S: Patient monitoring in ICCU using syntactic approach. In: Engineering in Medicine and Biology Society, 1995 and 14th Conference of the Biomedical Engineering Society of India. An International Meeting, Proceedings of the First Regional Conference. IEEE 1995; 1–2
Sadeghi S, Barzi A, Sadeghi N, King B: A Bayesian model for triage decision support. Int J Med Inform 2006; 75:403–11
Chua CP, Garvey J, Redmond S, Heneghan C, McNicholas WT: Towards Automated Sleep State Estimation using a Holter-Oximeter. Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE 2007; 3998–4001
Auerbach AD, Landefeld CS, Shojania KG: The tension between needing to improve care and knowing how to do it. N Engl J Med 2007; 357:608–13
Pronovost PJ: Tracking progress in patient safety: An elusive target. JAMA 2006 8; 296:696–9
Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL: Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293:1197–203
Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC, Bayir H, Orr RA: Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116:1506–12
Wachter RM, Pronovost PJ: The 100,000 Lives Campaign: A scientific and policy review. Jt Comm J Qual Patient Saf 2006; 32:621–7
Laine C, Goldman L, Soukup JR, Hayes JG: The impact of a regulation restricting medical house staff working hours on the quality of patient care. JAMA 1993; 269:374–8