Automated Continuous Noninvasive Ward Monitoring: Validation of Measurement Systems Is the Real Challenge

About half of all adverse events in hospitalized patients occur on the general care ward.¹  However, acute cardiorespiratory events do not occur out of the blue. Up to 60% of patients have at least one or more abnormal vital signs as early as 4 to 6 h before a cardiac arrest.²  Early detection of changes in cardiorespiratory physiology therefore is critical for preventative or therapeutic measures to be effective and to eventually improve patient outcomes. Automated continuous noninvasive ward monitoring may be a promising approach for improving surveillance of general care ward patients at risk for cardiorespiratory events. With numerous continuous ward monitoring devices flooding the healthcare market, it becomes crucial to rigorously test their measurement performance and validate them in different clinical settings before proceeding to larger clinical trials (fig. 1).

In this context, Breteler et al.³  need to be commended for their clinical study investigating the measurement performance of novel wireless sensors for heart rate and respiratory rate monitoring published in this issue of Anesthesiology. They compared heart rate and respiratory rate measured using two wearable patch sensors, a bed-based system, and a patient-worn monitor against current intensive care unit standard bedside monitoring (multiparameter monitor) in 25 high-risk postsurgery patients after admission to a step-down unit.³  The authors used several statistical tests to compare these noninvasive sensors with the bedside monitor used as the reference method. The primary outcome was the absolute agreement of heart rate and respiratory rate measurements between the test methods and the reference method. By using Bland–Altman analysis accounting for repeated measurements within subjects, the authors assessed the mean of the differences and the limits of agreement. A clinically acceptable agreement was defined as ±10%, ±5 beats/min, or ±3 breaths/min in comparison with the reference method. Although the mean of the differences (often termed “bias”) was low, the relatively wide limits of agreement indicate that the precision of agreement of the test methods in comparison with the reference method still need to be improved.

In addition, as a secondary endpoint, the clinical relevance of measurement differences was assessed using error-grid analysis that provides information about the consequences of incorrect treatment decisions triggered by measurements with the test method. An error grid is a scatter plot with the test method on one axis and the reference method on the other. Different risk zones within the error grid indicate the risk for the patient to be harmed by inappropriate treatment due to measurements with the test methods in comparison with the “true” reference measurement. Visual inspection of the error grid and calculation of the absolute and relative number of measurements within the risk zones enable the clinical relevance of measurement differences to be assessed. The risk zones need to be defined differently for each measured physiologic variable and should be based on pathophysiologic rationale or expert consensus. Error grid analysis is established to validate methods for the measurement of blood glucose⁴  and blood pressure⁵  but not for heart rate or respiratory rate. In the absence of established error-grid risk zones for heart rate or respiratory rate, the authors chose to define the risk zones based on the cutoff boundaries of the Modified Early Warning Score.⁶  Because of the huge amount of data (the authors analyzed more than 700 h of monitoring), the authors used color coding to illustrate superimposing pairs of measurements.

Furthermore, because the ability to follow changes in vital signs over time is an important feature of ward monitoring devices Breteler et al.³  also analyzed the ability of each of the ward monitoring devices to follow changes in vital signs over time using four quadrant plots and concordance rates using an exclusion zone (clinically irrelevant changes) of 1 beat/min for heart rate and 1 breath/min for respiratory rate. The concordance rate is the ratio (percentage) of measurements assessed by the test method and the reference method that change in the same direction (decrease or increase) to the sum of all changes. To describe the technical performance and signal stability, the authors also quantified data loss with any of the devices and demonstrated that 90% of episodes with data loss did not exceed 15 min.

This intricate analysis of four continuous vital signs monitoring devices is timely and of substantial importance to both the clinician and the data scientist. Although current practice patterns show that most patients are still being monitored on an intermittent “spot check” basis on hospital wards, there are already data showing that continuous ward monitoring for early detection of changes in vital signs may improve patient outcome.^7,8  However, before the use of ward monitoring systems in large-scale interventional outcome studies or their implementation across hospital systems, several challenges need to be addressed.^9,10  Ward monitoring systems should use small wireless sensors, be easy to use, filter artifacts to decrease alarm fatigue, have seamless integration with most electronic patient records along with artificial intelligence platforms, and allow real-time synchronization of vital sign data. Additionally, sensors should be able to record and process different vital signs to minimize the number of different sensors that need to be attached to the patient. However, most importantly, healthcare providers need to have confidence in using innovative ward monitoring systems. This confidence can only be built if ward monitoring systems provide vital sign readings with “clinically acceptable” accuracy and precision.

Therefore, meticulously performed validation studies such as this work by Breteler et al.³  are of crucial importance for future ward monitoring research and a meaningful implementation in daily clinical practice. However, we are still missing guidelines or at least consensus statements on how validation studies of ward monitoring systems should be performed and analyzed. Such guidelines could include a checklist outlining how to perform and report validation studies—as for example the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)¹¹ —and even more challenging a priori definitions of “clinically acceptable” measurement performance based on different statistical tests. In the absence of any guidelines on how to investigate and validate innovative ward monitoring systems, authors need to carefully consider which statistical tests to use to analyze measurement performance and data quality and how to interpret the findings. After meticulous validation, the impact of continuous ward monitoring on patient-centered outcomes needs to be investigated in large interventional clinical trials.

In summary, before their use in interventional outcomes research studies or clinical practice, automated continuous ward monitoring devices need to be meticulously validated regarding their measurement performance. A critical appraisal of the measurement performance and of the clinical impact are prerequisites for a sustainable long-term implementation of automated continuous ward monitoring eventually allowing a change in the culture of patient surveillance on the general care ward.