Impact of the Choice of Risk Model for Identifying Low-risk Patients Using the 2014 American College of Cardiology/American Heart Association Perioperative Guidelines

MORE than 200 million major surgical procedures are performed worldwide each year.¹  Eight percent of patients undergoing major noncardiac surgery experience myocardial injury.²  Perioperative mortality is the third leading cause of death in the United States,³  and cardiac complications cause one third of all perioperative deaths.⁴  Current clinical practice guidelines endorsed by the American College of Cardiology and the American Heart Association recommends a stepwise approach to the cardiac assessment of patients scheduled to undergo noncardiac surgery. According to these guidelines, patients whose estimated perioperative risk of major adverse cardiac event is less than 1% can proceed to surgery without undergoing further cardiac testing.⁵  The use of a validated risk-prediction tool to predict the risk of perioperative major adverse cardiac event is a Class IIa recommendation.⁵  Three risk models are recommended by the American College of Cardiology/American Heart Association guidelines for predicting the risk of major adverse cardiac event: (1) the Revised Cardiac Risk Index⁶ ; (2) The American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator;⁷  and (3) the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest (myocardial infarction or cardiac arrest) calculator.⁸ 

Risk stratification, using one of these three prediction models to separate low-risk (less than 1%) from elevated-risk (greater than or equal to 1%) patients, is a key decision point in the American College of Cardiology/American Heart Association perioperative guideline. In theory, prediction models allow clinicians to more accurately estimate risk rather than relying on clinical judgement alone. However, while drugs and therapies must undergo U.S. Food and Drug Administration approval prior to clinical use, the decision to incorporate clinical prediction models in patient care is based on peer-review alone. The American College of Cardiology/American Heart Association guideline does not specify which model to use for risk stratification. Twenty years ago, Iezzoni published a seminal paper The Risks of Risk Adjustment showing that hospital quality is partly a function of the choice of the risk adjustment model.⁹  Our goal in this exploratory study is to determine how often the three prediction models recommended in the American College of Cardiology/American Heart Association perioperative guidelines agree on the classification of patients scheduled to undergo major noncardiac surgery as low risk versus elevated risk. We will use data from the American College of Surgeons National Surgical Quality Improvement Program database to calculate the predicted risk of major adverse cardiac event for individual patients using each of these three prediction models. We expect to find that predicted patient risk is also, in part, a function of the choice of prediction models. Our findings may help inform the process of developing best practices for incorporating prediction models in clinical practice guidelines.

This analysis was based on data from the American College of Surgeons National Surgical Quality Improvement Program participant use data file for patients undergoing noncardiac surgery in 2012. The database includes comprehensive clinical information on patient demographics, preoperative risk factors, and 30-day postoperative mortality and complications.¹⁰  We used the National Surgical Quality Improvement Program definition of major adverse cardiac event as cardiac arrest or myocardial infarction (appendix).⁷  Myocardial infarction is defined by the National Surgical Quality Improvement Program as either (1) “documentation of electrocardiogram changes indicative of acute myocardial infarction (ST elevation > 1 mm in two contiguous leads, new left bundle branch block, or new Q waves in two or more contiguous leads)”; (2) “new elevation of troponin greater than three times the upper level of the reference range in the setting of suspected myocardial ischemia”; or (3) “physician diagnosis of myocardial infarction.¹⁰ ” These data elements are prospectively collected by trained Surgical Clinical Reviewers using chart abstraction and through extraction from hospitals’ information systems.¹⁰  Data quality is verified using auditing comparing chart data with data submitted by participating sites. Systematic case sampling is used to avoid bias in case selection.¹⁰ 

We first identified 71,575 patients in the American College of Surgeons National Surgical Quality Improvement Program participant use data file who underwent one of the major noncardiac surgical procedures tracked by the Centers for Disease Control and Prevention National Healthcare Safety Network, excluding very low-risk procedures (e.g., appendectomy, breast procedures, pacemakers), and cardiac cases. We excluded cases from hospitals submitting a minimal data set which did not include data elements for history of myocardial infarction and stroke; these data elements are necessary to calculate the risk of major adverse cardiac event using the Revised Cardiac Risk Index (fig. 1). Procedures were then divided into high-risk or low/intermediate-risk surgeries based on the approach used in the derivation of the Revised Cardiac Risk Index ⁶  and the procedure stratification described in the American College of Cardiology/American Heart Association 2007 guidelines.¹¹  We then constructed a stratified sample of 10,000 records in which we over-sampled patients undergoing high-risk procedures because we assumed that surgeons are more likely to obtain cardiology consultation for higher-risk surgeries. To construct our analytic data set, we excluded American Society of Anesthesiologists (ASA) physical status V and emergency cases (emergency cases do not undergo further cardiac testing in the American College of Cardiology/American Heart Association guidelines), records with missing ASA physical status or functional status, and records missing American College of Surgeons National Surgical Quality Improvement Program risk of major adverse cardiac event (fig. 1). For the purpose of our analysis, we assumed that patients with no recorded value of serum creatinine had normal renal function, unless they were coded as having renal failure or dialysis (n = 880). A priori, we limited the size of the analytic data set to 10,000 observations so that the number of inquiries submitted to the web-based risk calculator did not interfere with the ability of clinicians to use the American College of Surgeons risk calculator for clinical purposes. We chose to reduce our sample size by including about 10% of the intermediate- or low-risk surgery cases, while retaining most of the high-risk procedures. This decision was driven by our assumption that the American College of Cardiology/American Heart Association algorithm is more often used by clinicians for patients undergoing higher-risk surgeries, and that our findings would thus have more clinical relevance when applied to a population that reflected the clinical use of the American College of Cardiology/American Heart Assocation guideline. We did, however, perform a sensitivity analysis (described in “Statistical Analysis”) to determine whether our findings were altered by our sampling strategy.

The risk of major adverse cardiac event was calculated for the Revised Cardiac Risk Index and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest models using published coefficients.^8,12  We submitted the records in our analytic data set to the publicly available American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator website to obtain risk estimates based on the American College of Surgeons National Surgical Quality Improvement Program prediction model.¹³  The risk of major adverse cardiac event predicted by the American College of Surgeons National Surgical Quality Improvement Program and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest prediction tools are continuous measures, whereas the Revised Cardiac Risk Index risk of major adverse cardiac event can only take on one of four discrete values: 0.4, 0.9, 6.6, and 11%.

The goal of this study was to examine the degree of agreement among these three risk-prediction tools. We performed two analyses. In the first analysis, we quantified the agreement in the patient-level risk estimates of major adverse cardiac event between each pair of the three risk-prediction tools using the intraclass correlation coefficient: (1) Revised Cardiac Risk Index versus National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest; (2) Revised Cardiac Risk Index versus American College of Surgeons National Surgical Quality Improvement Program; and (3) National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest versus American College of Surgeons National Surgical Quality Improvement Program. The intraclass correlation coefficient measures the interrater reliability and reflects the agreement between two risk-prediction scores (i.e., Revised Cardiac Risk Index vs. American College of Surgeons National Surgical Quality Improvement Program) on the risk of major adverse cardiac event for the same sample of patients. The intraclass correlation coefficient is based on a two-way mixed effects model, where the dependent variable y is the predicted risk of major adverse cardiac event, the risk score S is an indicator variable specifying the risk score (i.e., Revised Cardiac Risk Index vs. National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest), patient is specified as a random effect P, and ε is the error term:

The intraclass correlation coefficient is calculated using¹⁴ :

where is the variance of the fixed effect indicating which score is being used, and is the variance of the error term.

If the scores (e.g., Revised Cardiac Risk Index and American College of Surgeons National Surgical Quality Improvement Program) are in close agreement on the value of the of the calculated patient risk, then and , and the intraclass correlation coefficient will be close to 1. Essentially, the intraclass correlation coefficient represents the signal-to-noise ratio.¹⁵  The closer the intraclass correlation coefficient is to “1,” the stronger the level of agreement between the 2 score. Intraclass correlation coefficient values less than 0.4 suggest poor reliability, between 0.4 to 0.59 represent fair reliability, between 0.60 and 0.74 good, and between 0.75 and 1.00 excellent reliability.¹⁶ 

Since the key decision point in the American College of Cardiology/American Heart Association algorithm is to separate patients into low-risk (less than 1%) and high-risk (greater than or equal to 1%) groups, we performed a second analysis in which we examined the extent to which the three risk-prediction tools agreed on the classification of patients into low- and high-risk groups. Patient records were first dichotomized into low risk of major adverse cardiac event (less than 1%) versus high risk of major adverse cardiac event (greater than or equal to 1%), and then kappa analysis was used to examine the level of agreement for all three risk models. Bootstrapping was used to construct 95% CI around the kappa statistic. Using the kappa statistic, values less than 0 suggest poor agreement, 0.00 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect agreement.¹⁷ 

We performed three sensitivity analyses. First, we repeated our analyses after excluding observations with missing creatinine values: complete case analysis. Second, we performed weighted kappa analysis to examine the agreement between the three risk-prediction tools, using the STATA-defined weight w. In the third sensitivity analysis, we over-sampled observations for intermediate- and low-risk surgeries to create an analytic sample that more closely resembles the ratio of high-risk to intermediate/low-risk surgeries (1:20) in original analytic sample (fig. 1). We created 10 bootstrap samples of the observations in the intermediate/low-risk surgery patients using sampling with replacement (62,060) and appended this to the sample of high-risk surgery patients (2,809). We then used the intraclass correlation coefficient and kappa analysis to examine the agreement between the three risk-prediction tools in this new analytic data set.

The goal of this exploratory study was not to perform an independent validation of these three risk-prediction tools. We did, however, perform a secondary analysis in which we examined the performance of each of these models using measures of discrimination (C statistic) and calibration (calibration curve, Hosmer–Lemeshow statistic). The C statistic is the probability that a patient selected at random who experienced a major adverse cardiac event had a higher predicted risk of a major adverse cardiac event compared to a randomly selected patient who did not experience a major adverse cardiac event.¹⁷  If the C statistic is 0.5, then the risk-prediction tool is no better than the flip of a coin. Higher C statistics indicates that a risk model is better able to discriminate between patients who experienced an event (i.e., major adverse cardiac event) and those that did not.¹⁸  The Hosmer–Lemeshow statistic assesses goodness-of-fit by first ranking the observations by the predicted probability of the event (major adverse cardiac event) into deciles of risk, and then comparing the observed and the predicted number of events within each decile using a chi-square–like statistic. P values greater than 0.05 for the Hosmer–Lemeshow statistic indicate acceptable model calibration.¹⁸  Since the Revised Cardiac Risk Index definition of major adverse cardiac event is somewhat different than the American College of Surgeons National Surgical Quality Improvement Program and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest definition, and not all of the components of the Revised Cardiac Risk Index definition of major adverse cardiac event major adverse cardiac event are included in the National Surgical Quality Improvement Program data, these comparisons must be interpreted with caution (appendix). These differences in the definition of major adverse cardiac event do not affect the validity of the primary analyses since the American College of Cardiology/American Heart Association guidelines assume that each of these three recommended prediction tools predicts the same clinical outcome.

We would also like to highlight the fact that we compared the Revised Cardiac Risk Index risk of major adverse cardiac event (which has only four possible values) to models (American College of Surgeons National Surgical Quality Improvement Program and National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest) which allow nearly continuous measures of predicted risk. By design, Lee et al.⁶  converted their multivariable risk prediction model to a risk index to make it easier for clinicians to use in daily practice. However, converting a regression model to a risk index (i.e., Revised Cardiac Risk Index) reduces the accuracy of predictions. It is possible that the extent of agreement between the predictions based on the model used to create the Revised Cardiac Risk Index and the other two risk-prediction tools would have been higher than that with the Revised Cardiac Risk Index. Nonetheless, we decided a priori to examine the extent of agreement between the Revised Cardiac Risk Index predictions, as opposed to model-based predictions, to the two other risk-prediction tools because the Revised Cardiac Risk Index is used as a risk index in clinical practice. Of note, the full set of model coefficients necessary to calculate the model-based predictions for the Revised Cardiac Risk Index model are not published, only the risk index is available.

The main analyses used in this study were specified prior to initiating our analyses. We modified our original analytic plan to use the intraclass correlation coefficient (instead of linear regression analysis) to examine the agreement between the risk-prediction tools. Our original protocol did not specify that the analytic set would be limited to about 10,000 observations in order to limit the demands placed on the National Surgical Quality Improvement Program calculator website.

Data management and statistical analyses were performed using STATA SE/MP Version 14.2 (Stata Corp, USA). All statistical tests were two-tailed and P values less than 0.05 were considered significant. This study was determined to be exempt from Institutional Review Board review (RSRB00066296).

Table 1 summarizes patient demographics. Sixty percent of the patients were under 65 yr of age and 97% were functionally independent. Over half of the patients were ASA physical status III or IV. Six percent of the patients had a history of diabetes requiring insulin and 6.5% had a history of cerebrovascular disease. The percentage of patients with a history of ischemic heart disease and congestive heart failure was 0.8% and 0.5%, respectively. One percent of the patients experienced major adverse cardiac event.

Figure 2 displays box plots comparing risk prediction using: (1) Revised Cardiac Risk Index versus American College of Surgeons National Surgical Quality Improvement Program risk calculator; and (2) Revised Cardiac Risk Index versus National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculator. Figure 3 displays scatter plots of: (1) Revised Cardiac Risk Index versus American College of Surgeons National Surgical Quality Improvement Program risk calculator; (2) Revised Cardiac Risk Index versus National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculator; and (3) American College of Surgeons National Surgical Quality Improvement Program risk calculator versus National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculator. There is good agreement between American College of Surgeons National Surgical Quality Improvement Program and National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest estimates of major adverse cardiac event (intraclass correlation coefficient = 0.68, 95% CI: 0.66, 0.70), and poor agreement between (1) American College of Surgeons National Surgical Quality Improvement Program and Revised Cardiac Risk Index (intraclass correlation coefficient = 0.37; 95% CI: 0.34, 0.40), and (2) National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest and Revised Cardiac Risk Index (intraclass correlation coefficient = 0.26; 95% CI: 0.23, 0.30). We found similar values for the intraclass correlation coefficient using complete case analysis (table 2). In the second sensitivity analysis in which we over-sampled intermediate- and low-risk surgeries, we found slightly worse agreement (table 2).

The three prediction models disagreed 29% of the time in which patients were low risk (29.2%; 95% CI: 28.1 to 30.0; table 3). Kappa analysis demonstrated fair agreement between the three scores (kappa 0.34; 95% CI: 0.33 to 0.36; table 3). The American College of Surgeons National Surgical Quality Improvement Program risk calculator exhibited fair agreement with the Revised Cardiac Risk Index (kappa = 0.22; 95% CI: 0.20, 0.25), while the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculator exhibited slight agreement with the Revised Cardiac Risk Index (kappa = 0.18; 95% CI: 0.16, 0.20; tables 3 and 4). We obtained similar results when we performed a weighted kappa analysis based on the original analytic sample, and when we performed a complete case analysis (table 2). We found worse agreement when we over-sampled intermediate- and low-risk surgeries (table 2 and Supplemental Digital Content table, https://links.lww.com/ALN/B758).

In our secondary analysis, we found that the American College of Surgeons risk calculator exhibited the best discrimination (C statistic 0.81; 95% CI: 0.77, 0.85) compared to both the Revised Cardiac Risk Index (C statistic 0.68; 95% CI: 0.62, 0.73) and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest calculator (0.73; 95% CI: 0.68, 0.77). The American College of Surgeons National Surgical Quality Improvement Program calculator exhibited acceptable calibration (Hosmer–Lemeshow statistic 17.14; P = 0.07), whereas the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest calculator (Hosmer–Lemeshow statistic 42.1; P < 0.001) and Revised Cardiac Risk Index (Hosmer–Lemeshow statistic 18.6; P < 0.001) were not as well calibrated. Calibration graphs are shown in the Supplemental Digital Content (https://links.lww.com/ALN/B758).

We examined the extent to which the risk-prediction tools in the 2014 American College of Cardiology/American Heart Association perioperative clinical practice guideless agreed on the results of risk stratification. The Revised Cardiac Risk Index, which is the most widely used tool to risk stratify patients at risk of major adverse cardiac event,¹⁹  exhibited only fair-to-poor agreement with two newer tools created using data from the American College of Surgeons. We found that these risk models disagreed about 30% of the time on which patients were at low risk versus elevated risk of major adverse cardiac event. According to the American College of Cardiology/American Heart Association algorithm, patients stratified as low risk should proceed directly to surgery, and would not be considered for possible stress testing or coronary revascularization.⁵  Our findings suggest that the calculated risk of major adverse cardiac event, and the need for further cardiac testing, is at least partly a function of which risk-prediction tool is used to assess patient risk in the 2014 American College of Cardiology/American Heart Association perioperative practice guideline.

The goal of predictive analytics is to help tailor diagnostic and treatment interventions to patient risk. Historically, physicians estimated patient risk using knowledge acquired during training, clinical experience, and from the shared experiences of colleagues.²⁰  In the past, the American College of Cardiology/American Heart Association perioperative guidelines did not include a risk-prediction tool.²¹  In theory, risk-prediction tools can more precisely assess patient risk using multivariable models that can simultaneously account for multiple risk factors. Predictions, however, are a function of which patient risks are included in the prediction model, and different models will produce different estimates of patient risk. There is a large body of literature showing that the same hospitals can be classified either as high-quality or low-quality, depending on which risk adjustment model is used.^9,22  In the current study, we found that the same patient could be classified as low risk or not low risk depending on which risk-prediction tool was used.

Selecting a best-in-class risk-prediction tool should take several factors into account, including face validity, statistical performance, and generalizability. Face validity is subjective, and in this case, implies that the model includes risk factors that would reasonably be associated with the risk of major adverse cardiac event (e.g., cardiac risk, surgical complexity). Risk predictions are only as good as the underlying models; the statistical performance of risk models is evaluated using measures of discrimination and calibration.²³  Most models are internally validated using data from the same database that was used to create the models. External validation using a different database is more desirable because it establishes whether a risk model accurately predicts outcomes beyond the database used to create the model; it addresses the generalizability of the risk-prediction tool.²⁴  Selecting a best-in-class risk calculator for the 2014 American College of Cardiology/American Heart Association perioperative guidelines from one of the currently available models may not be possible for several different reasons. First, the American College of Surgeons National Surgical Quality Improvement Program surgical risk calculator is not in the public domain. Second, the only head-to-head comparison of the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculator and the Revised Cardiac Risk Index was conducted with data used to develop the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest calculator, conferring a performance advantage to the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculator. Third, the Revised Cardiac Risk Index, the American College of Surgeons National Surgical Quality Improvement Program risk calculator, and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculators are based on two different definitions of major adverse cardiac event, and are thus, strictly speaking, not comparable. The Revised Cardiac Risk Index defines major adverse cardiac event as myocardial infarction, pulmonary edema, ventricular fibrillation or primary cardiac arrest, or complete heart block.⁶  In comparison, the National Surgical Quality Improvement Program definition of major adverse cardiac event, on which the American College of Surgeons National Surgical Quality Improvement Program risk calculator and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculator are based, only includes perioperative myocardial infarction and cardiac arrest.

Our study has several important limitations. A priori, one would expect that since the National Surgical Quality Improvement Program risk calculators were developed using National Surgical Quality Improvement Program data, while the Revised Cardiac Risk Index was developed using non–National Surgical Quality Improvement Program data, the National Surgical Quality Improvement Program risk calculators would more accurately predict outcomes than the Revised Cardiac Risk Index in the National Surgical Quality Improvement Program data set. However, it is not likely that this is the main reason that the three risk calculators disagree 30% of the time about which patients were at low-risk of major adverse cardiac event. Instead, it is more likely that these prediction tools disagree because they are fundamentally different models. The Revised Cardiac Risk Index is based on a relatively small sample of patients (4,135 patients) from a single tertiary care center using data that is more than 20 yr old. Further, the Revised Cardiac Risk Index is based on six risk factors, and categorizes procedure-specific risk into high-risk versus not high-risk surgery.⁶  The American College of Surgeons National Surgical Quality Improvement Program risk calculator, on the other hand, was developed using data from 1.4 million patients in nearly 400 hospitals,⁷  has 20 risk factors, classifies procedure-specific risk using a continuous scale, and is updated on a regular basis.

Second, mapping National Surgical Quality Improvement Program data to Revised Cardiac Risk Index risk factors is not perfect, and this could lead to less accurate Revised Cardiac Risk Index predictions, thus contributing to the lack of agreement between the National Surgical Quality Improvement Program risk-prediction tools and the Revised Cardiac Risk Index. However, each of the Revised Cardiac Risk Index risk factors (ischemic heart disease, history of congestive heart failure, history of cerebrovascular disease, insulin therapy for diabetes, and preoperative serum creatinine greater than 2) can be easily mapped to corresponding data elements in National Surgical Quality Improvement Program (e.g., congestive heart failure is a data field in National Surgical Quality Improvement Program). We chose to use American College of Surgeons National Surgical Quality Improvement Program data because it contains the risk factors used in all three risk-prediction tools, and more accurately reflects real world use of these risk-prediction tools than using administrative data in population-based all-payer data (e.g., Nationwide Inpatient Sample). Since the National Surgical Quality Improvement Program is a convenience sample of hospitals, our results are not necessarily generalizable to all U.S. hospitals.

Third, clinicians using risk-prediction tools may have access to more detailed clinical data than the data available in the American College of Surgeons National Surgical Quality Improvement Program database. Many of the risk factors used in these risk-prediction tools are somewhat subjective, and are assigned by surgical clinical reviewers instead of the treating clinician. Fourth, our analysis is incomplete because we cannot compare the results of these risk-prediction tools to clinical judgement alone. Many clinicians may use clinical judgement alone to decide which patients are low risk and can proceed to surgery without further cardiac testing. Nonetheless, our findings have face validity, especially since differences in patient risk estimates across different models are also the root of the reported variability in hospital performance evaluations based on different risk adjustment models.

It is also important to recognize that elevated-risk patients (risk of major adverse cardiac event greater than 1%) will not undergo further testing if: (1) they have moderate or greater (greater than or equal to 4 metabolic equivalents) functional capacity; or (2) if further testing is not expected to impact clinical decision making or perioperative care. In other words, the clinical significance of “elevated risk” is conditional on a patient’s functional capacity and other factors.⁵  Thus, incorrect estimates of major adverse cardiac event may, in some instances, have no impact on clinical decision making.

The lack of agreement between risk-prediction tools recommended in the American College of Cardiology/American Heart Association guidelines could lead clinicians applying these guidelines to make different clinical decisions depending on which risk-prediction tool is used to stratify patients. Whether these findings are generalizable to other clinical guidelines incorporating more than one risk-prediction tool is an empirical question that requires further investigation. However, it is well recognized that quality metrics based on different risk adjustment models can lead to different conclusions on hospital quality with some hospitals classified as high quality and others as low quality, depending on which risk adjustment model is used.^9,22  The National Quality Forum evaluates the scientific acceptability of quality measures, focusing on the validity of risk adjustment, before endorsing a quality metric.²⁵  In particular, the National Quality Forum endorses a single best-in-class measure when several quality metrics with the same focus and target population are submitted for evaluation.²⁶  This avoids the possibility that hospital “quality” will differ for the same outcome and patient population depending on which quality metric is used. Ideally, guideline developers might similarly evaluate the scientific validity of risk-prediction tools (using the available literature) before they are incorporated into clinical algorithms, and attempt to identify a single best-in-class prediction model, whenever possible, to reduce the possibility that a practice guideline might provide clinicians with different recommendations depending on which guideline-recommended prediction tool a clinician selects.

Our findings highlight the need for more accurate prediction modeling for cardiac risk stratification. The venerable Revised Cardiac Risk Index, still the gold standard for cardiac risk stratification, is based on a relatively small patient sample and reflects clinical practice from more than two decades ago. Nonetheless, the Revised Cardiac Risk Index has been extensively validated and exhibits moderate diagnostic accuracy in a recent a meta-analysis based on studies published between 2001 and 2008.²⁷  The American College of Surgeons National Surgical Quality Improvement Program and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest risk calculators, although derived using a very large and contemporary clinical registry, are limited by the narrow definition of major adverse cardiac event (which does not include important cardiac complications such as congestive heart failure) and the possibility that many cardiac complications are not accurately recorded due to incomplete surveillance. Additionally, the National Surgical Quality Improvement Program risk calculators do not include important risk factors such as history of ischemic heart disease or recent myocardial infarction, and have not undergone external validation.

We found wide variability in the predicted risk of major adverse cardiac event for individual patients depending on which risk-prediction tool is used in the 2014 American College of Cardiology/American Heart Association Perioperative Guideline. Including more than one prediction tool in clinical guidelines could lead to differences in decision-making for some patients depending on which risk calculator is used.

Supported by funding from the Department of Anesthesiology at the University of Rochester School of Medicine, Rochester, New York.

The authors declare no competing interests.