Effectiveness of Written and Oral Specialty Certification Examinations to Predict Actions against the Medical Licenses of Anesthesiologists

PHYSICIANS are frequently assessed during training, and their subsequent performance in practice can be used to evaluate the validity of these assessments. However, it remains challenging to measure physician performance on a large scale. Although patient-level outcomes have been used in studying the effect of physician experiences and qualifications,^1–6  appropriate outcomes may be difficult to ascertain. Potential confounding factors, such as the practice environment, further complicate interpretations. Actions against physician medical licenses represent one tool to ascertain performance deficiencies. Previous studies examining associations between this outcome and factors such as performance during medical school and residency training and performance on specialty certification examinations found in general that lower performance is associated with an increased risk of disciplinary actions.^7–12 

All 24 specialty member boards of the American Board of Medical Specialties (ABMS) administer written examinations as part of their primary certification processes. The American Board of Anesthesiology (ABA) and 13 other ABMS member boards also administer oral examinations. The ABA written examination is primarily a test of knowledge,¹³  whereas the oral examination evaluates candidates’ ability to describe patient management in clinical scenarios. Specific attributes assessed by the oral examination include sound judgment in decision-making and management of surgical and anesthetic complications, appropriate application of scientific principles to clinical problems, adaptability to unexpected changes in clinical situations, and logical organization and effective presentation of information.¹⁴  In terms of Miller’s Pyramid of Assessment, a commonly used conceptual framework in education, the written examination focuses on whether the candidate “knows,” whereas the oral examination attempts to assess whether the candidate “knows how.”^15,16  ABA candidates must pass the written examination to be eligible for the oral examination and must pass both written and oral examinations to be certified. The oral examination requires significant resources to administer but is thought by the ABA to capture domains related to clinical judgment and communication that are important for physician performance. However, because performance on written and oral examinations tends to be correlated, it is not clear whether the oral certification examination indeed improves the validity to predict later physician performance beyond that provided by the written examination.

The primary purpose of this study was to test the hypothesis that the lifetime risk of a disciplinary action against a physician’s medical license, a marker of professionalism and professional standing, is lower in those who pass both written and oral anesthesiology certification examinations (e.g., those who are certified by the ABA) compared with those who pass only the written examination. The secondary purposes were to describe the basis and severity of the disciplinary actions in physicians who entered anesthesiology training from 1971 to 2011 and to determine the secular trend in the incidence of these actions during the study period.

This study was deemed exempt from review by the Mayo Clinic Institutional Review Board (Rochester, Minnesota).

This retrospective cohort study initially included all physicians who entered anesthesiology training in an Accreditation Council for Graduate Medical Education (ACGME)–accredited program from 1971 to 2011. The follow-up time was to May 2014. The Federation of State Medical Boards (FSMB) provided data on medical licenses and disciplinary actions from state medical and osteopathic boards over the study period. Of the 53,614 physician records that the ABA provided to the FSMB, 51,907 (96.8%) were successfully matched using common identifiers. After further review, 129 matching physicians were excluded because of inconsistencies in name, date of birth, sex, medical school name, or medical school graduation year; 677 were excluded because their first training year was unknown or occurred before 1971; 782 were excluded because their training programs were not ACGME accredited; and 1,041 were excluded because they did not have an active medical license in any given year from 1971 to 2014 in the FSBM database or the ABA database. Thus, the final study population included 49,278 physicians.

The FSMB aggregates disciplinary action information from all of the U.S. state medical and osteopathic boards and disseminates this information to the ABMS member boards as alerts through the Disciplinary Action Notification Service (DANS). Each DANS alert includes an action, which is classified into one of four categories in order of decreasing severity by the FSMB: loss of license or license privileges (LL), restriction of license or license privileges (RL), other prejudicial action (OP), and nonprejudicial actions. A license action incident case was defined as the first alert that a physician received in the action categories of LL, RL, or OP. For cases with more than one action, the most severe action classification code was used to characterize the case. Physicians with only actions classified as nonprejudicial were not treated as cases.

Each DANS alert also has one or more bases explaining the reason that a physician is disciplined as reported by the individual state medical and osteopathic boards and compiled by the FSMB. These were classified into one of 11 basis categories, as follows (Supplemental Digital Content 1, https://links.lww.com/ALN/B409): substance use, malpractice, fraud, inappropriate prescribing, criminal activity, impairment, license/board violation, records violation, failure to supervise, unprofessional conduct, and other inappropriate activity.

The denominator for calculating secular trends in the incidence rate of license actions was the number of physicians who had started anesthesiology residency training and had an active medical license in a given year. The primary source of the license data for this study was the FSMB Physician Data Center, supplemented by any additional license information available from the ABA database. Of the 130,113 total license records received from the FSMB, 3,878 were excluded because of missing or invalid license issue or expiration dates.

Sex, medical school country (American vs. international), primary certification examination results, and residency training performance were considered a priori as potential predictors of license actions.

Candidates for the ABA primary certification are required to pass both a Part 1 (written) examination and a Part 2 (oral) examination. To evaluate the effect of examination results on license actions, a variable was defined describing performance on these examinations using 7 categories: passed both examinations on the first attempt, passed both examinations but required multiple attempts to pass Part 1, passed both examinations but required multiple attempts to pass Part 2, passed both examinations but required multiple attempts for both, passed only Part 1 (i.e., did not pass the Part 2), passed neither examination, or did not finish training (i.e., not eligible to take either examination).

The ABA collects clinical competence committee (CCC) reports for residents from the ACGME-accredited programs. Each report is classified as “S” for satisfactory, “U” for unsatisfactory, and “L” for leave of absence. Less than half (44%) of the study population had available CCC reports, because they only became systematically available via electronic data sources for all residents in 2000.

The first set of analyses described characteristics and incidence of license actions. The frequencies of action severity and basis classification were summarized for all incident license action cases. License action incidence rates were calculated for each year of the study period, and secular trends over the years were plotted. Cumulative incidence rates were also calculated by sex, medical school country, ABA certification status, primary examination results, and residency training performance.

The second set of analyses aimed to identify factors associated with incident license action cases. Cox proportional hazards model tested the association between incident license actions and each of the potential predictors. The time scale was number of years since the beginning of clinical anesthesiology training. For the noncases, May 31, 2014, was used as the end of the observation window. Due to the nature of the available data, individuals who died or left practice were not accounted for in these analyses. Univariate models were first fitted for each predictor, and then multivariable models were constructed using all factors defined a priori. Because CCC reports were available for less than half of the population, two multivariable models were constructed. Model 1 used data from the entire population, and residency training performance was not included as a predictor. Model 2 used data from the subpopulation that had available CCC reports, and residency training performance was included as a predictor. All statistical analyses were performed in R version 3.1.3 (R Foundation for Statistical Computing, Austria).

Of the overall population of 49,278 physicians analyzed, 46,566 (94.5%) finished or are still in anesthesiology residency training, and 2,712 (5.5%) left training before completion.

Between 1977 and May 2014, 2,142 incident license action cases were identified, of which 1,065 (50%) were LL cases, 435 (20%) were RL cases, and 642 (30%) were OP cases. The most common basis category for LL and RL cases was substance use, followed by license/board violation, malpractice, and unprofessional conduct (table 1). For OP cases, the most common category was license/board violation, followed by malpractice and records violation. Within the license/board violation category, the three most common basis codes were reciprocal as a result of action taken by another board or agency, failure to disclose required information, and violation of statute or rule of the board.

The incidence rate of license actions was relatively stable from 1983 to 2013 (fig. 1), with approximately 2 to 3 new cases per 1,000 person-years. The cumulative incidence of license actions was nearly twice as high in men as in women (table 2), and more than threefold higher in noncertified physicians compared with board-certified physicians. With respect to primary certification examination results, cumulative license action incidence was lowest among those who passed both Part 1 and Part 2 examinations on their first attempt (28,300 physicians, 57% of the overall cohort), intermediate among those who passed both examinations but not on their first attempt (5,026 [10%] required multiple attempts for Part 2 examination, 5,371 [11%] required multiple attempts for Part 1, and 2,991 [6%] required multiple attempts for both), and highest among those who passed Part 1 examination only (n = 1,842 [4%]), passed neither examination (n = 3,036 [6%]), or did not finish training (n = 2,712 [6%]). Compared with those with no unsatisfactory CCC report during residency training, physicians with an unsatisfactory report had a more than fivefold higher cumulative incidence of license actions.

To place these rates in perspective, Kaplan–Meier analysis was used to calculate the cumulative incidences of license actions every 5 yr, starting at the beginning of clinical anesthesiology training according to sex and primary examination results (table 3). For simplicity, the three groups that passed both examinations but required multiple attempts were combined, given that they had similar license action incidence rates. Over a typical 30-yr career, estimated cumulative incidence ranged from 32.4 per 1,000 physicians for women who passed both examinations on the first attempt to 171.1 per 1,000 physicians for men who did not finish training, an approximately fivefold variation.

In univariate Cox models, sex, board certification status, primary examination results, and residency training performance were significantly associated with license action incidence (table 4; fig. 2).

In the multivariable model that included all physicians (model 1), the risk of license actions was higher in men than women and lower in international medical school graduates than American medical school graduates (table 4). License action incidences were significantly higher in all groups who did not pass their Part 1 and Part 2 examinations on their first attempts compared with the group who passed both on their first attempt. Requiring more than one attempt to pass the Part 1 or Part 2 examinations modestly increased the risk compared with passing both on the first attempt (hazard ratios [HRs] from 1.19 to 1.59 with those passing both on first attempt as a reference group). Those passing only Part 1 and those passing neither examination had a similar risk of license actions (HR = 3.51 [95% CI, 2.87 to 4.29] and HR = 3.60 [95% CI, 3.14 to 4.13], respectively; P = 0.81).

In the multivariable model that included only those physicians with CCC reports available (model 2), the results regarding the effects of sex, medical school location, and primary examination results were qualitatively similar (table 4), except that the risk of license actions in the group who passed both examinations but required multiple attempts for Part 2 examination was no longer significantly different from the group who passed both examinations on their first attempts (P = 0.052). For the additional variable in this model, at least one unsatisfactory CCC report during residency independently increased the risk of license actions (HR = 2.96 [95% CI, 2.28 to 3.83]).

The major findings of this study are (1) the incidence rate of actions against the licenses of physicians who enter anesthesiology training programs has been stable for the past 30 yr; (2) the cumulative incidence of license actions is higher among men, American medical school graduates, and those who do not pass both primary anesthesiology certification examinations (i.e., those who are not board certified); and (3) the risk of license actions is similar between physicians who do not pass the written examination and those who pass the written examination but do not pass the oral examination.

Several previous studies have attempted to determine the association between specialty board certification and physician performance.^1–6  In studies using patient outcomes as a measure of performance, outcomes are generally more favorable among patients receiving care from board-certified physicians. Specific to anesthesiologists, Silber et al.⁶  examined 30-day mortality and failure to rescue (rate of death after in-hospital complications), looking at Medicare claims records for patients undergoing general surgical or orthopedic procedures who were cared for by midcareer anesthesiologists (11 to 25 yr from graduation). After covariate adjustment, risks of both outcomes were approximately 15% higher in patients receiving care from noncertified physicians. A major limitation of this and similar studies is that patient outcomes depend on many practice factors, which may differ systematically in locations where certified versus noncertified physicians practice.

Other work has used adverse license actions as a measure of physician performance, examining cumulative incidences.^7–12  There are two limitations to this approach. First, there can be variations among state medical and osteopathic boards with regard to what physician behaviors prompt which board actions, so the criteria or threshold for actions may differ in some instances. Second, this is a dichotomous measure that captures only the lower bound of physician performance and does not reflect variations in performance among the majority of physicians who do not have disciplinary actions against their license. However, the major advantage of this approach is the ability to almost completely ascertain this outcome among all licensed physicians, thanks to data available through the FSMB. The incidence rate of actions has been remarkably stable over the past 30 yr. The single most common cause of actions for anesthesiologists was related to substance use. Substance use is relatively less prominent in previous case series of internists and physicians from all specialties,^7,12  which may support previous speculation that substance use is particularly problematic among anesthesiologists.¹⁷ 

Several studies have examined the ability of assessments made during training to predict license actions. Consistent with our findings, previous studies demonstrated that physicians who are specialty-board certified (a dichotomous outcome) are consistently less likely to experience license actions compared with those who are not.^7,9,10,12  In an analysis of the association between performance on the written certification examination of the American Board of Internal Medicine (analyzed as a continuous variable) and license actions, higher scores were associated with fewer actions.⁷  When analyzed as a dichotomous variable (pass/fail), passing the written examination alone did not decrease the risk of license actions in our analysis. We did not assess the predictive validity of the written examination as a continuous variable, because not all written examination scores were electronically available in our data set. In addition, scale scores from different years are not directly comparable, because they were not equated across the entire study period. Other studies show associations between increased risk of license actions and assessments during medical and specialty training, including low professionalism ratings during residency training,⁷  unprofessional behavior during medical school,⁸  lower grades in medical school,⁸  and lower scores on medical school admissions tests.⁸  Our finding that deficiencies in residency training performance, as assessed by unsatisfactory CCC reports, were independently associated with license actions is consistent with the previous work.

The previous study most comparable to ours examined patient complaints to medical regulatory authorities in Ontario or Quebec as an outcome.¹¹  The certifying process in these provinces includes both a traditional written examination and a clinical skills examination (CSE) using an objective structured clinical examination (OSCE), with the latter scored in domains of communication, data acquisition, and problem solving. Both the written examination score and the communication score of the CSE (but not the other CSE domains) were associated with patient complaints. The association between the communication score and complaints persisted even after adjustment for written examination performance. Although the formats and scoring of the CSE and the ABA Part 2 examination differ, both include assessment of communication skills, and both provide incremental predictive validity over written examination. These findings lend support to a central role for communication skills in physician performance and are consistent with previous literature on the contribution of communication failures to malpractice claims.¹⁸  The ability of an oral examination to assess domains important to clinical performance beyond those assessed in written examinations is further supported by a recent study that used a validated clinical performance score assessed during anesthesiology residency training.¹⁹  Both clinical performance scores and Part 1 (written) examination scores were independently associated with Part 2 (oral) examination scores, suggesting that Part 2 performance is related to aspects of clinical performance not accounted for by Part 1 performance. Our results provide additional evidence for the validity of oral examination as a measure of physician performance and support its continued use by the ABA in its primary certification process. The ABA will be adding an OSCE component to the current oral examination in 2018. The intent of this addition is to move up Miller’s pyramid to a higher level of assessment (“shows how”) and to potentially capture additional attributes related to professionalism thought to be important to physician performance.^15,16  It will be of interest to determine whether the addition of the OSCE will further differentiate among physicians in measures of their performance.

In previous studies, other factors associated with license actions include male sex,^7,10,12  advanced age,^10,12  and training at an international medical school.^7,10  We confirm the finding that men are at greater risk. Our analysis did not directly examine the impact of age. Nevertheless, the segmental slopes of the Kaplan–Meier curve, representing an incremental increase in the proportion of physicians in the entire cohort receiving a license action for every 5-yr interval from the beginning of training, remains relatively constant over the study period (Supplemental Digital Content 2, https://links.lww.com/ALN/B410), suggesting that the rate of license actions does not noticeably accelerate with age.²⁰  Thus, from a career point of view, there was no obvious evidence for any particularly risky career stage. In contrast to previous work, we find that training at an international medical school is associated with a reduced risk of license actions. In our previous work examining the risk of substance use disorder among anesthesiology residents,¹⁷  we observed that international medical school graduates were less likely to develop a substance use disorder. Indeed, in the current cohort, the proportion of license actions related to substance use was greater in American compared with international medical graduates (0.72 and 0.25 per 1,000 person-years, respectively; Supplemental Digital Content 3, https://links.lww.com/ALN/B411). Thus, a subanalysis was carried out, excluding license actions related to substance use (Supplemental Digital Content 4, https://links.lww.com/ALN/B412). Although the effects of other factors remained similar, location of the medical school was no longer a significant predictor of license actions in the multivariable model (HR = 0.87 [95% CI, 0.67 to 1.14]). This finding is consistent with our previous work, suggesting that the lower rates of license actions among international medical graduates in this analysis can be explained at least in part by their lower probability of being cited for substance use.

This study has several other limitations beyond those noted previously. Due to the nature of the available data, individuals who may have died or left practice were not accounted for in the survival analyses. This factor would only affect the results to the extent that death or leaving practice depended on performance on certification examinations. Also, the oral examination assesses multiple domains, and our results do not provide insights into whether deficits in any specific domain are associated with license actions. Finally, different residency programs may use different criteria to determine unsatisfactory performance,²¹  which may affect precision of the estimate for how unsatisfactory residency performance affects the risk for subsequent license actions.

This study supports the hypothesis that an oral examination assesses domains important to anesthesiologist performance that are not fully assessed in a written examination and provides further evidence that physicians who are certified by an ABMS specialty board are markedly less likely to develop pronounced performance deficiencies in their medical careers.

The authors thank Frann Holmes, M.S., former business intelligence specialist from the Federation of State Medical Boards, for matching American Board of Anesthesiology records with Federation of State Medical Boards records to provide the physician licensure and disciplinary actions data.

Supported by the American Board of Anesthesiology, Raleigh, North Carolina.

Drs. Harman, Sun, and Zhou are staff members of the American Board of Anesthesiology (ABA); Dr. Young is a staff member of Federation of State Medical Boards; Drs. Culley and Warner are ABA Directors and receive a stipend for their participation in ABA activities.