Association between Performance in a Maintenance of Certification Program and Disciplinary Actions against the Medical Licenses of Anesthesiologists

THE 24 member boards of the American Board of Medical Specialties (Chicago, Illinois) issue certificates to physicians who successfully complete board certification requirements. These certificates are now all time-limited, requiring periodic renewal through a maintenance of certification program. In 2000, the American Board of Anesthesiology (Raleigh, North Carolina) began issuing time-limited certificates that require renewal every 10 yr. American Board of Medical Specialties standards mandate a four-part framework for maintenance of certification: professionalism and professional standing (part I); lifelong learning and self-assessment (part II); assessment of knowledge, judgment, and skills (part III); and improvement in medical practice (part IV). Although previous literature supports the association of achieving primary certification and subsequent measures of physician performance,^1–6  there is little direct evidence of how participation in maintenance of certification, or performance in various elements of maintenance of certification, may be associated with physician performance. This lack of evidence has contributed to controversy surrounding the value of maintenance of certification.

Assessing this value is challenging as it requires assessing the performance of large populations of physicians. Some studies examining the association between primary certification and physician performance have employed various quality measures related to patient care as outcomes.^1,3–5,7,8  Other studies focus on actions by state medical and osteopathic boards against physician medical licenses, which reflect pronounced performance deficiencies.^2,6,9,10  In general, primary board certification is associated with a reduced risk of license actions regardless of specialty, supporting the utility of this outcome. Regarding maintenance of certification, lower performance on a cognitive written examination used to fulfill the part III requirement is associated with a higher rate of license actions in diplomates of the American Board of Internal Medicine¹¹  (Philadelphia, Pennsylvania) and the American Board of Anesthesiology.¹²  However, it is not known whether the inability to pass a cognitive written examination (requirement of part III) indicates an increased likelihood of license actions.

The overall goal of this study was to explore the association between participation and performance in the Maintenance of Certification in Anesthesiology program and postcertification physician performance as measured by license actions. We tested three hypotheses: (1) the incidence of license actions was lower in physicians with time-limited certificates (i.e., certified in 2000 or later) than in physicians with non–time-limited certificates (i.e., certified before 2000); (2) among physicians with non–time-limited certificates, participating in maintenance of certification voluntarily is associated with a lower incidence of license actions; and (3) among physicians with time-limited certificates, the incidence of license actions was higher among those who did not complete their maintenance of certification requirements within 10 yr of certification than among those who did.

This study was deemed exempt from review by the Mayo Clinic Institutional Review Board (Rochester, Minnesota).

A previous report details our method to ascertain license actions.⁶  To summarize, the Federation of State Medical Boards (Euless, Texas) gathers information from all U.S. state medical and osteopathic boards on license actions and disseminates this information to the American Board of Medical Specialties member boards via the Disciplinary Action Notification Service. License actions are classified as prejudicial (e.g., loss or restriction of license) or nonprejudicial (e.g., lifting of probation conditions). In this study, a license action incident case was defined as receiving at least one prejudicial action; nonprejudicial actions were not included. For a physician receiving multiple prejudicial actions, only the first one was considered an incident case.

Two sets of analyses were conducted to address the hypotheses.

The first set of analyses included all anesthesiologists whose primary certificates were awarded in calendar years 1994 to 1999 (n = 9,529) and 2000 to 2005 (n = 6,060). The periods were chosen to bracket the introduction of time-limited certificates in 2000 and to provide sufficient follow-up time to ascertain license actions. Although each cohort included all those awarded a certificate over a 6-yr period, the more recent cohort (i.e., time-limited certificate holders) had fewer physicians due to reductions in residency enrollment.¹³  Incidences of license actions were compared between the 1994 to 1999 cohort (who were issued non–time-limited certificates and thus not required to participate in maintenance of certification) and the 2000 to 2005 cohort (who were issued time-limited certificates and thus required to participate in maintenance of certification). In an additional analysis, the 1994 to 1999 cohort was split into two subgroups—those who had and those who had not chosen to voluntarily participate in maintenance of certification—and incidences of license actions for each subgroup were compared to that of the 2000 to 2005 cohort. This additional analysis excluded 69 (0.7%) American Board of Anesthesiology volunteers (e.g., standardized oral examination examiners, question authors, and examination committee members) in the 1994 to 1999 cohort who are required to participate in maintenance of certification as a condition of their service.

The second set of analyses included all physicians receiving time-limited certificates between 2000 and 2005 who were continuously enrolled in maintenance of certification within their 10-yr maintenance of certification cycles and did not fail maintenance of certification requirements because of reasons directly related to license actions (e.g., substance use disorder). Incidences of license actions were compared between those who met maintenance of certification requirements within 10 yr of issuance and those who did not. Additionally, to fully explore whether passing the anesthesiology maintenance of certification examination (necessary to meet the part III requirement) predicted license actions, incidences of license actions were compared for various subgroups within those who did not meet maintenance of certification requirements: (1) physicians who passed the anesthesiology maintenance of certification examination but did not meet at least one other maintenance of certification requirement, (2) physicians who either did not take or did not pass the anesthesiology maintenance of certification examination, (3) physicians who took but did not pass the anesthesiology maintenance of certification examination, and (4) physicians who failed to complete maintenance of certification requirements solely because they failed the anesthesiology maintenance of certification examination. The anesthesiology maintenance of certification examination could be taken beginning in year 7 of the 10-yr maintenance of certification cycle, and those who did not achieve a passing score were allowed to retake the examination until the final year of their cycle. Passing the examination was required to meet the part III requirement.

From the 6,060 physicians certified between 2000 and 2005, these analyses excluded those who died (n = 45), retired (n = 11), or had their certificates revoked by the American Board of Anesthesiology before the end of their first 10-yr maintenance of certification cycle (and thus dropped out of maintenance of certification; n = 10), and those who had a shorter than 10-yr maintenance of certification cycle (due to a history of disciplinary issues such as substance use disorder for whom certificate issuance was delayed from the date they passed the certifying examination; n = 25). Also excluded were those who voluntarily withdrew from maintenance of certification because they only wanted to maintain a subspecialty certificate (e.g., pain medicine; n = 9), those who were living in a foreign country and did not maintain a U.S. medical license (n = 5), those who left practice due to health reasons (n = 4), those who were planning to retire and opted not to complete maintenance of certification (n = 3), and those who could not complete maintenance of certification requirements (but were continuously enrolled in maintenance of certification) because of reasons directly related to license actions (e.g., substance use disorder; n = 10). The final study population for these analyses included 5,938 physicians.

Demographic characteristics were compared for the 1994 to 1999 and the 2000 to 2005 cohorts with a two-sample Student’s t test for continuous variables (age at certification) or a chi-square test for categorical variables (sex and medical school country). The incidence of license actions was analyzed with survival analysis, with the time to event defined as the time elapsed from the date of certification to the time of first prejudicial license action. Incident license action cases occurring before the date of primary certification were excluded from analysis. For the non-license action cases, a physician was censored from the analysis on the date of death if deceased, the date of retirement if retired, or the end of follow-up (i.e., December 31, 2016) if alive and not retired. If the date of death or the date of retirement was unknown, December 31, 2016, was used as the censoring date.

For each set of analyses, the cumulative incidence of license actions was visualized for each of the groups being compared with Kaplan–Meier curves, with the x-axis representing number of years since primary certification and the y-axis representing cumulative proportion of individuals who have not had a license action. Group differences in the incidence of license actions were tested with Cox proportional hazards models. Sex and medical school country (American medical graduates vs. international medical graduates) were considered a priori as covariates in multivariable Cox models on the basis of a previous study that demonstrated their association with license actions under some circumstances.⁶ 

This study is based on population data, and sample size was not designed with a priori statistical power calculation. A P value less than 0.05 was considered to indicate statistical significance in all the analyses. All statistical analyses were performed in R version 3.3.1 (R Foundation for Statistical Computing, Vienna, Austria; available at https://www.r-project.org/; accessed September 14, 2016).

In the overall population of 15,589 physicians for the first set of analyses, 690 (4.4%) incident license action cases were identified. Of these, 103 cases occurring before the date of primary certification were excluded. In the remaining 15,486 physicians, among the 9,463 in the 1994 to 1999 cohort (non–time-limited certificates) there were 420 cases (4.4%), and among the 6,023 in the 2000 to 2005 cohort (time-limited certificates) there were 167 cases (2.8%; table 1; Supplemental Digital Content, https://links.lww.com/ALN/B739). Compared to the 1994 to 1999 cohort, the 2000 to 2005 cohort was older at certification and had a greater proportion of both women and international medical graduates (table 2). By the end of follow-up, 195 physicians were known to have died (nine had an unknown date of death) and 42 were known to have retired (one had an unknown date of retirement).

In univariate analysis, the incidence of license actions was not significantly different between the two cohorts (table 3; hazard ratio = 1.18; 95% CI, 0.98 to 1.42; for non–time- limited certificate holders, 1994 to 1999 cohort, compared with time-limited certificate holders, 2000 to 2005 cohort). Results were similar in multivariable analysis, which included sex and medical school country (table 3, multivariable model 1). In this analysis, the incidence of license actions was significantly higher in men than in women but not significantly different between American medical graduates and international medical graduates.

Within the 1994 to 1999 cohort (excluding 69 American Board of Anesthesiology volunteers), 953 physicians voluntarily participated in maintenance of certification and 8,441 physicians did not. The proportion of women was higher in those who participated than those who did not (table 2). Compared to the 2000 to 2005 cohort, the incidence of license actions was significantly lower in those voluntarily participating in maintenance of certification (hazard ratio = 0.60; 95% CI, 0.38 to 0.94) and significantly higher in those not participating (hazard ratio = 1.22; 95% CI, 1.01 to 1.48), after adjusting for the effects of sex and medical school country (table 3, multivariable model 2; fig. 1).

Of the 5,938 physicians who received time-limited certificates between 2000 and 2005 and were considered for the second set of analyses, 168 incident license action cases were identified. Of these, 21 cases occurring before the date of certification were excluded. In the remaining 5,917 physicians, 5,329 (90.1%) completed maintenance of certification requirements by the end of their 10-yr cycle and 588 (9.9%) did not. There were 99 (1.9%) and 48 (8.2%) cases, respectively, among those who did and did not complete maintenance of certification requirements on time. By the end of follow-up, 12 were known to have died and 12 retired.

In univariate analysis, the incidence of license actions was significantly higher in physicians who did not complete their maintenance of certification requirements within 10 yr (hazard ratio = 4.63; 95% CI, 3.28 to 6.54; table 4). Results were similar in multivariable analysis, which included sex and medical school country (table 4). In the multivariable analysis, the incidence of license actions was significantly higher in men than in women but not significantly different between American medical graduates and international medical graduates. According to Kaplan–Meier analysis, the estimated cumulative incidence at 15 yr after certification was 2.0% (95% CI, 1.6 to 2.4%) for those who completed maintenance of certification requirements and 8.5% (95% CI, 6.2 to 11.2%) for those who did not (fig. 2).

Among the 588 physicians who did not complete maintenance of certification requirements on time, 414 (70%) did not meet the part II requirement for lifelong learning and self-assessment, 521 (89%) did not meet the part III requirement for passing the anesthesiology maintenance of certification examination, and 391 (66%) did not meet the part IV requirement for improvement in medical practice. Most physicians not meeting requirements had deficits in more than one element of maintenance of certification (456 physicians, 78%).

Sixty-seven physicians (11.4% of those not completing maintenance of certification requirements) passed the anesthesiology maintenance of certification examination (i.e., met part III requirement) but did not meet at least one other maintenance of certification requirement. There were 6 (9.0%) license action cases in these physicians, compared with 42 (8.1%) cases in the 521 physicians who either did not take or did not pass the anesthesiology maintenance of certification examination (hazard ratio = 1.17; 95% CI, 0.50 to 2.75; P = 0.72, univariate). Of those who took but did not pass the anesthesiology maintenance of certification examination (n = 134), there were 6 (4.5%) cases (hazard ratio = 0.43; 95% CI, 0.14 to 1.35; P = 0.15, univariate, compared with the 67 physicians who passed the examination but did not meet at least one other requirement). Finally, there were 48 physicians (0.8% of the 5,917 physicians issued time-limited certificates) who failed to complete maintenance of certification solely because they failed the anesthesiology maintenance of certification examination (i.e., they met all other requirements). There was 1 (2.1%) case among these 48 physicians (hazard ratio = 0.19; 95% CI, 0.02 to 1.60; P = 0.13, univariate, compared with the 67 physicians who passed the examination but did not meet at least one other requirement).

The major findings of this study were that (1) the introduction of time-limited certificates in 2000 was not associated with a difference in the postcertification incidence of license actions between physicians certified before and after 2000, (2) voluntary participation in maintenance of certification was associated with a lower incidence of license actions, and (3) not completing maintenance of certification requirements in a timely fashion was associated with a higher incidence of license actions. Initial board certification after residency training (i.e., primary certification) is associated with better physician performance, assessed with patient outcomes,^1,3–5,7,8  quality measures,^7,8  and license actions.^2,6,9,10  Unlike primary certification, little is known regarding how physician participation and performance in maintenance of certification is associated with physician performance. Three studies found an association between scores on the American Board of Internal Medicine maintenance of certification examination and various measures of care quality.^14–16  However, none attempted to determine whether quality measures differed between physicians who passed and failed the examination, providing little information to patients since examination scores are not known to the public. In a previous analysis,¹²  we found that a history of license actions was associated with lower scores on the anesthesiology maintenance of certification examination; comparisons were not made between those who passed and failed the examination. To our knowledge, there are no published studies regarding how performance on the other elements of maintenance of certification, such as lifelong learning and self-assessment (part II) or improvement in medical practice (part IV) may be related to physician performance.

One approach to evaluate the impact of maintenance of certification is to compare the performance of physicians certified before and after the introduction of time-limited certificates. Previous work from the American Board of Internal Medicine found that the introduction of their maintenance of certification process was not associated with a change in ambulatory care–sensitive hospitalizations, but was associated with a small reduction in the growth rate of annual healthcare costs.¹⁷  We found little effect of the introduction of time-limited certificates in 2000 on the incidence of license actions among anesthesiologists. Although this finding could be interpreted as indicating that maintenance of certification did not improve this measure of physician performance, several factors need to be considered. First, the four-part requirements for maintenance of certification in anesthesiology evolved over time, not reaching relative stability until 2009, which provided a short follow-up time to gauge its effectiveness in improving performance. Second, those certified between 1994 and 1999 had the option to voluntarily participate in maintenance of certification. Approximately 1 in 10 physicians did, and they were significantly less likely to receive license actions. This could have occurred because participation improved performance or because participation was a marker for those physicians who already were at a lower risk for license actions. For example, those who participated may already be better performers or may be more likely to consistently comply with rules of any kind. Indeed, excluding physicians with non–time-limited certificates who voluntarily participated in maintenance of certification from consideration, the introduction of the time-limited certificates (with the attendant requirement to participate in maintenance of certification) was associated with a small but significantly lower risk of license actions (see table 3, multivariable model 2). Again, this may represent participation bias. Third, although we adjusted for the effects of sex and medical school country, it is possible that the two cohorts may have differed in other characteristics or abilities, further confounding any cause-and-effect (or lack thereof) interpretation for maintenance of certification introduction. For example, the proportion of candidates who passed both part I and part II examinations on their first attempts were 67.6% and 64.7% for the 1994 to 1999 cohort and the 2000 to 2005 cohort, respectively (chi-square test, P = 0.0002). Although the certification examinations have evolved over time and pass rates in different years may not be directly comparable, success in passing the anesthesiology certification examinations on the first attempt is associated with a reduced risk of license actions.⁶  Thus, the higher rate of passing in the earlier cohort could bias against finding a lower incidence of license actions after the introduction of maintenance of certification.

We also found that timely completion of maintenance of certification requirements was associated with a lower incidence of license actions; this result, to our knowledge, represents the first demonstration in any specialty of an association between meeting maintenance of certification requirements and a measure of physician performance. This finding was based on physicians who continuously enrolled in maintenance of certification (i.e., did not have their license or certificate revoked during the 10-yr maintenance of certification cycle). The majority of those not completing requirements were deficient in multiple elements of maintenance of certification. This finding could reflect physicians who simply decided not to participate, so did not complete reporting of their maintenance of certification requirements.

We were particularly interested in the effectiveness of the anesthesiology maintenance of certification examination in predicting physician performance. Although this exploratory analysis was limited due to the relatively small number of physicians who did not pass the examination, we found little evidence of an association between passing the anesthesiology maintenance of certification examination and license actions. Indeed, among those physicians who did not complete maintenance of certification requirements solely because they failed the anesthesiology maintenance of certification examination, only one license action case was reported. This finding may reflect a relatively high pass rate for the anesthesiology maintenance of certification examination, averaging 94% for first-time takers over the period of study. Nonetheless, it suggests that achieving a passing score on the examination alone has little value in predicting the pronounced practice deficiencies implied by license actions. The anesthesiology maintenance of certification examination was replaced in 2016 by the Maintenance of Certification in Anesthesiology Minute pilot program, a longitudinal assessment of knowledge that applies principles of adult learning theory in an effort to retain and enhance knowledge of the participants that provides 120 questions online each year.¹⁸  Future research will investigate whether participation and performance in this program are better predictors of license actions than the anesthesiology maintenance of certification examination.

We have previously reported a detailed analysis of incident license actions in anesthesiologists who entered training in U.S. programs from 1971 to 2011, showing that achieving primary certification was associated with a markedly lower risk of actions.⁶  The risk of license actions was higher in men than women and lower in international medical graduates than in American medical graduates. The current analysis (which included only physicians who achieved primary certification over a more limited date range) confirms the finding of lower risk among women but does not find a significant difference according to medical school country. In this previous analysis, incidence rates for license actions were largely stable over time at approximately 2 to 3 new cases per 1,000 person-years,⁶  such that approximately 1 in every 20 board-certified anesthesiologists were estimated to experience a license action over the course of a 30-yr career. Accordingly, in the current study 587 individuals (3.8% of the 15,486 anesthesiologists who had not experienced license actions before primary certification) experienced at least one prejudicial license action after primary certification. To put this cumulative incidence in context, a previous report concerning California physicians¹⁰  found that the incidence rate of license actions for all California physicians (2.4 new cases per 1,000 person-years) is comparable to what we had previously observed for U.S. anesthesiologists.⁶  In a case-control analysis of California physicians, anesthesiologists had a similar risk compared with surgeons and internal medicine specialists.⁹  Further work would be necessary to evaluate hypotheses related to any differential risk of license actions across medical specialties.

This analysis has several limitations. Although a major advantage of using license actions is the ability to ascertain this outcome in all licensed physicians, license actions only capture the lower bound of physician performance and do not reflect variations in physician performance above this threshold. The relatively low rates of license actions also may have affected our statistical power to detect a true difference between the 1994 to 1999 cohort and the 2000 to 2005 cohort. Another limitation of using license actions is variability among state boards regarding the criteria for license actions. Unmeasured variables may have confounded the results, especially in the direct comparison of the two cohorts before and after the introduction of maintenance of certification. Due to the relatively high pass rate for those taking the anesthesiology maintenance of certification examination for the first time, we did not account for the number of attempts to pass this examination in the primary analysis (only 3.2% of those passing the examination required more than one attempt to do so). Nevertheless, there was little evidence that the incidence of license actions differed between those who did and did not require multiple attempts to pass the examination (hazard ratio = 1.09; 95% CI, 0.35 to 3.44; P = 0.88, univariate).

In summary, although the introduction of time-limited certificates and the maintenance of certification requirement in 2000 was not associated with a difference in the incidence of license actions among board-certified anesthesiologists, both voluntary participation in maintenance of certification and completion of maintenance of certification requirements in a timely fashion were associated with a lower incidence of such actions. These findings suggest that timely completion of maintenance of certification requirements serves as an indication to the public that a physician is less likely to be disciplined by a state licensing agency. These observational data cannot determine whether the association arises because maintenance of certification participation actually improves physician performance or simply serves as a marker for other physician characteristics that reduce the risk of license actions.

Support was provided solely from institutional and/or departmental sources.

Drs. Harman, Sun, Wang, and Zhou are staff members of the American Board of Anesthesiology, Raleigh, North Carolina. Drs. Keegan, Macario, Patterson, and Warner are American Board of Anesthesiology Directors and receive a stipend for their participation in American Board of Anesthesiology activities. Dr. Minhaj is a nondirector member of the American Board of Anesthesiology Research Committee.