Association between Participation in an Intensive Longitudinal Assessment Program and Performance on a Cognitive Examination in the Maintenance of Certification in Anesthesiology Program^®

IN alignment with the standards of the American Board of Medical Specialties^® (ABMS; Chicago, Illinois), the American Board of Anesthesiology (ABA; Raleigh, North Carolina) requires participation in a Maintenance of Certification in Anesthesiology Program^® (MOCA^®) for diplomates with a time-limited certificate. The MOCA program is designed to address six ABMS/Accreditation Council for Graduate Medical Education core competencies within a four-part framework that addresses (1) professionalism and professional standing; (2) lifelong learning and self-assessment; (3) assessment of knowledge, judgment, and skills; and (4) improvement in medical practice.

The assessment of knowledge, judgment, and skills has been performed via the MOCA Cognitive Examination, a secure multiple-choice examination that has to be passed once every 10 yr. According to the new 2015 Standards for the ABMS Program for MOC, the stated purpose of this examination is to “…provide an assurance that the diplomate has maintained the necessary commitment to lifelong learning” and to “…drive learning both through preparing for the examination and through targeted learning in response to examination results.” However, recent evidence raises questions regarding the effectiveness of infrequent periodic testing to promote learning. For example, massed study immediately before testing produces poor long-term retention of study materials.^1,2  Rather, techniques including spaced study, interleaving of topics, frequent testing that requires information retrieval, and immediate feedback have been shown to be effective in a variety of settings.^3–6  Thus, a single test administered once every decade may not be optimal to promote learning or assess working knowledge, and recent surveys of ABA diplomates reveal significant concerns about its utility.⁷ 

Accordingly, the ABA is in the process of redesigning its MOCA program. The goal for this effort, known as MOCA 2.0, is to provide a platform for intensive longitudinal assessment and learning that will help identify and address knowledge gaps with the goal of ultimately improving patient care. In January 2014, the ABA launched the pilot MOCA Minute program, a web-based application designed to help diplomates achieve better mastery of the topics included in the MOCA Cognitive Examination. In this intensive longitudinal assessment, eligible diplomates receive one question per week via email in topics where the majority of diplomates have not performed well on previous MOCA Cognitive Examinations. The participants have 1 min to answer the question once accessed. Whether the question is answered correctly or incorrectly, the correct answer, a rationale, and a link to additional resource materials are displayed on a subsequent screen.

The purpose of this observational study was to test the hypothesis that voluntary enrollment and participation in the MOCA Minute program is associated with improved performance in the subsequent MOCA Cognitive Examination when compared with the performance of individuals who do not participate.

This study was deemed exempt from review by the Mayo Clinic Institutional Review Board (Rochester, Minnesota).

This web-based application was developed by the ABA. Every Monday morning, participants receive an email inviting them to answer a MOCA Minute question (appendix 1). Each question is available for 1 week. When the participants click the appropriate link embedded in the email, they are taken to a multiple-choice question and have 1 min to respond. The majority of these questions are context rich so that they involve a scenario and application of knowledge. After they respond, a subsequent screen provides the correct answer, a brief explanation of why the keyed response is correct, appropriate additional background information about the topic, and links to references or resources pertinent to this topic. Questions are written and edited by an editorial group of volunteer anesthesiologists specifically for this application.

During the study period, the diplomates had opportunities to voluntarily participate in the MOCA Minute pilot program and to take the secure MOCA Cognitive Examination. This observational study included two phases. Phase 1 evaluated how MOCA Minute participation affected performance on the July 2014 Cognitive Examination, and phase 2 aimed to determine the replicability of phase 1 findings, evaluating how MOCA Minute participation affected performance on the January 2015 Cognitive Examination.

In December 2013, all diplomates who were eligible to take the July 2014 MOCA Cognitive Examination and had not already passed the examination within their current 10-yr cycle were invited via email to enroll in the MOCA Minute program. Of the 2,800 diplomates who received the invitation in December 2013, 1,408 (50%) enrolled (fig. 1). By enrolling, the diplomates agreed to have their responses tracked over time.

For this phase of the study, questions were prepared from 16 topics in which the majority of diplomates answered questions incorrectly on previous MOCA Cognitive Examinations (appendix 2). Three similar questions were created in each of the topics. For the first 17 weeks, 1 of 3 questions from the same topic was randomly sent to the participating diplomates (2 different sets of 3 questions were generated from 1 of the 16 topics). When an original question was answered incorrectly by a diplomate, a similar question on the same topic was sent out to the same diplomate in later weeks to solidify learning.

Diplomates in phase 1 were eligible to take the MOCA Cognitive Examination administered in July 2014. It included 185 items chosen according to the published blueprint for the examination, and 15 items covering 15 of the 16 topics included in the MOCA Minute. These 15 items were not scored for the examination. Thus, this phase evaluated the association between MOCA Minute participation and subsequent performance in MOCA examination items both related to and not related to the MOCA Minute questions.

In June 2014, 1,046 diplomates who were approaching their seventh year of the MOCA cycle in 2015 (and thus could register for the Cognitive Examination) were also invited to enroll in the MOCA Minute. Of the 1,046 diplomates who were invited, 330 (32%) enrolled and were added to the question distribution list (fig. 1). For this phase, MOCA Minute questions covered an additional 22 topics (appendix 2; questions from previous topics were repeated in 2 weeks, and no questions were sent during 2 holiday weeks).

Those who enrolled in phase 1 but did not pass the July 2014 Cognitive Examination and those newly enrolled diplomates in phase 2 were eligible to take the January 2015 Cognitive Examination (fig. 1). This examination included 200 items chosen based on the MOCA examination blueprint without regard for whether they covered topics included in the MOCA Minute. Thus, this phase served to validate the association between MOCA Minute participation and subsequent performance on the MOCA examination when MOCA Minute questions were not targeted to specific topics in the examination.

In May 2015, survey invitations were sent via email to MOCA Minute participants in the two phases to assess their perceptions and attitudes. The survey consisted of the rating of agreement (5-point Likert scale) with seven brief statements, including the usefulness of the MOCA Minute questions, their relevance to practice, usefulness of the key points and the references provided, preference for answering MOCA Minute questions versus taking the current MOCA Cognitive Examination, and helpfulness of MOCA Minute for diplomates to provide better patient care and stay current in anesthesiology. Participants were asked to provide any free-text comments about the MOCA Minute program.

No a priori sample size calculation was conducted as we considered unethical to withhold a potentially beneficial program for those randomized to a control group, and the consequences of not passing the MOCA Cognitive Examination could be as serious as losing Board certification.

Subjects of this analysis are the 616 diplomates who took the MOCA Cognitive Examination for the first time in July 2014 (phase 1 analysis) and the 684 diplomates who took the examination for the first time in January 2015 (phase 2 analysis), regardless of their enrollment status in the MOCA Minute pilot. Those who were enrolled in the MOCA Minute pilot but were not taking the examinations for the first time or did not take either of these examinations were not included in the analysis (fig. 1). The results of this examination are reported as a standardized scale score with a mean of 250 and a SD of 50, with those who take the MOCA examination for the first time within their current 10-yr cycle under standard conditions as the calibration group. Approximately half of the diplomates taking the two examinations (55% for phase 1 and 50% for phase 2) were enrolled in the MOCA Minute program (table 1).

In each phase of the analysis, demographic characteristics and first-time primary certification examination outcomes were summarized for diplomates who were enrolled and not enrolled in MOCA Minute and were compared using independent sample Student’s t test for continuous variables (e.g., age, year of first MOCA attempt within the 10-yr cycle, and examination scores) and chi-square test for categorical variables (e.g., sex, medical school country, and examination pass/fail outcomes). This comparison is of particular interest because diplomates were not randomized to receive or not receive the MOCA Minute.

The association between MOCA examination score and each of several covariates was first assessed using simple regression. Most of these covariates were chosen on the basis of our previous work examining factors associated with first-attempt MOCA scores.⁸  Covariates included results from the first-time Part 1 (written examination for primary certification, pass vs. fail) and Part 2 (oral examination for primary certification, pass vs. fail) examinations, sex, age, international (vs. U.S.) medical school, year of first MOCA attempt within the 10-yr MOCA cycle, history of a Disciplinary Action Notification System alert (DANS alert; indicative of an action against the medical license), and MOCA Minute participation level. Those who were enrolled in the MOCA Minute pilot answered a median of 44 questions (interquartile range, 19). Participants who answered less than 10 questions were a priori considered as “not actively participating,” and those who answered 10 or more questions were considered as “actively participating.” Those not enrolled served as the reference group. According to the a priori analysis plan, statistically significant covariates (i.e., P < 0.05) in univariate analysis were included in multiple linear regression; and covariates with a P > 0.05 in univariate analysis were excluded in the multiple regression for the purpose of model parsimony. The same model was used in both phases of analysis. Sensitivity analysis was then conducted to estimate the impact of eliminating insignificant variables in the univariate analysis. All the statistical analyses were performed in R version 3.1.3 (https://cran.r-project.org; accessed August 5, 2016).

Among the 616 diplomates analyzed in phase 1, those who enrolled in MOCA Minute were slightly younger, were more likely to be female, and took the first-time MOCA examination earlier in the 10-yr cycle than the group not enrolled (table 1). Very few in either group had a history of a DANS alert. Neither first-time pass rates nor examination scores differed significantly between MOCA Minute participation groups for the Part 1 or Part 2 examinations. Those who had enrolled in MOCA Minute scored an average of 9.9 points higher on the July 2014 MOCA examination than the nonenrollees (P = 0.03); pass rates were not significantly different (P = 0.25, Fisher exact test).

In univariate analysis, July 2014 MOCA Cognitive Examination scores were significantly associated with passing both the Part 1 and Part 2 examinations on the first attempt, sex, age, medical school country, and MOCA Minute participation (table 2). Specifically, those who passed the Part 1 examination and the Part 2 examination on their first attempt scored higher on July 2014 MOCA examination. In addition, female anesthesiologists scored lower than male anesthesiologists, and international medical school graduates scored lower than the American medical school graduates. The MOCA examination score decreased as diplomate age increased. Those who were actively participating in the MOCA Minute pilot scored 10.4 points higher (95% CI, 1.1 to 19.7) than those who were not enrolled.

The variables that were significant in the simple regression were included in multiple regression analysis (table 2). Controlling for other variables in the model, those who passed the Part 1 examination on their first attempt scored higher; but passing Part 2 on the first attempt was not significantly predictive. Female anesthesiologists scored lower than male anesthesiologists, and international medical school graduates scored lower than American medical school graduates. Compared to the group not enrolled in MOCA Minute, those who actively participated scored 9.9 points (95% CI, 0.8 to 18.9) higher on the MOCA examination when compared with those not enrolled (P = 0.03). Those who enrolled but did not actively participate demonstrated no improvement in MOCA Cognitive Examination scores (table 2).

To reiterate, these MOCA examination scores reflected performance on items querying topic areas not covered by the MOCA Minute topics administered during phase 1. Performance on the items covering the topics included in the MOCA Minute (and not scored for the July 2014 MOCA Cognitive Examination) was also analyzed. Performance on these items was significantly higher among the actively participating group than the not enrolled group (7.4 ± 2.5 vs. 6.1 ± 1.9 correct answers, respectively (mean ± SD), P < 0.0001). Scores on these items for those who enrolled but did not actively participate (6.3 ± 1.5 correct answers) were not significantly different from the group not enrolled.

Among the 684 diplomates analyzed in phase 2, those who did and did not enroll in the MOCA Minute were similar in most respects except that those enrolled were more likely to be female (table 1).

In univariate analysis, January 2015 MOCA Cognitive Examination scores were significantly associated with passing Part 1 and Part 2 examinations on the first attempt, sex, age, medical school country, the year of first MOCA examination within the 10-yr cycle, and MOCA Minute participation (table 2). Specifically, those who passed their first Part 1 and Part 2 examinations scored higher on the January 2015 MOCA examination. In addition, female anesthesiologists scored lower than their male counterparts, international medical school graduates scored lower than American medical school graduates, and those who had history of a DANS alert scored lower than those who did not. The MOCA examination score decreased as diplomate age increased. Those who were actively participating scored 8.8 points higher (95% CI, 1.2 to 16.3) than those who were not enrolled in the MOCA Minute.

As this phase of the study served to determine the replicability of the findings in phase 1, the same predictors in the multiple regression in phase 1 were retained in the phase 2 analysis (table 2). Controlling for other variables in the model, those who passed their first Part 1 and Part 2 examinations scored higher and female anesthesiologists scored lower than male anesthesiologists. The MOCA examination score decreased as diplomate age increased. Compared to the group not enrolled in the MOCA Minute pilot, those who enrolled but did not actively participate demonstrated no improvement in MOCA Cognitive Examination scores (table 2). Those who actively participated scored 9.3 points (95% CI, 2.3 to 16.3) higher on their first attempt MOCA Cognitive Examination compared with those not enrolled; pass rates were not significantly different (P = 0.11, Fisher exact test).

Of the diplomates invited, 57% completed the MOCA Minute survey. Table 3 summarizes the respondents’ ratings of the degree to which they agreed with each of the survey statements. Overall, the respondents expressed positive attitudes about the MOCA Minute program, with 75 to 97% agreeing or strongly agreeing to the seven favorable statements.

Of the respondents, 25% provided free-text responses regarding the MOCA Minute program. Approximately half explicitly stated that they preferred these frequent assessments to an examination administered every 10 yr. Approximately a quarter volunteered that they enjoyed the MOCA Minute process. A few respondents suggested improvements, including linking to learning modules on specific topics, developing a similar program for anesthesia subspecialties, and offering questions more frequently than once a week.

The major findings of this study are that (1) active voluntary participation in the MOCA Minute program was independently associated with subsequent better performance on the MOCA Cognitive Examination and (2) most participants expressed satisfaction with the program.

Standardized examinations can be a useful means of evaluation in education, professional certification, and many other fields. However, it is debatable how effectively a periodic examination for every 10 yr reconsolidates knowledge and promotes lifelong learning. A common strategy for preparing for examinations is massed study right before the examination, which may lead to short-term improvements in examination performance but is less likely to result in long-term retention of the study materials.^1,2  Conversely, spaced study sessions over a longer period of time have positive effects on long-term memory. This advantage has been demonstrated in various educational settings,^4,6  including teaching of surgical skills in a surgical residency program.⁵ 

Thus, the form of assessment, if designed and used appropriately, could be a powerful learning tool. Compared to repeated exposure to material alone (e.g., via reading), assessment involves a process of retrieving information from memory and thus constitutes better practice for later efforts to retrieve, especially when the process is more difficult.⁹  Even if the learner fails to correctly recall information on an assessment, retrieval efforts themselves benefit learning.¹⁰  The value of testing-enhanced learning has been known for at least the last century (see a comprehensive review of laboratory and classroom studies by Roediger and Karpicke¹¹ ), including in medical education.^12,13  A key ingredient in the success of assessment as a learning tool is frequent retrieval at spaced intervals (e.g., frequent quizzing throughout a course). In addition, feedback is an amplifier for the effects of assessment. Not only does feedback strengthen retention more than assessment alone, it also reduces potential negative effects of multiple-choice questions caused by exposure to misinformation in distractors in the absence of any immediate feedback.¹⁴  Indeed, making incorrect alternatives competitive with correct answers can induce learners to retrieve why each incorrect alternative is incorrect—information provided as immediate feedback in the MOCA Minute process.¹⁵  Of interest, learners may not appreciate the benefit of these techniques, expressing greater confidence in massed study even when it is demonstrably less effective.¹⁶ 

While some learning mechanisms universally apply to learners of all ages, other aspects of learning can be quite different for people of different ages. Compared to children, adult learners are seen as more internally motivated and self-directed; they have a foundation of experience and knowledge on which to build, are more goal oriented and relevancy oriented, behave in a more practical manner, and seek out environments that they view as respectful.^17,18  Because of their high level of autonomy, adult learners are more often engaged in what is called self-regulated study,²  which involves many decisions they make about how to study and when to study. Unfortunately, findings from survey research show that adults do not always choose good study strategies.²  For example, they usually do not plan spaced study sessions but rather attend to whatever is most urgent. In addition, infrequent high-stakes assessments may be viewed as threatening rather than a viable learning strategy.

The MOCA Minute program was conceived as a means to help diplomates continually evaluate and improve their knowledge in content areas relevant to their practices. It incorporates several features demonstrated to support effective learning, including (1) spaced study, (2) frequent assessment that requires information retrieval, (3) interleaving of topics, and (4) immediate feedback. In addition to these potential benefits to learning, a frequent assessment with feedback approach also has several other potential advantages, including the ability to (1) customize learning to practice by allowing learners to designate their primary practice areas and focusing questions on these topics; (2) rapidly disseminate new knowledge through questions addressing topics of emerging clinical relevance, and (3) provide opportunities for intensive longitudinal assessment of diplomate knowledge that could be used to help learners identify gaps in knowledge and potentially for summative assessments. In addition, most MOCA Minute questions involve a case presentation and require application of clinical knowledge, a format recently shown to improve medical students’ learning.¹⁹  The current study sought to evaluate the efficacy of this approach in improving performance on the MOCA Cognitive Examination, a measure of knowledge and judgment.

As demonstrated in a previous analysis,⁸  many factors influence performance on the MOCA Cognitive Examination, so it was necessary to control for these. For the most part, factors associated with MOCA examination performance are consistent with this previous work, with the most important predictor (perhaps not surprisingly) being performance on primary certification examinations. When these other factors were taken into account, participation in MOCA Minute was associated with a modest but significant improvement in performance, not only for those items querying topics covered by MOCA Minute questions but also for items in other topic areas. The latter result was robust across both study phases. Differences in pass rates were not significant, perhaps reflecting the relatively high absolute pass rates or the relatively small number of MOCA Minute questions that were included in this pilot. The finding that performance improved on items querying topics not covered by MOCA Minute may imply that participation in the program motivated general self-study in preparation for the MOCA Cognitive Examination.

Controlling for these factors in the multiple regression involved using univariate analysis to guide selection of covariates. To evaluate if this method of selecting covariates would affect the results, sensitivity analyses were performed for the analyses of both phases. For phase 1, including the variables that were not significant in the univariate analysis (i.e., years of first MOCA attempt within the 10-yr MOCA cycle and history of a DANS alert) increases the variance of the outcome variable explained by the predictors (R²) from 7.5 to 7.7%, increases the standard error of the coefficient of the actively participating group from 4.62 to 4.77, and decreases the coefficient of the actively participating group from 9.86 to 8.67, approximately a quarter of the standard error of that coefficient. For phase 2, including the same two variables above increases R² from 17.6 to 18.0%, has no impact on the standard error of the coefficient of the actively participating group (3.57 in both models), and decreases the coefficient of the actively participating group from 9.26 to 9.22, approximately 1% of the standard error of that coefficient. Thus, inclusion of variables not significant in simple regression had little impact on multiple regression analysis in either phase.

The major limitation of this analysis is that subjects were not randomized to participate in the MOCA Minute, such that participation could simply be a marker for those who are already more motivated to prepare for the examination and would thus have performed better in any event. As it was hypothesized that MOCA Minute participation would improve performance, it would have been unethical to randomize recruited diplomates into the participation and nonparticipation groups; we felt that it was necessary to make the program available to all diplomates eligible for the high-stakes MOCA Cognitive Examination because a passing grade is required to maintain certification. However, in other respects, the two groups were well matched, with the exception being more females enrolled in the MOCA Minute pilot program. In particular, there was no evidence that performances on the primary certification examinations, the most important predictors of MOCA Cognitive Examination performance, were different between the two groups. The fact that more females enrolled would be consistent with previous research suggesting that women are more prone to approach opportunities to gain information about their abilities in testing settings.²⁰  Some studies also suggest that female medical students may have lower self-efficacy even when their performance was comparable or higher than their male counterparts^21,22 —if this is the same case for mid-career anesthesiologists, female anesthesiologists may be more likely to take advantage of preexamination interactive learning opportunities. Although the majority of those who enrolled also actively participated, those who enrolled but did not actively participate afforded the opportunity to evaluate the effect of enrollment only without significant participation—and there was little evidence of benefit. Nonetheless, because diplomates were not randomized to participation, this study does not establish a causal relationship between MOCA Minute participation and improved performance in MOCA Cognitive Examination.

Finally, given the cognitive science and the adult learning principles presented above, it is possible that a more sophisticated program (e.g., distributing questions customized to the diplomates’ practice profile, refeeding questions based on previous answering patterns, giving diplomates more flexibility in the schedules of taking questions with a time frame, more choices of the platform in answering the questions) could even more profoundly affect physician learning. These approaches are being actively pursued.

In summary, this analysis provides evidence that voluntary enrollment and participation in a program featuring frequent knowledge assessments accompanied by targeted learning resources is associated with improved performance on a high-stakes cognitive examination and that those who participate perceive the program to be valuable. This information could help inform efforts to improve the education and evaluation of practicing physicians participating in the Maintenance of Certification process.

Support was provided solely from institutional and/or departmental sources.

Drs. Harman, Sun, and Zhou are staff members of the American Board of Anesthesiology (ABA), Raleigh, North Carolina; and Drs. Culley, Lien, and Warner are ABA Directors and receive a stipend for their participation in ABA activities.

January to June 2014

1. Antibiotic selection: Penicillin rash

2. Cerebral vasospasm: Management

3. Drug complications: Hyperkalemia

4. Electro surgery unit: Complications

5. Ethics: Refusing therapy

6. Hormonal stress response

7. Hyperbaric oxygen therapy: Indications

8. Laser endotracheal tube: Fire prevention

9. Liposuction complications

10. Pacemaker mode nomenclature

11. Poor larynx visualization: Risk factors

12. Postobstructive pulmonary edema: Management

13. Renal function tests

14. Resuscitation: Crystalloid versus colloid

15. Substance abuse: Relapse risk

16. Wall oxygen failure: Signs

July to December 2014

1. Anesthesia for tonsillectomy

2. Antiemetics: Dopaminergic side effects

3. Automated implanted cardiac defibrillator: Perioperative management

4. Cardiovascular function: Aging

5. Circle system: Dead space

6. Codeine resistance: Differential diagnosis

7. Contrast reaction: Treatment

8. Cor pulmonale: Preoperative findings

9. Electrocardiogram interpretation: Delta waves

10. Hypoxemia during laparoscopy: Differential diagnosis

11. Hypoxia: Traumatic brain Injury

12. Local anesthetic systemic toxicity

13. Malignant hyperthermia: Dantrolene indications

14. Neuromuscular blockade: Recovery

15. Postoperative renal failure: Differential diagnosis

16. Predictors of difficult intubation

17. Preterm labor: Treatment

18. Pulse pressure variation

19. Syndrome of inappropriate antidiuretic hormone: Diagnosis

20. Temperature regulation

21. Uterine artery blood flow: Effects of pH

22. Volatile agents: Production of carbon monoxide