ACADEMIC anesthesiology programs wrestle constantly with many aspects of resident education. Each year, programs in the course of ensuring academic excellence exert a substantial amount of time and effort toward recruiting the best applicants, ensuring that the same applicants will be successful and, ultimately, measuring and predicting this success. And, although it may seem unrelated to the more pressing daily concerns of the readers in private practice, they also need to know that training programs are giving adequate attention to the training process and ultimately turning out high-quality professionals available for hire. In this issue, McClintock and Gravlee evaluated a measure of success that is of interest to both the training programs and the private practice clinician alike, predicting subsequent board certification or the ultimate success of an anesthesiology trainee.1
Success is a broad term encompassing many facets; measuring success (or competency) is an issue that the Accreditation Council of Graduate Medical Education continues to attempt to define for the nation, and for all academic programs, to ensure that we, as physicians, are holding ourselves accountable to an appropriate level of professional standards.*We do not yet have a consensus or a well-defined and validated set of metrics by which we can measure the overall competency of our residents. Some of the more difficult standards to measure remain the humanistic qualities and traits that are subjective but important professionalism and interpersonal/communication skills.
Fortunately, one of the six core competencies, as defined by the Accreditation Council of Graduate Medical Education, is medical knowledge, which we have the most experience in assessing and measuring. For most specialties, one of the concrete metrics used to assess this competency is board certification. Toward this end, how do the academic programs ensure that the residents accomplish this goal without “teaching for the test” and how can they predict who needs a little more help with this objective and who is going to pass without issue? As in most disciplines of medicine, our professional societies (the American Board of Anesthesiology [ABA] in collaboration with the American Society of Anesthesiology) have created an interim set of examinations to help us measure the progress of our residents, the In-Training Exam (ITE). And, although all programs have different approaches to defining an at-risk score that suggests possible failure of the actual board certification examination and how to handle the at-risk score, sufficient emphasis rests with the ITE to define it as a high-stakes metric. For such a tool, whose use and application has significant impact on the structure and process of an educational program, we need to ensure that it is a reliable and valid measure. Until now, we had limited information and literature to support the validity and reliability of the ITE in predicting eventual success (or at least board certification) of our residents.
In this article, the authors provide a thorough and complete assessment of the ITE data available from 2000 to 2002, compared with board pass rates for the same residents who then graduated in 2002–2004. In their initial analysis, they define and provide a regression model to predict ABA Part 1 board scores based on the ITE score and a few other variables. The application of this model has its limitations, but it provides programs with more direction and insight into the predictability of the ITE scores than any information previously available. The statistics are easy to follow and the authors even demonstrate the application of this model back to the initial ITE scores in a simple graph that outlines the likelihood of an accurate prediction, all of which has definite utility for programs as they interpret ITE scores. In their final analyses, Drs. McClintock and Gravlee generate a regression model that allows one to predict the likelihood of full certification in the shortest time period (i.e. , passing both ABA Part 1 and 2 on the first attempts). Again, the application and resultant likelihood of accuracy is demonstrated for us in a straightforward method.
As with any metric associated with human behavior, we know we will never have the perfect tool that has 100% accuracy but this study gives us a starting point, both from the standpoint of infusing more scientific method into the process, as well as generating ideas for further studies to try and delineate that components of the educational process and experience have the most impact on such results as board certification and eventual success. As noted, the authors do look at other variables that were available to them, such as number of years of accreditation of any given resident's program as well as gender, foreign medical graduate status, and the number of unsatisfactory reports for any residents. The interpretation of some of this data is intriguing and might also provide stimulation for future research and investigation. One important issue to note and consider is that both the ABA Part 1 examination and the ITE have changed their formats since the collection of these data; ABA Part 1 is now computerized, both examinations have removed k-type questions and have fewer overall questions, and the ITE remains a pencil and paper examination.2Hopefully, we will see a follow-up study once the ABA has enough data on the newer examinations.
In conclusion, this article is worthy of perusal for all journal readers and, hopefully, we will see more like this in future issues. As the current environment of public concern and attention remains elevated toward all disciplines of health care and medicine, we need to reassure the public, and ourselves, that we are applying due diligence to our process and trying to ensure the best possible education and training for the future physicians in all specialties. I am certain that I join all the anesthesiology programs and program directors across the country in sending a profound thanks to the authors for providing us with a small amount of “method to our madness.”
Department of Anesthesiology, Wake Forest University School of Medicine, Winston-Salem, North Carolina. pnagle@wfubmc.edu
