Facilitation of Resident Scholarly Activity: Strategy and Outcome Analyses Using Historical Resident Cohorts and a Rank-to-Match Population

RESEARCH is indispensable to the future growth of anesthesiology. Research training of resident physicians will lay the foundation for the development of future clinician scientists.^1,2  More fundamentally, providing all anesthesiology residents with educational opportunities in the form of scholarly activities is a key responsibility of any anesthesiology program to nurture the next generation of academic anesthesiology faculty members. Indeed, the Accreditation Council for Graduate Medical Education (ACGME) Anesthesiology Residency Review Committee recently reported that lack of faculty scholarly activity was the second biggest reason for citations in their 2012 program review.^*

Despite the Anesthesiology Residency Review Committee’s^† emphasis on residents’ research and scholarly activities, methods to achieve this goal have not been fully established. Resulting outcomes of these initiatives have scarcely been reported in the field of anesthesiology.³  This is an concerning contrast to other medical specialties in which a number of resident research initiatives have been reported, including internal medicine,^4–13  family medicine,^14–20  radiology,^21–23  emergency medicine,^24,25  pediatrics,^26,27  psychiatry,^28,29  rehabilitation medicine,³⁰  orthopedics,³¹  general surgery,³²  and urology.³³ 

We hypothesized that implementing structured initiatives to promote resident research in an anesthesiology department would: (1) increase anesthesiology residents’ participation in research, (2) improve residents’ research productivity, and (3) facilitate faculty members’ involvement in resident research. Evaluation of the effectiveness of any education initiative that could affect the entire residency program and even the culture of the department is often difficult due to the lack of an appropriate control group.³⁴  In this study, the effectiveness of education initiatives was evaluated not only by evaluating the historical resident class cohort in the department, but also by assessing the rank-to-match population as a control group.¹² 

The University of Pittsburgh Institutional Review Board approved the study as an exemption (Institutional Review Board #PRO10120290).

In the first part of the study, prospectively recorded data, including scholarly activities of anesthesiology residents graduating between 2003 and 2011 from the University of Pittsburgh Medical Center (UPMC), were reviewed and analyzed. “Class of 2003” refers to residents who entered the 3-yr anesthesiology residency program on July 1, 2000 and graduated on June 30, 2003, and so forth.

In the second part of the study, a rank-to-match population was compared with residents who conducted research at UPMC. The study group consisted of residents who graduated from the UPMC anesthesiology residency program from 2009 to 2011. The control group consisted of rank-to-match residents,¹²  candidates who were on our department’s residency ranking list and who would have matched into our program had they ranked our program high enough on their list. The list of rank-to-match residents was retrieved from our department’s residency program administration office and included the candidate’s full name, medical school, and name of the residency program with which he or she matched.

The following research initiatives were implemented in our department during the study period. For all anesthesiology residents, (1) an annual research introductory lecture (1-h lecture of basic grantsmanship, steps in research activity, and introduction of potential faculty research mentors) was added at postgraduate year (PGY)-3 as part of the mandatory resident lecture series in academic year (AY) 2006 (July 1, 2005 to June 30, 2006), (2) meeting schedules and abstract deadlines for local/state/national anesthesiology meetings have been publicized to residents since AY2006, (3) residents’ research achievements (publications, abstract presentations, and grants) were announced on the department’s Web site since AY2006, (4) in AY2007, a faculty member was appointed director of the resident research rotation (RRR) who is responsible for up to 6 month-long ACGME-approved research elective rotation undertaken mainly during PGY-4, and (5) a 90-min research problem–based learning discussion (PBLD) was developed and has been presented annually since AY2010 as part of a mandatory resident PBLD series to teach residents how to conduct a randomized clinical trial or write a case report and conduct a retrospective case series study.³⁵ 

The following initiatives were implemented in AY2010 for residents electing the RRR: (1) a formal application process to the rotation includes a presubmission consultation with the rotation director, submission of a formal application letter by the resident with justification for the months requested based on the number and the quality of intended projects, and submission of a commitment letter by the faculty research mentor(s), (2) attendance is mandated at a weekly research meeting with the rotation director during the rotation period, (3) self-announcement of a weekly research milestone for each research project is mandated during the weekly meeting with the rotation director and its achievement over the next week is evaluated, (4) preparation of an abstract and an article in the research month(s) is encouraged and monitored by the director, and (5) submission of a research abstract to local and state resident research competitions is strongly recommended and monitored by the director.

The department was awarded a National Institutes of Health/National Institute of General Medical Science T32 postdoctoral training grant in AY2008 (“Research Training in Anesthesiology and Pain Management”, T32 GM075770: Program Director, Yan Xu, PhD: July 01, 2007 to June 30, 2017) with the “goal of developing clinician-scientists who will be leaders in the field of anesthesiology research, by providing rigorous postdoctoral research training with an emphasis on hypothesis-driven laboratory or clinical research…the research and training are specifically designed to promote a research career addressing problems in anesthesiology and provides opportunities to enhance their research training with a clinical perspective.” This research fellowship has been offered to residents who are interested in pursuing an academic career. Although the department previously looked favorably upon residency candidates with an interest in research, since AY2006, there has been a more concerted effort to identify and recruit into the anesthesiology residency program those residency candidates who have a research degree/background. In AY2009, a one and a half day research training symposium was held to educate residents “about the practical aspects of research from the development of a research theme, literature review, project development, and data analysis to scientific writing, oral presentation, grantsmanship, and research integrity.” This program served as the basis for the research PBLD instituted in AY2010. In AY2009, the T32 program created a departmental research day to highlight current research in the department. In AY2010, the event expanded into a Multi-Departmental Trainees’ Research Day, which was held in collaboration with the departments of critical care medicine, emergency medicine, and physical medicine and rehabilitation. The Trainees’ Research Day has been held annually since.

For department faculty members, (1) research mentorship of anesthesiology residents has been recognized as part of a productivity-based faculty compensation system since AY2004,³⁶  (2) an annual research mentorship award was established to recognize an outstanding faculty mentor in AY2006, and (3) faculty mentorship has been recognized when residents’ research achievements (publications, abstract presentations, and grants) are featured on the department Web site.

In this study, resident scholarly activity was defined and evaluated based on the Scholarly Activity Point (SAP) system,³⁷  in which residents are given points reflecting the quantity and quality of their scholarly products. The SAP system is an objective scale used in our department to convert each scholarly product to points. This system was created based on the faculty merit matrix system used in our department since AY2004.³⁶  In brief, scholarly activities included in this study were mainly in the domain of “the scholarship of discovery,” with some inclusion of the other three areas of scholarship (the scholarship of integration, the scholarship of application, and the scholarship of teaching) advocated by Boyer,³⁸  Glassick,³⁹  and Hutchings and Shulman.⁴⁰  Scholarly products included were abstracts accepted for presentation in local, regional, national, and international meetings, articles (case reports and original research reports) published in peer-reviewed journals, grants awarded by intramural and extramural funding agencies, book chapters, published books, and Institutional Review Board–approved research protocols. The quality of each scholarly product was evaluated using the SAP system, which takes into account the resident’s contribution (i.e., rank of authorship), product impact (i.e., podium presentation, national presentation, award, peer-reviewed publication journal impact factor), and product complexity (i.e., case report vs. original research). This system converts each scholarly product into a single SAP value. Scholarly productivity of each resident and class was described by the sum of SAPs.

Medically challenging case presentations at the meetings of the American Society of Anesthesiologists, the International Anesthesia Research Society, or the Post Graduate Assembly were counted as case report presentations at national meetings. Multiple submissions of the same abstract to local/state/national meetings were counted independently.

Intramural presentations at subspecialty rotations, grand rounds, mortality and morbidity rounds, and journal clubs were excluded from resident scholarly activity in this study.

Only scholarly activities based on works during internship (PGY-1) and anesthesiology residency (PGY-2 to PGY-4) were included in the study. The follow-up period for including scholarly activities was at least 2 yr after graduation from the residency program; therefore, the time period for which scholarly activities by Class of 2011 residents were considered was extended to June 30, 2013. Scholarly activities based on works performed before the commencement of the PGY-1 were excluded (i.e., original research articles accepted for publication during the anesthesiology residency, but work was based on research performed during medical school).

Given the difficulty of collecting all scholarly activity data from the rank-to-match control group, only peer-reviewed publications based on work completed during the residency and internship were included in this study. The PubMed Web site^‡ was used to search for peer-reviewed articles using last name, first, and middle initials, and name of the residency program. Each citation was retrieved, and the full name of the authors and the affiliated institutions were identified as rank-to-match residents. Publication dates between July 1 of the PGY-2 yr and 1 yr after graduation from the residency were used to determine whether the publication was based on work completed during the residency. Publications before and after the above period were further reviewed. Only when the physician’s position was listed as “resident” was the work considered to have occurred during the residency period and included for further analysis. To include original research publications, a minimum follow-up period of 2 yr was ensured.^41,42 

The following data were collected from residency graduates: (1) Abstracts: total number of presentations (including multiple presentations of the same abstract at different meetings, including local, regional, and national meetings); number of independent projects (excluding multiple presentations); number of original studies in independent projects, and number of independent projects with first authorship, (2) Articles: total number of publications, number of original studies, number of first-authored publications, (3) Other scholarly activities including book chapters, research grants, and others (e.g., letters to the editor, study Institutional Review Board approvals, book authorship, creation of educational materials). Each scholarly product was also converted to SAPs. Overall residents’ involvement in research activity was represented by the number of residents with SAPs.

In terms of peer-reviewed articles, additional analyses were made to find the percentage of resident-authored peer-reviewed publications in a given AY (July 1 to June 30) and the number and percentage of resident first authorship among these publications. The percentage of original study articles was analyzed among all department publications as well as in resident-authored publications. The percentage of peer-reviewed articles on which a resident served as the first author was also analyzed in the resident-authored publications.

The number of residents who elected the RRR was collected. The number of peer-reviewed publications, first-authored publications, and original study publications by RRR residents and by non-RRR residents were compared.

The number of anesthesiology faculty members who mentored resident research in a given AY was also collected.

Residents’ postgraduate work types were collected and categorized as either fellowships, academic practices, private practices, or military obligations. Updated information on postfellowship/military obligation workplace was sought and categorized as academic practice or nonacademic practice.

The following factors were compared between the UPMC residents and the rank-to-match residents: sex, the number of U.S. medical graduates, and the number of residents with peer-reviewed publications at the time of residency application. The following main outcomes of residency scholarly activity were also compared: (1) the number of residents who published peer-reviewed article(s) based on work completed during residency; (2) the quality of each peer-reviewed publication indicated by SAPs,³⁷  which were calculated as follows: SAP = 150 × (1 [original article], 0.75 [review article], or 0.5 [case report]) × (1 [first author] or 0.5 [other author]) × (impact factor of the year of publication or 0.5 [if impact factor < 0.5]); and (3) the scholarly productivity of resident calculated as the sum of the publication SAPs.

The impact factor of each journal available at the time of publication of a given article was used, which was verified using Journal Citation Reports^® (ISI Web of Knowledge^SM; Thomson Reuters, New York, NY).^§

The data were described as mean ± 1SD, median with range (minimum–maximum), or the number with percentage, as appropriate. Categorical variables were analyzed using Pearson chi-square test or Fisher exact test, as indicated. For continuous variables, comparison was performed using the unpaired Student t test for data with parametric distribution or the Mann–Whitney U test for data with nonparametric distribution. Comparison of continuous values among the classes was performed using the analysis of variance with post hoc test using Bonferroni method or Kruskal–Wallis test with post hoc test using Dunn multiple comparison method for the data with nonparametric distribution. The level of significance was set at P value less than 0.05. Statistical analysis was performed using GraphPad Prism 6 (GraphPad Software, Inc., La Jolla, CA).

To analyze the trend of scholarly activity output before and after the initiatives, we performed a segmented regression analysis based on the method described by Wagner et al.⁴³  Using the mean SAP scores per graduated classes, we fit a model to predict mean SAP using three variables: (1) Year (“ClassYear”) which we interpreted as the baseline trend, (2) Intervention (“Year07orLater,” a binary variable), which we interpreted as a one-time change immediately after the intervention, and (3) Time after intervention (“TimeAfter07”), which we interpreted as the trend after the intervention. First, we fit a full model containing all three of these variables. Then we further analyzed the trend with a parsimonious model by using stepwise selection to remove nonsignificant terms. The level of significance was set at P value less than 0.05. This statistical analysis was performed using IBM^® SPSS^® Statistics 20 (IBM^®, New York, NY).

Scholarly activities by class members who graduated from our program increased in the number of abstract presentations, authorships on articles, and authorships on book chapters (table 1). Residents’ overall involvement in research activity significantly increased in the Postinitiative group (Class 2007 to Class 2011) compared with the Preinitiative group (Class 2003 to Class 2006) (89.2% [58 in 65 residents] vs. 64.8% [35 in 54]; P = 0.0013).

Each scholarly activity was converted to SAPs for the segmented regression analysis.⁴³  First, we found that the base line trend or Year07orLater was not significant (P = 0.88 and P = 0.15, respectively) nor was TimeAfter07 (P = 0.057). A further parsimonious model analysis by using stepwise selection to remove nonsignificant terms revealed TimeAfter07 statistically significant (P = 0.002), suggesting that after the intervention, there was a significant increase in mean SAP scores. The coefficient was 80.6, meaning mean SAP scores rose by that amount on average after Class Year 2007 (fig. 1).

The total number of peer-reviewed publications from the department overall increased, as well as the rate of original research articles among all the publications. As for residents, the number and the rate of authorship in these publications showed a similar trend of increase (table 2). An increase trend was observed in the rate of resident first authorship since AY2008 and the rate of resident original research articles since the AY2010.

The residents who elected the RRR significantly increased from 7.4% (4 residents of 54) in the Preinitiative group (Class 2003 to Class 2006) to 32.3% (21 of 65) in the Postinitiative group (Class 2007 to Class 2011) (P = 0.0012). Overall, the RRR residents (n = 25) published 36 articles whereas the non-RRR residents (n = 94) published 33 articles. The RRR residents authored more original articles than the non-RRR residents (83.3 vs. 33.3%; P < 0.0001). Of note, there was no statistical difference in the frequency of first authorship among articles by both resident groups (52.8 vs. 57.6%; P = 0.69).

The number of anesthesiology faculty members who provided research mentorship to residents in a given AY increased from 9.2% (55 faculty members of 595) in the Preinitiative group (AY2003 to AY2006) to 23.9% (161 of 673) in the Postinitiative group (AY2007 to AY2010) (P < 0.0001) (fig. 2).

Each graduate’s initial work destination upon completion of the residency program was summarized and compared between the Preresearch Initiative (Classes 2003–2006) and the Postresearch Initiative (Classes 2007–2011) (table 3). A statistically significant greater number of Postresearch Initiative residents entered fellowships (42.6 vs. 70.8%; P = 0.002). However, upon fellowship completion, the number of residents who entered academic practice did not show a statistical difference (40.7 vs. 50.8%; P = 0.28).

Demographic characteristics did not differ significantly between the UPMC residents (n = 38) and the rank-to-match residents (n = 220): male sex (25 [65.8%] vs. 142 [64.5%]; P = 0.89); U.S. medical graduates (34 [89.5%] vs. 188 (85.5%]; P = 0.50); and applicants with preresidency publications (4 [10.5%] vs. 29 [13.2%]; P = 0.088).

The percentage of residents whose article(s) were published in peer-reviewed journals during residency was significantly higher for the UPMC residents (55.3% [21] vs. 13.2% [29]; P < 0.0001; odds ratio 8.1 with 95% CI of 3.9 to 17.2). The total number of publications was 28 (0.7 publication per resident) versus 42 (0.2 per resident). As measured with SAPs, the quality of each publication and the overall productivity of residents with publications did not significantly differ between the UPMC residents and the rank-to-match residents (P = 0.44, P = 0.37, respectively); the median SAPs of each publication was 184 (99 to 245 [interquartile range]) versus 146 (60 to 272), and the median SAPs earned per resident who wrote articles published in peer-reviewed journal(s) was 206 (99 to 426) versus 156 (60 to 411).

The research initiatives adopted in a single U.S. anesthesiology department over 7 yr resulted in an overall increase in resident research involvement and productivity compared with the historical resident cohort. This increase in research productivity was especially notable in residents electing to participate in the ACGME-approved research rotation; they were more productive not only in overall publications but also in original research projects compared with those who did not elect the research rotation. The number of faculty mentors also increased. An additional evaluation was performed using rank-to-match residents as the control group to minimize a potential historical selection bias of residents entering our residency program. This evaluation further confirmed that the research initiatives resulted in a greater number of residents who authored peer-reviewed publications than the control group.

Systematic surveys given to program directors of various medical specialties have shown the importance of a structured resident research program. In 1996, Alguire et al.⁴  reported that only 37% of ACGME-accredited internal medicine residency programs had an organized, comprehensive research curriculum. In 2005, Levine et al.⁷  reported that internal medicine programs supported resident scholarship through research curricula (47%), funding (46%), and protected time (32%). On the basis of a recent survey given to all 450 U.S. family medicine program directors, Crawford and Seehusen¹⁹  identified five factors associated with increased resident participation in research projects: (1) resident recognition of scholarship, (2) dedicated resident time for research, (3) local research day, (4) academic advancement linked to scholarship, and (5) residency director performs research. In nonanesthesiology medical specialties, a number of single-institutional research initiatives have been implemented to facilitate scholarly activity by resident physicians in various disciplines with success,^{5,6,8–16,20,21,24,26–29,31,32}  trying to address the well-known barriers to successful implementation of resident scholarly activities, including lack of faculty mentoring and time,⁴  competing resident clinical responsibility,^4,44  and funding limitations.^4,45  The status of anesthesiology residency research education has only recently been investigated. A survey of U.S. anesthesiology residency programs showed 32% of programs had a structured resident research education program.³  Structured programs were more likely to be curriculum based, require resident participation in a research project, and provide specific training in presentation and writing skills; these programs were associated with higher resident research productivity; 40% of structured programs had more than 20% of trainees with publications in the last 2 yr compared with 14% of departments with unstructured programs (difference, 26%; 99% CI of 8 to 51%; P = 0.01).³ 

Our research initiatives include several key factors for improving resident engagement in research and productivity: an introductory lecture, an interactive research PBLD, and a RRR with a rotation director.⁶  Recognition of resident scholarship and faculty mentors¹⁹  has been stressed in department Web announcements and at the residency graduation ceremony. Up to 6 months of dedicated research time¹⁹  is provided for senior residents. An annual local research presentation opportunity¹⁹  is available. Residency directors have performed research¹⁹  in the fields of clinical anesthesiology and education. Furthermore, the T32 fellowship program was heavily involved in the establishment of the research PBLD, initiated an annual departmental Research Day, and created an impetus for the residency program to recruit more residents with a propensity toward research. Of note, the residency program announced no expectation to the residents that research activity based on the above initiatives was mandatory. The residency program has recognized other traditional scholarly activities (i.e., authorship on book chapters, presenting at departmental grand rounds) as fulfillment of the ACGME scholarly activity requirement. Therefore, resident engagement with any research activity has been voluntary. A research elective rotation of up to 6 months, which has been approved as a formal anesthesiology residency rotation by the ACGME, had long been in place before this research initiative.

We do not think the increase in resident research productivity and increase in research mentorship observed in our program should be attributed to the incentives provided by these activities. For the faculty, a productivity-based compensation system has been in place since AY2004; therefore, the compensation plan could not be responsible for the increase in quality and quantity of resident research activities seen in recent years. Moreover, resident research mentorship is one of many ways to earn merit matrix points to regain salary at risk for academic faculty members.³⁶  Department leadership has neither particularly stressed nor mandated resident research mentorship per se for faculty members. For the residents, especially those who elected the RRR, freedom from clinical duties and presenting an abstract at multiple meetings might be viewed as incentives. However, such an ACGME-approved research elective had been available for a long period of time for any resident who wished to do so. Furthermore, our program did not offer additional meeting dates or extra funding for such meeting presentations.

In general, methods to evaluate system-wide education initiatives have not been well-established. A long-term effect of department-wide initiatives could change the culture in the department and does not allow establishment of a well-defined control group.³⁴  In our study, the education initiatives could impact the resident candidate application and selection process. Potentially, an increased number of applicants with propensity toward research would apply to our program and/or applicants with research affinity would be favored in the selection process. Unfortunately, many studies on research initiatives only relied on historical cohort data, which may have intrinsic limitations. To address this issue, we used rank-to-match residents as the control group. This method uses a population of residents who could have entered into the residency program as the control group.¹²  This population is an ideal counterpart to residents exposed to research initiatives, because the overall quality of the rank-to-match residents should be similar to, or even theoretically better than, that of the study residents. In this comparison, we found that significantly more residents published during residency under the educational initiative, while rank-to-match residents with peer-reviewed journal articles as medical students continued to publish as anesthesiology residents; however, those without publications did not publish during residency. This finding may indicate the effectiveness of the educational initiative and the importance of medical school students’ early exposure to research activity. It is important to note that the rank-to-match analysis carries an important asymmetrical bias; we have complete information on residents within the program, whereas we have a high potential to be missing information on rank-to-match residents. For example, a resident who gets married and changes one’s name, suffers an untimely death, or changes specialty will not be found in the publication search. This bias may appear to increase the differences between the two groups in favor of the program residents.

This study has several important limitations. First, it is difficult to quantify each initiative’s effect on the positive impact on this residency program. These initiatives have worked synergistically to enhance the culture in this department to promote resident research activity. Second, reproducibility of these research initiatives at other institutions and their effect has not been verified. Third, the scholarly activities measured in this study did not necessarily entirely cover the four areas of scholarship.^38–40  We acknowledge that we focused on research output, which is traditionally well defined and easier to identify and evaluate. Fourth, we were not able to collect and evaluate the previous research knowledge and the in-training experience of the rank-to-match group. Therefore, the study does not address residents’ other attributes, such as knowledge of basic sciences, statistics, writing, and other skills that residents may need as well as other program attributes that may influence a resident’s scholarly activity. Fifth, there could be a concern for potential abuse of such a faculty compensation system where faculty members unprofessionally list residents as authors on their publications. The fact that the majority of our residents were listed as first authors on abstracts and publications should show that our practice has been based on sound professionalism. Still, departmental oversight might be required to check this type of faculty compensation practice. Sixth, the long-term effect of the initiatives on residents has not been evaluated, given the limited duration of the follow-up period. Last, whether these research initiatives increase National Institutes of Health anesthesiology research funding remains to be seen. Obviously, such introductory research initiatives by no means match the research training requirement for candidates to pursue careers as independent researchers. However, the initiatives may lead to more residents seeking training grant opportunities, which are designed for further training to develop independent physician-scientists.

In conclusion, implementing structured research initiatives at an anesthesiology department is feasible and these research initiatives showed increased resident research involvement, research productivity, and faculty mentorship. The research initiatives were further evaluated using rank-to-match residents as the control group and demonstrated that these initiatives increased the number of residents who published articles in peer-reviewed journals.