Assessing resident autonomy: What tools are available?

Training competent, confident, and autonomous surgeons is the goal of surgical residency and fellowships programs. However, the readiness of new graduates to transition to independent practice continues to be a topic of debate.¹ The Halsted model—wherein trainees receive increasing responsibility with each advancing year—was first introduced by William S. Halsted, MD, FACS, in 1904. This model continues to be the backbone of surgical training, particularly with regard to resident autonomy. However, a distinction must be made between resident independence and resident autonomy.

Independence is associated with unique opportunities within the patient’s care that allow for specific acts of decision making, while autonomy is the compilation of these opportunities that result in the resident leading the care of the patient, the operation, and the surgical patient care team.² This last decade has been marked by a progressive erosion of resident operative independence and autonomy as a result of multiple factors, such as increased scrutiny of surgical outcomes, pressure to reduce operative times, potential medicolegal action, and duty-hour restrictions.

Nearly one-fourth of graduating residents have stated they feel unprepared for independent practice.³ Only half of the faculty who have hired recent residency graduates assert these residents were sufficiently trained for clinical practice. However, by the end of the first year, these new hires are generally deemed ready to practice independently for elective and on-call cases.⁴

At the root of surgical training autonomy is discordance in the level of independence perceived by residents and attending physicians. Logging a case as a junior surgeon is generally indicative of a high level of involvement in the patient’s care. However, there is significant variability between training programs regarding how residents log cases as a surgeon. In 10 percent of these programs, residents log cases as junior surgeons if they perform 25 percent of the case, in 80 percent of programs if they complete 50 percent or more of the case, and in 10 percent if they do 75 percent or more of the case.⁵ This range in defining operative independence among residents is augmented by the varied perception of autonomy between attending physicians and residents. A multi-institutional study by Meyerson and colleagues revealed that both attending physicians and residents expect high levels of resident autonomy, but observe low levels of resident autonomy in the actual operating room (OR) setting. Correlation on the amount of autonomy given also differed, with attending surgeons perceiving a higher degree of autonomy afforded to the residents than the degree of autonomy perceived by the residents.⁶

The increasing recognition of “the autonomy gap” has spurred novel strategies to improve resident education in the perioperative setting. Some of these strategies include the use of surgical simulation,⁷ resident-run minor surgery clinics,⁸ and the study of resident and faculty behaviors.⁹ Validated and reliable assessment tools are necessary to understand and develop techniques that increase resident autonomy. This article reviews strategies and tools that measure resident autonomy in the perioperative setting.

Tools to assess resident autonomy

Similar to the preoperative evaluation of patients, residents and faculty surgeons need to engage in a dialogue that includes preoperative planning, intraoperative strategies, and postoperative care. Ideally, this discussion occurs days before the procedure for elective cases, yet these exchanges often take place at the scrub sink just before cases are to start. Faculty should engage residents before initiating each case and ask questions about the case to assess residents’ level of preparation. These queries should center on key steps of the procedure, relevant intraoperative anatomy, surgical pitfalls to avoid, possible complications, and postoperative care. This interaction allows faculty to assess resident readiness and prior experience, and it fosters trust between the faculty member and the resident.

Residents also may initiate these conversations regarding how they would like to approach the case. A three-phase observational study published in 2013 by Anderson and colleagues measured surgeons’ perioperative instruction before and after the implementation of standardized perioperative briefings, and found an increase in resident operative preparation and faculty teaching after these discussions were standardized. Additionally, the authors reported a 43.2 percent increase in intraoperative technique demonstration, a 29 percent increase in constructive feedback provided to the resident, and a 20 percent increase in nonverbal feedback, with an overall increase in perioperative teaching by faculty and residents.¹⁰

Briefing, intraoperative teaching, debriefing model

A comparable method to perioperative instruction is the briefing, intraoperative teaching, debriefing (BID) model. Introduced by Roberts and colleagues, the BID model was created to fit within an attending surgeon’s current practice.¹¹ This model takes advantage of teaching moments that occur before, during, and at the end of any operation.

The briefing is a short interaction that takes place at the scrub sink, and serves to assess resident knowledge level and preparation and to establish objectives for the case based on resident experience. Setting objectives specific to the learner allows faculty to tailor their intraoperative teaching approach.

The focus of intraoperative teaching is to guide the resident through the procedure, emphasizing the learning objectives set in the briefing. Intraoperative teaching focuses on immediate feedback and direct guidance on the task being performed.

The debriefing may occur after the procedure is complete or during incision closure. This last step of the BID model is divided into four components: reflection, rules, reinforcement, and correction. The debriefing should focus on the learning objectives set before the case. Debriefing allows residents to reflect on their intraoperative performance and for attending physicians to provide feedback. This interaction also should include positive reinforcement of appropriate clinical knowledge and technical skills, correction of mistakes, and a guiding principle or takeaway for future operations.

Table 1. Zwisch scale of progressive autonomy

The Zwisch scale and SIMPL

An additional tool that is commonly used to measure resident progression of autonomy in the OR is the Zwisch scale.¹² This four-level scale rates resident operative autonomy based on the level of assistance required: show and tell, active help, passive help, and supervision only (see Table 1). George and colleagues demonstrated minimal disruption to surgical workflow with the use of the Zwisch scale on a smartphone platform that allows for timely evaluation of resident autonomy and operative performance.¹³ The Zwisch scale is applicable to any procedure and to all levels of training. The tool offers residents a reliable rating across evaluators that has been demonstrated to correlate with postgraduate year level. However, rating is highly dependent on resident experience with the procedure and may not provide a global picture of a resident’s level of autonomy based on individual experience with the procedure.

The Zwisch scale has been applied and incorporated into many smartphone assessment applications, including the System for Improving and Measuring Procedural Learning (SIMPL) tool.¹⁴ SIMPL integrates the validated Zwisch scale to assess the level of faculty guidance provided and level of autonomy granted to the resident during an operation into an easy-to-use smartphone interphase. SIMPL, developed by the Procedural Learning and Safety Collaborative, allows for intraoperative performance assessment in a quick and real-time manner. SIMPL is a software platform that facilitates resident intraoperative evaluation for every procedure in which they participate. The application consists of a three-question assessment for both the resident and the attending physician (see Table 2). In addition to incorporating the Zwisch scale, SIMPL adapted and integrated operative performance ratings to measure resident readiness for independent practice, practice-ready performance being the target goal for surgical training. SIMPL also takes into account increasing operative complexity to avoid confounding the assessment of resident autonomy with resident operative performance.

Bohnen and colleagues showed how SIMPL can be integrated into surgical training programs to improve the frequency and timeliness of resident intraoperative performance assessment.¹⁴ A resident or attending physician may initiate a SIMPL assessment. To do so, information on who performed the operation, the procedure performed, and the date the procedure was performed is necessary. The three-question assessment is completed by both the resident and attending physician, independent of each other, and the attending physician may add a dictation as feedback for the resident. Of note, once a SIMPL assessment is initiated, it expires in 72 hours to avoid memory bias as evaluations performed after three days lack necessary clarity and details for a complete observation.¹⁵

Table 2. SIMPL questions

Surgical Procedure Feedback Rubric

The Surgical Procedure Feedback Rubric (SPR) evaluates resident intraoperative performance during a single operative encounter and in the setting of competency-based education.¹⁶ SPR measures three factors: OR preparation, technical skills, and intrinsic competencies. This tool differs from the other assessment instruments described in this article because it intertwines operative performance with the level of case complexity, and therefore it can be used to assess resident performance throughout the entire training process, not only during a single operative encounter. Using one tool throughout the entirety of training would increase consistency of feedback provided to residents.

Within SPR, each individual operative encounter is divided into multiple stages or attributes (a list of attributes assessed is listed in Table 3). Furthermore, each stage of resident performance is evaluated as needing attention, developing, or achieving the set goal. Most of the attributes evaluated in SPR fall within the OR preparation factor because of the vast knowledge and clinical skills involved in this process. SPR was initially evaluated in a single-institution study. A follow-up study incorporating two different academic institutions within the study sample demonstrated that SPR is able to discriminate between residents’ intraoperative performances by training year, although this tool was initially developed as part of a competency-based training model. As with other workplace-based assessments, additional studies on the validity and applicability of SPR are needed.

Table 3. List of attributes assessed in SPR

O-SCORE

Another rubric-based assessment tool that has been incorporated into multiple residency programs is the Ottawa Surgical Competence Operating Room Evaluation (O-SCORE).^17,18 O-SCORE was developed at the University of Ottawa’s division of orthopaedic surgery to assess trainee competence independent of postgraduate year. The O-SCORE incorporates different facets of tools discussed in this article, such as the Zwisch scale, to determine the level of assistance provided by the faculty surgeon (see Table 4). The O-SCORE evaluates resident preoperative preparation by assessing procedural indication, patient positioning, operative approach, key instruments, knowledge of operative steps, and pitfalls to avoid, as well as visuospatial skills (see Table 5). The resident’s postoperative plan, operative efficiency, and ability to communicate with the operative team are also taken into account in this assessment.

As with SPR, the O-SCORE incorporates case complexity, and also offers supervising surgeons the opportunity to summarize residents’ ability to safely perform procedures independently, which is similar to SIMPL’s dictation feature and allows faculty to add comments on resident performance. Unique to the O-SCORE is the opportunity for evaluators to provide constructive criticism by listing one aspect of the procedure that was done well and one that needs improvement, thereby providing an organized, comprehensive way to assess resident autonomy.

Table 4. Five-point competency scale to assess resident surgical competence

Table 5. O-SCORE items rated on the five-point competency scale

Conclusion

Assessing and tracking resident autonomy in the perioperative setting is critical to understanding the resident’s overall competency level during training and when entering practice. Given the current concern regarding the decrease in resident autonomy during training, surgery programs should consider implementing tools to assess perioperative resident autonomy as a means of evaluating residents for promotion. Smartphone-based platforms seem especially useful to assess operative independence, as they are accessible for both faculty surgeons and residents. However, comparisons of these assessment tools are lacking in the literature, and further studies are necessary prior to standardization.

References

1. Hashimoto DA, Bynum WE, Lillemoe KD, Sachdeva AK. See more, do more, teach more: Surgical resident autonomy and the transition to independent practice. Acad Med. 2016;91(6):757-760.
2. Sandhu G, Magas CP, Robinson AB, Scally CP, Minter RM. Progressive entrustment to achieve resident autonomy in the operating room. Ann Surg. 2017;265(6):1134-1140.
3. Coleman JJ, Esposito TJ, Rozycki GS, Feliciano DV. Early subspecialization and perceived competence in surgical training: Are residents ready? J Am Coll Surg. 2013;216(4):764-771.
4. Nakayama DK, Taylor SM. SESC Practice Committee survey: Surgical practice in the duty-hour restriction era. Am Surg. 2013;79(7):711-715.
5. Patel M, Bhullar JS, Subhas G, Mittal V. Present status of autonomy in surgical residency—a program director’s perspective. Am Surg. 2015;81(8):786-790.
6. Meyerson SL, Sternbach JM, Zwischenberger JB, Bender EM. Resident autonomy in the operating room: Expectations versus reality. Ann Thorac Surg. 2017;104(3):1062-1068.
7. Kim SC, Fisher JG, Delman KA, Hinman JM, Srinivasan JK. Cadaver-based simulation increases resident confidence, initial exposure to fundamental techniques, and may augment operative autonomy. J Surg Educ. 2016;73(6):e33-e41.
8. Wojcik BM, Fong ZV, Patel MS, et al. The resident-run minor surgery clinic: A pilot study to safely increase operative autonomy. J Surg Educ. 2016;73(6):e142-e149.
9. Torbeck L, Wilson A, Choi J, Dunnington GL. Identification of behaviors and techniques for promoting autonomy in the operating room. Surgery. 2015;158(4):1102-1110.
10. Anderson CI, Gupta RN, Larson JR, et al. Impact of objectively assessing surgeons’ teaching on effective perioperative instructional behaviors. JAMA Surgery. 2013;148(10):915-922.
11. Roberts NK, Williams RG, Kim MJ, Dunnington GL. The briefing, intraoperative teaching, debriefing model for teaching in the operating room. J Am Coll Surg. 2009;208(2):299-303.
12. DaRosa DA, Zwischenberger JB, Meyerson SL, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ. 2013;70(1):24-30.
13. George BC, Teitelbaum EN, Meyerson SL, et al. Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ. 2014;71(6):e90-e96.
14. Bohnen JD, George BC, Williams RG, et al. The feasibility of real-time intraoperative performance assessment with SIMPL (System for Improving and Measuring Procedural Learning): Early experience from a multi-institutional trial. J Surg Educ. 2016;73(6):e118-e130.
15. Williams RG, Chen XP, Sanfey H, Markwell SJ, Mellinger JD, Dunnington GL. The measured effect of delay in completing operative performance ratings on clarity and detail of ratings assigned. J Surg Educ. 2014;71(6):e132-e138.
16. Toprak A, Luhanga U, Jones S, Winthrop A, McEwen L. Validation of a novel intraoperative assessment tool: The Surgical Procedure Feedback Rubric. Am J Surg. 2016;211(2):369-376.
17. Frank JR, Snell L, Sherbino J, editors. CanMEDS 2015 Physician Competency Framework. Ottawa: Royal College of Physicians and Surgeons of Canada; 2015.
18. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): A tool to assess surgical competence. Acad Med. 2012;87(10):1401-1407.