The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP®) Surgical Risk Calculator was developed in 2013 as a decision support tool to “provide accurate, patient-specific risk information to guide both surgeon decision making and informed consent.”1-3 Since the introduction of the ACS NSQIP Surgical Risk Calculator, several studies have been published validating its use for a range of surgical procedures.4-7 Little has been published, however, about how this important tool may also be used for surgical education in addition to quality improvement in clinical practice.
This article describes our experience with incorporating the ACS NSQIP Surgical Risk Calculator (see Figure 1) into weekly morbidity and mortality (M&M) conferences at the department of surgery, New York Presbyterian Hospital Weill Cornell Medicine, NY.
Figure 1. ACS NSQIP Surgial Risk Calculator
What we did
Each week, the chair of quality improvement in our department selects cases for presentation at the M&M conference. The decision regarding which cases will be presented is based on several criteria, including perceived educational value, opportunities to improve patient care, and whether the cases highlight systems-related or multidisciplinary issues that may arise in our institution. Beginning in June 2014, all residents were asked to include the ACS NSQIP Surgical Risk Calculator in each M&M case presentation. We conducted a retrospective chart review from June 2014 to December 2015 to determine the implementation of the risk calculator in our M&M conferences.
What we found
We reviewed 124 M&M cases during the 18-month study period. Of those cases, 13 (11 percent) cases did not include use of the ACS NSQIP Surgical Risk Calculator. A total of seven of those 13 procedures had Current Procedural Terminology (CPT) codes that could not be accurately captured using the risk calculator. Three additional cases (2 percent) used the calculator for the wrong procedure. After excluding these 16 M&M presentations, 108 (87 percent) cases were deemed appropriate for analysis.
The median age of patients discussed in these M&M presentations was 59 years old. A total of 73 cases (68 percent) were elective while 35 cases were classified as urgent/emergent (32 percent). Of the 90 M&M cases (83 percent) that involved intra-abdominal operations, 58 (64 percent) were open procedures, and the remaining 32 (36 percent) were laparoscopic or endoscopic cases. Residents used the “surgeon adjustment” function in 21 (19 percent) cases to estimate that their patients were actually at higher risk than the calculator had determined independently.
The ACS NSQIP Surgical Risk Calculator estimated that 61 patients were at “above average” risk (56 percent) for the primary complication they developed. In contrast, 29 patients (27 percent) were estimated to be at “below average” risk, and the remaining 18 (17 percent) were estimated to be at “average” risk (see Figure 2). Of the 29 patients who were at “below average” risk, the most common complications were return to operating room (10 patients, or 34 percent) and venous thromboembolism (seven patients, or 24 percent). Eight of the “below average” cases involved procedures that had CPT codes that could not be accurately captured using the ACS NSQIP Surgical Risk Calculator, including single incision laparoscopy (2), robotics (2), and laparoscopic conversion to open (4).
Figure 2. ACS NSQIP Surgical Risk Calculator estimation of postoperative risk among M&M case presentations
What it means
To our knowledge, this is the first article to describe the use of the ACS NSQIP Surgical Risk Calculator during weekly M&M conference as a tool for educating surgery residents on risk assessment and quality improvement. Our findings confirm that it is feasible to have surgery residents incorporate the risk calculator in M&M presentations and that the ACS NSQIP Surgical Risk Calculator may serve as an important tool for educating surgery residents about the importance of risk assessment and quality improvement. This innovative approach to M&M arguably touches on all six of the Accreditation Council for Graduate Medical Education (ACGME) core competencies of resident education: patient care and technical skills, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice.8 An important next step of this study is to measure the effect on resident education of using the risk calculator during M&M presentation.
Another important finding of the study is related to the fact that most of these M&M cases were considered to have an “above average” risk for the complication ultimately developed by the patient. Several “below average” cases, as well as those M&M cases that did not involve the ACS NSQIP Surgical Risk Calculator, were for procedures that did not have CPT codes that could be accurately captured using the risk calculator. Future studies should focus on validating this tool’s ability to estimate the risk of complications for particular circumstances, including laparoscopic conversion to open, single-incision laparoscopic surgery, and robotic surgery.
Another possible next step for future research is to use the ACS NSQIP Surgical Risk Calculator as part of M&M case selection. Whereas our study found the highest proportion of M&M cases was composed of above-average risk complications, perhaps the risk calculator should be used to identify below-average risk cases for M&M presentation. These below-average risk cases may indicate postoperative complications that are worth discussing to highlight areas for quality improvement.
It should be noted that our study had several limitations. First, it involved a retrospective chart review, and selection of cases for M&M presentation was subjective, leading to a potentially biased cohort. Furthermore, the study was not designed to validate the ACS NSQIP Surgical Risk Calculator or detect any statistically significant differences in complication rates. This was not a stated objective of our study, however, because our primary goal was to provide a qualitative description of our experience using the risk calculator.
Nevertheless, our study demonstrates the feasibility of using the ACS NSQIP Surgical Risk Calculator as part of surgical education during M&M presentations. Future studies are needed to determine the effect of this important tool on uptake of the ACGME core competencies.
This topic was presented as a poster and oral presentation at the 2016 ACS NSQIP Annual Conference Monday, July 18, in San Diego, CA.
- American College of Surgeons. New ACS NSQIP Surgical Risk Calculator offers personalized estimates of surgical complications. Bull Am Coll Surg. 2013;98(10):72-73.
- Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: A decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217(5):833-842.
- Paruch JL, Ko CY, Bilimoria KY. An opportunity to improve informed consent and shared decision making: The role of the ACS NSQIP Surgical Risk Calculator in oncology. Ann Surg Oncol. 2014;2(1):5-7.
- Samson P, Robinson CG, Bradley J, et al. The National Surgical Quality Improvement Program risk calculator does not adequately stratify risk for patients with clinical stage I non-small cell lung cancer. J Thorac Cardiovasc Surg. 2016;15(3):697-705.
- Ventral Hernia Outcome Collaborative, Mitchell TO, Holihan JL, et al. Do risk calculators accurately predict surgical site occurrences? J Surg Res. 2016;203(1):56-63.
- Prasad KG, Nelson BG, Deig CR, Schneider AL, Moore MG. ACS NSQIP Risk Calculator: An accurate predictor of complications in major head and neck surgery? Otolaryngol Head Neck Surg. 2016;155(5):740-742.
- O’Neill AC, Bagher S, Barandun M, Hofer SO, Zhong T. Can the American College of Surgeons NSQIP surgical risk calculator identify patients at risk of complications following microsurgical breast reconstruction? J Plast Reconstr Aesthet Surg. 2016;69(10):1356-1362.
- Accreditation Council for Graduate Medical Education. Resident resources. Available at: www.acgme.org/Residents-and-Fellows/Welcome. Accessed October 14, 2016.