Examining the validity of the ACS-NSQIP Risk Calculator in plastic surgery: lack of input specificity, outcome variability and imprecise risk calculations
==========================================================================================================================================================

* Cassandra Johnson
* Insiyah Campwala
* Subhas Gupta

## Abstract

American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) created the Surgical Risk Calculator, to allow physicians to offer patients a risk-adjusted 30-day surgical outcome prediction. This tool has not yet been validated in plastic surgery. A retrospective analysis of all plastic surgery-specific complications from a quality assurance database from September 2013 through July 2015 was performed. Patient preoperative risk factors were entered into the ACS Surgical Risk Calculator, and predicted outcomes were compared with actual morbidities. The difference in average predicted complication rate versus the actual rate of complication within this population was examined. Within the study population of patients with complications (n=104), the calculator accurately predicted an above average risk for 20.90% of serious complications. For surgical site infections, the average predicted risk for the study population was 3.30%; this prediction was proven only 24.39% accurate. The actual incidence of any complication within the 4924 patients treated in our plastic surgery practice from September 2013 through June 2015 was 1.89%. The most common plastic surgery complications include seroma, hematoma, dehiscence and flap-related complications. The ACS Risk Calculator does not present rates for these risks. While most frequent outcomes fall into general risk calculator categories, the difference in predicted versus actual complication rates indicates that this tool does not accurately predict outcomes in plastic surgery. The ACS Surgical Risk Calculator is not a valid tool for the field of plastic surgery without further research to develop accurate risk stratification tools.

*   Risk
*   Morbidity
*   Evidence-Based Medicine
*   Academic Medical Centers

## Introduction

Prediction of postoperative risk is vital to every surgeon, both for his practice and the health of his patients. Assessing potential outcomes offers the patient the opportunity to assess risk and allows informed decision-making. Previously, this has been estimated based on surgeon's experience adjusted by outcomes in published research, such as clinical trials and case reports. However, this allows the administration of individual bias and may not account for all associated risks.1

Seeing this need, the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) collected clinical data from almost 400 hospitals, combining preoperative risk factors in order to predict postoperative risk of complication.2 They used the data to create the Surgical Risk Calculator, which allows physicians to offer patients a risk-adjusted 30-day surgical outcome prediction. Outcomes include 21 defined morbidities (including the following categories: wound, respiratory, urinary tract, central nervous system, cardiac, and five others), as well as mortality. Patients and physicians can simply enter patient information—gathered from the history, physical, and previous surgical information—into the website, and receive their risk of complication. The outcomes are offered as a percentage of the average person's risk of complication with the given procedure.3

While mortality and morbidity have declined and patient satisfaction rates have risen in certain populations since the creation of NSQIP,2 ,4 several limitations to this predictor have been documented.

The data collected for the creation of the Surgical Risk Calculator are from 393 hospitals, which make up only 10% of hospital in the USA. There is likely variation between hospitals, as far as outcomes by different surgeons and different programs, which is not accounted for. Also, only the variables that are considered by the NSQIP will factor into the calculation. While surgeons are allowed to adjust this score using the Surgeon Adjustment Score (SAS), there is no evidence that this allows more accurate predictions.5

While the ACS-NSQIP offers the only multidisciplinary surgical care predictor, we question its application to the field of plastic surgery. Surgical morbidity that is monitored is the same for all procedures across all specialties, and is not specific for concerns within plastic surgery. Its external validity has been tested in other subspecialties,6 which have found that the predictor overestimates risks of complications.7 ,8 While its external validity has been tested in other subspecialties, it has not yet been validated in plastic surgery procedures. The abstract for this manuscript was published in the *Journal of Investigative Medicine* in January 2016 and presented at the Western Student and Resident Medical Research Forum in January 2016.

## Materials and methods

After appropriate approval from the Institutional Review Board at our level I trauma center was obtained, a retrospective analysis of all intradepartmental, plastic surgery-specific complications from a thorough review of a quality assurance database from September 2013 to July 2015 was performed. Patient information, including preoperative risk factors, was entered into the ACS Surgical Risk Calculator, and predicted outcomes were compared to actual morbidities.

Risk factors included in analysis were:

<colgroup span="1" id="colgroup-1"><col align="left" span="1" id="column-1" class="table-left"></col><col align="left" span="1" id="column-2" class="table-left"></col><col align="left" span="1" id="column-3" class="table-left"></col><col align="left" span="1" id="column-4" class="table-left"></col><col align="left" span="1" id="column-5" class="table-left"></col><col align="left" span="1" id="column-6" class="table-left"></col><col align="left" span="1" id="column-7" class="table-left"></col></colgroup>
| Age group                        | Sex                             | Functional status     | Emergency class                       | ASA class | Wound class         | Chronic steroid use |
|:-------------------------------- | ------------------------------- | --------------------- | ------------------------------------- | --------- | ------------------- | ------------------- |
| Ascites within 30 days           | Systemic sepsis within 48 hours | Ventilator dependence | Disseminated cancer                   | Diabetes  | Hypertension        | Cardiac event       |
| Congestive heart failure 30 days | Dypsnea                         | Smoker 1 year         | Chronic obstructive pulmonary disease | Dialysis  | Acute renal failure | Body mass index     |

*   ASA, American Society of Anesthesiologists.

### **V**ariables

Morbidity, the primary end point, was defined as death, unplanned return to operating room (OR), or any of the NSQIP-recorded cardiac, pulmonary, infectious, neurological, or bleeding complications within 30 days of surgery.9 The outcomes are as follows:

<colgroup span="1" id="colgroup-2"><col align="left" span="1" id="column-8" class="table-left"></col><col align="left" span="1" id="column-9" class="table-left"></col><col align="left" span="1" id="column-10" class="table-left"></col><col align="left" span="1" id="column-11" class="table-left"></col></colgroup>
| Serious complication          | Any complication              | Pneumonia                              | Cardiac complication              |
|:----------------------------- | ----------------------------- | -------------------------------------- | --------------------------------- |
| Surgical site infection (SSI) | Urinary tract infection (UTI) | Venous thromboembolism                 | Renal failure                     |
| Return to OR                  | Death                         | Discharge to nursing or rehab facility | Predicted length of hospital stay |

A serious complication was defined as cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, pulmonary embolism (PE), deep vein thrombosis (DVT), return to the OR, deep incisional SSI, organ space SSI, systemic sepsis, unplanned intubation, UTI, or wound disruption. Any complication was defined as superficial incisional SSI, deep incisional SSI, organ space SSI, wound disruption, pneumonia, unplanned intubation, PE, DVT, ventilator >48 hours, progressive renal insufficiency, acute renal failure, UTI, stroke, cardiac arrest, myocardial infarction, return to the OR, or systemic sepsis.

The surgical calculator's output included patients’ risk of complication, as well as a comparison to the average risk of the given complication for each procedure. Patients’ risks were listed as below average, average, and above average.3

The percentage of correctly predicted complications was calculated by dividing the number of patients who had been assigned above average risk and had suffered complication by the total number of subjects. This was compared with the average of the predicted risks of complication calculated by NSQIP to assess the validity of the calculator as it applies to outcomes in a broad academic plastic surgery practice.

## Results

Within the study population of patients with complications (n=104), the calculator predicted an above average risk for 14 patients who had also underwent a serious complication (20.90% accuracy; table 1). The average of the predicted risks of serious complications was 6.25%, while the actual incidence of serious complications in our practice was 1.36%. The calculator predicted an 8.86% average risk for any complication in our population; actual incidence of any complication within the 4924 patients treated in our plastic surgery practice from September 2013 to June 2015 was 1.89%. For SSIs, the average predicted risk for the study population was 3.30%; this prediction was proven only 24.39% accurate. The ACS-NSQIP calculator achieved 0% accuracy in its prediction of venous thromboembolisms, and was 24.14% accurate with its average predicted risk of return to the OR of 4.89%. Our actual incidence in practice is only 1.18%.

View this table:
[Table 1](/content/65/3/722/T3)

Table 1 
Rates of predicted and actual morbidity

## Discussion

The ACS-NSQIP Surgical Risk Calculator studied in this investigation was created based on 1,414,006 patients encompassing 1557 unique CPT codes from subspecialties including general surgery, gynecology, neurosurgery, orthopedics, otolaryngology, plastic surgery, cardiothoracic surgery, urology, and vascular surgery. Of these, only 2.1% had the primary service coded as ‘plastic surgery’.5 As such, we have found that the ACS-NSQIP Surgical Risk Calculator is not an adequate predictor specifically within the realm of plastic surgery.

Previous literature indicates several limitations of the Surgical Risk Calculator. Edelstein *et al*6 found that the universal risk calculator does not accurately predict complications on an individual patient basis based on the lack of variables that are of particular relevance to lower extremity arthroplasty. From their study of 5570 proctectomy patients, Sherman and colleagues concluded that the calculator underestimates proctectomy morbidity, predicting an overall morbidity of 23%—significantly less than the actual 40.2% morbidity rate. One major shortcoming of the calculator for proctectomy was its failure to include bleeding—the most frequent complication in this practice—as a complication.7 Other studies have found that the risk calculator was inaccurate in its prediction of outcomes when serious complications occur,8 and often had a tendency for predicted risk to be overestimated for lowest and highest risk patients and underestimated for moderate-risk patients.10

Others have engineered their own procedure-specific risk calculators and compared their accuracy to that of the ACS-NSQIP calculator. McMillan *et al*11 created a procedure-specific model incorporating Fistula Risk Score (FRS) and surgeon/institution factors for pancreatoduodenectomy, which outperformed the universal calculator for 30-day mortality, 90-day mortality, serious complications, and reoperation. An adjusted NSQIP was developed for head and neck reconstructive surgery. On seeing deficiencies in NSQIP, procedure-specific preoperative, intraoperative, and postoperative variables were added, such as smoking pack years, alcohol abuse, feeding tube dependence, type of reconstruction and donor site closure details.12 Within pediatric plastic surgery, the ACS-NSQIP was compared with traditional morality and mortality conferences. Zhang *et al*13 concluded that the programme misses ∼50% of all pediatric cases, many of which could be high-complication type cases.

From our own study, we see that the calculator's surgery input does not include many plastic surgery-specific procedures, which reduces reliability of risk predictions. For example, nasal fracture repair with complex laceration repair had to be entered as ‘unlisted craniofacial surgery’.

Certain risk factors appeared on the patient chart that did not factor into postsurgical risk. Tobacco use has been proven to be a major factor for dehiscence in plastic surgery procedures.14 Yet, smoking was factored in only if the patient had been a smoker within the past 1 year, although many patients had a large pack year history of smoking. Also, hyperlipidemia, alcohol use, and altered mental states such as anxiety and depression were not considered risk factors. History of infections were also not indicated, which has been shown to be a major predictor of postsurgical infection.15 Age categories included <65, 65–74, and >75, allowing no modification for the younger patient population within plastic surgery.

The most common complications in plastic surgery include seroma and/or hematoma, dehiscence, and flap-related complications.16 The ACS Risk Calculator does not present rates for these specific risks for plastic surgery. While most frequent outcomes fall into general risk calculator categories of ‘serious complication’, ‘any complication’ or ‘SSI’, the calculated differences in predicted versus actual complication rates (table 1) indicates that this tool does not accurately predict outcomes in plastic surgery. Furthermore, the large range of calculated risk labeled as ‘above average’ may not definitively indicate the cause for postponing surgery. Every patient's risk displayed an above average chance of any complication. This lowers the validity of the calculator and the chance that the physician will take the risk seriously.

The ACS-NSQIP universal Surgical Risk Calculator is not an effective tool for predicting postsurgical complications for the field of plastic surgery. Its limited procedure databank, risk factors, and potential outcome complications are unable to provide an accurate picture of postoperational conditions in their specialty. Further work should be carried out in creating a plastic surgery-specific risk calculator.

## Footnotes

*   Contributors CJ and IC contributed to the conception, design, data collection, data analysis and interpretation, and writing of the article. SG contributed to the conception, design, data analysis and interpretation, critical revision of the article, and final approval of the article.

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

## References

1.  Volk ML, Roney M, Merion RM. Systematic bias in surgeon predictions of donor-specific risk of liver transplant graft failure. Liver Transpl 2013;19:987–90. [doi:10.1002/lt.23683](http://dx.doi.org/10.1002/lt.23683)
    
    [PubMed](/lookup/external-ref?access_num=23784716&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

2.  Ingraham AM, Richards KE, Hall BL, et al. Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg 2010;44:251–67. [doi:10.1016/j.yasu.2010.05.003](http://dx.doi.org/10.1016/j.yasu.2010.05.003)
    
    [CrossRef](/lookup/external-ref?access_num=10.1016/j.yasu.2010.05.003&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=20919525&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

3.  About the ACS Risk Calculator. 2015. [http://www.riskcalculator.facs.org/Home/About/](http://www.riskcalculator.facs.org/Home/About/) (accessed 23 Jun 2015).
    
    

4.  Walker JA. What is the effect of preoperative information on patient satisfaction? Br J Nurs 2007;16:27–32. [doi:10.12968/bjon.2007.16.1.22712](http://dx.doi.org/10.12968/bjon.2007.16.1.22712)
    
    [CrossRef](/lookup/external-ref?access_num=10.12968/bjon.2007.16.1.22712&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=17353832&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

5.  Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aide and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217,833–42.e1-3. [doi:10.1016/j.jamcollsurg.2013.07.385](http://dx.doi.org/10.1016/j.jamcollsurg.2013.07.385)
    
    [CrossRef](/lookup/external-ref?access_num=10.1016/j.jamcollsurg.2013.07.385&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=24055383&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

6.  Edelstein AI, Kwasny MJ, Suleiman LI, et al. Can the American College of Surgeons Risk Calculator predict 30-day complications after knee and hip arthroplasty? J Arthroplasty 2015;30(9 Suppl):5–10. [doi:10.1016/j.arth.2015.01.057](http://dx.doi.org/10.1016/j.arth.2015.01.057)
    
    [CrossRef](/lookup/external-ref?access_num=10.1016/j.arth.2015.01.057&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=26165953&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

7.  Sherman SK, Hrabe JE, Charlton ME, et al. Development of an improved risk calculator for complications in proctectomy. J Gastrointest Surg 2014;18:986–94. [doi:10.1007/s11605-013-2448-2](http://dx.doi.org/10.1007/s11605-013-2448-2)
    
    [CrossRef](/lookup/external-ref?access_num=10.1007/s11605-013-2448-2&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=24395071&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

8.  Cologne KG, Keller DS, Liwanag L, et al. Use of the American College of Surgeons NSQIP Surgical Risk Calculator for laparoscopic colectomy: how good is it and how can we improve it? J Am Coll Surg 2015;220:281–6. [doi:10.1016/j.jamcollsurg.2014.12.007](http://dx.doi.org/10.1016/j.jamcollsurg.2014.12.007)
    
    [PubMed](/lookup/external-ref?access_num=25617914&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

9.  NSQIP, ACS. User guide for the 2011 participant use data file. Chicago: American College of Surgeons, 2012.
    
    

10. Liu Y, Cohen ME, Hall BL, et al. Evaluation and enhancement of calibration in the American College of Surgeons NSQIP Surgical Risk Calculator*.* J Am Coll Surg 2016;223:231–9. [doi:10.1016/j.jamcollsurg.2016.03.040](http://dx.doi.org/10.1016/j.jamcollsurg.2016.03.040)
    
    [PubMed](/lookup/external-ref?access_num=27212006&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

11. McMillan MT, Allegrini V, Asbun HJ, et al. Incorporation of procedure-specific risk into the ACS-NSQIP Surgical Risk Calculator improves the prediction of morbidity and mortality after pancreatoduodenectomy. Ann Surg Published Online First: 26 May 2016. doi: 10.1097/SLA.0000000000001796.
    
    

12. Lewis CM, Aloia TA, Shi W, et al. Development and feasibility of a specialty-specific National Surgical Quality Improvement Program (NSQIP): the head and neck-reconstructive surgery NSQIP. JAMA Otolaryngol Head Neck Surg 2016;142:321–7. [doi:10.1001/jamaoto.2015.3608](http://dx.doi.org/10.1001/jamaoto.2015.3608)
    
    [PubMed](/lookup/external-ref?access_num=26892756&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

13. Zhang JX, Song D, Bedford J, et al. What is the best way to measure surgical quality? Comparing the American College of Surgeons National Surgical Quality Improvement Program versus traditional morbidity and mortality conferences. Plast Reconstr Surg 2016;137:1242–50. [doi:10.1097/01.prs.0000481737.88897.1a](http://dx.doi.org/10.1097/01.prs.0000481737.88897.1a)
    
    [PubMed](/lookup/external-ref?access_num=27018679&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

14. Karamanos E, Osgood G, Siddiqui A, et al. Wound healing in plastic surgery: does age matter? An American College of Surgeons National Surgical Quality Improvement Program study. Plast Reconstr Surg 2015;135:876–81.
    
    [CrossRef](/lookup/external-ref?access_num=10.1097/PRS.0000000000000974&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=25415275&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

15. Tahiri Y, Fischer J, Wink J, et al. Analysis of risk factors associated with 30-day readmissions following pediatric plastic surgery: a review of 5376 procedures. Plast Reconstr Surg 2015;135: 521–9. [doi:10.1097/PRS.0000000000000889](http://dx.doi.org/10.1097/PRS.0000000000000889)
    
    [CrossRef](/lookup/external-ref?access_num=10.1097/PRS.0000000000000889&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=25357160&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom) 

16. Biskup N, Workman AD, Kutzner E, et al. Perioperative safety in plastic surgery: Is the World Health Organization Checklist useful in a broad practice? Ann Plast Surg 2016;76:550–5. [doi:10.1097/SAP.0000000000000427](http://dx.doi.org/10.1097/SAP.0000000000000427)
    
    [PubMed](/lookup/external-ref?access_num=25664411&link_type=MED&atom=%2Fjim%2F65%2F3%2F722.atom)