Assessment and comparison of 3 mortality prediction models SAPS II, APACHE II and SOFA for prediction of mortality in patients of sepsis

INTRODUCTION

Severe sepsis and septic shock are major reasons for intensive care unit (ICU) admission and leading causes of mortality in non-coronary ICUs.^1-3 Apart from in the West, little is known about outcomes of patients admitted to the ICU with severe sepsis and septic shock, despite the seriousness of sepsis as a public health problem in developing countries. Understanding sepsis outcome studies is hampered by lack of an agreed severity of illness scoring system for sepsis patients. In the absence of such a system, it would be difficult to interpret sepsis outcome studies.⁴ Mortality prediction systems have been introduced as tools for assessing the performance of ICUs.^5-8 Some of these systems have been customized for patients with specific conditions such as sepsis, and liver transplantation and long-stay ICU patients.^9,10 If these systems are proved to predict accurately mortality in severe sepsis and septic shock, then they will have the advantage of being readily available and easily incorporated into general ICU databases without additional data collection.

Scoring systems for use in intensive care unit (ICU) patients have been introduced and developed over the last 30 years. They allow an assessment of the severity of disease and provide an estimate of in-hospital mortality. This estimate is achieved by collating routinely measured data specific to a patient. A weighting is applied to each variable, and the sum of the weighted individual scores produces the severity score. Various factors have been shown to increase the risk of in-hospital mortality after admission to ICU, including increasing age and severity of acute illness, certain pre-existing medical conditions (e.g. malignancy, immunosuppression, and requirement for renal replacement therapy), and emergency admission to ICU.

Most critical care severity scores are calculated from the data obtained on the first day of ICU admission [e.g. the APACHE, the SAPS, and the mortality prediction model (MPM)]. Other scoring systems are repetitive and collect data sequentially throughout the duration of ICU stay or over the first few days. Examples of repetitive systems are the SOFA and multiple organ dysfunction score (MODS).

The objective of the present study is to assess and compare the validity of 3 mortality prediction systems for the severe sepsis and septic shock patient population

METHODS

One hundred patients of Sepsis were selected after applying the inclusion and exclusion criteria. Informed consent was taken from the patients or their relatives after explaining the nature of study and the risks involved in participation of the study. The clinical and demographic profile at the time of admission to medicine ward including the age, sex, associated chronic illnesses like hypertension and diabetes, and hyperlipidemia were recorded for all the study subjects.

A careful and detailed history was recorded to assess the onset and duration of clinical events and the probable risk factors for the same. A detailed general physical examination was performed. Blood sampling for complete blood count, renal function test, liver function test and Arterial Blood Gas Analysis was done. SAPS 2, APACHE II and SOFA scores were calculated on the day of admission.

The results obtained were subjected to standard statistical methods for analysis and relevant conclusions were drawn from them. The correlation between the mortality models on admission and outcome was assessed by AUROC curves. An independent t-test was used to evaluate the differences between SAPS 2, APACHE II and SOFA scores between the survivors and non survivors.

Further univariate analysis was done to assess the effect of different physiological parameters on prediction of mortality. A p-value of less than 0.05 was considered to be statistically significant throughout the study.

Inclusion criteria

As per International Guidelines for Management of Severe Sepsis and Septic Shock: 2012 (Society of Critical Care Medicine).

Patients diagnosed with sepsis: 2 out of the following criteria

• • Temperature higher than 38°C (100.4°F) or lower than 36°C (96.8°F).
• • Heart rate (HR) higher than 90 beats/min.
• • Respiratory rate (RR) higher than 20 breaths/min or arterial carbon dioxide tension (PaCO₂) lower than 32 mmHg.
• • White blood cell (WBC) count higher than 12,000/µl or lower than 4000/µl or with 10% immature (band) forms.
• • Significant edema or positive fluid balance >20ml/Kg over 24 hours.
• • Hyperglycemia (plasma glucose >140mg/dl in absence of diabetes.

And at least one of the following

• • Alteration in mental state.
• • Hypoxemia (arterial oxygen tension [PaO₂] <72mmHg at a fraction of inspired oxygen [FIO₂] of 0.21, with overt pulmonary disease not being the direct cause of hypoxemia).
• • Oliguria (urine output <30ml or 0.5 ml/kg for at least 2 hours despite adequate fluid resuscitation.
• • Coagulation abnormality INR >1.5.

Exclusion criteria

• • Burns
• • Coronary artery disease
• • Stroke

RESULTS

This study was a prospective observational study done with the aim to predict mortality in patients of sepsis using SAPS2, APACHE II and SOFA models and also compare these 3 mortality prediction models for prediction of mortality in patients of sepsis. The study comprised of 100 patients admitted to the Medicine Department who were enrolled in the study after taking informed consent from patient and family over a period of 2014-2016.

Table 1 shows the distribution of aetiologies of sepsis in the study population.

Table 2 compares the physiological parameters and mortality. It was found that the age was higher in the mortality group as compared to the discharge group.

The mean heart rate in the discharge group was 106±14, whereas in the mortality group it was higher, 125±18. This difference was found to be statistically significant (p-value≤0.001).

The respiratory rate in the discharge group was lower as compared to the mortality group. It was found that this difference is statistically significant (p-value <0.001).

The mean blood pressure in the discharge group was 114.2±15.6/75.7±10.6, whereas the mean blood pressure in the mortality group was 90.7±23.1/55.6±16.5. This difference was statistically significant for both systolic and diastolic blood pressure (p-value <0.001).

The PaO₂ and PaCO₂ in the discharge group were 79.7±33 and 24.6±8.8, whereas the readings were 78.1±8.8 and 28.6±14.4. The difference for both PaO₂ and PaCO₂ were statistically insignificant.

The pH in the discharge group was higher as compared to the pH in the mortality group. It was 7.34±0.091 in the discharge group and 7.22±0.15. This difference was found to be statistically significant (p-value <0.001).

The total leucocyte count was lower in the discharge group. However, this showed no correlation between discharge and mortality. The p-value was statistically not significant. Platelet count was higher in the discharge group as compared to the mortality group. This difference however was statistically significant (p-value=0.009). The haematocrit was lower in the discharge group as compared to the mortality group. This was found to be statistically significant (p-value=0.004).

The difference in total urine output, blood urea nitrogen and serum creatinine between the two groups was also found to be statistically significant (p=value<0.001).

Mean serum sodium levels was 134.8±5.8 in the discharge group, whereas in the mortality group mean sodium level was 133±10. Serum sodium levels were found not to affect the mortality. The difference was found to be statistically not significant. Mean Serum Potassium level was 4±0.8 in the discharge group and 4.9±1.3 in the mortality group. This difference was found to be highly significant (p-value<0.001).

The mean bicarbonate levels in the discharge and mortality groups were 21.8±5 and 17.9±9.5. This was found to be statistically significant (p-value=0.008).

The difference in the mean Glasgow Coma Scale between the discharge and mortality group was also found to be statistically significant (p-value<0.001).

The predicted mortality of SAPS 2, APACHE II and SOFA was found to be 30, 38 and 25 respectively. It was found that SAPS2 and SOFA predicted a slightly lower mortality rate than the actual mortality, whereas APACHE II predicted a higher mortality than the actual mortality.

It was found that SAPS 2 and SOFA had a 100% positive predictive value whereas APACHE II had a PPV of 88%. The NPV was highest in APACHE II score (100%), followed by SAPS 2 (94%) and SOFA (88%). It was also found that SAPS 2 and SOFA had 100% specificity whereas APACHE II had a specificity of 93.9%. However, it was observed that APACHE II had 100% sensitivity as a model for mortality prediction.

The ROC analysis shows that the best discrimination was provided by SAPS 2 score (AUROC=0.981), followed by APACHE II (AUROC=0.978) and SOFA (AUROC=0.911).

However, the difference between SAPS 2 and APACHE II score was statistically not significant p-value=0.723. However, the discriminatory powers of SAPS 2 and SOFA scoring showed a statistically significant difference (p-value=0.0253).

DISCUSSION

The three commonly used severity scoring systems, compared in this study, have been developed using large cohorts of critically ill patients admitted to American and European ICUs. Indian hospitals were not included in any of these cohorts. Validation is essential before routine application of any predictive model in a group of patients different from the one originally used for model development. In India, performances of these severity models have been tested only on individual basis in a few studies.^11-15

In the present study, mortality was 34%, which is comparable to that reported in previous studies from India (40.3%) and Indonesia (39.8%).¹⁶ It is higher compared with those reported in Germany (9%), Australia (16%), and Saudi Arabia (31.6%).^17-19 The high mortality can be attributed to shortage of ICU beds in our centre (ICU bed: hospital bed ratio is 0.006). It has been documented that physicians tend to be more selective in their ICU admissions during times of bed shortages, with patients having higher severity of illness being admitted.²⁰ Critically ill patients usually deteriorate when managed outside ICU while waiting for availability of ICU beds. Poor nutritional status at ICU admission may also be one of the important causes contributing to high mortality rate in India. Poor nutrition leads to decline in immunity and thereby a rise in acquired infection rate. The reason for poor nutrition is that most of the patients, who come to government sector for free treatment, belong to a lower socioeconomic class. Moreover, anaemia and infectious diseases (e.g. tuberculosis, malaria, and dengue) are prevalent problems among patients coming to Indian ICU. Therefore, proper daily assessment of nutritional state of each patient and subsequent intervention is imperative.

In present study, the best discrimination was provided by SAPS 2 score (AUROC=0.981), followed by APACHE II (AUROC=0.978) and SOFA (AUROC=0.911). Several studies have compared the different outcome prediction scoring systems. For example, in a study of 10,393 patients from Scottish ICUs, Livingston and colleagues compared the APACHE II, an APACHE II using United Kingdom-derived coefficients (UK APACHE II), SAPS II, and MPM₀ and MPM₂₄.²¹ These authors reported that all models showed good discrimination, although observed mortality was significantly different from that predicted by all models. SAPS II had the best performance overall.

In the present study, we evaluated the ability and validity of APACHE II and SAPS II systems to accurately predict hospital mortality. Both models showed excellent discrimination, although we found that discrimination was better for SAPS2 than for APACHE. Good discrimination of both models has been reported in previous studies.^22-24

The area under the ROC of both systems in the present study was higher than that in the other reports. Previously reported area under the ROC curve of APACHE II and SAPS II included 0.839 and 0.870 in Greece, 0.787 and 0.817 in Portugal, 0.83 and 0.79 in Saudi Arabia, 0.819 and 0.840 in Tunisia and 0.88 and 0.87 in Hong Kong.^24-28

In a study by Georgescu AM et al the APACHE II, SOFA and SAPS II scores were determined prospectively, in the first 24 hours after admission, for all 56 patients with septic shock who were included in this study.²⁹ Data were statistically evaluated; the discriminating power regarding survivors versus deceased patients was calculated based on the receiver operating characteristic curves. The overall mortality of the 56 patients with septic shock was 60.71% (34 deaths). The average APACHE II score was 25.36±7.477. The average SOFA score was 7.679±3.197. The average SAPS II score was 44.45±16.97. For the APACHE II and SOFA scores the differences when deceased and survivors were compared were not statistically significant (APACHE II: 26.76±6.742 versus 23.18±8.175 respectively for SOFA: 8.029±3.099 versus 7.136±3.342). For the SAPS II score the values are: 49.12±16.61 in deceased versus 37.23±15.20 in survivors, the difference being statistically significant (p=0.0092). Similarly, in present study the difference between the mean SAPS 2 score in the survivors and deceased was statistically significant (68.70±19.58 in deceased and 21.87±13.26 in survivors with p-value<0.001) The areas under ROC for the three scores are 0.622 for APACHE II, 0.575 for SAPS II and 0.705 for SOFA.

The study concluded that the SAPS II score was superior to the other scores for predicting survival in patients with septic shock.

Simplified acute physiology score II (SAPS II) is a complex score calculated during the first 24 hours of hospitalization in the ICU. Rating values obtained from areas under the curve chart show SAPS II score to be superior in predicting patients’ survival as compared with the other two scores (APACHE II and SOFA). A ROC value above 0.7 qualifies a score in discriminating survivors versus deceased although strong discrimination occurs in a ROC over 0.8.

The results of present study differ from those of Kim who identified the SOFA score as being the most predictive in patients poisoned with organophosphates.³⁰ An explanation for the observed difference may stem from the more complex nature of septic shock as compared with poisoning both in onset and evolution. Moreover, the simplicity of the SOFA score, in particular the absence of parameters related to associated diseases makes it more relevant to organ dysfunctions, unlike SAPS II and APACHE II that evaluate multiple organ dysfunction specific to the critically septic patient. While the APACHE II score is a predictor for morbidity, the SAPS score is a predictor of mortality. The use of these scores as predictors must take into account the heterogeneity of the population tested. There are, on one hand, significant geographical variables, the homogenization trend of confirmation by excluding certain population groups (for example, age, length of admission in ICU) and on the other different statistical methods for assessing prediction. The parameters taken into account may lead to evaluation errors. Tunnell et al demonstrated what are the most common bias causes when applying scores in the ICU: heart rate, arterial pressure, respiratory rate, oxygenation, pH value and blood glucose.³¹ For these reasons and taking into account the above mentioned limitations, ICU scores can be taken into consideration only to evaluate ICU performance in order to improve the quality of medical care even if the existence of a single output- mortality may influence both the decision for ICU admission as well as the time of discharge. Obviously, this criterion can be influenced by the case-mix and the level of care in a specific ICU as well. A high case-mix, specific to high performing ICU services is the consequence of complex pathologies, with unfavourable prognostic scores. The limitations of the study are the low number of studied cases that were concentrated within a single centre. Moreover, the study does not take into account deaths that could have occurred immediately after discharge. These scores are complementary and have certain limitations. They do not provide individual prediction but may assist therapeutic and managerial decisions. For a complex pathology such as sepsis, a more complex score could be more informative.

CONCLUSION

We conclude that the SAPS II score was superior to the APACHE II and SOFA scores for predicting survival in patients with septic shock but a combination of factors must be taken in consideration to estimate the prognosis in the ICU.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the institutional ethics committee

REFERENCES

1. 1. Rangel-Frausto MS. The epidemiology of bacterial sepsis. Infect Dis Clin North Am. 1999;13:299-312.
2. 2. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303-10.
3. 3. Angus DC, Wax RS. Epidemiology of sepsis: an update. Crit Care Med. 2001;suppl:S109-16.
4. 4. Friedman G, Silva E, Vincent JL. Has the mortality of septic shock changed with time? Crit Care Med. 1998;26:2078-86.
5. 5. Lemeshow S, Le Gall Jr. Modeling the severity of illness of ICU patients. JAMA. 1994;272:1049-55.
6. 6. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II. A severity of disease classification system. Crit Care Med. 1985;13:818-29.
7. 7. Gall LJR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multi center study. JAMA. 1993;270:2957-62.
8. 8. Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J. Mortality probability models (MPM II) based on an international cohort of intensive care unit patients. JAMA. 1993;270:2478-86.
9. 9. Suistomaa M, Niskanen M, Kari A, Hynynen M, Takala J. Customized prediction models based on APACHE II and SAPS II scores in patients with prolonged length of stay in the ICU. Intensive Care Med. 2002;28:479-85.
10. 10. Angus DC, Clermont G, Kramer DJ, Linde-Zwirble WT, Pinsky MR. Short-term and long-term outcome prediction with the acute physiology and chronic health evaluation II in system after orthotopic liver transplantation. Crit Care Med. 2000;28:150-6.
11. 11. Rapsang AG, Shyam DC. Scoring systems in the intensive care unit: A compendium. Indian Journal of Critical Care Medicine: Peer-reviewed, Official Publication of Indian Society of Critical Care Medicine. 2014;18(4):220-8.
12. 12. Apostolopoulou E, Charcharidou M, Karga M, Kaliakmanis A. The relationship of scoring systems and mortality in the intensive care unit. Icus Nurs Web J. 2003;13:1-8.
13. 13. Khwannimit B, Geater A. A comparison of APACHE II and SAPS II scoring systems in predicting hospital mortality in Thai adult intensive care units. J Med Assoc Thai. 2007;90:643-52.
14. 14. Marra AR, Bearman GM, Wenzel RP, Edmond MB. Comparison of severity of illness scoring systems for patients with nosocomial bloodstream infection due to Pseudomonas aeruginosa. BMC Infect Dis. 2006; 6:132.
15. 15. Rao MH, Marella P, Kath B. Assessment of severity and outcome of critical illness. IJA. 2008;52:652-62.
16. 16. Zygun DA, Laupland KB, Fick GH, Sandham JD, Doig CJ. Limited ability of SOFA and MOD scores to discriminate outcome: a prospective evaluation in 1,436 patients. Can J Anaesth. 2005;52:302-8.
17. 17. Boyd O, Grounds RM. Physiological scoring systems and audit. Lancet. 1993;341:1573-4.
18. 18. Lemeshow S, Hosmer DW Jr. A review of goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol. 1982;115:92-106.
19. 19. Moreno R, Matos R. The 'new' scores: what problems have been fixed, and what remain? Curr Opin Crit Care. 2000;6:158-65.
20. 20. Strauss MJ, LoGerfo JP, Yeltatzie JA, Temkin N, Hudson LD. Rationing of intensive care unit services. An everyday occurrence. JAMA. 1986;255:1143-6.
21. 21. Livingston BM, MacKirdy FN, Howie JC, Jones R, Norrie JD: Assessment of the performance of five intensive care scoring models within a large Scottish database. Crit Care Med. 2000;28:1820-7.
22. 22. Castella X, Artigas A, Bion J, Kari A. A comparison of severity of illness scoring systems for intensive care unit patients: results of a multicenter, multinational study. The European/North American Severity Study Group. Crit Care Med. 1995;23:1327-35.
23. 23. Capuzzo M, Valpondi V, Sgarbi A, Bortolazzi S, Pavoni V, Gilli G, et al. Validation of severity scoring systems SAPS II and APACHE II in a single center population. Intensive Care Med. 2000;26:1779-85.
24. 24. Arabi Y, Haddad S, Goraj R, Al-Shimemeri A, AlMalik S. Assessment of performance of four mortality prediction systems in a Saudi Arabian intensive care unit. Crit Care. 2002;6:166-74.
25. 25. Katsaragakis S, Papadimitropoulos K, Antonakis P, Strergiopoulos S, Konstadoulakis MM, Androulakis G. Comparison of acute physiology and chronic health evaluation II (APACHE II) and simplified acute physiology score II (SAPS II) scoring systems in a single Greek intensive care unit. Crit Care Med. 2000;28:426-32.
26. 26. Moreno R, Morais P. Outcome prediction in intensiveive care: results of a prospective, multicentre, Portuguese study. Intensive Care Med. 1997;23:177-86.
27. 27. Nouira S, Belghith M, Elatrous S, Jaafoura M, Ellouzi M, Boujdaria R, et al. Predictive value of severity scoring systems: comparison of four models in Tunisian adult intensive care units. Crit Care Med. 1998;26:852-9.
28. 28. Tan IK. APACHE II and SAPS II are poorly calibrated in a Hong Kong intensive care unit. Ann Acad Med Singapore. 1998;27:318-22.
29. 29. Georgescu AM, Szederjesi J, Copotoiu SM, Azamfirei L. Predicting scores correlations in patients with septic shock-a cohort study. Rom J Anaesth Int Care. 2014;21:95-8.
30. 30. Kim YH, Yeo JH, Kang MJ, Lee JH, Cho KW, Hwang S, et al. Performance assessment of the SOFA, APACHE II scoring system, and SAPS II in intensive care unit organophosphate poisoned patients. J Korean Med Sci. 2013;28:1822-6.
31. 31. Tunnell RD, Millar BW, Smith GB. The effect of lead time bias on severity of illness scoring, mortality prediction and standardised mortality ratio in intensive care- a pilot study. Anaesthesia. 1998;53:1045-53.