https://orcid.org/0000-0002-2248-8145
Figures
Abstract
Background
Systemic inflammatory response syndrome (SIRS) and sepsis are the most common causes of in-hospital death. However, the characteristics associated with the improvement in the patient conditions during the ICU stay were not fully elucidated for each population as well as the possible differences between the two.
Goal
The aim of this study is to highlight the differences between the prognostic clinical features for the survival of patients diagnosed with SIRS and those of patients diagnosed with sepsis by using a multi-variable predictive modeling approach with a reduced set of easily available measurements collected at the admission to the intensive care unit (ICU).
Methods
Data were collected from 1,257 patients (816 non-sepsis SIRS and 441 sepsis) admitted to the ICU. We compared the performance of five machine learning models in predicting patient survival. Matthews correlation coefficient (MCC) was used to evaluate model performances and feature importance, and by applying Monte Carlo stratified Cross-Validation.
Results
Extreme Gradient Boosting (MCC = 0.489) and Logistic Regression (MCC = 0.533) achieved the highest results for SIRS and sepsis cohorts, respectively. In order of importance, APACHE II, mean platelet volume (MPV), eosinophil counts (EoC), and C-reactive protein (CRP) showed higher importance for predicting sepsis patient survival, whereas, SOFA, APACHE II, platelet counts (PLTC), and CRP obtained higher importance in the SIRS cohort.
Conclusion
By using complete blood count parameters as predictors of ICU patient survival, machine learning models can accurately predict the survival of SIRS and sepsis ICU patients. Interestingly, feature importance highlights the role of CRP and APACHE II in both SIRS and sepsis populations. In addition, MPV and EoC are shown to be important features for the sepsis population only, whereas SOFA and PLTC have higher importance for SIRS patients.
Author summary
Sepsis is defined as the dysregulated host response to infection causing a significant increase in patients’ mortality, thus resulting in an important global health problem. Systemic inflammatory response syndrome (SIRS), an exaggerated response of the body to a noxious stressor, and sepsis, an organ dysfunction caused by a dysregulated host response to infection, are two of the most critical conditions in the intensive care unit showing high patients’ mortality and resulting in an important global health problem. However, the major differences leading to an improvement in the patient conditions during SIRS and sepsis are not fully elucidated.
In this study, we assess the role of simple and easily available routinely collected blood count parameters as predictors for patients’ prognosis in 1,257 patients with SIRS or sepsis. We applied and compared the performance of five distinct machine-learning models when predicting ICU patient survival. Furthermore, we investigated the feature importance of the best-performing models for each population to highlight the major differences between the two populations. Results highlight the role of C-reactive protein and APACHE II score in both populations, whereas mean platelet volume and eosinophil counts show higher importance in sepsis patients and SOFA score and platelet count show higher importance for SIRS patients.
Citation: Mollura M, Chicco D, Paglialonga A, Barbieri R (2024) Identifying prognostic factors for survival in intensive care unit patients with SIRS or sepsis by machine learning analysis on electronic health records. PLOS Digit Health 3(3): e0000459. https://doi.org/10.1371/journal.pdig.0000459
Editor: Nan Liu, Duke-NUS Medical School, SINGAPORE
Received: February 9, 2023; Accepted: February 5, 2024; Published: March 15, 2024
Copyright: © 2024 Mollura et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The dataset is publically available at the following URL: https://figshare.com/articles/dataset/_C_Reactive_Protein_and_Hemogram_Parameters_for_the_Non_Sepsis_Systemic_Inflammatory_Response_Syndrome_and_Sepsis_What_Do_They_Mean_/1644426.
Funding: The work of DC was funded by the European Union – Next Generation EU program, in the context of The National Recovery and Resilience Plan, Investment Partenariato Esteso PE8 “Conseguenze e sfide dell’invecchiamento”, Project Age-It (Ageing Well in an Ageing Society) and partially supported by Ministero dell’Università e della Ricerca of Italy under the “Dipartimenti di Eccellenza 2023-2027” ReGAInS grant assigned to Dipartimento di Informatica Sistemistica e Comunicazione at Università di Milano-Bicocca. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Patient’s outcome has long been used as primary endpoint for trials in critical care as well as for determining the patient’s prognosis after treatments. Patient mortality and survival are indeed the major clinical outcomes, and they are main targets for assessing prognostic factors driving the patient conditions and the effectiveness of clinical interventions [1, 2], especially in the intensive care unit (ICU) where admitted patients are usually in very critical conditions and require constant monitoring and treatment.
In this context, sepsis represents an important global health problem accounting for about one-third of ICU deaths and its reported incidence is still increasing [3–6] and a proper and precise description of sepsis is still not available. Indeed, the definition of sepsis was subjected to several revisions during the years [7–9], and according to the Third International Consensus Definitions for Sepsis and Septic Shock [9] it is currently defined as a life-threatening organ dysfunction caused by a dysregulated host response to infection. This last update of the sepsis definition abandons the use of Systemic Inflammatory Response Syndrome (SIRS) criteria, which were recognized as presenting a lack of specificity, whereas it focuses on the life-threatening condition and the presence and progression of organ failure. However, the major differences between SIRS and sepsis leading to a positive or negative patient outcome are still not fully elucidated in the medical literature.
In particular, Gucyetmez et al. [10] evaluated the ability of hemogram parameters, a set of medical laboratory tests providing information about the cells in a person’s blood, and C-reactive protein (CRP) to distinguish non-sepsis SIRS from sepsis patients. The authors found that the combinations of CRP, lymphocytes count (LymC), and platelet count (PLTC) can be used to determine the likelihood of sepsis, however without exploring the association of these parameters with the patient survival for each population. This information can provide significant indications about the most important prognostic factors specifically for non-sepsis SIRS and sepsis patients. Also, the authors did not investigate the predictive power of the observed variables.
Especially for this last task, machine learning approaches have shown a good ability in the early identification of sepsis with data collected both from electronic health records [11–13] and from physiological vital signs monitoring [14], also providing insights about the role of each feature in a multi-variable setting. Several studies focused on predicting ICU patient outcomes focusing on mortality or survival prediction task [15–22], but multi-variable prognostic models estimating and comparing SIRS and sepsis outcomes are still lacking. The stratification of patients’ risk in particular for patients undergoing infections and with sepsis is important. In fact, these patients often require prompt management and interventions, like the initiation of antibiotic therapy and the administration of fluid and vasopressors for maintaining adequate tissue perfusion and hemodynamic stability [23]. These aspects led to an increasing interest toward the prediction of the patient outcome specifically for sepsis patients, in the last few years [24–29].
In this context, simple and easily available laboratory measurements of blood cell counts (for example platelet, eosinophil, neutrophil, and lymphocyte counts) can be useful tools for patients’ risk stratification.
The goal of this study is to further explore the ability of hemogram parameters in estimating the survival of ICU patients with non-sepsis SIRS and with sepsis, by applying machine learning techniques in order to estimate the survival probability of ICU patients and to investigate the role of the different parameters in a multi-variable prediction setting. Specifically, our study makes further use of the features proposed by Gucyetmez and colleagues [10] to explore the predictive power of hemogram parameters in estimating the survival probability of patients with non-sepsis SIRS and with sepsis, by comparing different machine learning approaches.
Multi-variable feature importance of the best performing models is applied to further assess the role of each feature and to highlight differences between non-sepsis SIRS and sepsis cohorts.
Dataset
In this study, we use data retrospectively collected from 1,257 eligible medical and surgical patients admitted to the ICU’s of Acibadem International Hospital and Atasehir Memorial Hospital between 1 January 2006 and 31 December 2013, Istanbul, Turkey, and made available by Gucyetmez et al. [10]. The considered cohort includes 816 (64.9%) non-sepsis SIRS and 441 (35.1%) sepsis patients.
The dataset contains the following features for each patient: Age, sex, APACHE II and SOFA scores, diagnosis (medical, elective, and emergency surgery), length of ICU stay (LOS-ICU), mortality, CRP, WBCC, NeuC, LymC, NLCR, EoC, PLTC, MPV. A detailed description of the data here used is provided in Table 1. The target variable for our analysis was survival, indicating whether the patient survived (1) or died (0) in the ICU. A quantitative description of the distribution of each numeric and categorical feature for the non-sepsis SIRS (SIRS) and sepsis (SEPSIS) cohorts are reported in Tables 2 and 3. From the analysis of the target variable (Survival) it is possible to observe that both cohorts are unbalanced, with stronger unbalance in the SIRS (3.07% not survived) than in the SEPSIS (23.64% not survived) cohort.
EC: Elective, AC: Emergency, and M: Medical. E: Male and K: Female.
DIAG.: Diagnosis, S: Survived, NS: Not Survived, EC: Elective, AC: Emergency, and M: Medical. E: Male, and K: Female.
Methods
In this retrospective study, we developed predictive models of patient survival using machine learning algorithms and we evaluated the importance of features associated with patient survival using machine learning and biostatistics approaches for both the SIRS and SEPSIS populations separately (Fig 1). All the analyses were performed with the Python 3.8.3 programming language, and scikit-learn 1.0 and SciPy 1.7.1 software packages. Observations with missing information (three patients) were removed.
MCCV: Monte Carlo Cross-Validation; SMOTE: Synthetic Minority Oversampling Technique.
Associations between features and survival
The association between the input features and patient survival was also explored with classical statistical approaches. Specifically, differences in numeric features between survived and deceased groups in each cohort were tested with the Mann-Whitney U test, whereas differences in categorical features were assessed with χ2-test [30]. Statistical significance was defined for p < 0.005 as advocated by Benjamin et al. [31], which also accounts for multiple testing adjustments.
Survival prediction models
We trained five machine learning models with the goal of predicting patient’s survival considering the following features: Age, Sex, SOFA, APACHE II, CRP, WBCC, NeuC, LymC, EOC, NLCR, PLTC, and MPV, for both SIRS and SEPSIS cohorts. The approach consisted of 100 runs of Monte Carlo stratified Cross-Validation with 80%-20% train-test split as already proposed by Chicco et al. [32]. At each iteration, 80% of the data were used as a training set and 20% as a test set keeping constant the ratio between survived and dead patients. In order to limit the effect of class imbalance, we applied Synthetic Minority Oversampling Technique (SMOTE) [33] to the training set. Features were rescaled before feeding them to the classifier by removing the median and dividing by the interquartile range, as estimated on the training set [37], and five machine learning classifiers were used to develop patient survival prediction models. We considered the following classifiers: Logistic Regression (LR), Support Vector Machine (SVM) [34], Decision Tree (Tree), Random Forest (RF) [35] and XGBoost (XGB) [36] (evalution metric: logloss and objective function: binary/logistic). To evaluate the performance of the classifiers, Matthews correlation coefficients (MCC) [38] on the cross-validated test sets were considered because its proven ability to summarize results from contingency tables and its invariance to class swapping [39–41, 60]. Specifically, the MCC can take values ranging from –1 to +1, where –1 represents the misclassification of all observations, 0 represents the random association, and 1 perfect classification. Average Receiver Operating Characteristic (ROC) curves and Precision-Recall curves (PRC) are also used to quantitatively assess the average model performances. Further details are reported in the supplementary material in Text A of S1 Appendix where additional sensitivity analyses summarizing model calibration on the test set (Text D of S1 Appendix) and the model performance with hyperparameters optimization (Text F of S1 Appendix) are also reported.
Feature importance
The best-performing model for each cohort was selected and feature importance was estimated through single feature elimination (SFE) approach, that is by evaluating the MCC obtained after removing one variable at a time. In this case, the smaller the resulting MCC, the higher the importance of the variable which generated that observed drop in performances. Feature importance analysis was executed for 100 runs of Monte Carlo stratified cross-validation partitions with 80%/20% train-test split [42]. The resulting MCCs for each run are obtained from the test set observations. Finally, we used the Spearman correlation coefficient and the Kendall coefficient [43] to quantify the correlation between the obtained ranks. Both coefficients range from –1 to 1 (from anticorrelation to perfect matching) whereas the absence of correlation is given by a 0 coefficient.
Results
Associations between features and survival
Results of the statistical analysis are reported in Table 4. It can be observed that APACHE II and SOFA scores showed significant (p<0.0001) association with survival in both SIRS and SEPSIS cohorts. EoC resulted significantly associated (p<0.0001) with survival in SIRS cohort only.
Differences in numeric features between survived and deceased groups in the two cohorts were tested with the Mann-Whitney U test [44], whereas differences in categorical features were assessed with the χ2-test.
Survival prediction
Survival prediction performances for SEPSIS and SIRS cohorts are graphically summarized in Fig 2. Median MCCs, accuracy, sensitivity, specificity, F1-scores, positive predictive value, negative predictive value, areas under precision-recall and receiver operating characteristic curves, and the respective interquartile ranges are reported in the supplementary material (Text C of S1 Appendix). SVM and LR obtained the highest MCCs in predicting sepsis patient survival, that is 0.533 and 0.533, respectively. Random Forest performed as second best model in this cohort with MCC equal to 0.516 whereas XGBoost and Tree obtained the lowest results 0.459 and 0.368, respectively. LR was chosen as the best performing because of the highest third quartile.
Each violin plot shows the distribution of the data, whereas the small boxplot inside each violin plot shows the median and the first and third quartiles, the whiskers indicate 0.05 and 0.95 quantiles.
The best score when predicting survival on the SIRS cohort was achieved with the XGB method that reached 0.489. The second and third best-performing models in the SIRS cohort were RF with MCC equal to 0.39 and LR showing MCC equal to 0.379. SVM and Tree showed scores equal to 0.378 and 0.289, respectively.
Fig 3 shows the median ROC and PRC curves for SIRS and SEPSIS cohorts as an overall summary of models’ performances across all Monte Carlo runs.
Receiver Operating Characteristic (ROC) curves for SIRS (panel (a)) and SEPSIS (panel (b)) cohorts showing the median performances of each model on the test sets generated during Monte Carlo cross-validation. Panels (c) and (d) depict Precision-Recall Curves (PRC) for SIRS and SEPSIS cohorts, respectively, showing the median performances of each model on the test sets generated during Monte Carlo cross-validation.
Feature ranking
This section describes the results obtained after the SFE approach performed on the models with the highest performance in the prediction task on each of the two cohorts. Median values and interquartile ranges for the resulting MCCs are reported in the supplementary material (Text E of S1 Appendix). A graphical representation of feature importance is shown in Fig 4a for the SEPSIS cohort and in Fig 4b for the SIRS cohort where features were ordered from lowest to the highest importance.
Features were ranked according to importance from left to right. Each violin plot shows the distribution of the data, whereas the small boxplot inside each violin plot shows the median and the first and third quartiles, the whiskers indicate 0.05 and 0.95 quantiles.
Specifically, APACHE II showed the highest importance, that is the lowest resulting median MCC equal to 0.436 (–0.097), in predicting SEPSIS patient survival with a Logistic Regression model. MPV ranked second in terms of feature importance for this specific cohort with MCC equal to 0.484. The other features did not induce a notable decrease in the model’s performance. Feature ranking for the survival prediction of SIRS patients with XGB algorithm showed that SOFA has the highest importance with resulting MCCs equal to 0.381 (–0.108) when the feature is removed.
Results with Spearman coefficient and Kendall distance did not show a significant correlation between the two series, with correlation equal to –0.091 (p = 0.737) and –0.007 (p = 0.983), respectively.
Discussion
Gucyetmez et al. [10] collected the data used in this study for exploring the ability of hemogram and CRP in discriminating between SIRS and SEPSIS cohorts. However, the authors did not investigate the prognostic role of the selected features within each cohort, therefore, our study aimed to investigate more in detail the importance of these features and the possible differences between the considered cohorts. Specifically, we performed the evaluation of the ability of hemogram parameters in predicting the survival of ICU patients diagnosed with SIRS or SEPSIS, using a set of parameters usually available in the patient clinical records. We used widely available features like patient sex, illness severity scores commonly measured and recorded at admission in the ICU, C-reactive protein, and blood cell count measurements. Patients’ comorbidities were not available in the patient’s records shared by Gucyetmez et al. despite they are commonly available in an ICU setting, which represents a significant lack of information. The developed models would have certainly benefited from more information about the patient’s history and this could have led to a more precise identification of differences in the prognostic factors. Therefore, future studies will focus on the extension of these analyses on more complete data including patients’ comorbidities.
The survival prediction models were developed and tested on SIRS and SEPSIS cohorts, with better performances observed in SEPSIS cohort.
Specifically, among all trained ML models, linear-based models like LR and SVM showed higher performances in the SEPSIS cohort, whereas RF and XGB performed better on the SIRS cohort.
Although average calibration curves are sub-optimal, which is likely due to the reduced sample size, the best-performing models show improved calibration with respect to the worst ones as expected.
This behavior suggests that a bigger population would allow for a proper calibration adjustment and translation of the model’s output score to an even more precise individualized patient survival probability. Also, this approach might account for a possible covariate shift due to changes in patient characteristics, without the need for the development of a new model. Therefore, we do consider that the relationships between the available variables both intra- and inter-population can be considered a reliable multivariable comparison of the major factor predicting survival for both SIRS and sepsis patients. As it can be observed in Text F of S1 Appendix, the sensitivity analysis implementing the hyperparameter optimization shows results very close to those observed without hyperparameter optimization, thus highlighting the robustness of the proposed framework.
The feature importance analysis attributed the highest importance to APACHE II and SOFA scores for SEPSIS and SIRS cohorts, respectively, thus confirming the importance of a preliminary assessment of patients’ risk at the admission in the ICU [45]. This result is also confirmed by statistical analysis, as shown in Table 4.
SEPSIS cohort
Results on the SEPSIS cohort showed that MPV was the second most important variable in predicting survival. This result is in line with the observed association of a higher MPV with an increased mortality risk as well as its predictive role [46–48]. Our analysis ranked EoC and CRP as third and fourth most important features. In the literature, a lower EoC has been associated with mortality in critically ill medical patients [49], in patients admitted with an exacerbation of chronic obstructive pulmonary disease [50] and in patients with pneumonia [51]. Interestingly, although non-significant, our cohort showed an increase in EoC in deceased SEPSIS patients. An epidemiological study [52] pointed out that eosinophilia is a predictor of all-cause mortality and that an increased number of peripheral blood eosinophils may reflect an increased inflammatory response, resulting in tissue injury, a condition that may reflect our cohort. CRP was the fourth most important variable in our ML model. Of note, CRP had already shown the potential of being a predictor of survival of ICU patients [53], and more in general a predictor of mortality in ML frameworks [54].
SIRS cohort
Interestingly, the third most important variable in predicting survival for the SIRS cohort was platelet counts, with a smaller median value for the non-survived patients than for the survived group. Indeed, Vanderschueren et al. [55] found that Thrombocytopenia was associated with a higher risk of death in a septic cohort, in line with our results considering the definition of sepsis (sepsis-1) used in 2000 which only required two SIRS criteria. CRP ranked fourth in predicting patient survival with non-sepsis SIRS and similar considerations as for the SEPSIS cohort can be done, moreover, its importance in predicting survival of a non-sepsis SIRS cohort was already observed in animal studies [56]. The fifth and sixth-ranked features were lymphocytes and eosinophils. In literature, Lymphocytes counts were found to be associated with increased mortality risk in general ICU patients [57], heart failure [58], and COVID-19 patients [59]. Eosinophils count significantly differed between survived and deceased groups with an increase in the deceased one. Similar considerations as for the SEPSIS cohort can be done also for eosinophil counts, where we already pointed out that this apparently opposite behavior with respect to literature might be due to the specific cohort of our study with undergoing inflammatory response [52].
General considerations and applicability
The developed models show the ability to predict patient survival and specifically, this study can be considered as an important integration of the study performed by Gucyetmez et al. [10] so that once a patient with inflammatory response has been identified as septic or not the corresponding model can give us the possibility to immediately assess the likelihood of survival. Also, the feature importance analysis proposed in our study gives a clue on the main features that contributed to the developed cohort-specific score, and it suggests to clinicians which of the considered variables is more informative for a patient falling in the SIRS or SEPSIS cohort. It is important to notice that the SFE method presents some limitations when features are highly interdependent, since the contribution of a feature that is very important may still be underestimated due to the effect of other covariates that depend on it.
Finally, it is worth mentioning that we are not aware of whether these data were collected for administrative health reasons or whether they are commonly used for clinical practice, which might limit the general applicability of the results. However, the employment of data like these for scientific analyses based on computational intelligence can allow new scientific discoveries that otherwise would be impossible with traditional hospital technologies.
This study presents an original application of a statistical framework aimed at predicting patient survival. As the approach is mainly limited by the reduced sample size of the cohort, it is expected that a larger collection of data would allow for a more effective model calibration and optimization that would further improve the model generalizability, thus providing a more precise estimate of patient survival probability.
Conclusions
The proposed study applies an original machine learning paradigm for processing clinical information at admission in the ICU to predict patient survival. The proposed approach relies on a multi-variable predictive modeling approach based on information gathered at the ICU admission, and aimed at predicting the likelihood of patient survival for patients with SIRS and with SEPSIS. Results provide insights into the differences of the most relevant variables between the two groups. A Monte Carlo Cross-Validation procedure was further applied to have robust estimates of the obtained scores. The performed sensitivity analysis showed that results did not notably vary with hyperparameter tuning thus confirming the need for a larger cohort to advance to a fully calibrated deployable model.
In this context, Logisitic Regression and XGBoost algorithms are the best-performing models for SEPSIS and SIRS cohorts, respectively. Moreover, feature importance analysis revealed a high importance of APACHE II score and a comparable important role of C-reactive protein in both cohorts. Also, MPV and EoC were revealed to be important predictors of survival mainly in the SEPSIS cohort, whereas they showed a secondary role in the SIRS cohort.
SIRS cohort showed greater importance of SOFA and platelets count features which instead ranked last in SEPSIS.
Importantly, beyond Gucyetmez et al. [10] findings, the proposed framework addresses the question of whether a patient has sepsis or not, and our models give clinicians the possibility to estimate patient’s survival, as well as to identify the most important features involved in the stratification of patients’ risk with SIRS or SEPSIS, and that also led to the proposed survival estimates.
Of note, to our knowledge, this is the first study where the ability of hemogram parameters in predicting patient survival at the admission in the ICU and the role of the considered features are investigated to highlight differences between SIRS and SEPSIS patients.
Supporting information
S1 Appendix.
Text A: Formulas of the confusion matrix rates. Text B: Model Hyperparameters. Text C: Model performances. Text D: Model Calibration. Text E: Single Feature Elimination. Text F: Sensitivity Analysis: Hyperparameter Optimization.
https://doi.org/10.1371/journal.pdig.0000459.s001
(PDF)
S1 Fig. Average calibration curves across the MCCV runs for the best (left column) and worst (right column) for SEPSIS (upper row) and SIRS (lower row).
https://doi.org/10.1371/journal.pdig.0000459.s002
(EPS)
References
- 1. Veldhoen R.A., Howes D., Maslove D.M.: Is mortality a useful primary end point for critical care trials? Chest 158(1), 206–211 (2020). pmid:31790654
- 2. REMAP-CAP Writing Committee for the REMAP-CAP Investigators, Effect of antiplatelet therapy on survival and organ support–free days in critically ill patients with COVID-19: a randomized clinical trial JAMA 327, 1247–1259 (2022). pmid:35315874
- 3. The EPISEPSIS Study Group: EPISEPSIS: a reappraisal of the epidemiology and outcome of severe sepsis in French intensive care units. Intensive Care Medicine 30(4), 580–588 (2004). pmid:14997295
- 4. Gaieski D.F., Edwards J.M., Kallan M.J., Carr B.G.: Benchmarking the incidence and mortality of severe sepsis in the United States. Critical Care Medicine 41(5), 1167–1174 (2013). pmid:23442987
- 5. Yébenes J.C., Ruiz-Rodriguez J.C., Ferrer R., Clèries M., Bosch A., Lorencio C., Rodriguez A., Nuvials X., Martin-Loeches I., Artigas A.: Epidemiology of sepsis in Catalonia: analysis of incidence and outcomes in an European setting. Annals of Intensive Care 7(1) (2017). pmid:28220453
- 6. Kaukonen K.-M., Bailey M., Suzuki S., Pilcher D., Bellomo R.: Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012. JAMA 311(13), 1308 (2014). pmid:24638143
- 7. Bone R.C., Balk R.A., Cerra F.B., Dellinger R.P., Fein A.M., Knaus W.A., Schein R.M.H., Sibbald W.J.: Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Chest 101(6), 1644–1655 (1992). pmid:1303622
- 8. Levy M.M., Fink M.P., Marshall J.C., Abraham E., Angus D., Cook D., Cohen J., Opal S.M., Vincent J.-L., Ramsay G.: 2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference. Critical Care Medicine 31(4), 1250–1256 (2003). pmid:12682500
- 9. Singer M., Deutschman C.S., Seymour C.W., Shankar-Hari M., Annane D., Bauer M., Bellomo R., Bernard G.R., Chiche J.-D., Coopersmith C.M., Hotchkiss R.S., Levy M.M., Marshall J.C., Martin G.S., Opal S.M., Rubenfeld G.D., van der Poll T., Vincent J.-L., Angus D.C.: The third international consensus definitions for sepsis and septic shock (SEPSIS-3). JAMA 315(8), 801–810 (2016). pmid:26903338
- 10. Gucyetmez B., Atalan H.K.: C-reactive protein and hemogram parameters for the non-sepsis systemic inflammatory response syndrome and sepsis: what do they mean? PLOS One 11(2), 1–9 (2016). pmid:26863002
- 11. Desautels T., Calvert J., Hoffman J., Jay M., Kerem Y., Shieh L., Shimabukuro D., Chettipally U., Feldman M.D., Barton C., Wales D.J., Das R.: Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Medical Informatics 4(3), 28 (2016). pmid:27694098
- 12. Fleuren L.M., Klausch T.L.T., Zwager C.L., Schoonmade L.J., Guo T., Roggeveen L.F., Swart E.L., Girbes A.R.J., Thoral P., Ercole A., Hoogendoorn M., Elbers P.W.G.: Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Medicine 46(3), 383–400 (2020). pmid:31965266
- 13. Chicco D., Oneto L.: Data analytics and clinical feature ranking of medical records of patients with sepsis. BioData Mining 14(1) (2021). pmid:33536030
- 14. Mollura M., Lehman L.-W.H., Mark R.G., Barbieri R.: A novel artificial intelligence based intensive care unit monitoring system: using physiological waveforms to identify sepsis. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 379 (2212) (2021). pmid:34689614
- 15. Kim Sujin, P. R.W. Kim Woojae: A comparison of intensive care unit mortality prediction models through the use of data mining techniques. Healthcare Informatics Research 17(4), 232–243 (2011). pmid:22259725
- 16. Silva I., Moody G., Scott D.J., Celi L.A., Mark R.G.: Predicting in-hospital mortality of ICU patients: The PhysioNet/Computing in cardiology challenge 2012. In: 2012 Computing in Cardiology, pp. 245–248 (2012) pmid:24678516
- 17. Pirracchio R., Petersen M.L., Carone M., Rigon M.R., Chevret S., van der Laan M.J.: Mortality prediction in intensive care units with the super ICU learner algorithm (SICULA): a population-based study. The Lancet Respiratory Medicine 3(1), 42–52 (2015). pmid:25466337
- 18. Awad A., Bader-El-Den M., McNicholas J., Briggs J.: Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. International Journal of Medical Informatics 108, 185–195 (2017). pmid:29132626
- 19.
Johnson, A.E.W., Pollard, T.J., Mark, R.G.: Reproducibility in critical care: a mortality prediction case study. In: Doshi-Velez, F., Fackler, J., Kale, D., Ranganath, R., Wallace, B., Wiens, J. (eds.) Proceedings of MLHC 2017—the 2nd Machine Learning for Healthcare Conference. Proceedings of Machine Learning Research, vol. 68, pp. 361–376. PMLR, Boston, Massachusetts, USA (2017)
- 20. Purushotham S., Meng C., Che Z., Liu Y.: Benchmarking deep learning models on large healthcare datasets. Journal of Biomedical Informatics 83, 112–134 (2018). pmid:29879470
- 21. Nistal-Nuño B.: Developing machine learning models for prediction of mortality in the medical intensive care unit. Computer Methods and Programs in Biomedicine 216, 106663 (2022). pmid:35123348
- 22. Hong C., Chen J., Yi F., Hao Y., Meng F., Dong Z., Lin H., Huang Z.: CD-Survdr: a contrastive-based model for dynamic survival analysis. Health Information Science and Systems 10(1) (2022). pmid:35494891
- 23. Levy M.M., Evans L.E., Rhodes A.: The surviving sepsis campaign bundle: 2018 update. Intensive care medicine 44(6), 925–928 (2018) pmid:29675566
- 24. Taylor R.A., Pare J.R., Venkatesh A.K., Mowafi H., Melnick E.R., Fleischman W., Hall M.K.: Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data–driven, machine learning approach. Academic Emergency Medicine 23(3), 269–278 (2016). pmid:26679719
- 25. Tansakul V., Li X., Koszalinski R., Paiva W., Khojandi A.: Prediction of sepsis and in-hospital mortality using electronic health records. Methods of Information in Medicine 57, 185–193 (2018). pmid:30248708
- 26. Chicco D., Jurman G.: Survival prediction of patients with sepsis from age, sex, and septic episode number alone. Scientific Reports 10(1) (2020). pmid:33051513
- 27. Zhang K., Zhang S., Cui W., Hong Y., Zhang G., Zhang Z.: Development and validation of a sepsis mortality risk score for sepsis-3 patients in intensive care unit. Frontiers in Medicine 7 (2021). pmid:33553206
- 28. van Doorn W.P.T.M., Stassen P.M., Borggreve H.F., Schalkwijk M.J., Stoffers J., Bekers O., Meex S.J.R.: A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis. PLOS One 16(1), 0245157 (2021). pmid:33465096
- 29. Selcuk M., Koc O., Kestel A.S.: The prediction power of machine learning on estimating the sepsis mortality in the intensive care unit. Informatics in Medicine Unlocked 28, 100861 (2022).
- 30.
Glantz S.A.: Primer of Biostatistics. The Mc Graw-Hill Companies Inc., California (2012).
- 31. Benjamin D.J., Berger J.O., Johannesson M., Nosek B.A., Wagenmakers E.-J., Berk R., Bollen K.A., Brembs B., Brown L., Camerer C., Cesarini D., Chambers C.D., Clyde M., Cook T.D., De Boeck P., Dienes Z., Dreber A., Easwaran K., Efferson C., Fehr E., Fidler F., Field A.P., Forster M., George E.I., Gonzalez R., Goodman S., Green E., Green D.P., Greenwald A.G., Hadfield J.D., Hedges L.V., Held L., Hua Ho T., Hoijtink H., Hruschka D.J., Imai K., Imbens G., Ioannidis J.P.A., Jeon M., Jones J.H., Kirchler M., Laibson D., List J., Little R., Lupia A., Machery E., Maxwell S.E., McCarthy M., Moore D.A., Morgan S.L., Munafó M., Nakagawa S., Nyhan B., Parker T.H., Pericchi L., Perugini M., Rouder J., Rousseau J., Savalei V., Schönbrodt F.D., Sellke T., Sinclair B., Tingley D., Van Zandt T., Vazire S., Watts D.J., Winship C., Wolpert R.L., Xie Y., Young C., Zinman J., Johnson V.E.: Redefine statistical significance. Nature Human Behaviour 2(1), 6–10 (2017).
- 32. Chicco D., Jurman G.: Arterial disease computational prediction and health record feature ranking among patients diagnosed with inflammatory bowel disease. IEEE Access 9, 78648–78657 (2021).
- 33. Chawla N.V., Bowyer K.W., Hall L.O., Kegelmeyer W.P.: SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16, 321–357 (2002)
- 34. Cortes C., Vapnik V.: Support-vector networks. Machine Learning 20(3), 273–297 (1995).
- 35. Breiman L.: Random forests. Machine Learning 45(1), 5–32 (2001).
- 36.
Chen, T., Guestrin, C.: XGBoost. Proceedings of KDD’16—the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016).
- 37.
Géron A., Hands-on machine learning with scikit-learn and TensorFlow: Concepts, Tools, and Techniques to build intelligent systems. O’Reilly Media (2017).
- 38. Matthews B.W.: Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA)—Protein Structure 405(2), 442–451 (1975). pmid:1180967
- 39. Chicco D., Jurman G.: The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21(1) (2020). pmid:31898477
- 40. Chicco D., Tötsch N., Jurman G.: The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Mining 14(1) (2021). pmid:33541410
- 41. Chicco D., Warrens M.J., Jurman G.: The Matthews correlation coefficient (MCC) is more informative than Cohen’s Kappa and Brier score in binary classification assessment. IEEE Access 9, 78368–78381 (2021).
- 42. Chicco D.: Ten quick tips for machine learning in computational biology. BioData Mining 10(1), 1–17 (2017). pmid:29234465
- 43.
Chicco, D., Ciceri, E., Masseroli, M.: Extended Spearman and Kendall coefficients for gene annotation list correlation. In: Di Serio, C., Liò, P., Nonis, A., Tagliaferri, R. (eds.) Computational Intelligence Methods for Bioinformatics and Biostatistics, CIBB 2014 Revised Selected Papers, vol. 8623, pp. 19–32. Springer, Cambridge, England, United Kingdom (2015).
- 44.
MacFarland T.W., Yates J.M.: Mann–Whitney U test. In: Introduction to Nonparametric Statistics for the Biological Sciences Using R, pp. 103–132. Springer, Cham, Switzerland (2016).
- 45. Bertsimas D., Lukin G., Mingardi L., Nohadani O., Orfanoudaki A., Stellato B., Wiberg H., Gonzalez-Garcia S., Parra-Calderón C.L., Robinson K., Schneider M., Stein B., Estirado A., a Beccara L., Canino R., Dal Bello M., Pezzetti F., Pan A.: COVID-19 mortality risk assessment: an international multi-center study. PLOS One 15(12), 0243262 (2020). pmid:33296405
- 46. Tajarernmuang P., Phrommintikul A., Limsukon A., Pothirat C., Chittawatanarat K.: The role of mean platelet volume as a predictor of mortality in critically ill patients: a systematic review and meta-analysis. Critical Care Research and Practice 2016, 1–8 (2016). pmid:26966574
- 47. Vardon-Bounes F., Gratacap M.-P., Groyer S., Ruiz S., Georges B., Seguin T., Garcia C., Payrastre B., Conil J.-M., Minville V.: Kinetics of mean platelet volume predicts mortality in patients with septic shock. PLOS One 14(10), 0223553 (2019). pmid:31622365
- 48. Vélez-Páez J.L., Legua P., Vélez-Páez P., Irigoyen E., Andrade H., Jara A., López F., Pérez-Galarza J., Baldeón L.: Mean platelet volume and mean platelet volume to platelet count ratio as predictors of severity and mortality in sepsis. PLOS One 17(1), 0262356 (2022). pmid:34990467
- 49. Abidi K., Belayachi J., Derras Y., Khayari M.E., Dendane T., Madani N., Khoudri I., Zeggwagh A.A., Abouqal R.: Eosinopenia, an early marker of increased mortality in critically ill medical patients. Intensive Care Medicine 37(7), 1136–1142 (2011). pmid:21369810
- 50. Holland M., Alkhalil M., Chandromouli S., Janjua A., Babores M.: Eosinopenia as a marker of mortality and length of stay in patients admitted with exacerbations of chronic obstructive pulmonary disease. Respirology 15(1), 165–167 (2010). pmid:19947999
- 51. Echevarria C., Hartley T., Nagarajan T., Tedd H., Steer J., Gibson G.J., Bourke S.C.: 30 day mortality and eosinopenia in patients with pneumonia. European Respiratory Journal 44(Suppl 58) (2014)
- 52. Hospers J.J., Schouten J.P., Weiss S.T., Postma D.S., Rijcken B.: Eosinophilia is associated with increased all-cause mortality after a follow-up of 30 years in a general population sample. Epidemiology 11(3), 261–268 (2000). pmid:10784241
- 53. Karagoz I.: Does hemogram biomarkers predict mortality in intensive care population. Experimental Biomedical Research 2(4), 163–168 (2019).
- 54. Gebhardt C., Hirschberger J., Rau S., Arndt G., Krainer K., Schweigert F.J., Brunnberg L., Kaspers B., Kohn B.: Use of C-reactive protein to predict outcome in dogs with systemic inflammatory response syndrome or sepsis. Journal of Veterinary Emergency and Critical Care 19(5), 450–458 (2009). pmid:19821886
- 55. Vanderschueren S., De Weerdt A., Malbrain M., Vankersschaever D., Frans E., Wilmer A., Bobbaers H.: Thrombocytopenia and prognosis in intensive care. Critical Care Medicine 28(6), 1871–1876 (2000). pmid:10890635
- 56. Gebhardt C., Hirschberger J., Rau S., Arndt G., Krainer K., Schweigert F.J., Brunnberg L., Kaspers B., Kohn B.: Use of C-reactive protein to predict outcome in dogs with systemic inflammatory response syndrome or sepsis. Journal of Veterinary Emergency and Critical Care 19(5), 450–458 (2009). pmid:19821886
- 57. Gutierrez-Rodriguez R., Aguilar-Alonso E., Arias-Verdu M., Castillo-Lorente E., Lopez-Caler C., Rojas-Amezcua M., De la Fuente-Martos C., Rivera-Fernandez R., Quesada-Garcia G.: Lymphopenia assessment in icu patients and relationship with mortality. Intensive Care Medicine Experimental 3(S1) (2015).
- 58. Terpos E., Ntanasis-Stathopoulos I., Elalamy I., Kastritis E., Sergentanis T.N., Politou M., Psaltopoulou T., Gerotziafas G., Dimopoulos M.A.: Hematological findings and complications of COVID-19. American Journal of Hematology 95(7), 834–847 (2020). pmid:32282949
- 59. Majmundar M., Kansara T., Park H., Ibarra G., Marta Lenik J., Shah P., Kumar A., Doshi R., Zala H., Chaudhari S., Kalra A.: Absolute lymphocyte count as a predictor of mortality and readmission in heart failure hospitalization. IJC Heart & Vasculature 39, 100981 (2022). pmid:35281758
- 60. Chicco D., Jurman G.: The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Mining 16(1), 1–23 (2023). pmid:36800973