Abstract
This study aims to establish a new scoring system based on biomarkers for predicting in-hospital mortality of children admitted to the pediatric intensive care unit (PICU). The biomarkers were chosen using the least absolute shrinkage and selection operator (LASSO)-logistic regression in this observational case-control study. The performance of the new predictive model was evaluated by the area under the receiver operating characteristic curve (AUC). Calibration plot was established to validate the new score accompanied by the Hosmer-Lemeshow test. There were 8818 patients included in this study. Finally, six predictors were included in the LASSO-regression model. Albumin <40 g/L, lactate dehydrogenase >452 U/L, lactate >3.2 mmol/L, urea >5.6 mmol/L, arterial PH <7.3 and glucose >6.9 mmol/L were treated as risk factors for higher mortality. The new score ranged from 1 to 6 among all the included patients. In the training set, the AUC of the probability of in-hospital mortality for the new predictive model was 0.81 (95% CI 0.79 to 0.84), which is larger than for the Pediatric Critical Illness Score (PCIS) (0.69, 95% CI 0.66 to 0.72). Similarly, in the validating set, the AUC of the probability of in-hospital mortality was larger for the new score (0.80, 95% CI 0.77 to 0.84) than for PCIS (0.67, 95% CI 0.63 to 0.72). The calibration plot and Hosmer-Lemeshow test showed excellent calibration. The calculated ORs showed a trend that higher scores indicated higher risk of death (p value for trend <0.001). In summary, this study develops and validates a totally biomarker-based new score to predict in-hospital mortality for pediatric patients admitted to PICU. More attention and more positive care and treatment should be given to children with a higher score.
Significance of this study
What is already known about this subject?
Previous studies suggested that many biomarkers such as serum creatinine, platelet, serum albumin (ALB), lactate dehydrogenase (LDH) and and so on, may be predictors of mortality, but the sensitivity and specificity was not high enough.
There are still many challenges in the early recognition of poor prognosis for critically ill children using laboratory values.
What are the new findings?
This is the first study establishing a simple-to-use scoring system for critically ill pediatric patients to predict the prognosis at admission.
The new scoring system is totally biomarker-based to avoid clinicians’ objective judgement of symptoms and signs. The in-hospital mortality ascends with the increase of the new score. ALB<40 g/L, LDH>452 U/L, lactate>3.2 mmol/L, urea>5.6 mmol/L, arterial PH <7.3 and glucose>6.9 mmol/L were treated as risk factors for higher mortality.
The accuracy of the new score was evaluated for different diagnoses and the results showed the new score could be more adaptive for patients with diseases of the digestive system and genitourinary system.
How might these results change the focus of research or clinical practice?
As this new score is totally biomarker-based, it could be implemented in clinical practice by development of an online calculator or implementation in an electronic medical record system. With such a tool, the predicted probability of mortality for the individual patient can be easily generated right after obtaining the six predictors.
Introduction
The pediatric intensive care unit (PICU) is responsible for providing comprehensive monitoring and life support for critically ill children. According to reports, the observed mortality rate in the PICU is 5.3%–37.35% in developing countries,1 2 which is higher than in developed countries.3 It brings a heavy economic burden on the family and society.4 Therefore, in order to optimise the diagnostic approach, care and outcome for critically ill children, the early identification of the development of poor prognosis is considered imperative.
Up to now, a series of scoring systems have been developed to predict the severe condition and poor prognosis in critically ill children such as the Pediatric Critical Illness Score (PCIS)5 and the Pediatric Risk of Mortality III.6 Especially, PCIS was widely used in China. However, they often need information that is difficult to retrieve (eg, Glasgow Coma Scale Score, cardiac function, comorbidity, underlying disease) and accurate judgement of experienced clinicians is required. Moreover, some needed vital signs such as heart rate, respiratory rate and blood pressure often fluctuate violently even in the short term. To our knowledge, researchers have attached importance to the value of laboratory biomarkers in the prognosis prediction in recent decades. Previous studies suggested that many biomarkers such as serum creatinine, platelet, serum albumin (ALB), lactate dehydrogenase (LDH) and so on, may be predictors of mortality, but the sensitivity and specificity was not high enough.7–9 There are still many challenges in the early recognition of poor prognosis for critically ill children using laboratory values.
Therefore, in order to evaluate disease severity early and effectively, we aim to establish a new scoring system with a combination of biomarkers that are easily available at admission and verify the accuracy of the new score for predicting the in-hospital mortality of children admitted to PICU.
Methods
The results of the present study were reported following the Strengthening the Reporting of Observational Studies in Epidemiology statement.10
Study population
This is an observational case-control study conducted using data from the Children’s Hospital of Zhejiang University, a 1900-bed children’s hospital in the south of China. The clinical data of patients admitted to PICUs was used over a period of 8 years from 2010 to 2018, to construct the Paediatric Intensive Care (PIC) database, a freely accessible pediatric-specific critical care database. The establishment of the PIC database has been reported in detail elsewhere.11 Briefly, a total of 12 881 patients with 13 449 admissions were recorded in the database, all of whom were admitted into PICUs and 971 (7.2%) patients died in hospital. Structured clinical data including patient demographics, symptoms, vital signs, comprehensive laboratory results, microbiological information, prescriptions and surgeries were all collected. The requirement for individual patient consent was waived because the project did not impact clinical care, and all protected health information was deidentifed. We accessed the PIC database under a data use agreement from the database managers.
Data extraction
Patients, except neonates, were included if they were admitted to PICU for the first time. Furthermore, we excluded the patients whose information was seriously absent or wrongly recorded. The data collected included age, gender, diagnoses of disease (including diseases of the respiratory system, nervous system, circulatory system, digestive system, genitourinary system and neoplasms), whether surgery was performed or not in this hospital stay, common laboratory values, the length of PICU stay and in-hospital mortality. In addition, PCIS5 was calculated. The clinical data and laboratory values were all obtained within 24 hours of hospital admission.
Statistical analysis
In this study, continuous variables were compared using the Mann-Whitney U test and were presented as medians with IQRs because most of variables were not normally distributed. Categorical variables were analyzed by the χ2 test or Fisher’s exact test, as appropriate. They were expressed as numbers (n) with percentages (%).
We divided the included patients into training set (70% data) and validating set (30% data) with ‘sample()’ function in R software. In the training set, all extracted laboratory values were included in the least absolute shrinkage and selection operator (LASSO)-logistic regression with 10-fold cross-validation for selecting the optimal factors.12 The performance of the new predictive model was evaluated and compared with that of PCIS by the area under the receiver operating characteristic (ROC) curve (AUC) in both the training set and the validating set. A calibration plot was established to validate the new score accompanied by the Hosmer-Lemeshow test in the validating set. The ROC curve was calculated for each included variable and Youden’s Index was calculated to determine their optimal cut-off values.13 To develop the new score, we divided each continuous variable into two categories in terms of the cut-off point of the ROC analysis. A value of ‘+1’ was assigned when the effects were higher risk, while ‘0’ was assigned when the effects were lower risk. The new score was calculated by summing up the score of each item. The ROC curves of the new score were drawn and compared for the population groups with different diagnoses. In dose-response analyses, we fitted unadjusted logistic models with the new score as a restricted cubic spline with four knots. The Wald test was used to determine whether there was a linear or non-linear relationship between the new score and in-hospital mortality.14 To test whether a trend across scores of the new score existed for risk estimates, the logistic regression was used to calculate the ORs and 95% CIs for the new score with the in-hospital mortality, with the lowest score as the reference group. Covariates were selected based on clinical experience and the change-in-estimate method.15 In the change-in-estimate method, a variable is considered to be adjusted if its inclusion in the regression model changes the regression coefficient by ≥10%. Subgroup analyses were planed to be performed to evaluate whether the observed association of the new score with in-hospital mortality was modified by age, gender or surgical treatment. Finally, decision curve analysis (DCA) was performed to evaluate the clinical utility of the new predictive model by calculating the net benefits at different threshold probabilities in both the training and the validating sets.16
A value of p<0.05 was considered statistically significant, and all tests were two-sided. Data extraction was conducted using PostgreSQL. Statistical analyses were performed using Stata V.15.1 software and R V.3.61 software.
Results
Participant characteristics
According to the inclusion criteria of this study, there were 8818 patients finally included in this study, with a total of 471 cases included in the non-survival group and 8344 cases in the survival group.
All patients were between 1 month and 18 years old. Patients in the non-survival group were on average younger than those in the survival group (p=0.037). There was a larger proportion of male patients in the non-survival group than in survival group (p=0.004). The proportion of patients with diseases of the respiratory system, digestive system, genitourinary system and neoplasms were significantly different between the two groups (all p<0.05). More patients had surgical experience in the survival group (p<0.001).
The difference in most of the laboratory values was significant between the two groups (all p<0.05). The PCIS Score was significant higher in the survival group than in the non-survival group (p<0.001). Patients in the non-survival group had a longer length of PICU stay than those in the survival group (p<0.001). The descriptive statistics of all included patients are presented in table 1.
Development and validation of a new score
After random sampling, there were 6172 patients finally in the training set and 2646 in the validating set. Nineteen potential variables that might predict in-hospital mortality were analyzed. Finally, six predictors (ALB, LDH, urea, arterial potential of hydrogen (PH), lactate and glucose) were included in the LASSO-regression model. A cross-validated error plot of the LASSO-regression model is shown in online supplemental figure 1a). The path of the coefficients included in this model with varying log-transformed λ values is shown in online supplemental figure 1b). In the training set, the AUC of the probability of in-hospital mortality for the new predictive model was 0.81 (95% CI 0.79 to 0.84), which is bigger than that for PCIS (0.69, 95% CI 0.66 to 0.72) (figure 1A). Similarly, in the validating set, the AUC of the probability of in-hospital mortality was bigger for the new predictive model (0.80, 95% CI 0.77 to 0.84), than for PCIS (0.67, 95% CI 0.63 to 0.72) (figure 1B). The calibration plot in the validating set showed excellent calibration as the plot followed the 45 degree line of perfect calibration (online supplemental figure 2). This result was confirmed by a non-significant Hosmer-Lemeshow test (p=0.18).
Supplementary data
To establish the new score, each continuous variable was divided into two categories in terms of the cut-off point of ROC analysis (table 2). ALB <40 g/L, LDH >452 U/L, lactate >3.2 mmol/L, urea >5.6 mmol/L, arterial PH <7.3 and glucose >6.9 mmol/L were treated as risk factors for higher mortality. A value of ‘+1’ was assigned when the effects were higher risk while ‘0’ was assigned when the effects were lower risk. Finally, each patient obtained a score that was calculated by summing up the score of each item. In summary, the new score ranged from 1 to 6 among all the included patients.
The new score in predicting in-hospital mortality in different groups
The AUC-ROC showed that the performance of the new score in predicting in-hospital mortality was different in groups with diseases of the respiratory system (0.67, 95% CI 0.62 to 0.72), diseases of the nervous system (0.77, 95% CI 0.70 to 0.84), diseases of the circulatory system (0.77, 95% CI 0.70 to 0.84), diseases of the digestive system (0.87, 95% CI 0.81 to 0.93), diseases of the genitourinary system (0.95, 95% CI 0.90 to 1.00) and neoplasms (0.72, 95% CI 0.64 to 0.80)(online supplemental figure 3).
Association between the new score and in-hospital mortality
The dose-response analysis was conducted between the new score and in-hospital mortality. The result showed a non-linear relationship between the new score and in-hospital mortality (p value for non-linearity <0.05) (figure 2). The calculated ORs showed there was a trend that higher scores indicated higher risk of death (p value for trend <0.001) (table 3). Subgroup analyses showed that the observed association between the new score and in-hospital mortality could be modified by surgical factors (p value for interaction <0.001). However, age and gender were not factors that could modify in-hospital mortality prediction (all p values for interaction >0.05) (figure 3).
The clinical value of the new score
The clinical value of the new score was evaluated by conducting DCA. DCA is a novel method for evaluating alternative predictive strategies. The DCA curves showed the new score had more obvious net benefits than PCIS in both the training set and the validating set (online supplemental figure 4).
Discussion
In this study, we successfully constructed and validated a simple-to-use new scoring system only containing biomarkers, which provided excellent performance in predicting the severity of the disease for patients admitted to PICU. Higher score indicates higher risk of death in hospital. Compared with PCIS, the discriminatory ability and clinical value of the new scoring system is better in predicting in-hospital mortality. Furthermore, the non-linear dose-response trend indicates that in-hospital mortality would change in a non-parallel manner with changes in the new score.
The scoring system containing six easily and reliably retrievable biomarkers in clinical practice included ALB, LDH, urea, arterial PH, lactate and glucose. ALB <40 g/L, LDH >452 U/L, lactate >3.2 mmol/L, urea >5.6 mmol/L, arterial PH <7.3 and glucose >6.9 mmol/L were treated as risk factors for higher mortality. The included factors are all markers of inflammation which are often changed in the inflammatory process and play major roles in regulating inflammation and innate immunity. Previous studies have also suggested that these biomarkers were predictors for mortality in different conditions,8 9 17–20 which is consistent with the results of this study.
The PCIS was the most widely used scoring system for predicting the prognosis of critically ill pediatric patients in China.5 It has suboptimal discriminatory power and clinical value than the newly established score with lower AUC-ROC in predicting in-hospital mortality in this study. To our knowledge, PCIS includes vital signs such as heart rate, blood pressure and respiratory rate. However, the values of these factors were not reliable enough and would fluctuate with different age ranges and may change rapidly even in a short period for certain values. Moreover, PCIS also includes arterial PO2, which may not be an appropriate factor for prognosis prediction because many children were given oxygen treatment. In addition, clinicians may have difficulties in the judgement of disorders in the gastrointestinal system for critically ill children at an early stage of the disease, which usually leads to inaccurate scoring in PCIS. On the contrary, the newly established score was totally biomarker based. It is objective and scientific enough to obtain an accurate total score which infers the disease severity in critically ill children.
Interestingly, the differences in predictive ability of the new score among different diagnoses were observed in terms of AUC-ROC. It seems that the new scoring system is more adaptive for critically ill children with diseases of the digestive or genitourinary systems. The reason for this was not clear and further studies are needed to validate this observation. In subgroup analysis we found that the positive association between the new score and in-hospital mortality could be modified by surgical factors. Although these factors did not reverse the predictive effect of the new score on in-hospital mortality, a better predictive ability of the new score was observed in children who had surgical treatment during this hospital stay. In addition, the results of dose-response analysis indicated that mortality increased with an increase in the new score, but it changed in a non-linear manner. It may be inferred that the risk of death is not simply superimposed with more predictors in the new score; this may be related to the different degrees of inflammation of organs and systems in the body.
Strength and limitations
Our study has several strengths that should be highlighted. First, this is the first study establishing a simple-to-use scoring system for critically ill children to predict the prognosis at an early stage. Second, the new scoring system is totally biomarker based to avoid clinicians’ objective judgment of symptoms and signs. Third, the huge sample size guarantees accuracy and stability of the results. Fourth, the accuracy of the new score was evaluated for different diagnoses and the results showed the new score could be more adaptive for patients with diseases of the digestive and genitourinary systems.
There were some shortcomings in this study also. First, the actual AUC-ROC of the new score is not high enough. Further studies should explore more biomarkers to improve the performance of the scoring system, with a larger sample size. Second, the included patients were all Chinese children, which may require the new score to be validated in different populations. Third, we did not obtain permission to collect information of more children hospitalized in intensive care units in other places. As a result, only internal validation was done and there was no external validation in this study, which led to a limited evaluation of the extrapolative utility of the new model.
Clinical implications
As this new score is totally biomarker based, it could be implemented in clinical practice by the development of an online calculator or implementation in an electronic medical record system. With such a tool, the predicted probability of mortality for the individual patient can be generated easily right after obtaining the six predictors.
Conclusion
In summary, this study develops and validates a totally biomarker-based new score to predict in-hospital mortality for pediatric patients admitted to a PICU. The in-hospital mortality ascends with the increase of the new score. ALB <40 g/L, LDH >452 U/L, lactate >3.2 mmol/L, urea >5.6 mmol/L, arterial PH <7.3 and glucose >6.9 mmol/L were treated as risk factors for higher mortality. More attention and more positive care and treatment should be given to children with higher scores. Future studies should be conducted to clarify whether in-hospital mortality could be modified by age, surgical factors, diseases of the respiratory, digestive or genitourinary systems, and explore the possible reason.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This project was approved by the Institutional Review Board of the Children’s Hospital, Zhejiang University School of Medicine (Hangzhou, China).
Acknowledgments
The authors thank the Children’s Hospital of Zhejiang University and Professor Haomin Li for access to PIC data.
Footnotes
Contributors YZ conceptualized and designed the study, supervised data collection, carried out the initial analyses, drafted the initial manuscript. QS designed the data collection instruments, collected data. GZ and XL coordinated and supervised data collection, assisted in the statistical analysis and carried out the initial analyses. JL and ZF coordinated and supervised data collection, and critically reviewed the manuscript for important intellectual content. JD conceptualized and designed the study, supervised data collection, reviewed and revised the manuscript. All authors read and approved the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.