Comparison of a new predictive model with other critical scores for predicting in-hospital mortality among children with pneumonia-related bacteremia
=====================================================================================================================================================

* Jilei Lin
* Yin Zhang
* Anchao Song
* Nan Yang
* Linyan Ying
* Jihong Dai

## Abstract

Prediction of mortality in children with pneumonia-related bacteremia is necessary for providing timely care and treatment. This study aims to develop and validate a nomogram and compare it with Pediatric Risk of Mortality III (PRISM III), Brighton Pediatric Early Warning Score (Brighton PEWS) and Pediatric Critical Illness Score (PCIS), which are widely used in predicting in-hospital mortality in children with pneumonia-related bacteremia. This retrospective study collected clinical data of hospitalized children with pneumonia-related bacteremia in Chongqing, China (January 2013–May 2019). The nomogram was built using multivariate logistic regression analysis. The nomogram was compared with PRISM III, PEWS and PCIS in accuracy and clinical benefits in predicting in-hospital mortality in children with pneumonia-related bacteremia. A total of 242 children were included. The nomogram including time to first positivity of blood cultures (TTFP), serum albumin (ALB) and lactate dehydrogenase (LDH) was established. The area under the receiver operating characteristic curve of the nomogram was 0.84 (95% CI 0.77 to 0.91) in the training set and 0.82 (95% CI 0.71 to 0.93) in the validating set. Good consistency was observed between the predictions and the actual observations, and the decision curve analysis showed that the nomogram was clinically useful. The results showed that the nomogram significantly performed better than the three critical scores. In conclusion, a nomogram-illustrated model incorporating TTFP, ALB and LDH for predicting in-hospital mortality in children with pneumonia-related bacteremia at the early stage was established and validated. It performed better than PRISM III, PEWS and PCIS.

*   bacteria
*   critical care
*   prognosis
*   respiratory tract diseases

### Significance of this study

#### What is already known about this subject?

*   A quarter to a third of pneumonia and 27.3% of severe pneumonia in children are caused by bacteria.

*   Prediction of mortality in children with pneumonia-related bacteremia is necessary for providing timely care and treatment.

*   There is no predictive model for predicting in-hospital mortality in children with pneumonia-related bacteremia at the early stage.

#### What are the new findings?

*   Laboratory indexes including time to first positivity of blood cultures (TTFP), serum albumin (ALB) and lactate dehydrogenase (LDH) were closely related to in-hospital mortality in children with pneumonia-related bacteremia.

*   The nomogram including TTFP, ALB and LDH was established for predicting in-hospital mortality in children with pneumonia-related bacteremia.

*   The newly established model is better than the three traditional critical scores.

#### How might these results change the focus of research or clinical practice?

*   This nomogram-illustrated model may be a new approach to predicting in-hospital mortality in children with pneumonia-related bacteremia.

*   Clinicians may be able to provide more timely care and treatment with the help of this model.

## Introduction

Pneumonia is one of the leading causes of hospitalization in children, especially in developing countries.1 2 The mortality of severe pneumonia is approximately 6.4% in children younger than 5 years old.3 A quarter to a third of pneumonia and 27.3% of severe pneumonia in children are caused by bacteria.3 A significant number of patients suffered from bloodstream infections (bacteremia), resulting in life-threatening conditions such as sepsis, shock and even death.4 Mortality approaches 10%–20% for children with bacteremia admitted to the pediatric intensive care unit.5 

Some laboratory variables such as white cell count, C reactive protein (CRP), procalcitonin (PCT), etc, have been used to indicate the possibility of bacterial infection. Some studies have suggested that high levels of PCT and CRP are closely related with severity of bacterial infection.6–8 Our previous study showed that the time to first positivity of blood cultures (TTFP) was associated with in-hospital mortality of children with pneumonia-related bacteremia.9 However, Parlato *et al*’s10 work pointed out that the value of individual laboratory indicators is limited and the predictive effect of these variables is uncertain at the early stage of bacterial bloodstream infection. Therefore, it may be useful to combine available clinical manifestations, laboratory indicators and radiological features to predict the prognosis of children with pneumonia-related bacteremia. To our knowledge, several critical scores for evaluating the conditions of critically ill children are widely used, such as the Pediatric Risk of Mortality III (PRISM III),11 Brighton Pediatric Early Warning Score (Brighton PEWS)12 and Pediatric Critical Illness Score (PCIS).13 However, these critical scores were not exclusively designed for children with pneumonia-related bacteremia, and there was no study evaluating their value in early recognition of poor outcomes in children with pneumonia-related bacteremia.

Thus, this study aims to develop and validate a predictive model for in-hospital mortality of children with pneumonia-related bacteremia at early stage. We will also evaluate the value of PRISM III, PEWS and PCIS in predicting mortality in such patients and compare the predictive model with these critical scores.

## Methods

This study is conducted in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology statement.14 

### Data source

This retrospective study was conducted in a tertiary teaching hospital in Chongqing, China. Hospitalized children with pneumonia-related bacteremia from January 2013 to May 2019 were enrolled retrospectively. The requirement for obtaining informed consent was waived due to the retrospective design of the study and all patients’ information was handled anonymously.

### Study population

The following were the eligibility criteria: (1) hospitalized children had pneumonia; (2) age between 1 month and 18 years old; (3) bacteremic isolates likely represented pathogens according to the judgment of clinicians; and (4) they were diagnosed as pneumonia-related bacteremia with systemic inflammation reaction syndrome status (see the Definitions section). The exclusion criteria were any of the following: (1) the outcomes were caused by other diseases but not directly by pneumonia-related bacteremia, such as acute hemorrhage, surgical factors, etc; (2) the guardian refused to continue the treatment; or (3) the needed information was not available.

Children who died in the hospital at this hospitalization were included in the death group, while those who did not die in the hospital at this hospitalization were included in the control group. A study on the same group of patients has been reported previously.9 

### Data extraction

Data were extracted from the electronic medical record system of the Children’s Hospital of Chongqing Medical University, Chongqing, China, by two data collectors who were totally unaware of the design of the study. The clinical symptoms, signs, laboratory values and radiological data were collected once positivity of blood cultures was reported.

Detailed information on data extraction has been described in our previous study9; additionally, the scores on PRISM III, PEWS and PCIS were included in this study.

### Definitions

The definitions of pneumonia-related bacteremia, TTFP, congenital heart disease and others are shown in online supplemental file.

### Supplementary data

[[jim-2020-001688supp001.pdf]](pending:yes) 

### Statistical analysis

Continuous data were compared by Mann-Whitney U test and were presented as median with IQR for variables that were almost not normally distributed. Categorical data were analyzed by χ2 test or Fisher’s exact test and expressed as numbers (n) and percentages (%). In this study, the included children were divided into the training set (65% data) and the validating set (35% data) by random sampling using R software with the ‘sample()’ function. In the training set, the least absolute shrinkage and selection operator (LASSO) regression method was used to determine the potential predictors of in-hospital mortality in children with pneumonia-related bacteremia.15 The picked predictors were used to build the multivariable logistic regression model and presented with a nomogram to predict in-hospital mortality. The accuracy of the nomogram was assessed by discrimination ability and calibration plot. The area under the curve (AUC) of receiver operating characteristic (ROC) was used to evaluate the discrimination ability of the nomogram. Calibration plot was established to validate the model accompanied with the Hosmer-Lemeshow test. Decision curve analysis (DCA) and clinical impact curve (CIC) were performed to evaluate the clinical utility of the nomogram by calculating the net benefits at different threshold probabilities.16 17 In order to compare the predictable ability and clinical utility value of the nomogram with the three critical scores (PRISM III, PEWS and PCIS), the AUC and DCA curve of the nomogram and that of the three critical scores were compared. A p value less than 0.05 was considered statistically significant and all tests were two-tailed. All statistical analyses were performed using R V.3.6.1 software and SPSS Statistics V.25.0.

## Results

### Clinical characteristics of included children

The flow chart of including patients and the clinical characteristics of the included children have been described in a previous study.9 The significant differences of PRISM III and PCIS scores between the death group and the control group indicated the overall conditions were worse in the death group. However, PEWS was not significantly different between the two groups. Detailed information on PRISM III, PCIS and PEWS for all included children is presented in online supplemental table S1. We divided these patients into two sets by random sampling in R software with the ‘sample()’ function. One hundred and fifty-seven children were finally divided into the training set and 85 children were divided into the validating set. Briefly, there was no significant difference in age, gender, weight and underlying diseases between the death group and the control group in the two sets. In-hospital mortality was similar in the subsets (20.4% in the training set and 21.2% in the validating set). Detailed information on the subsets is presented in online supplemental table S2.

### Developing a nomogram in the training set

The LASSO regression method was used to extract the most important predictive factors from the training set. Thirty variables of patients were analyzed in the LASSO regression model, the most regularized and parsimonious model with a cross-validated error within 1 SE of the minimum included three variables (TTFP, albumin (ALB) and lactate dehydrogenase (LDH)). A cross-validated error plot of the LASSO regression model is shown in figure 1A. The path of the coefficients included in this model with varying log-transformed lambda values is shown in figure 1B.

![Figure 1](/https://d3hme472k3gd2d.cloudfront.net/content/jim/69/7/1339/F1.medium.gif)

[Figure 1](/content/69/7/1339/F1)

Figure 1 
Selection of predictors of in-hospital mortality using the LASSO regression method. (A) A 10-fold cross-validation was used in the LASSO regression; (B) LASSO coefficient profiles of 30 potential variables. A coefficient profile plot was produced against the log (lambda) sequence. LASSO, least absolute shrinkage and selection operator.

A new model incorporating these three predictors was established by multiple variables logistic regression, and the nomogram of this logistic regression model for visualization was drawn by the ‘regplot’ package in R software (figure 2). The value of every predictor corresponded to a specific point, and the total points of all predictors corresponded to the risk of in-hospital mortality. The values of ALB, LDH and TTFP are marked on the axis with corresponding points. The total points of this model are obtained by adding the points of individual factors together, corresponding to ‘probability’, which indicates the risk of in-hospital mortality. The AUC of the probability of in-hospital mortality in the nomogram was 0.84 (95% CI 0.77 to 0.91) in the training set, which is higher than the three individual predictors (TTFP: 0.78, 95% CI 0.68 to 0.87; ALB: 0.71, 95% CI 0.61 to 0.81; LDH: 0.74, 95% CI 0.64 to 0.84) (figure 3).

![Figure 2](/https://d3hme472k3gd2d.cloudfront.net/content/jim/69/7/1339/F2.medium.gif)

[Figure 2](/content/69/7/1339/F2)

Figure 2 
The nomogram established for predicting in-hospital mortality in the training set. The values of ALB, LDH and TTFP are marked on the axis with corresponding points. The total points of this model are obtained by adding the points of individual factors together, corresponding to ‘probability’, which indicates the risk of in-hospital mortality. ALB, albumin; LDH, lactate dehydrogenase; TTFP, time to first positivity of blood culture. glm, generalized linear regression.

![Figure 3](/https://d3hme472k3gd2d.cloudfront.net/content/jim/69/7/1339/F3.medium.gif)

[Figure 3](/content/69/7/1339/F3)

Figure 3 
Area under the curve (AUC) of the probability of in-hospital mortality for the nomogram, time to first positivity of blood culture, albumin and lactate dehydrogenase in the training set.

### Validation of the nomogram

Validation of the predictive nomogram was performed with a 1000 bootstrap analysis. The Harrell’s concordance index in the validating set was 0.82. In the validating set, the AUC of the probability of in-hospital mortality was 0.82 (95% CI 0.71 to 0.93). The calibration curves of the predictive nomogram showed good probability consistencies between the prediction and the observation in the training set (online supplemental figure S1A) and in the validating set (online supplemental figure S1B). Hosmer-Lemeshow goodness-of-fit tests indicated no significant deviation between observed and predicted events in the validating set (p=0.91).

The clinical value of the predictive nomogram was conducted by DCA and CIC analyses. DCA is a novel method for evaluating alternative predictive strategies, which has advantages over the ROC.16 17 The DCA curve showed obvious net benefits of the predictive nomogram. The CIC of this predictive model visually showed the estimated number that would be declared as high risk for each risk threshold and the proportion of those who were cases (true positives). The DCA curve and CIC for the predictive model in the validating set are presented in online supplemental figure S2.

### Comparison with three critical scores

For all included children, PRISM III, PEWS and PCIS had an AUC of probability of in-hospital mortality of 0.66 (95% CI 0.58 to 0.75), 0.56 (95% CI 0.46 to 0.66) and 0.62 (95% CI 0.53 to 0.72), respectively (online supplemental figure S3). The results indicated that the predictable abilities of the three critical scores were not good enough.

In order to determine the ability in predicting in-hospital mortality and the clinical utility of the nomogram, the AUC and DCA curve of the nomogram were compared with that of the three critical scores in the validating set. Consequently, PRISM III, PEWS and PCIS had an AUC of probability of in-hospital mortality of 0.64 (95% CI 0.51 to 0.78), 0.55 (95% CI 0.37 to 0.74) and 0.62 (95% CI 0.46 to 0.79), respectively (figure 4A). The AUC of probability of in-hospital mortality of the nomogram was significantly bigger than PEWS, PCIS and PRISM III (all p<0.05). The DCA curves also showed the nomogram had more obvious net benefits than the PRISM III, PEWS and PCIS (figure 4B).

![Figure 4](/https://d3hme472k3gd2d.cloudfront.net/content/jim/69/7/1339/F4.medium.gif)

[Figure 4](/content/69/7/1339/F4)

Figure 4 
Comparison of the nomogram and the three critical scores in the validating set: (A) receiver operating characteristic and (B) decision curve analysis curves. AUC, area under the curve; PCIS, Pediatric Critical Illness Score; PEWS, Brighton Pediatric Early Warning Score; PRISM III, Pediatric Risk of Mortality III.

## Discussion

In this study, we established and validated a simple-to-use nomogram as a new approach to predicting in-hospital mortality in children with pneumonia-related bacteremia at the early stage. The predictive nomogram incorporates three predictors including TTFP, ALB and LDH. It shows good accuracy and discrimination, which indicates that the nomogram may have good utility in clinical practice. In comparison with PRISM III, PEWS and PCIS, the nomogram demonstrated superior prognostic accuracy for in-hospital mortality in children with pneumonia-related bacteremia at the early stage.

Bacteremia denotes a serious infection which can lead to significant morbidity and mortality if not treated appropriately. The laboratory indexes (TTFP, ALB and LDH) were closely related to in-hospital mortality in children with pneumonia-related bacteremia. CRP and PCT were applied widely in bacterial infection, and these two indexes were higher in the death group compared with the control group in this study. However, CRP and PCT were not included in the predictors by the LASSO regression model, which may be attributed to TTFP being more sensitive and specific in predicting in-hospital mortality in children with pneumonia-related bacteremia. Previous studies have already shown that TTFP has good accuracy in the prediction of poor prognosis in patients with bacteremia.18 19 

In this study, antibiotics, especially ß-lactam antibiotics, were used in many patients in the outpatient department before admission. However, the results showed that ß-lactam antibiotics could not reduce the mortality of the included children with pneumonia-related bacteremia. It indicates that a lot of severe bacterial infections may resist ß-lactam antibiotic therapy nowadays. Of the patients, 42.1% were using appropriate antibiotic therapy, but no significant difference was obtained between the death and the control group. This suggests that mortality was not reduced even with appropriate antibiotic therapy. It indicates that the vitro drug-sensitive test may not be absolutely consistent with vivo pharmacodynamics. Therefore, early recognition of poor prognosis in children with pneumonia-related bacteremia is vitally important since more active treatment and care are necessary for children at high risk of in-hospital mortality.

The findings of this study may have implications for clinical practice. First, shorter TTFP, lower ALB level and higher LDH level may indicate higher in-hospital mortality. Second, we should assess the patient’s status by synthetical analysis since combination of predictors may be better than the individual variables. Third, the commonly used critical scores may not perform well in predicting in-hospital mortality for children with pneumonia-related bacteremia; therefore, a newly established predictive model designed for these patients is necessary.

This study had a number of strengths. This is the first study to establish a nomogram-illustrated model for predicting in-hospital mortality in children with pneumonia-related bacteremia at the early stage. The LASSO regression method is suitable for regression of highly dimensional data and avoids overfitting. Furthermore, the predictors (TTFP, ALB and LDH) are easy to obtain and routinely collected in clinical practice. In comparison with PRISM III, PEWS and PCIS, the nomogram demonstrated superior prognostic accuracy for in-hospital mortality in the early period. We validated this nomogram not only in accuracy and discrimination but also in clinical value by DCA and CIC. The results showed that the nomogram performed better in predicting in-hospital mortality than the existing critical scores.

Limitations of this study should be noted. First, this is a single-centered retrospective study with a small sample size; thus, the conclusion should be drawn with caution and the nomogram should be further validated in different centers in the future. Moreover, our study mainly focused on the clinical, laboratory and radiological features of patients, but we could not obtain the very detailed information about the antibiotic therapy before admission, which might influence the result of TTFP.

In conclusion, this retrospective study establishes and validates a nomogram-illustrated model which incorporates TTFP, ALB and LDH to predict in-hospital mortality in children with pneumonia-related bacteremia at the early stage. It demonstrates superior prognostic accuracy compared with PRISM III, PEWS and PCIS. However, the nomogram should be further validated in the future due to the small sample size.

## Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

## Ethics statements

### Patient consent for publication

Not required.

### Ethics approval

The study design was approved by the ethics committee of the Children’s Hospital of Chongqing Medical University and conducted according to the Declaration of Helsinki guidelines (no. 2019221).

## Acknowledgments

We thank Professor Shu Yang from Chengdu University of Traditional Chinese Medicine, Chengdu, China, for the comprehensive statistical review of this study.

## Footnotes

*   Contributors Conceptualization: JL, JD and YZ. Methodology: JL, JD and YZ. Validation: YZ and JL. Formal analysis: JL and AS. Investigation: LY and NY. Data curation: LY and NY. Writing–original draft preparation: JL and YZ. Writing–review and editing: JD.

*   Funding The study was funded by the Youth Program of the National Natural Science Foundation of China (no. 81700017).

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

## References

1.   Jain S , Williams DJ , Arnold SR , et al . Community-Acquired pneumonia requiring hospitalization among U.S. children. N Engl J Med 2015;372:835–45.[doi:10.1056/NEJMoa1405870](http://dx.doi.org/10.1056/NEJMoa1405870) pmid:http://www.ncbi.nlm.nih.gov/pubmed/25714161 
    
    [CrossRef](/lookup/external-ref?access_num=10.1056/NEJMoa1405870&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=25714161&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

2.   Liu L , Oza S , Hogan D , et al . Global, regional, and national causes of under-5 mortality in 2000-15: an updated systematic analysis with implications for the sustainable development goals. Lancet 2016;388:3027–35.[doi:10.1016/S0140-6736(16)31593-8](http://dx.doi.org/10.1016/S0140-6736(16)31593-8) pmid:http://www.ncbi.nlm.nih.gov/pubmed/27839855 
    
    [CrossRef](/lookup/external-ref?access_num=10.1016/S0140-6736(16)31593-8&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=27839855&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

3.  Pneumonia Etiology Research for Child Health (PERCH) Study Group. Causes of severe pneumonia requiring hospital admission in children without HIV infection from Africa and Asia: the PERCH multi-country case-control study. Lancet 2019;394:757–79.[doi:10.1016/S0140-6736(19)30721-4](http://dx.doi.org/10.1016/S0140-6736(19)30721-4) pmid:http://www.ncbi.nlm.nih.gov/pubmed/31257127 
    
    [CrossRef](/lookup/external-ref?access_num=10.1016/S0140-6736(19)30721-4&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

4.   McMullan BJ , Bowen A , Blyth CC , et al . Epidemiology and Mortality of Staphylococcus aureus Bacteremia in Australian and New Zealand Children. JAMA Pediatr 2016;170:979–86.[doi:10.1001/jamapediatrics.2016.1477](http://dx.doi.org/10.1001/jamapediatrics.2016.1477) pmid:http://www.ncbi.nlm.nih.gov/pubmed/27533601 
    
    [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

5.   Jaramillo-Bustamante JC , Marín-Agudelo A , Fernández-Laverde M , et al . Epidemiology of sepsis in pediatric intensive care units: first Colombian multicenter study. Pediatr Crit Care Med 2012;13:501–8.[doi:10.1097/PCC.0b013e31823c980f](http://dx.doi.org/10.1097/PCC.0b013e31823c980f) pmid:http://www.ncbi.nlm.nih.gov/pubmed/22460772 
    
    [CrossRef](/lookup/external-ref?access_num=10.1097/PCC.0b013e31823c980f&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=22460772&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

6.   Amaro R , Liapikou A , Cilloniz C , et al . Predictive and prognostic factors in patients with blood-culture-positive community-acquired pneumococcal pneumonia. Eur Respir J 2016;48:797–807.[doi:10.1183/13993003.00039-2016](http://dx.doi.org/10.1183/13993003.00039-2016) pmid:http://www.ncbi.nlm.nih.gov/pubmed/27174880 
    
    [Abstract/FREE Full Text](/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiZXJqIjtzOjU6InJlc2lkIjtzOjg6IjQ4LzMvNzk3IjtzOjQ6ImF0b20iO3M6MTk6Ii9qaW0vNjkvNy8xMzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

7.   Albrich WC , Harbarth S . Pros and cons of using biomarkers versus clinical decisions in start and stop decisions for antibiotics in the critical care setting. Intensive Care Med 2015;41:1739–51.[doi:10.1007/s00134-015-3978-8](http://dx.doi.org/10.1007/s00134-015-3978-8) pmid:http://www.ncbi.nlm.nih.gov/pubmed/26194026 
    
    [CrossRef](/lookup/external-ref?access_num=10.1007/s00134-015-3978-8&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=26194026&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

8.   Hillas G , Vassilakopoulos T , Plantza P , et al . C-Reactive protein and procalcitonin as predictors of survival and septic shock in ventilator-associated pneumonia. Eur Respir J 2010;35:805–11.[doi:10.1183/09031936.00051309](http://dx.doi.org/10.1183/09031936.00051309) pmid:http://www.ncbi.nlm.nih.gov/pubmed/19717486 
    
    [Abstract/FREE Full Text](/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiZXJqIjtzOjU6InJlc2lkIjtzOjg6IjM1LzQvODA1IjtzOjQ6ImF0b20iO3M6MTk6Ii9qaW0vNjkvNy8xMzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

9.   Zhang Y , Lin J , Shi Q , et al . Diagnostic accuracy of time to first positivity of blood cultures for predicting severe clinical outcomes in children with pneumonia-related bacteremia. J Investig Med 2020;68:1241–9.[doi:10.1136/jim-2020-001473](http://dx.doi.org/10.1136/jim-2020-001473) pmid:http://www.ncbi.nlm.nih.gov/pubmed/32848047 
    
    [Abstract/FREE Full Text](/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamltIjtzOjU6InJlc2lkIjtzOjk6IjY4LzcvMTI0MSI7czo0OiJhdG9tIjtzOjE5OiIvamltLzY5LzcvMTMzOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10.  Parlato M , Philippart F , Rouquette A , et al . Circulating biomarkers may be unable to detect infection at the early phase of sepsis in ICU patients: the CAPTAIN prospective multicenter cohort study. Intensive Care Med 2018;44:1061–70.[doi:10.1007/s00134-018-5228-3](http://dx.doi.org/10.1007/s00134-018-5228-3) pmid:http://www.ncbi.nlm.nih.gov/pubmed/29959455 
    
    [CrossRef](/lookup/external-ref?access_num=10.1007/s00134-018-5228-3&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

11.  Pollack MM , Patel KM , Ruttimann UE . Prism III: an updated pediatric risk of mortality score. Crit Care Med 1996;24:743–52.[doi:10.1097/00003246-199605000-00004](http://dx.doi.org/10.1097/00003246-199605000-00004) pmid:http://www.ncbi.nlm.nih.gov/pubmed/8706448 
    
    [CrossRef](/lookup/external-ref?access_num=10.1097/00003246-199605000-00004&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=8706448&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 
    
    [Web of Science](/lookup/external-ref?access_num=A1996UL40200004&link_type=ISI) 

12.  Agulnik A , Méndez Aceituno A , Mora Robles LN , et al . Validation of a pediatric early warning system for hospitalized pediatric oncology patients in a resource-limited setting. Cancer 2017;123:4903–13.[doi:10.1002/cncr.30951](http://dx.doi.org/10.1002/cncr.30951) pmid:http://www.ncbi.nlm.nih.gov/pubmed/28881451 
    
    [PubMed](/lookup/external-ref?access_num=28881451&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

13.  Ren XX , Song GW . Pediatric risk of mortality III score and pediatric CriticalIllness score. Appl Clin Pediatr 2006;21:382–4.
    
    

14.  von Elm E , Altman DG , Egger M , et al . The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg 2014;12:1495–9.[doi:10.1016/j.ijsu.2014.07.013](http://dx.doi.org/10.1016/j.ijsu.2014.07.013) pmid:http://www.ncbi.nlm.nih.gov/pubmed/25046131 
    
    [CrossRef](/lookup/external-ref?access_num=10.1016/j.ijsu.2014.07.013&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=25046131&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

15.  Tibshirani R . Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society: Series B 1996;58:267–88.[doi:10.1111/j.2517-6161.1996.tb02080.x](http://dx.doi.org/10.1111/j.2517-6161.1996.tb02080.x) 
    
    [Web of Science](/lookup/external-ref?access_num=A1996TU31400017&link_type=ISI) 

16.  Vickers AJ , Elkin EB . Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006;26:565–74.[doi:10.1177/0272989X06295361](http://dx.doi.org/10.1177/0272989X06295361) pmid:http://www.ncbi.nlm.nih.gov/pubmed/17099194 
    
    [CrossRef](/lookup/external-ref?access_num=10.1177/0272989X06295361&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=17099194&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 
    
    [Web of Science](/lookup/external-ref?access_num=000242172200001&link_type=ISI) 

17.  Fitzgerald M , Saville BR , Lewis RJ . Decision curve analysis. JAMA 2015;313:409–10.[doi:10.1001/jama.2015.37](http://dx.doi.org/10.1001/jama.2015.37) pmid:http://www.ncbi.nlm.nih.gov/pubmed/25626037 
    
    [CrossRef](/lookup/external-ref?access_num=10.1001/jama.2015.37&link_type=DOI) 
    
    [PubMed](/lookup/external-ref?access_num=25626037&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

18.  Siméon S , Le Moing V , Tubiana S , et al . Time to blood culture positivity: an independent predictor of infective endocarditis and mortality in patients with Staphylococcus aureus bacteraemia. Clin Microbiol Infect 2019;25:481–8.[doi:10.1016/j.cmi.2018.07.015](http://dx.doi.org/10.1016/j.cmi.2018.07.015) pmid:http://www.ncbi.nlm.nih.gov/pubmed/30036664 
    
    [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom) 

19.  Lin J-J , Weng T-H , Tseng W-P , et al . Utility of a blood culture time to positivity-incorporated scoring model in predicting vascular infections in adults with nontyphoid Salmonella bacteremia. J Microbiol Immunol Infect 2018;51:652–8.[doi:10.1016/j.jmii.2018.01.004](http://dx.doi.org/10.1016/j.jmii.2018.01.004) pmid:http://www.ncbi.nlm.nih.gov/pubmed/29501434 
    
    [PubMed](/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjim%2F69%2F7%2F1339.atom)