External Validation of a Paediatric Smart Triage Model for Use in Resource Limited Facilities
Joyce Kigo, Stephen Kamau, Alishah Mawji, Paul Mwaniki, Dustin Dunsmuir, Yashodani Pillay, Cherri Zhang, Katija Pallot, Morris Ogero, David Kimutai, Mary Ouma, Ismael Mohamed, Mary Chege, Lydia Thuranira, Niranjan Kissoon, J. Mark Ansermino, Samuel Akech
Abstract
Models for digital triage of sick children at emergency departments of hospitals in resource poor settings have been developed. However, prior to their adoption, external validation should be performed to ensure their generalizability. We externally validated a previously published nine-predictor paediatric triage model (Smart Triage) developed in Uganda using data from two hospitals in Kenya. Both discrimination and calibration were assessed, and recalibration was performed by optimizing the intercept for classifying patients into emergency, priority, or non-urgent categories based on low-risk and high-risk thresholds. A total of 2539 patients were eligible at Hospital 1 and 2464 at Hospital 2, and 5003 for both hospitals combined; admission rates were 8.9%, 4.5%, and 6.8%, respectively.
Introduction
The global burden of child mortality remains high in low and middle-income countries (LMICs). Despite significant progress globally, Sub-Saharan Africa continues to record mortality rates of 74 (95% confidence interval (CI), 68–86) deaths per 1000 live births, which is approximately 14 times higher than the mortality rate of children in Europe and North America [1,2]. These numbers, accounting for paediatric deaths outside the neonatal period, are largely attributed to infectious diseases including malaria, pneumonia, and diarrhoea diseases which can be prevented or treated through simple interventions and training of healthcare workers [3].
Methods
Study Population and Design
The Smart Triage model was developed in a study conducted at the pediatric emergency department (ED) in Jinja Regional Referral Hospital (JRRH), a public hospital within the Uganda Ministry of Health, between April 2020 and March 2021 [9]. The Smart Triage model equation is a multiple logistic regression model that includes nine predictor variables which were selected using bootstrap stepwise regression and clinical expertise. The model can be used for rapid identification of critically ill children at triage and can be integrated into any digital platform.
Results
At Hospital 1, 2680 patients were screened for eligibility between 24th February 2021 and 6th November 2022; 2539 patients (94.7%) met the inclusion criteria and were included in analysis. Of those that were analyzed, 226 (8.9%) had a positive primary outcome and 79.8% of all participants were aged 5 years or younger. No participants had 25% of the predictor variables missing but 28 participants required imputation of missing values, 13 participants were missing the admission outcome and were thus excluded from analysis (Fig 1). The most common reason for admission was pneumonia, diagnosed using clinical signs criteria, which accounted for 55.8% of the admissions; 56.2% of all admission were male
Discussion
Performance of the Smart Triage model showed stable discrimination in all the three sets of data which suggests that for a pair of randomly selected children, the model would assign the higher risk score to the one with positive outcome compared to an individual with a negative outcome. On graphically assessing calibration of the predicted against observed outcomes, the graphical plot deteriorated in all the three sets of data shifting towards overprediction. Recalibration-in-the-large improved the calibration plot, but this required a change in risk thresholds to optimize sensitivity and specificity and organize patients into clinically manageable risk classification triage categories.
Conclusion
The Smart Triage model showed good discrimination on external validation but required modest recalibration to improve the graphical fit of the calibration plot. On recalibration, new site-specific set of thresholds were required to maintain the same sensitivity and specificity across the triage categories. There was no significant change in the distribution of patients into the three triage categories, which alleviates concerns about model updating for prediction if prioritization based on rank is all that is required. Future research could examine whether the Smart Triage model can be applied to different populations if only the risk thresholds are adjusted without recalibration. The Smart Triage model shows promise for wider application for use in triage for sick children in different settings.
Citation: Kigo J, Kamau S, Mawji A, Mwaniki P, Dunsmuir D, Pillay Y, et al. (2024) External validation of a paediatric Smart triage model for use in resource limited facilities. PLOS Digit Health 3(6): e0000293. https://doi.org/10.1371/journal.pdig.0000293
Editor: Ryan S. McGinnis, Wake Forest University School of Medicine, UNITED STATES
Received: June 5, 2023; Accepted: April 24, 2024; Published: June 21, 2024
Copyright: © 2024 Kigo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: We are unable to make the data publicly available, owing to the sensitive nature of clinical data. Access to the de-identified data is granted on a case-by-case basis and will require the signing of a data sharing agreement. Therefore the data will be made available upon request through KEMRI-Wellcome Trust Research Program data governance committee as stated in the manuscript. The KEMRI-Wellcome Trust Research Program data governance committee can be contacted at dgc@kemri-wellcome.org.
Funding: This research was supported by the Wellcome Trust (grant code: 215695/B/19/Z) to MA and SA. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
