Performance and safety of a fine-tuned small language model for pediatric emergency triage: A benchmark study

Eui Jun Lee, Jae Yun Jung, Do Kyun Kim, Joong Wan Park, Young Ho Kwak

Abstract:

Pediatric emergency triage is a safety-critical task, and recent studies have explored whether artificial intelligence, including language models, can support triage decision-making; however, evidence on fine-tuned open-weight language models remains limited. We conducted a retrospective benchmark study using de-identified triage records from a tertiary pediatric emergency department in Korea collected from January 2020 to April 2025.

Introduction

Triage serves as the initial phase of emergency care and plays a pivotal role in resource allocation, patient safety, mitigation of overcrowding, and timely treatment [1,2]. However, its effective implementation can be particularly challenging in specific clinical settings or patient populations. In pediatrics, communication is often limited, and assessments therefore rely largely on caregivers’ reports and clinicians’ observations [3,4].

Methods

This retrospective study analyzed de-identified triage records collected in the pediatric emergency department of a tertiary hospital in Korea. We screened all visits recorded in this setting between January 2020 and April 2025.

Results:

A total of 77,315 triage records were extracted from the hospital database. After exclusion of 3,145 encounters that did not meet cohort eligibility criteria or contained missing or clearly erroneous key triage fields, the final study cohort comprised 74,170 encounters. Cohort selection and dataset partitioning are summarized in Fig 1.

Discussion

In this study, we fine-tuned an open-weight 8-billion-parameter small language model with safety-oriented optimization for pediatric emergency triage and benchmarked it against a structured-data XGBoost comparator. XGBoost showed higher overall discrimination, whereas the fine-tuned model exhibited a different error profile, with greater within- ± 1-level agreement, fewer extreme errors, and lower strict under-triage, offset by higher over-triage.

Conclusion

In this real-world pediatric emergency department benchmark, the structured-data XGBoost comparator showed higher overall performance than the fine-tuned model. The fine-tuned model nevertheless showed a distinct error profile, with fewer extreme errors and lower strict under-triage in selected high-acuity groups, at the cost of higher over-triage.

Citation: Lee EJ, Jung JY, Kim DK, Park JW, Kwak YH (2026) Performance and safety of a fine-tuned small language model for pediatric emergency triage: A benchmark study. PLoS One 21(6): e0350770. https://doi.org/10.1371/journal.pone.0350770

Editor: Yong-Hong Kuo, University of Hong Kong, HONG KONG

Received: December 29, 2025; Accepted: May 18, 2026; Published: June 4, 2026

Copyright: © 2026 Lee et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly because of ethical and institutional restrictions on sensitive human participant information derived from electronic health records. Access to de-identified data may be granted to qualified researchers who meet the criteria for access to confidential data, upon reasonable request and with formal approval from the Institutional Review Board of Seoul National University Hospital (IRB No. E-2508-068-1665). Requests should be directed to the Institutional Review Board Office of Seoul National University Hospital at irb@snuh.org. The source code utilized for model training and evaluation is publicly available at: https://github.com/eklesia-lee/KTAS_GRPO.
Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.