Variable Importance Analysis with Interpretable Machine Learning for Fair Risk Prediction
Yilin Ning, Siqi Li, Yih Yng Ng, Michael Yih Chong Chia, Han Nee Gan, Ling Tiah, Desmond Renhao Mao, Wei Ming Ng, Benjamin Sieu-Hon Leong, Nausheen Doctor, Marcus Eng Hock Ong, Nan Liu
Abstract
Machine learning (ML) methods are increasingly used to assess variable importance, but such black box models lack stability when limited in sample sizes, and do not formally indicate non-important factors. The Shapley variable importance cloud (ShapleyVIC) addresses these limitations by assessing variable importance from an ensemble of regression models, which enhances robustness while maintaining interpretability, and estimates uncertainty of overall importance to formally test its significance. In a clinical study, ShapleyVIC reasonably identified important variables when the random forest and XGBoost failed to, and generally reproduced the findings from smaller subsamples (n = 2500 and 500) when statistical power of the logistic regression became attenuated.
Introduction
Understanding the impact of relevant factors on an outcome of interest is important in healthcare research, which generates evidence to inform intervention design and resource allocation. For example, the continuing efforts in understanding the impact of patient characteristics, emergency medical service (EMS) system and community interventions on outcomes (e.g., survival and neurological outcomes) after out-of-hospital cardiac arrest (OHCA) have contributed to improvements over the past decades via more responsive EMS systems, public cardiopulmonary resuscitation (CPR) training programs and beyond [1–3]. Existing evidence on variable importance to clinical outcomes is predominantly gathered using epidemiological and biostatistical approaches, e.g., from regression analyses of cohort data.
Results
In this study we analyzed variable importance to survival to discharge (or 30 days if not yet discharged) after OHCA among adult patients who had non-trauma etiology, were resuscitated and attained return of spontaneous circulation (ROSC), using information extracted from a nationwide data in Singapore. The final cohort analyzed in this study included 7490 OHCA patients with complete information on the outcome and 20 variables of interest, of which 1154 (15.4%) patients survived 30 days or to hospital discharge.
Discussion
Statistical modeling of associations between relevant factors and the outcome is an important way to understand underlying mechanisms of health-related outcomes and to uncover patterns for closer investigations in future studies. Compared to traditional regression analyses, ML methods are sometimes considered more powerful for such purposes due to their more complex and therefore more flexible model structures, and their promising predictive performance in some tasks. However, as highlighted in recent works [13,17–20], ML methods are not necessarily superior to traditional methods when working with structured and static clinical data. Complex ML models often require larger sample sizes to train and incur more cognitive burden when interpreting the findings.
Citation: Ning Y, Li S, Ng YY, Chia MYC, Gan HN, Tiah L, et al. (2024) Variable importance analysis with interpretable machine learning for fair risk prediction. PLOS Digit Health 3(7): e0000542. https://doi.org/10.1371/journal.pdig.0000542
Editor: Po-Chih Kuo, National Tsing-Hua University: National Tsing Hua University, TAIWAN
Received: November 9, 2023; Accepted: June 1, 2024; Published: July 12, 2024
Copyright: © 2024 Ning et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data are stored in a secure server environment hosted by Singapore Clinical Research Institute and can be accessed by researchers in the Pan-Asian Resuscitation Outcomes Study (PAROS) Clinical Research Network. For further information, please contact Singapore Clinical Research Institute (contact@scri.cris.sg).
Funding: This research received support from SingHealth Duke-NUS ACP Programme Funding (15/FY2020/P2/06-A79). YN was supported by the Khoo Postdoctoral Fellowship Award (project no. Duke-NUS-KPFA/2021/0051) from the Estate of Tan Sri Khoo Teck Puat. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: MEH Ong reports an advisory relationship with Global Healthcare SG, a commercial entity that manufactures cooling devices. MEH Ong has a licensing agreement and a patent filed (Application no: 13/047,348) with ZOLL Medical Corporation for a study titled "Method of predicting acute cardiopulmonary events and survivability of a patient". All other authors have no conflict of interests to declare.
