Jahan C. Penny-Dimri, Christoph Bergmeir, Christopher M. Reid, Jenni Williams-Spence, Andrew D. Cochrane, Julian A. Smith
Abstract
Machine learning (ML) is increasingly applied to predict adverse postoperative outcomes in cardiac surgery. Commonly used ML models fail to translate to clinical practice due to absent model explainability, limited uncertainty quantification, and no flexibility to missing data. We aimed to develop and benchmark a novel ML approach, the uncertainty-aware attention network (UAN), to overcome these common limitations. Two Bayesian uncertainty quantification methods were tested, generalized variational inference (GVI) or a posterior network (PN). The UAN models were compared with an ensemble of XGBoost models and a Bayesian logistic regression model (LR) with imputation. The derivation datasets consisted of 153,932 surgery events from the Australian and New Zealand Society of Cardiac and Thoracic Surgeons (ANZSCTS) Cardiac Surgery Database. An external validation consisted of 7343 surgery events which were extracted from the Medical Information Mart for Intensive Care (MIMIC) III critical care dataset.
Introduction
Machine learning (ML) is increasingly being applied to risk stratification and prediction of postoperative outcomes in cardiac surgery. The extreme physiological demands of cardiac surgery make the development of effective risk stratification tools an important strategy for improving patient care. Currently, the most widely used tools are clinical scores, which are derived from logistic regression (LR) models. Modern approaches, such as ensemble decision tree (EDT) models or deep neural networks, have had limited success improving upon the performance or the interpretability of the standard linear regression methods.
Methods:
The Australian and New Zealand Society of Cardiac and Thoracic Surgeons (ANZSCTS) Cardiac Surgery Database registry recorded 153,932 cardiac surgery events in 151,078 unique patients from June 2001 to December 2019, captured at 32 centers in Australia. As the Database stores sensitive patient information, it is not publicly available. Criteria for inclusion within the database was any patient undergoing cardiac surgery, other thoracic surgery using cardiopulmonary bypass (CPB), or pericardiectomy for constrictive pericarditis, where performed on or off CPB.
Results
All 153,932 cardiac surgical procedures in 151,078 unique patients were included from the ANZSCTS Database, and 7,343 cardiac surgical procedures in 6,748 unique patients were subsetted from the MIMIC III database. 149,988 procedures in 147,317 unique patients were further subsetted from the ANZSCTS Database for the partial dataset. Patient characteristics for the partial dataset and the MIMIC III dataset are presented in Table 1, and for the full dataset in S3 of the supplemental materials.
Discussion:
This is the first description of an uncertainty-aware attention-based neural network. The results presented demonstrate the UAN to have performance as good or better than traditional benchmarks, as well as acceptable uncertainty calibration. The success of ML across many industries has not translated to healthcare, as performance alone does not overcome clinicians’ lack of trust in black-box model predictions. Key to developing clinician trust and use is flexibility to incomplete data, interpretable predictions and uncertainty assessment. We have demonstrated that the UAN provides all of these features.
Conclusions
The UAN is a novel, interpretable, and readily available tool for clinicians to assist in risk stratification in cardiac surgery. The model outperforms current gold standards in important performance benchmarks. Further research needs to be conducted in improving uncertainty calibration and externally validating the model in new cohorts.
Citation: Penny-Dimri JC, Bergmeir C, Reid CM, Williams-Spence J, Cochrane AD, Smith JA (2023) Paying attention to cardiac surgical risk: An interpretable machine learning approach using an uncertainty-aware attentive neural network. PLoS ONE 18(8): e0289930. https://doi.org/10.1371/journal.pone.0289930
Editor: Guangyu Tong, Yale University, UNITED STATES
Received: April 28, 2022; Accepted: July 29, 2023; Published: August 30, 2023
Copyright: © 2023 Penny-Dimri et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The ANZSCTS Cardiac Surgery Database Program is funded by the Department of Health (Victoria), the Clinical Excellence Commission (NSW), Queensland Health (QLD), and funding from individual cardiac surgical units participating in the registry. ANZSCTS Database Research activities are supported through a National Health and Medical Research Council Principal Research Fellowship (APP 1136372) and Program Grant (APP 1092642) awarded to C.M. Reid. This particular study received no additional funding or financial support. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
