Validation of a deep learning, value-based care model to predict mortality and comorbidities from chest radiographs in COVID-19
Ayis Pyrros, Jorge Rodriguez Fernandez, Stephen M. Borstelmann, Adam Flanders, Daniel Wenzke, Eric Hart, Jeanne M. Horowitz, Paul Nikolaidis, Melinda Willis, Andrew Chen, Patrick Cole, Nasir Siddiqui, Momin Muzaffar, Nadir Muzaffar, Jennifer McVean, Martha Menchaca, Aggelos K. Katsaggelos, Sanmi Koyejo, William Galanter
Abstract
We validate a deep learning model predicting comorbidities from frontal chest radiographs (CXRs) in patients with coronavirus disease 2019 (COVID-19) and compare the model’s performance with hierarchical condition category (HCC) and mortality outcomes in COVID-19. The model was trained and tested on 14,121 ambulatory frontal CXRs from 2010 to 2019 at a single institution, modeling select comorbidities using the value-based Medicare Advantage HCC Risk Adjustment Model. Sex, age, HCC codes, and risk adjustment factor (RAF) score were used. The model was validated on frontal CXRs from 413 ambulatory patients with COVID-19 (internal cohort) and on initial frontal CXRs from 487 COVID-19 hospitalized patients (external cohort). The discriminatory ability of the model was assessed using receiver operating characteristic (ROC) curves compared to the HCC data from electronic health records, and predicted age and RAF score were compared using correlation coefficient and absolute mean error.
Introduction
Managed care, also known as value-based care (VBC) in the US, emphasizes improved outcomes and decreased costs by managing chronic comorbidities and grouping (ICD10) diagnosis codes with reimbursements proportioned to disease burden [1]. There is growing concern that these VBC models do not recognize radiology’s central role [1]. Radiologists are often unfamiliar with the complexity of VBC systems and their significance to clinical practice [2]. Furthermore, radiologists frequently receive limited relevant clinical information on the radiology request; instead, clinical information is buried in the non-radiology electronic health record (EHR) data which is at best awkward and time-consuming to retrieve and at worst simply un-obtainable.
Methods
This retrospective study was approved on scientific and ethical research basis by the institutional review boards of both institutions and was granted waivers of Health Insurance Portability and Accountability Act authorization and written informed consent. This research study was conducted retrospectively from data obtained for clinical purposes.
Results
Patient characteristics
A total of 900 patients were included in the study: 413 (46%) from the internal COVID+ test set and 487 (54%) from the external COVID+ validation set (Fig 1, Table 1). The mean age of the internal cohort was 49.9 years (median = 51, range = 16–97, IQR = 39); 221 were White (53%), 31 were Asian (7%), 96 were Hispanic (23%), 27 were African American (6.5%), and 38 were other or unknown (10%). There were 4 deaths.
The mean age of the external validation set was 56.3 years (median = 57, range = 18–95, IQR = 45); 32 were White (6.6%), 225 were African American (46.2%), 59 were Hispanic (12.1%), and 165 were other (33.9%). In the external validation set, there were 59 deaths.
Discussion
In this study we developed a DL model to predict select comorbidities and RAF score from frontal CXRs and then externally validated this model using a VBC framework based on HCC codes from ICD-10 administrative data. Because it was recognized that the internal validation set was skewed towards less ill patients, with 4 deaths in the internal validation set, a corresponding dataset with more severe disease was sought to ascertain validity over a broad spectrum of patient presentations. The DL model demonstrated discriminatory power in both hospitalized external and ambulatory internal cohorts but was improved when operating on the full spectrum of patients as would be seen across a region of practice with multiple different care settings and degrees of patient illness.
Citation: Pyrros A, Rodriguez Fernandez J, Borstelmann SM, Flanders A, Wenzke D, Hart E, et al. (2022) Validation of a deep learning, value-based care model to predict mortality and comorbidities from chest radiographs in COVID-19. PLOS Digit Health 1(8): e0000057. https://doi.org/10.1371/journal.pdig.0000057
Editor: Heather Mattie, Harvard University T H Chan School of Public Health, UNITED STATES
Received: January 21, 2022; Accepted: May 5, 2022; Published: August 1, 2022
Copyright: © 2022 Pyrros et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Source code available: https://zenodo.org/record/6587719#.YpD1zC-cZQI.
Funding: AP, NS and SK were funded by the Medical Imaging Data Resource Center, which is supported by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under contracts 75N92020C00008 and 75N92020C00021. JR-F and WG received funding from the University of Illinois at Chicago Center for Clinical and Translational Science (CCTS) award ULTR002003. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
