Accuracy of deep learning-based computed tomography diagnostic system for COVID-19: A consecutive sampling external validation cohort study

TatsuyoshiIkenoue, Yuki Kataoka, Yoshinori Matsuoka, Junichi Matsumoto, JunjiKumasawa, KentaroTochitatni, Hiraku Funakoshi, Tomohiro Hosoda, Aiko Kugimiya, MichinoriShirano, Fumiko Hamabe, Sachiyo Iwata, Shingo Fukuma, Japan COVID-19 AI team

Abstract
Ali-M3, an artificial intelligence program, analyzes chest computed tomography (CT) and detects the likelihood of coronavirus disease (COVID-19) based on scores ranging from 0 to 1. However, Ali-M3 has not been externally validated. Our aim was to evaluate the accuracy of Ali-M3 for detecting COVID-19 and discuss its clinical value. We evaluated the external validity of Ali-M3 using sequential Japanese sampling data. In this retrospective cohort study, COVID-19 infection probabilities for 617 symptomatic patients were determined using Ali-M3. In 11 Japanese tertiary care facilities, these patients underwent reverse transcription-polymerase chain reaction (RT-PCR) testing.

Introduction
A proper triage system is critical during the COVID-19 pandemic [1, 2]. An improper triage system may be disadvantageous to patients and lead to a waste of personal protective equipment (PPE). An increase in hospital infections through the admission of infected patients to healthcare facilities could result in the collapse of the medical system. Although reverse transcription-polymerase chain reaction (RT-PCR) tests have been developed, the delay in receiving RT-PCR results could hamper appropriate triage.

Materials and methods
This retrospective cohort study consisted of 11 Japanese tertiary care facilities that provided treatment for COVID-19 in each region of the country. The institutions from which the medical data were obtained are listed in S1 Table. We collected data from the medical records of each institution between April 15 and May 31, 2020. We partially followed the guidelines of the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis Statement to plan and report this study (S2 Table) [17]. The Institutional Review Board of each facility approved the study. The requirement to obtain written informed consent was waived as it was decided that this was an emergent study with public health implications. The accuracy and reliability of the data were confirmed by PMDA during the approval process of Ali-M3.

Results
In this validation study, we assessed 617 symptomatic patients. The characteristics of the study population for the main datasets are listed in Table 1. Overall, 289 patients (46.8%) were diagnosed with COVID-19 using RT-PCR. Thirteen patients required more than two RT-PCR tests before being diagnosed with COVID-19. The major symptoms were dry cough (37.6%), fever (33.5%), and sore throat (25.8%).

Discussion
In this external validation study, our results indicated that Ali-M3 could be useful for the immediate triage of suspected COVID-19 patients with symptoms at a lower cut-off value. In particular, greater accuracy was observed in patients with greater severity, a few days after symptom onset, and with images with a thinner reconstructed CT slice.

Currently, all patients with symptoms such as fever are triaged as COVID-19 patients. Therefore, medical practitioners must use PPE for each patient [24]. Additionally, bed zoning is essential to avoid contamination of non-infected patients [25]. On the other hand, under-triaging results in hospital infections through the admission of infected patients to health care facilities. This should be continued until a definitive diagnosis is established. Since Ali-M3 is available on the cloud, the physician can receive results immediately.

Conclusion
We conducted a retrospective cohort study for the external validation of Ali-M3 using symptomatic patient data from Japanese tertiary care facilities. Despite limited data analysis, our results indicated that AI-based CT diagnosis could be useful for a diagnosis of the exclusion of COVID-19 in symptomatic patients. This is particularly true in patients requiring oxygen and only a few days after symptom onset. Using Ali-M3 support can reduce PPE consumption and prevent hospital infections through the admission of covertly infected patients. Moreover, Ali-M3 also has the potential to support the diagnosis of RT-PCR in patients with suspected COVID-19. However, as Ali-M3 has some limitations in terms of development, further studies and learning are warranted to update this system.

Citation: Ikenoue T, Kataoka Y, Matsuoka Y, Matsumoto J, Kumasawa J, Tochitatni K, et al. (2021) Accuracy of deep learning-based computed tomography diagnostic system for COVID-19: A consecutive sampling external validation cohort study. PLoS ONE 16(11): e0258760. https://doi.org/10.1371/journal.pone.0258760

Editor: HaoranXie, Lingnan University, HONG KONG

Received: February 4, 2021; Accepted: October 6, 2021; Published: November 4, 2021

Copyright: © 2021 Ikenoue et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Chest CT images and individual clinical information could not be publicized because of restrictions imposed by the IRB and by Japanese domestic law and guidelines, which do not allow us to open our data according to Article 16 in "Act on the Protection of Personal Information".
(http://www.japaneselawtranslation.go.jp/law/detail/?id=2781&vm=04&re=01).

Funding: The authors did not receive financial funding for this study

Competing interests: At the start of the study, Ali-m3 existed as a tool produced by Alibaba Damo (Hangzhou) Technology Co., Ltd for research use that had not received any approval. With the results of this study, we have confirmed that the Ali-M3 has clinical benefits. Therefore, under a special expedited review in Japan, Ali-M3 has been approved by the Japanese Pharmaceuticals and Medical Devices Agency (PMDA) and licensed for use as a diagnostic tool in actual practice. For the license in Japan, Ali-m3 should have been a commercial tool. Therefore, it was not planned that M3 Inc would benefit from the commercialization of the Ali-M3 as a result of our research for which M3 provided the Ali-M3 and storage free of charge. The authors have no patents, products in development, or marketed products associated with this research to declare. This does not alter our adherence to PLOS ONE policies on sharing data and materials. In Japan, the M3.inc completely held the right of Ali-M3. This study was utterly free from Alibaba Damo Technology except Alibaba platform by the right of M3.inc.

Latest Issue
Get instant
access to our latest e-book
Sanofi UpperEdge - Healthcare ERP Transformation Medical Manufacturing Asia 2022 Medical Fair Asia 2022