Automated Screening of COVID-19-Based Tongue Image on Chinese Medicine

OBJECTIVE: Artificial intelligence-powered screening systems of coronavirus disease 2019 (COVID-19) are urgently demanding since the ongoing outbreak of SARS-CoV-2 worldwide. Chest CT or X-ray is not sufficient to support the large-scale screening of COVID-19 because mildly-infected patients do not have imaging features on these images. Therefore, it is imperative to exploit supplementary medical imaging strategies. Traditional Chinese medicine has played an essential role in the fight against COVID-19. METHODS: In this paper, we conduct two kinds of verification experiments based on a newly-collected multi-modality dataset, which consists of three types of modalities: tongue images, chest CT scans, and X-ray images. First, we study a binary classification experiment on tongue images to verify the discriminative ability between COVID-19 and non-COVID-19. Second, we design extensive multimodality experiments to validate whether introducing tongue image can improve the screening accuracy of COVID-19 based on chest CT or X-ray images. RESULTS: Tongue image screening of COVID-19 showed that the accuracy (ACC), sensitivity (SEN), specificity (SPEC), and Matthew correlation coefficient (MCC) of the improved AlexNet and Googlenet both reached 98.39%, 98.97%, 96.67%, and 99.11%. The fusion of chest CT and tongue images used a tandem multimodal classifier fusion strategy to achieve optimal classification, and the results and screening accuracy of COVID-19 reached 98.98%, resulting in a significant improvement of 4.75% the highest accuracy in 375 years compared with the single-modality model. The fusion of chest x-rays and tongue images also had good classification accuracy. CONCLUSIONS: Both experimental results demonstrate that tongue image not only has an excellent discriminative ability for screening COVID-19 but also can improve the screening accuracy based on chest CT or X-rays. To the best of our knowledge, it is the first work that verifies the effectiveness of tongue image on screening COVID-19. This paper provides a new perspective and a novel solution that contributes to large-scale screening toward fast stopping the pandemic of COVID-19.