Contactless blood oxygen estimation from face videos: A multi-model fusion method based on deep learning

Blood Oxygen ([Formula: see text]), a key indicator of respiratory function, has received increasing attention during the COVID-19 pandemic. Clinical results show that patients with COVID-19 likely have distinct lower [Formula: see text] before the onset of significant symptoms. Aiming at the shortcomings of current methods for monitoring [Formula: see text] by face videos, this paper proposes a novel multi-model fusion method based on deep learning for [Formula: see text] estimation. The method includes the feature extraction network named Residuals and Coordinate Attention (RCA) and the multi-model fusion [Formula: see text] estimation module. The RCA network uses the residual block cascade and coordinate attention mechanism to focus on the correlation between feature channels and the location information of feature space. The multi-model fusion module includes the Color Channel Model (CCM) and the Network-Based Model(NBM). To fully use the color feature information in face videos, an image generator is constructed in the CCM to calculate [Formula: see text] by reconstructing the red and blue channel signals. Besides, to reduce the disturbance of other physiological signals, a novel two-part loss function is designed in the NBM. Given the complementarity of the features and models that CCM and NBM focus on, a Multi-Model Fusion Model(MMFM) is constructed. The experimental results on the PURE and VIPL-HR datasets show that three models meet the clinical requirement(the mean absolute error [Formula: see text] 2%) and demonstrate that the multi-model fusion can fully exploit the [Formula: see text] features of face videos and improve the [Formula: see text] estimation performance. Our research achievements will facilitate applications in remote medicine and home health.