Real-time hand rubbing quality estimation using deep learning enhanced by separation index and feature-based confidence metric

Hand hygiene plays a crucial role in healthcare environments which can cease infections and diseases from spreading. It is also regarded as the second most effective way to control the transmission of COVID-19. The World Health Organization (WHO) recommends a 12-step guideline for alcohol-based hand rubbing. Compliance with this guideline is vital in order to clean the hands thoroughly. Hence, an automated system can help to improve the quality of this procedure. In this study, a large-scale and diverse dataset for both real and fake hand rubbing motions is collected as the first stage of building a reliable hand hygiene system. In the next stage, various pre-trained networks were analyzed and compared using a swift version of the Separation Index (SI) method. The proposed Swift SI method facilitates choosing the best pre-trained network without fine-tuning them on the whole dataset. Accordingly, the Inception-ResNet architecture achieved the highest SI among Inception, ResNet, Xception, and MobileNet networks. Fine-tuning the Inception-ResNet model led to an accuracy of 98% on the test dataset, which is the highest score in the literature. Therefore, from the proposed approach, a lightweight version of this model with fewer layers but almost the same accuracy is produced and examined. In the final stage, a novel metric, called Feature-Based Confidence (FBC), is devised for estimating the confidence of models in prediction. The proposed confidence measure is able to profoundly differentiate models with similar accuracy and determine the superior one. Based on the metrics results, the Inception-ResNet model is about 2x slower but 5% more confident than its lightweight version. Putting all together, by addressing the real-time application concerns, a Deep Learning based method is offered to qualify the hand rubbing process. The model is also employed in a commercial machine, called DeepHARTS, to estimate the quality of the hand rubbing procedure in different organizations and healthcare environments.