An optimized deep convolutional neural network for adaptive learning using feature fusion in multimodal data

The outbreak of COVID-19 has caused an unprecedented increase in the usage of e-Learning platforms. The closure of educational institutions globally has significantly impacted the traditional education system. Without physical interaction, engaging students in an e-Learning environment has become a major challenge for teachers and e-Learning platform providers. Unlike a face-to-face classroom, where teachers can easily monitor students’ behavior and adapt the learning content according to their needs, this is impossible in an e-Learning environment. This study introduces an optimized deep learning-based approach for detecting student engagement levels to ensure students remain connected to the learning process. The proposed optimized model uses facial emotion recognition and head movement detection to track real-time engagement levels for big data as real-time-based face and head datasets are analyzed. The system leverages facial landmark detection to monitor head movements and deep learning models such as VGG19 and ResNet50 for facial emotion recognition. The system combines the output of both approaches to accurately predict the student engagement state as either ‘engaged’ or ‘disengaged’ with an accuracy rate of 91.67%.