3.1 Facial Expression Recognition Algorithm Based on Improved Convolutional Neural Network

Due to the complexity and diversity of facial expressions, facial expression recognition places high demands on the accuracy and robustness. As a classic foundational network, CNN has derived various new recognition algorithms based on its excellent convolutional characteristics. A typical CNN consists of multiple modules, namely convolutional layer, pooling layer, and fully connected layer ^[14]. The convolution operation process is shown in Fig. 1.

Fig. 1. Convolutional operation process.

From Fig. 1, the input data is first partially multi-plied by elements. The multi-plied element data is slid according to their respective positions. By performing convolution operations on the output values of each position, a two-dimensional tensor map is formed sequentially, namely, the feature map ^[15]. The convolutional operation of convolutional layers can extract different feature information, such as texture, color, borders, etc. The pooling layer compresses the feature map spatially, reducing its dimension while preserving feature information. The pooling operation is shown in Fig. 2.

Fig. 2. Pooling operation.

In Fig. 2, by filtering the maximum, minimum, or average values of data information within a specific area, the computational complexity and network parameters are simplified. However, traditional CNN has some limitations in facial expression recognition tasks. Firstly, the different parts and dynamic changes in facial expressions play a crucial role in expression. Traditional CNNs do not have good modeling capabilities when processing these information. Secondly, due to the uneven distribution of training data sets and the similarity between categories, traditional CNNs are easily affected by sample bias in facial expression recognition tasks, resulting in a decrease in recognition accuracy ^[16]. In view of this, SCN is introduced, which evolves from CNN. Through dynamic learning and adaptive adjustment, self damage or errors are repaired while maintaining the normal operation of the network ^[17]. When dynamic learning manifests as the network detecting damage or errors, the SCN uses this information to update its own configuration to fix the problem. Adaptive adjustment is manifested as the ability of SCN to adjust its behavior based on new inputs and feedback, such as reallocating resources, reacquiring routing information, or adjusting parameter configurations. When the network detects damage or errors, it is able to self-repair using dynamic learning and adaptive tuning mechanisms. By re-configuring its own connections and weights, the network can return to its normal state. The network structure diagram of SCN is shown in Fig. 3.

Fig. 3. SCN architecture diagram.

In Fig. 3, the entire SCN structure consists of three main modules: self attention module, regularization module, and noise labeling. Firstly, feature extraction is performed on the image. Secondly, the importance of each part is weighted using a self attention weighting module. Then, the weighted features are sorted using a regularization module. The average weight is used as the threshold for regularization in each sorting process ^[18]. Finally, the noise labeling module is used to label the highly important features for subsequent screening and training. The importance weighting process of the self attention module is shown in equation (1).

In equation (1), $\sigma $ represents the Sigmoid function. $W_{\alpha }^{T} $ represents the $T$-th layer parameter under the attention mechanism. $\alpha _{i} $ represents the importance of the $i$-th sample. $x_{i} $ denotes the first $i$ element of the input sequence. The self-attention weighting module calculates the importance scores of each part of the image, such as pixels, regions. The Sigmoid function maps these scores between 0 and 1, which can assign weights to different parts of the input image. The main processing method of the regularization module is weight ranking, which divides various high weight feature samples and low weight feature samples to ensure the high and low difference between the two types of data ^[19]. The process is shown in equation (2).

In equation (2), $L$ represents the regularization function. $\alpha _{H} $ and $\alpha _{L} $ represent the average weight values of high and low weight restructurings, respectively. $\delta _{1} $ represents a fixed hyper-parameter. Noise labeling is used to determine the threshold of each grouped data in the form of labels. The threshold, in turn, is determined by a combination of performance metrics such as accuracy and recall after model training. If the predicted probability is greater than the threshold, the data is adjusted to a high weight feature sample. The process is shown in equation (3).

In equation (3), $\delta _{2} $ represents a predetermined threshold. $l_{\max } $ and $l_{org} $ represent labels that are greater than the predicted probability and the original labels, respectively. $P_{\max } $ and $P_{gt\ln d} $ represent the maximum prediction probability and the given prediction probability, respectively. Although SCN can effectively prevent sample uncertainty issues, repeatedly using a predetermined threshold reduces noise labels, thereby reducing the robustness of the network. In view of this, CS is introduced to reduce the impact of uncertain samples in model training ^[20]. Compared with other methods, the advantage of CS is that it can improve the performance and robustness of the model in tasks such as facial expression recognition by calculating sample calibration weights and resetting the loss function, as well as by applying calibration weights during the attention weighting process. This strategy can be roughly divided into two directions. The first is to calculate the sample calibration weight and reset the loss function based on the weight. Secondly, the calibration weights are assigned to the attention weighting process. The thresholds are adjusted timely by monitoring the performance of the model on the validation set or test set. This process is shown in equation (4).

In equation (4), $P_{gt\ln d} $ represents the importance weighting of the optimized $i$-th sample, while the rest of the algebra remains the same as before. The regularization process is prone to smaller losses when predicting images with higher weights. However, images with smaller weights result in significant losses when predicted incorrectly ^[21]. Regarding this issue, the CS is used to optimize the regularization process. The CS function is used to replace the regularization function, as shown in equation (5).

In equation (5), $L_{cs} $ represents the CS function. $M$ represents the number of incorrect samples. For the noise marking process, there may also be differences in weight values in groups with lower importance. It is unreasonable to use the established threshold again for judgment. It also takes into account that the CS can dynamically adjust the threshold according to the actual situation. Therefore, it better adapt to different data distributions and model performance, and improve the accuracy and reliability of the noise labeling. Therefore, the threshold for noise labeling is also replaced, as shown in equation (6) ^[22].

In equation (6), $\delta _{2} *cw$ represents the calibrated sample weight threshold. The rest of the algebra remains the same as before. On the basis of optimizing and improving the above modules, the final SCN-CS facial expression recognition model is proposed, as shown in Fig. 4.

In Fig. 4, the overall framework of the recognition model is still based on SCN, with only updates made to each part. Firstly, the student facial images are input and subjected to initial convolution operations using a CNN, which is then decomposed into multiple layers. Secondly, the SCN algorithm is used to perform self attention importance weighting, regularization ranking, and noise labeling operations on these images. After completion, CS performs threshold replacement on the above three steps, recalculates the results, and finally outputs them. The model effectively improves the robustness of a single SCN algorithm. By calibrating the strategy and dynamically adjusting the thresholds, the model is able to better adapt to the changes in the data. It automatically adjusts to cope with different data distributions and noise situations during the training process, which improves the robustness and performance of the model, making it more robust in facial expression recognition facing data fluctuations.

Fig. 4. SCN-CS model structure.

3.2 Construction of An Interactive Model Based on Emotional Compensation and Motivation

After constructing a facial expression recognition model, it is still difficult to provide targeted assistance for virtual online classroom teaching through recognition results ^[23]. Human emotions are rich and varied. According to dimensional analysis, human emotions can be divided into six categories: happiness, sadness, surprise, disgust, anger, and fear. Based on the six basic emotions mentioned above, the emotional space of four components is constructed, as shown in Fig. 5 ^[24].

Fig. 5. Four-direction emotion 3D model.

In Fig. 5, in three-dimensional space, emotions can be divided into 8 parts along three coordinates. Excitement, joy, happiness, and relaxation are positive emotions, while calmness, depression, tension, and anger are negative emotions. Positive emotions have a motivating effect on students' teaching emotions, while negative emotions have a restraining effect. The emotional space expression in this state is shown in equation (7).

In equation (7), $S$ represents the set of emotions. $s_{1} $, $s_{2} $ represent basic emotions. $A$ represents the probability set of a certain emotional state. $A_{1} $, $A_{2} $ represent the probability of emotional states. The probability of one emotion is shown in equation (8).

In equation (8), $A_{i} $ denotes the $i$th emotion. $N$ denotes the number of basic emotional states. Although emotions cannot be directly observed, preliminary judgments can be made by processing expression features through special means, such as Hidden Markov Model (HMM). Compared with other models of the same type, HMM is able to calculate the emotion transfer probability by building emotion sequences, thus effectively capturing the dynamic changes in the sequence data. It is suitable for describing the evolution of emotions over time. Therefore, the study utilizes HMM to construct the relational expressions of the above eight emotions, as shown in Fig. 6.

Fig. 6. HMM of 8 emotional states.

According to Fig. 6 combined with equation (8), the conversion probability between emotions of the same category is higher, while the conversion probability between emotions of different categories is lower. Therefore, to optimize the transmission mode between emotional states, the study utilizes the HMM to convert conditions into probabilistic evaluation problems for calculation ^[25]. The forward-backward algorithm is used to solve probability assessment problems ^[26]. The probability calculation of forward and backward variables is shown in equation (9).

In equation (9), $\lambda $ represents the parameters of the HMM emotion model. $O$ represents the observation sequence 0. $\alpha _{T} $ and $\beta _{T} $ represent variables that move forward and backward at time $T$. After continuous iteration, the parameters in the model tend to be rationalized. The optimal observation sequence judgment is shown in equation (10).

In equation (10), $\varepsilon $ represents the threshold. $\lambda _{0} $ represents the model parameters after multiple iterations. If the probability result satisfies the above equation, the probability of emotional transition is output at this time. Conscious stimulation is used to control emotions and stabilize learning state. The study defines this consciousness as motivation and dilution compensation ^[27]. Incentives act on students' emotional states through motivational factors, thereby achieving a positive guiding effect. Incentive factors can be further divided into four categories, namely reducing learning difficulty, removing difficult problems, enhancing classroom fun, and improving teaching quality ^[28]. For the convenience of subsequent calculations, the empirical value probability is used to express the calculation method for these four incentive factors and transition state probabilities, as shown in equation (11).

In equation (11), $\tau $ represents the empirical constant. $A_{k} (i,j)$ represents the transition state probability of the $i$-th state $j$. $I_{k1} ,I_{k2} ,I_{k3} ,I_{k4} $ represent four types of incentive factors, respectively. In addition, influenced by external stimuli, students emotions may also experience a fading process, as shown in equation (12).

In equation (12), $E$ represents the emotion in the ideal state. $E(t)$ represents the emotional state at moment $t$. $d$ represents the rate of change. $\psi $ represents the emotional dilution factor, which can directly reflect the rate of emotional dilution in students. Based on the above emotional state description, a Emotional Compensation and Encouragement Algorithm (ECEA) is proposed based on a facial expression recognition model ^[29]. This algorithm converts student emotions into the emotion axis through dimension reduction for judgment. The emotion that satisfies the optimal emotion region on the emotion axis is defined as the target emotion region, as shown in equation (13).

In equation (13), $TEA$ represents the target emotional region. $\pi $ represents the emotional state model. $C$ represents the dimension reduction coefficient of emotions. The condition for adopting emotional motivation strategies is shown in equation (14).

In equation (14), $P_{TEA} $ represents the optimal emotional region. $O_{TEA} $ represents the current emotional region. $\varphi $ represents a constant on the emotional axis, and $\varphi \in (0$, $1$, $2$, $\cdots $, $n)$. The interaction model will only actively adopt emotional motivation strategies when the current emotional state of the student is less than the optimal emotional state. In summary, the final Emotional Compensation and Encouragement Model (ECEM) is proposed. The interaction process of the model is shown in Fig. 7 ^[30].

Fig. 7. ECEM model interaction flow.

In Fig. 7, the entire process is roughly divided into four parts. Firstly, the image is recognized and classified using a facial expression recognition model. The recognition results are input into the interaction model, with initialization parameters set. The formula determines whether the current emotion is in the target emotion region. If it is satisfied, there is no need for a motivational factor, that is, a motivational factor of 0, and outputs the current emotion. If the target emotion region is not met, the current emotion is initialized with parameters. The probability of the next possible emotion is re estimated through a calculation formula. Then, based on the optimal emotional region on the emotional axis, if it meets the optimal emotional region, the incentive factor is output. If it is not satisfied, it is returned and the parameters are initialized to calculate the optimal emotional region until it is satisfied. In summary, this model can achieve emotional monitoring and interaction among students during remote virtual teaching, improving teaching quality.

4.1 Recognition Model Performance Testing

Three classic expression recognition databases are introduced, namely the Cohn Kanade (CK+) data set, Multi-media Multi-modal Interaction (MMI) data set, and Facial Expression Images (FEI) data set. CK+ is a widely used facial expression database that includes a series of static facial images and video sequences. MMI is a multi-modal facial expression database that combines facial expressions with sound and body movements. FEI is a database containing nearly 200 facial images for research on expression recognition and facial analysis. In addition, the specific equipment and parameters for the experiment are shown in Table 1.

After setting all testing parameters, the SCN-CS model is first subjected to ablation testing to explore the impact of each module on the overall algorithm performance, such as CNN, SCN, CS, and SCN-CS. The expression recognition accuracy of the four modules on the CK+, MMI, and FEI data sets with varying sample sizes are shown in Fig. 8.

Fig. 8. Test results of each module on different data sets.

Fig. 8 (a) shows the recognition rate test results of different modules on the CK+ data set. Fig. 8 (b) shows the recognition rate test results on the MMI data set. Fig. 8 (c) shows the results on the FEI data set. In Fig. 8, the recognition results of the SCN module were gradually increasing. With the parameter threshold optimization strategy of the CS module, the performance of the final SCN-CS module was much higher than the other three categories. The highest recognition accuracy was 95.8% in the MMI data set, which far exceeded the CNN module by nearly 34.6%. In addition, to verify the optimization degree of the CS for the SCN model and explore the superiority and robustness of the SCN-CS model, taking the MMI data set with better data performance as an example, several similar and smoother facial expression recognition models are introduced, namely Visual Gemmetry Group-Face (VGG-Face), DeepFace, and OpenFace. The iteration speed, label noise, and random occlusion are used as variables, and the misidentification rate is used as a reference indicator for testing. The test results are shown in Fig. 9.

Fig. 9. Performance test results of different facial expression recognition models.

Fig. 9 (a) shows the iteration rate test results of different recognition models. Fig. 9 (b) shows the error rate of different recognition models as label noise changes. Fig. 9 (c) shows the error rate of different recognition models as the occlusion changes. In Fig. 9, the mis-identification rate of the four models gradually decreased with the increase of iteration times. The error rate of the four models significantly increased with the increase of label noise and occlusion ratio. The SCN-CS model had the smallest increase in error rate slope. The performance was best when the label noise ratio was 70%, with an error rate of 33%. The model had the best performance when the occlusion ratio was 67%, with the lowest mis-identification rate of 34%. In summary, after comparing multiple models horizontally, the proposed model had the best overall performance, with better computational speed and lower recognition error rate. To compare the performance of the proposed model more vividly, seven different facial expressions are studied, namely anger, disgust, fear, happiness, surprise, sadness, and contempt. The chaotic matrices of DeepFace, OpenFace, and SCN-CS in the MMI data set are plotted. The test results are shown in Fig. 10.

Fig. 10 (a) shows the confusion test results of the DeeFace model on the MMI data set. Fig. 10 (b) shows the confusion test results of the OpenFace model on the MMI data set. Fig. 10 (c) shows the confusion test results of the SCN model on the MMI data set. In Fig. 10, among the three recognition models, the OpenFace model had the lowest recognition accuracy, with 5 recognition results above 80 points. Next was the DeeFace model, which had 6 recognition results with scores above 80. The proposed SCN-CS model still performed the best, which accurately recognized corresponding facial expressions. The recognition rate was generally above 90 points. In summary, the test results once again validated the efficiency and feasibility of the proposed model.

Fig. 10. Test results of confusion matrix for three facial expression recognition models.

4.2 Interaction Model Performance Testing

The IMF-DB facial emotion data set, RAF-DB data set, and AffectNet data set are the test data. These three data sets all contain nearly 1000 facial expression images, which are roughly divided into 7 categories: anger, disgust, fear, happiness, sadness, surprise, and neutrality. The study begins with ablation testing of the ECEM model to investigate the effect of each module on the overall performance of the model, including HMM, HMM-FBA, HMM-Motivation Factor and HMM-FBA-Motivation Factor, i.e. ECEM algorithm. Taking the emotion recognition accuracy and the number of emotion recognition as the test indexes, the test results are shown in Table 2.

From Table 2, the ablation test of the ECEM module revealed that the individual HMM had a maximum emotion recognition accuracy of 81.5% and 287 emotion recognition numbers on the three types of data sets. The HMM accuracy optimized by FBA was effectively improved by a maximum of about 4%. After introducing incentive factors, the model achieved a maximum improvement of 7% in recognition accuracy compared with the HMM-FBA. ECEM had the highest emotion recognition accuracy of 93.4%, and the number of emotion recognition was 692, which were effectively improved compared with the other three sub-modules. Therefore, the sub-modules in the ECEM model had a facilitating effect on the overall model performance, which was significant. Four emotional motivational factors for emotional dilution in the ECEM model are tested to explore the patterns of human emotional changes under the effects of positive single factor, positive double factor, negative single factor, and negative multi-factor. The emotional entropy is used as the testing indicator. This higher indicator value indicates that the individual state of the test object is chaotic. The test results are shown in Fig. 11.

Fig. 11 (a) shows the test results of incentive factors on the IMF-DB data set. Fig. 11 (b) shows the test results of the RAF-DB data set. Fig. 11 (c) shows the test results of the AffectNet data set. From Fig. 11, affected by both positive and negative factors, the emotional entropy of students gradually decreased and tended to stabilize. Secondly, the test data of multiple factors was significantly better than that of single factors. The fastest compensation time for positive multiple factors was 14 minutes. The fastest compensation time for negative multiple factors was 12 minutes. Compared with the single factor compensation, it was shortened by 4 minutes and 8 minutes, respectively. In addition, popular interaction models are introduced for comparison, such as the K-Nearest Neighbor (KNN) model, Naive Bayes (NB) model, and Decision Tree (DT) model. Taking emotional intensity as an indicator, 10 emotions, including warmth and joy are tested. The test results are shown in Table 3.

Fig. 11. Test results of different incentive factors on three data sets.

From Table 3, among the 14 virtual teaching emotion tests given, the lowest average value of emotion intensity was the NB, followed by the KNN and DT models. The average emotion intensity of the ECEM model was 0.074, which increased by 0.025 compared with the NB model. This data reflected the fact that the ECEM model could more accurately process and respond to diversified emotions, such as warmth, cheerfulness, etc. It helps the virtual teaching system to adjust teaching strategies in real time to match students' emotional states, and enhance teaching interactivity and personalized experience. Compared with NB, KNN and DT models, the high emotional intensity of ECEM meant that it was more effective in processing complex emotions, creating a more empathetic learning environment for students. It was crucial for improving the engagement and learning outcomes of virtual teaching. To compare the effectiveness of various interaction models more deeply, four models are tested using three analysis indicators, namely cognitive interaction, teaching interaction, and emotional interaction. Cognitive interaction is divided into low-level and high-level. The teaching interaction is divided into promoting dialogue and promoting reflection. The emotional interaction has negative and positive. The precision, recall, and F1 values of each model in different indicators are shown in Table 4.

In Table 4, the ECEM model had a maximum P value of 0.092, which was the negative interaction in affective interactions. The maximum R value was 0.094, which was the facilitated reflection in instructional interactions. The maximum F1 value was 0.091, which was the low-level interaction in cognitive interactions. The above data indicated that the model performed well in recognizing and processing cognitive activities. In contrast, the other three models performed more evenly, with no prominent facets. There were significant data differences with the ECEM model. The reason for this is that ECEM has a complex algorithmic structure that better captures and understands multi-modal information in interaction data. At the same time, the model effectively integrates different aspects of cognitive, pedagogical, and affective interactions, achieving excellent performance on all metrics. Finally, the scoring method is used to compare the NB model and the ECEM model with better performance. The test results are shown in Fig. 12.

Fig. 12 (a) shows the scores of various interaction indicators under the NB model. Fig. 12 (b) shows the scores of various interaction indicators under the ECEM model. In Fig. 12, the highest positive evaluation value of the NB model was 82, which meant that the teaching interaction promoted communication and cognitive interaction at a low level. The scores for various indicators in the ECEM model were around 90 points. The highest positive evaluation value was 93 points for the negative factors of emotional interaction. This result was basically consistent with the change pattern of data in Table 3, which demonstrated the absolute advantage of ECEM in cognitive interaction, instructional interaction and emotional interaction. It also confirmed that the strong item of NB model was instructional interaction and the weak item was emotional interaction. From the above results, the public's recognition and favorability towards the ECEM model are high, which further indicates the superiority of the interaction performance of ECEM model.

Fig. 12. Comparison of ratings between two models.