2.1 Mask-guided Attention Method

The mask-guided attention method is a simple but effective approach to address interference issues in multiple-person environments. It suppresses irrelevant features through the straightforward multiplication of convolutional feature maps and binary masks of a target object. The masks provide supervision or guidance for predicting pixel-wise spatial attention maps that highlight regions that are likely to be associated with the target object. For example, by using mask-guided attention methods, the performance of pedestrian re-identification tasks was notably improved in ^[16,18]. A re-identification task is challenging in that the model should distinguish each pedestrian’s unique gait patterns regardless of their clothing and background. Box-shaped coarse masks from the Faster R-CNN detector ^[17] were utilized to focus on each detected individual for the person-person occlusion problem effectively ^[16]. Binary semantic segmentation masks from the mask R-CNN ^[19] were utilized to reduce background clutter and prioritize the pedestrian’s body parts ^[18]. Our approach differs from these mask-guided methods ^[16,18], which heavily rely on the quality of the noisy pseudo-mask labels. Instead, we optimize auxiliary scaling weights for refined mask supervisions through bi-level optimization.

2.2 Bi-level Optimization Problem (BOP)

The goal in the bi-level optimization problem (BOP) is to find the optimal hyperparameters when the optimization task involves a hierarchical structure with two levels of optimization. A generative adversarial network (GAN) ^[32] is the most famous and successful example of the BOP and alternatively solves two optimization problems at different levels using discriminator and generator networks. In many studies, BOP has been used to optimize the hyperparameters of deep neural networks. BOP was utilized for class-incremental learning (CIL), in which the number of classes increased phase by phase ^[20]. To address the stability-plasticity dilemma between learning old and new classes, aggregation weights (i.e., hyperparameters) for adaptively balancing them are learned. The aggregation weights are learned alongside model parameters while carefully balancing the stability and plasticity building blocks by an alternative training strategy. This approach demonstrates improved results in varied CIL benchmarks.

BOP was utilized for the gradient regularization method, in which gradients are boosted when the gradients of both the training and validation sets agree in direction and are regulated otherwise ^[21]. The authors introduced scalar weights, which adaptively modulate the magnitude of the gradients during weight updates. These scalar weights were also considered as hyperparameters and were fine-tuned right after updating model parameters to minimize validation errors through a bi-level optimization strategy. They minimized training errors by updating model parameters with the scalar weights fixed and then switched to minimize validation errors. This bi-level approach resulted in consistent improvement in generalization across various image classification benchmarks.

Inspired by previous studies, we introduce scaling weights for re-weighting mispredicted attention maps. In order to train the proposed SMAGNet, we employed a bi-level optimization scheme to optimize scaling weights. The SMAGNet showed refined attention patterns, even when noisy pseudo-mask labels were used for training attention, as shown in Fig. 4. The overall architecture of the proposed SMAGNet is shown in Fig. 2.

Fig. 2. The proposed SMAGNet architecture.

3.1 Mask-guided Attention

Following a previous mask-guided attention operation ^[16,18], we defined generic mask-guided attention as element-wise multiplication of every base feature map channel in $f_{base}$ and attention mask $f_{mask}$ as:

where $i$ is the channel index, and $\otimes $ is element-wise multiplication. $f_{mask}\in \left[0,1\right]$ is defined as $f_{mask}=\sigma \left(W_{mask}*f_{base}\right)$, where $*$ is a convolution operator, $\sigma \left(x\right)=1/\left(1+\exp \left(-x\right)\right)$ is the sigmoid function, and $W_{mask}$ denotes the convolution filters of the mask prediction layer. The mask-guided approach assumes that a predicted attention mask would be helpful in identifying a target object and provide supervision for the base feature by optimizing attention loss $L_{att}$. $L_{att}$ is defined as the per-pixel binary cross entropy loss (BCE):

where $m_{gt}$ is coarse-level ground truth mask labels, which are obtained using the YOLO v8 object detector ^[14].

If a pixel lies within the target object’s detection box, it is annotated as 1. Otherwise, it is annotated as 0. When a pseudo-mask label $m_{gt}$ is used as a target label, it can be quite noisy, leading to invalid attention. To address this issue, we introduce a scaling layer with weights that are learned by backpropagation. This layer is parallel to the mask prediction layer, and its output is multiplied to refine the original mask predictions, as illustrated in Fig. 2.

3.2 Attention Module

In order to reduce attention errors from noisy pseudo-mask labels from the YOLO detector, we designed a simple and effective attention module that adaptively scales the intensity of the mispredicted attention mask in an explicit manner. As shown in Fig. 2, the attention module includes two parallel convolution layers: a mask layer and a scaling layer. These layers predict the attention mask $f_{mask}$ and scaling weights $\phi _{scale}$, respectively. The scaling layer was implemented with small stacks of convolutions with an architecture that is shared with the mask layer. Two $3\times 3\times 3$ convolution layers with rectified linear unit (ReLU) activation are used for feature extraction, and $1\times 1\times 1$ convolution and a sigmoid function are used for a single-channel output in the range of [0,1]. Then, we obtain refined features ${f}_{out}^{s}$ by re-weighting the base feature map using a scaled attention mask $\sqrt{f_{mask}\otimes \phi _{scale}}$:

To build SMAGNet, the ResNet-based 3D CNN R(2+1)D-18 ^[30] was employed as the base model. The proposed attention module was placed after each residual block (or stage) of the base model. By forwarding refined features ${f}_{out}^{s}$ across every residual stage, we predict gait variables in fully connected layer fc as $y_{pred}=W_{fc}{{f}_{out}^{s}}_{last}$, where $W_{fc}$ denotes the fc layer weights, and ${{f}_{out}^{s}}_{last}$ is the refined features in the last residual stage.

We employed the R(2+1)D-18 model in our study due to its inherent capability to effectively capture spatiotemporal characteristics. By factorizing 3D kernels into 2D and 1D kernels, the model is able to separately capture spatial and temporal information. This systematic kernel factorization method allows the R(2+1)D-18 model to accurately measure gait variables by comprehensively detecting both spatial and temporal changes. Also, since this architecture is easy to customize, we employed it architecture as our baseline model.

Our objective was to learn the optimal scaling weights $\phi _{scale}$ by suppressing misguided features that adversely affect the main task loss, resulting in refined attention maps. We used main task loss $L_{main}$ as smooth L1 loss for its robust nature against outlier samples:

where $y_{gt}$ denotes ground truth gait variables, and $\delta $ is a hyperparameter used for determining whether to use L1 loss or L2 loss adaptively, depending on the input. We followed the default PyTorch implementation and used 1.0 for ${\delta}$.

3.3 Attention Mask Refining Process

The proposed attention module has two key layers to optimize: 1) the mask prediction layer supervised with pseudo-mask labels and 2) the scaling layer that corrects mask prediction errors. In this work, we introduce scaling weights as new hyperparameters that adaptively re-weight mispredicted attention maps. To this end, we employed BOP, which alternately solves two different levels of problems, where one task (i.e., the lower-level task) is subject to the other task (i.e., the upper-level task). In our formulation, the lower-level task involves learning attention maps using pseudo-mask labels, whereas the upper-level task involves fine-tuning the scaling weights (i.e., hyperparameters) to achieve optimal mask predictions for improved validation prediction performance, given previously learned attention maps. These two steps are alternately performed to balance the mask and scaling layers until convergence. This is accomplished through two-round refinement steps, where attention masks are progressively refined by alternatively switching to optimize the two groups of parameters $\theta _{1}$ and $\theta _{2}$ of SMAGNet. Each subscript number represents the parameter group index.

In round 1, we jointly optimize our main task loss $L_{main}$ and attention loss $L_{att}$ for the training dataset. We backpropagate the gradients with respect to both $L_{main}$ and $L_{att}$ through layers of SMAGNet while keeping the scaling layers fixed. Thus, in round 1, we learn to approximate attention maps using pseudo-mask labels by updating the model parameters $\theta _{1}=\left[W_{base},W_{mask},~ W_{fc}\right]$ with a learning rate of $\gamma _{1}$. Each element of the list represents the model parameter: the base model ($W_{base}$), mask layer ($W_{mask}$), and fully connected output layer ($W_{fc}$).

In round 2, we subsequently optimize the scaling weights $\phi _{scale}$ on the top of predicted attention masks $f_{mask}$, which are used for the refined feature maps ${f}_{out}^{s}$ defined in Eq. (3). The optimization involves learning the optimal scaling weights to refine misguided attention caused by invalid supervision of pseudo-labels, which could lead to bad main-task performance. The parameters $\theta _{2}=\left[W_{scale}\right]$ are optimized with a learning rate of $\gamma _{2}$, where $W_{scale}$ represents the model parameter of the scaling layer.

In contrast to round 1, we only update the scaling layers while all other model weights of SMAGNet remain fixed and minimize the main task loss for the validation sets. This enables the model to learn optimal scaling weights for enhanced generalization capability in the main task, which is equivalent to hyperparameter optimization. Thus, the proposed bi-level optimization serves as a unified method for the continuous optimization of both the model parameters and hyperparameters for better generalization, which is akin to the procedure in cross validation. The proposed two-round refinement process is repeated alternately until the model converges (i.e., round-1 model parameters $\theta _{1}=\left[W_{base},W_{mask},W_{fc}\right]$ are updated in the n-th epoch, and the round-2 model parameters $\theta _{2}=\left[W_{scale}\right]$ are updated in the (n+1)-th epoch).

4.1 Dataset

4.2 Experimental Settings

R(2+1)D-18 CNN models ^[30] pre-trained on Kinetics-400 were used as our baseline models. The official implementations from the torchvision library were utilized. We implemented the mask layer and scaling layer in parallel, as shown in Fig. 2, and integrated them into each residual stage of the R(2+1)D-18 model. Both the mask layer and scaling layer were constructed using two stacks of convolution layers with the sigmoid function as the output activation to predict binary attention masks and scaling weights, respectively.

For all models, Adam ^[33] was used as an optimizer with the momentum set to 0.9 and weight decay set to 0.001. All models were trained for 150 epochs using a batch size of 32 and a learning rate of 0.0001. To implement the BOP-based attention mask refinement algorithm, which requires two optimizers, we used the same settings for both optimizers. We allocated 80% of the dataset for training and 20% for testing. Since gait analysis can be performed multiple times for each patient, we carefully split the training/testing data to ensure that the same patients were not included in both splits.

4.3 Preprocessing

As a preprocessing step, we obtained detection boxes of all individuals using YOLO v8 ^[14] and tracked each detection box's trajectories using the DeepSort algorithm ^[15]. We utilized DeepSort to minimize track ID switching caused by occlusion or varying viewpoints. Considering that our videos were recorded from a frontal viewpoint, we chose a trajectory that varied the most in the y-direction (vertical) compared to the x-direction (horizontal). The process of the patient detection is shown in the upper part of Fig. 2.

From each video, we uniformly sampled 64 frames. We applied standard data augmentation methods, such as random cropping and horizontal flipping. Moreover, we stacked 64 images with a resolution of 112${\times}$112 pixels to use them as input for our model. We used 64 images because they empirically fit our GPU memory capability (NVIDIA TITAN V; 12 GB). We also employed a frame resolution of 112x112 because the R(2+1)D-18 model has been pretrained using video frames with this resolution for the Kinetics-400 dataset ^[30]. As shown in Table 4, we used nine types of gait variables as our targets with different units and scales. Therefore, we applied standard scaling for data normalization.

4.4 Results

Table 1 presents the overall evaluation results on our GAITRite dataset. We compared the root mean squared error (RMSE) and mean absolute percentage error (MAPE) using the same base model of R(2+1)D-18. To evaluate the overall performance, we computed the average RMSE and MAPE of all gait variables. In the results, our proposed SMAGNet demonstrated significant improvements in both RMSE and MAPE, with values dropping from 4.11 to 2.17 and from 10.0% to 6.63%, respectively. This result is notable since we only utilized pseudo-mask labels obtained from the YOLO v8 object detector to train SMAGNet’s attention capability without any labeling costs. Furthermore, our method requires light modifications to the modeling pipeline, including the insertion of attention modules and the implementation of our proposed bi-level optimization strategy for attention mask refinements.

Next, we conducted an ablation study to demonstrate the effectiveness of each proposed method: the insertion of the attention modules and the BOP-based mask refinement process. The resulting RMSE reduction ratio (${\Delta}$) is the relative change in RMSE before and after applying some method and is reported in Table 2. In the results, simply inserting the attention module had the most significant impact on improving performance, showing a reduction ratio of 44.9% compared to the baseline method. The BOP-based mask refinements enhanced the results, showing a reduction ratio of 22.4% compared to the case of optimizing only the attention module without any refinement. These results suggest that learning attention masks for the target patients, even when using pseudo-mask labels, is critical for accurate gait analysis in multi-person scenarios. In addition, they also indicate that by utilizing our proposed bi-level strategy to refine potentially mispredicted mask predictions that are trained with noisy pseudo-mask labels, we can achieve better results.

As shown in Table 3, we inspected the MAPE values for each gait variable for error analysis. The gait variable named ``Toe In-Out (degree)'' is defined as the angle between the foot and the gait direction. The MAPE value was reduced most drastically by 20% for this variable. Analyzing this variable is challenging as it requires accurately capturing the shape of the foot, which occupies only a small portion of the video frame. Furthermore, for gait variables that require precise detection of the gait cycle, such as ``Swing Percent (%)'', ``Stance Percent (%)'', and ``Double Support Percent (%)'', SMAGNet also demonstrated a notable reduction in MAPE. These results suggest that SMAGNet has the ability to capture gait cycles by paying more attention to the target patient in our multi-person scenarios effectively while preventing interference from non-target individuals walking alongside the target patient.

In order to investigate the linear correlation between predictions and ground truth from the GAITRite, we calculated Pearson correlation coefficients using SciPy’s stats module and compared the proposed methods with the baseline. As shown in Fig. 3, our full methods (denoted as ``w/BOP) showed the best correlations with all ground truth gait variables. However, when incorporating attention modules without BOP-based mask refinements (denoted as ``w/o BOP''), the correlation coefficients were poor for the angle-related gait variable ``Toe In-Out (degree)''. We suspect that the proposed method of learning attention maps supervised with box-shaped masks might not be sufficient to provide details of foot shapes. After applying BOP-based mask refinements (see w/BOP), we observed improved results for all gait variables, including the ``Toe In-Out (degree)''. The proposed method can overcome the limitations of using box-shaped attention masks by producing more complex and detailed attention masks through refinements, leading to better results.

Lastly, as shown in Fig. 4, we conducted a qualitative investigation of the attention capability of our proposed method by visualizing intermediate convolutional features for the baseline method, our approach before BOP-based refinements, and our approach after the refinements. To examine the attention capability for challenging samples, we analyzed samples providing inaccurate mask labels to the model. To find these samples, we applied a simple K-nearest neighbor outlier detection algorithm to the detection box coordinate vectors (i.e., width, height, x_center, y_center).

In our baseline model, R(2+1)D-18 ^[30], we observed less meaningful results with high-level attention values on all moving individuals, regardless of whether they were patients or not. In contrast, after simply incorporating the attention module (denoted as ``before BOP'' in the figure), the attention shifted toward patients’ lower body parts in most cases. Intriguingly, as illustrated in the rightmost column, after applying BOP-based attention mask refinements, we noticed even better attention results that presented more reasonable attention maps, which particularly focused on the actual foot strides of the target patient, which is crucial for precise gait-cycle and foot-shape analysis. These results support the enhanced performance of our proposed SMAGNet compared to before applying BOP-based refinements, particularly in foot-angle-related gait variables and gait-cycle-related variables.

Fig. 3. Correlation coefficient comparison for all gait variables. All results showed p<0.05, which represents statistical significance of correlation coefficients.

Fig. 4. Impact of the proposed method on attention capability in multi-person environments: convolutional features are visualized where the detection of the target patient is not accurate. From left to right are the inaccurate mask labels using detection boxes from YOLO v8, convolutional features by R(2+1)D-18 [30], our method without BOP refinement, and our full method (red color: higher values; blue color: lower values).

4.5 Comparison with State of the Art

Table 5 shows comparison results for a recent state-of-the-art gait recognition model named GaitBase ^[31] using our GAITRite dataset. GaitBase is a recently proposed method that has achieved strong performance in many gait-based person re-identification benchmarks in wild conditions and contains diverse noisy visual factors, such as clothing, carrying, etc. Accordingly, silhouette images with fewer visuals are utilized as inputs. We followed the official codebase provided by the official OpenGait repository (\url{https://github.com/ShiqiYu/OpenGait}) to implement preprocessing, including person tracking and extracting silhouette images of each person. After tracking, we selected patient silhouettes using the same selection criterion as our method.

In this experiment, we used the GaitBase model, which was pretrained on the largest public gait dataset named GREW ^[22]. Robust silhouette embeddings were extracted from the pretrained GaitBase and fed to the fc layer. The same regression loss $L_{main}$ defined in Eq. (4) was adopted to finetune the entire model weights. We employed the same training strategy as our method. RMSE and MAPE(%) are compared in Table 5.

Table 5 shows that the proposed SMAGNet significantly outperforms GaitBase across all evaluation metrics. The lower performance of GaitBase could arise from its heavy reliance on the quality of silhouette inputs. As shown in Fig. 5, preprocessed silhouette images are too simple or noisy to provide enough information for precise gait analysis. As a result, the GaitBase model, which only uses silhouette images as inputs, cannot cope with such irreversible bad inputs and performs worse than the proposed model, even though it was fine-tuned with the large-scale dataset GREW.