2.1 Action Recognition Models

Deep learning-based computer vision technology is being studied to classify the action of objects. Algorithms, such as CNN-based R-CNN ^[4] and YOLO ^[5], make it possible to classify the object types at a fast speed. On the other hand, as for the classification of the action of objects, Two-Stream neural networks, such as the SlowFast network ^[8], have been achieving high performance. Fig. 1 presents the structure of a Two-Stream neural network. The two-stream neural network has a high AUC in object behavior classification because it identifies different types of characteristics through the temporary and spatial streams and utilizes them for behavior classification. Nevertheless, it is slow because Two-Stream neural network-based object action classification models use two neural networks simultaneously.

Biparva, M. et al. ^[9] predicted the lane change of vehicles and compared the performance results through a two-stream network. It provides robust prediction of lane changes for nearby vehicles, demonstrating great accuracy in the temporal domain ranging from 1-2 seconds. The video used is a traffic video that records the front situation of a vehicle. The Two-Stream Convolutional Networks, Two-Stream Inflated 3D Convolutional Networks (I3D) ^[10], Spatiotemporal Multiplier Networks ^[11], and SlowFast Networks ^[8] are used for performance evaluation. Based on the model comparison results, the SlowFast network showed the highest AUC compared to the observation horizon. Regarding the SlowFast network, an excessive overhead occurs, and this causes a GPU memory problem. Therefore, the author finally classified the lane change of vehicles through the Spatiotemporal Multiplier Networks. Based on the performance valuation results. They confirmed that the NLC (No Lane Change) achieved excellent classification performance. On the other hand, the LLC (Left Lane Change) and RLC (Right Lane Change) was limited because they showed a precision of 60\textendash{}70\% and a recall of 60\textendash{}70\%. In addition, the data used for testing requires two seconds of observation, even though the image had its vehicle object section cropped in advance. This serves as a factor that slows the processing to be performed by the existing system used for pre-processing, such as object recognition and cropping. Therefore, this study developed a plan to reduce the required model performance while maintaining the AUC of the SlowFast network.

Fig. 1. Structure of Two-Stream neural network.

2.2 Dynamic Framerate

A deep learning model can improve object detection and classification of AUC. The required computing performance improves because of the characteristics of deep learning. In addition, there has been an increase in cases where various deep learning models are used in conjunction for AUC improvement purposes. Therefore, an efficient data processing technique capable of reducing the required performance of a deep learning model is necessary. Videos shot on the road in diverse environments have differences in their dynamic level. Therefore, unnecessary frames are inputted when the same content is repeated continuously. Therefore, the aim is to improve performance by applying the frame skip method and reducing the data entered into the model. Park. J. W. et al. ^[12] proposed a faster object detection using the frame skip method. Whether to input a frame into YOLO or keep the result of the previous frame is determined by temporal subtraction and ORB feature matching between two consecutive frames. First, the temporal subtraction of two adjacent frames was performed. The temporal subtraction mask did not provide the output if the two adjacent frames were identical. In this case, the frame was not entered into the YOLO model. Instead, the output results of the YOLO model drawn from the previous frame were used. The object detection speed was enhanced using such a process, and the AUC was enhanced. The limitation is that it is unsuitable for applying to road traffic data because whether to skip a frame is determined only when the two adjacent frames are identical. As for road traffic data, it is rare to have completely identical frames due to vibration. In addition, the frame will not be skipped when only the illuminance value is changed, as it is with tunnels. Therefore, instead of simply comparing the illuminance values, it was necessary to develop a method that determines how similar the context of the images per frame. Therefore, this study used a measuring technique that focuses more on the structural differences of images than on the change in illuminance values.

Image similarity measuring techniques include techniques using the differences between the pixels and techniques comparing the structural differences of pixels. NRMSE (Normalized Root Mean Square Error) ^[13] calculates the difference at the same location and then normalizes it. On the other hand, it is difficult to calculate the similarity between images accurately because the technique cannot calculate the structure of image pixels. An image similarity measuring technique performed through histogram makes a comparison through color distribution. Nevertheless, the structural characteristics of images or the differences in pixel positions are not considered. SSIM (Structural Similarity Index Measure) ^[14] is an image similarity technique that considers brightness, contrast, and structure to resolve the problem demonstrated by the NRMSE. Therefore, the SSIM prioritizes applying structure to similarity over applying brightness and contrast to similarity.

Fig. 2 shows the similarity scores derived through various image similarity measurement techniques for images that have undergone color tone and brightness change. SSIM, which is highly similar, is suitable for this study because it does not change the image structure but only changes the color tone or brightness.

Fig. 2. Various types of image similarity measuring techniques.

3.1 Data Collection and Preprocessing

This study collected the dataset through the PREVENTION (PREdiction of VEhicles iNTentIONs) dataset ^[16]. This dataset is manufactured according to the need to predict the driver's intention and the future trajectory of the vehicle. It included the videos recorded with a camera facing the front as the vehicle is driven and included vehicle trajectory information, category information, and line information as labels. The events include cut-in, cutout, left/right lane change, and risk maneuver. Each video had a resolution of 1920 ${\times}$ 920 and an FPS of 10 frames per second. The dataset consisted of 11 videos recorded on five mutually different days, and the videos were 377 minutes long in total. In this study, only the data related to lane change were used. Table 1 lists the number of videos and the average video length per lane change event included in the dataset.

For road-driving images of the original PREDICTION dataset, it is inappropriate to enter it because it is in the SlowFast Network because there is too much ambient information. Accordingly, it is possible to construct an image dataset in which surrounding information is not excessive by detecting only vehicle objects to classify the behavior through the YOLO model and performing a crop ^[17]. Hence, a lane change image dataset was configured to crop only the vehicle objects for each image frame, resize them to 256${\times}$256 size and input them to the SlowFast Network. The lane change image dataset consisted of cropping only objects moving in the lane using the learning data and the validation data of the bounding box information of the object. The data for learning and verification of SlowFast Network was constructed by cropping the image through the lane change point of the label and the bounding box information of the object. In this case, the cropped image was inverted left and right for use as class data in the opposite situation and solve the data imbalance and shortage. In addition, the vehicle was cropped from the image when the lane change did not occur to derive an image of a general situation not during the lane change. Among the original PREDICTION datasets, the Days 1, 2, and 3 images; Day 4 images; Day 5 images were configured as learning data. The validation data was also composed of the remaining images. After extracting the data, the lane change image dataset consisted of 668 images for each label of the left lane change, right lane change, and no lane change, consisting of 2,004 images. One thousand five hundred images were configured as learning data for the performance evaluation, and the remaining 504 images were configured as validation data. Table 2 lists the configuration of a lane change image dataset for learning and verifying the SlowFast Network.

This study detected objects from driving images and cropped them through YOLOv5 models pre-trained via MS COCO (Microsoft Common Objects in Context) dataset ^[18]. The MS COCO Dataset is an image dataset for object detection, segmentation, key-point detection, and captioning. The YOLOv5 model detects objects by receiving images. The model is very fast compared to existing object detection algorithms. The YOLOv5 model pre-trained through the MS COCO dataset derives performance with a mean average precision (mAP) of approximately 45 or higher due to the performance evaluation on the dataset. Through this, it is possible to detect quickly and accurately only the vehicle objects to classify behavior in the original image through the pre-trained YOLOv5 model. In this case, the Region of Interest (RoI) was adjusted to include some surrounding information.

3.2 Similarity Estimation

Regarding the pre-existing SlowFast network, the framerate was fixed. In this study, however, the similarity between frames within each unit frame of the video was analyzed, and the similarity between unit frames was assessed before model input. The following limitation was improved using such as process. The road traffic data has a limitation in that unnecessary frames entered as the same contents are repeated continuously in most cases. When the similarity in both cases exceeds the threshold, the similarity is acknowledged, and the input to the model is skipped. The SSIM (Structured Similarity Image Matching) ^[14] is used for similarity analysis.

The SSIM divides an image into NxN windows and applies Eq. (1) to each window. In Eq. (1), ${\mu}$ represents average and ${\sigma}$ represents dispersion. c is a positive constant that prevents the denominator from being 0. Unlike other image similarity analysis techniques, the SSIM focuses on structural differences. Therefore, it makes it possible for the similarity to focus more on the change in the objects than on the change in external environments. This is suitable for drawing similarities from road driving data containing various illuminance changes resulting from changes in sunlight and tunnel entrances during driving. Fig. 5 shows the similarity between the frames of the first-day video.

In each plot, the x-axis represents the image frame of the comparison point. The y-axis is the similarity score derived by comparing the image of the corresponding frame with the image of the previous frame through a similarity measurement technique. Fig. 5(a) shows the SSIM; Fig. 5(b) shows the NRMSE; Fig. 5(c) shows the similarity scores measured through a histogram. In the case of the NRMSE and histogram, they were so insensitive to image changes that the similarity is constantly high. Therefore, it is appropriate to skip the frame after measuring the similarity through the SSIM.

Fig. 5. Plots that derive the similarity between frames of the first-day video.

3.3 Dynamic Framerate SlowFast Network

Using the Dynamic Framerate SlowFast network proposed in this study, the video of the initial unit frame was entered into the SlowFast network. Starting with the second unit frame, where the possibility of one class exceeds a certain level, the involved class is drawn as an action classification result. The video similarity between the next unit frame video and the previous unit frame video was analyzed when an action classification result was drawn. The involved unit frame was entered into the SlowFast network, and the action of objects was classified when the similarity was below the set threshold. On the other hand, the involved unit frame was skipped and not entered into the SlowFast network when the similarity exceeded the set threshold. At that point, the model output used the action classification results from the previous unit frame. Through such a process, provided that minimum movements existed, the frame was entered into the SlowFast network. Fig. 6 shows a pseudo-code of skipping unit frame process.\begin{enumerate}[1.]

1. Confirming whether the probability of one class exceeds a certain level after action classification

2. Confirming whether the similarity exceeds a certain level after similarity analysis

3. Skipping the frame when 1. and 2. apply

Fig. 6. Process of skipping a unit frame.

An algorithm is entered in units of 10 frames. A UnitFrame variable refers to the unit frame being entered. The SlowFast method receives unit frame inputs, and the CheckSimilarity () classifies the similarity between two unit frames. The similarity threshold is entered into the threshold variable, serving as the hyperparameter. Fig. 7 shows the final output results of the model proposed in this study.

Fig. 7. Final output results of the proposed model.

In the video, information is displayed per object. First, the type of object is displayed and the recognition ranking of the involved object (how quickly the involved object is recognized) is displayed. Second, the distance between the camera and the object is displayed. At that point, a higher number means that the distance is closer. Lastly, the current action taken by the object drawn through the SlowFast network is displayed.

4.1 Similarity Estimation Method

The software in the experimental environment of the proposed model used Ubuntu 18.04, CUDA version 11.2, Python 3.9.7, and the deep learning framework PyTorch. The hardware consisted of Intel(R) Xeon(R) Silver 4210R CPU @ 2.40 GHz, NVIDIA RTX 3090, and 24 GB RAM. In the case of a dataset, the learning data consisted of images on Days 1, 2, 3, and 4 and the images on Day 5. The validation data was also composed of the remaining images. When the image length was converted into a ratio, the ratio of the learning data and the validation data was approximately 5;1. In the case of lane change image data derived through this, 2,004 images were composed. For performance evaluation, 1,500 images were configured as learning data, and the remaining 504 images were configured as validation data. The performance evaluation used the AUC of the proposed model and compared it with the existing model. The AUC is the area under the ROC (Receiver Operating Characteristic) Curve. It represents the width of the bottom area of the ROC Curve ^[19,20]. A value closer to 1 indicates that the sensitivity and specificity are close to 1, indicating that the model performance is superior.

Table 3 shows the performance evaluation results per technique used for similarity measurement. The performance evaluation results are acquired by comparing the AUC and the percentage of skipped frames between the case where the similarity analysis technique is applied and the case where the pre-existing models are applied. Based on the AUC comparison results, it was confirmed that the histogram method showed an AUC of 0.5923 and that NRMSE showed an AUC of 0.5782. SSIM used in this study showed an AUC of 0.7126, which is higher than those of histogram and NRMSE. In addition, compared to the histogram and NRMSE, SSIM showed a greater number of skipped frames when the same threshold was applied. Through such a process, it was confirmed that SSIM is the similarity measuring technique that enables skipping more frames while maintaining AUC in the road environment.

4.2 Dynamic Framerate SlowFast Network

The performance was evaluated by comparing the level of speed improvement and the level of AUC performance maintenance when the frames were skipped using the SSIM. Table 4 lists the results acquired by comparing the speed and AUC between the case where the proposed frame skip technique was applied and the case where the proposed frame skip technique was not applied.

In addition, it shows the AUC changes made when the similarity threshold was adjusted. First, based on the AUC comparison results per similarity threshold, the highest AUC was confirmed when the threshold was 0.75. Through this process, 0.75, with a smaller AUC reduction than the speed increase, was the most suitable threshold. In addition, the FPS(Frame Per Second) was drawn to analyze the video analysis speed. Compared to when the pre-existing system was applied when the proposed frame skip technique was applied, the image analysis speed increased approximately 1.5 fold to 0.7912 FPS with Video 5. In addition, the AUC was 0.7126, which was on 0.04 decrease compared to the pre-existing AUC. Through the proposed similarity measuring technique, the frame skip technique can maintain its AUC while reducing its analysis time.