1. Introduction

Recent advances in artificial intelligence have significantly impacted various fields, with multi-object tracking (MOT) being one of the most widely used applications in image processing. MOT involves monitoring multiple objects simultaneously within continuous frames. It is utilized for tracking objects in embedded systems from diverse fields to detect abnormal behavior ^[1], and plays a crucial role in autonomous driving ^[2].

An essential objective is to identify and differentiate multiple objects. Achieving this requires not only effective detection of specific objects but continuous and prolonged tracking of their presence. It is crucial that a detected object remains distinguishable from other objects throughout the tracking process. Temporary non-detection during tracking, or the inability to confirm its identity as the same object from the previous image, results in recognizing it as a different object, significantly diminishing accuracy from the tracking process.

Leveraging diverse design strategies, deep learning networks have demonstrated outstanding performance in object detection. Their superior performance ensures robust capabilities in simultaneously and persistently tracking multiple objects. This tracking-by-detection method involves receiving the coordinate values of the object from its detection in every frame. State-of-the-art (SOTA) tracking networks employ one-stage detectors ^[3,4] to enhance accuracy, capitalizing on their speed and high precision ^[5-7].

While existing deep learning tracking networks primarily focus on videos featuring static surroundings and well-defined object boundaries, such as pedestrians on roads ^[8,9], this paper introduces a unique dataset centered on swimming ^[10,11]. In this dataset, the surrounding environment is highly dynamic, and occlusion is particularly severe. Several distinctions set the swimming dataset apart from conventional MOT datasets. First, frequent occlusion is not only caused by other objects but also arises from dynamic surroundings such as water and intense spray. Secondly, even mild spray can introduce noise, posing challenges for object detection. Third, swimmers exhibit substantial posture deformations owing to rapid movements, especially during departure and turns, and even while swimming in a straight line. These distinctive characteristics are responsible for a more significant degradation in tracking accuracy compared to other datasets.

In this paper, the FairMOT ^[12] tracking-by-detection method is employed as a baseline architecture. In addition to FairMOT’s object detection and Re-ID responsible for adjusting object IDs, a scene detection network accounting for environmental changes to object detection and tracking is integrated into CenterNet ^[4] (the feature extraction network of FairMOT). This additional network adjusts the value of the IoU metric at the object detection stage and the predicted object location of the Kalman filter.

In object detection, a modified IoU metric (cIoU) transforms the existing IoU metric to allow consideration of not only the overlap between the detected bounding box and the predicted bounding box but also additional factors such as center point distance and aspect ratio. In cIoU, a scene-specific metric is achieved by assigning weights to each input parameter based on detected scene information from each frame. Performance is further enhanced by adaptively adjusting the ratio reflecting the predicted object location of the Kalman filter to the detected object bounding box from the network.

2. Related Work

3. The Proposed Method

3.1 Scene Detection Network

We incorporated a scene detection head into the existing FairMOT network to classify the corresponding scene in every frame. Following the backbone network, the FairMOT network was segmented into heatmap, box size, offset, and Re-ID heads, respectively, to generate output values. Additionally, a head for classifying swimming event scenes was introduced, and class labeling for each frame was applied to the swimming dataset to train the network to classify current input images. The approximate network architecture is shown in Fig. 1. Classes were categorized into Swimming, Turning, Diving, Finish, and On-block. The Swimming class (Fig. 2), encompasses scenes after swimmers dive into the water (i.e., bodies submerged); they swim until just before the return point, swim again after the turn, and finish swimming at the endpoint. However, since swimmers race at varying speeds, the moment they initiate the turn differs. Consequently, in that scene, the class switches from Swimming to Turning.

Fig. 1. The Architecture of the Entire Network.

Fig. 2. An Example of the Swimming Class.

In the freestyle competition, when swimmers reach the return point (Fig. 3), the Turning class is from the moment swimmers begin to turn their bodies while putting their heads into the water until last swimmer reaches that point. For the breaststroke competition, the Turning class starts at the point where swimmers reach out and touch the wall until they push off the return point with their feet and resume the breaststroke.

Fig. 3. An Example of the Turning Class.

As depicted in Fig. 4, Diving is classified as such from the moment the players' hands fall off the blocks until their whole bodies enter the water.

Fig. 4. An Example of the Diving Class.

As shown in Fig. 5, the Finish class is from the moment swimmers stop at the end of the race through to the movement they stand up while reaching out to the wall.

Fig. 5. An Example of the Finish Class.

The On-block class is from the moment swimmers wait on the starting blocks before the start of the race until their hands fall off the blocks, as shown in Fig. 6. In the backstroke, the preparation posture differs in the On-block class because swimmers hold the rods under the blocks and wait, as shown in Fig. 7, up until the moment their hands stop holding the rods.

Fig. 6. An Example of the On-block Class.

Fig. 7. An Example of the Backstroke On-block Class.

4. Experiments

As shown in Table 2, we initially conducted experiments with sequence videos classified by using the existing FairMOT and measured with tracking performance through the metrics in multi-object tracking.

In SWIM-02 and -03, which consist solely of the Swimming class, the MOTA score was high at 85.4% and 83.4%, respectively. In SWIM-02 and -03, swimmers move consistently without significant changes in movements, allowing for stable tracking.

In SWIM-07 and -08 (sequences where swimmers reach the return point and turn), MOTA scores were 47.5% and 53.5%, respectively. The lower MOTA scores compared to the Swimming class can be attributed to the camera angle being from the side, as shown in Fig. 2. Players farther from the camera are less visible than those who are closer, which makes tracking them challenging. Additionally, when swimmers turn, they are covered by spray from movements in the water, leading to ID switching due to the significant changes in motion.

SWIM-27 and -31, with MOTA scores of 34.9% and 51.3%, respectively, are sequences that include On-block, Diving, and Swimming classes. In these sequences, the most ID switching occurred when transitioning the class from On-block to Diving. When swimmers extend their bodies from a crouching position into the dive, significant changes in features and IoU take place, leading to failed matches in both cascade matching and IoU matching in Deep SORT. Due to significant changes in motion, the IDs of some swimmers switched not only when transitioning from On-block to Diving but also during the diving motion.

SWIM-31 exhibited a higher MOTA score compared to SWIM-27. The reason is that when transitioning from On-block to Diving, the IDs of swimmers changed, and these IDs were subsequently well-maintained. The ID changed at the moment the class switched due to the change in IoU, but afterward, tracking was maintained through Re-ID features. The difference in SWIM-27 was that it was easy to find Re-ID features from having more data on topless male swimmers during training. On the other hand, SWIM-27 had relatively more difficulty finding Re-ID features because female swimmers wore full-body swimsuits.

SWIM-06 and -14 are sequences that contain scenes at the end of the swimming competition, and the sequences were completed with the Finish class. The two sequences showed MOTA scores of 52.2% and 47.2%, respectively. When shooting the Finish class, the composition changed to filming next to the swimming pool, resulting in one swimmer being covered by the referee, leading to ID switching. There was also a case where ID switching occurred due to spray in the swimming scene, and further ID switching occurred due to changes in motion and features as the Swimming class transitioned to the Finish class.

We experimented by changing the IoU formula to the cIoU formula in IoU matching by Deep SORT, and by weighting the aspect ratio. We first used a grid search to find the weight of the optimal aspect ratio for each class. As a result, Diving and On-block classes showed optimal MOTA values when the weight was 41, and the rest of the classes showed the highest values when it was 1. Based on these values, we set the optimal weight value according to the scene through the scene detection network. We show the results in Table 3. SWIM-02, -03, -06, and -14 do not appear to have affected IoU matching, and the MOTA score slightly increased in SWIM-07, -27, and -31. SWIM-02, -03, -06, and -14 are sequences in which significant IoU changes do not occur and were not affected by cIoU formula changes. SWIM-07, -27, and -31 have classes with large IoU changes, such as On-block, Diving, and Turning, so the cIoU formula affected performance.

In addition, through the scene detection network, hyper-parameters of the Kalman filter, the position, and the velocity can be found and applied to each scene, as shown in Table 1. We present the results in Table 4. SWIM-02 and -03, which only have the Swimming class, showed improved performance in IDF1 and MOTA scores by applying hyper-parameters optimal for the Swimming class. In SWIM-07 and -08, where Turning and Swimming appear, IDF1 showed a significant improvement in performance, and in SWIM-07, the MOTA score improved by 5.2 compared to the original FairMOT. In SWIM-07 and -27, adjustment of the Kalman filter's hyper-parameters was not significantly effective. As seen in Table 1, the optimal value for Diving and On-block classes was not significantly affected by the hyper-parameter weight of the Kalman filter, and the fact that the hyper-parameter value was set to an inappropriate value in switching from Diving to Swimming caused performance degradation. In SWIM-06 and -14, performance improvement was achieved in IDF1 through hyper-parameter adjustment, and ID was maintained even when the swimmer was occluded by the referee. Additionally, ID switching also showed improvement at the moment Swimming changed to Finish.

5. Conclusion

In this paper, we proposed a method to enhance performance by incorporating additional factors. We achieved this by modifying the IoU formula in the IoU matching of Deep SORT into the cIoU formula, thereby improving tracking performance on swimming datasets using the FairMOT tracking network. MOTA scores were enhanced by up to 0.6% in specific sequences by using this method. Additionally, the MOTA score saw an improvement of up to 5.2% through the adjustment of hyper-parameters in the Kalman filter based on class.

In future studies, we intend to enhance multi-object tracking performance by individually analyzing and considering the scene characteristics of each object, especially when dealing with swimmers detected in various states of motion, such as Turning.