1. Introduction

In the manufacturing process, combined with advanced technologies, the Artificial Intelligence (AI) model for collecting sensor data in various industrial domains for fault prediction, defect detection, and anomaly detection has been developed. Accurate descriptions of defects, faults, and conditions are needed to apply an AI model to manufacturing equipment. Given the sensitivity, multi-variance, irregularity, and time-series features according to external environmental factors, it is necessary to suggest a plan for securing quality. Unfortunately, there are practical difficulties in meeting the execution conditions of industrial systems ^[1]. In industrial domains, anomaly detection is performed by comparing with training data and checking if the data distribution is consistent. Hence, overfitting can occur with data imbalances in which data for a specific class fail to be distributed evenly ^[2]. Overfitting can also come about if the goodness-of-fit of a model is overemphasized in the production line. In this respect, it is essential to reduce the number of feature extractions of sensing data, or to make normalization through linear regression or logistics regression ^[3]. In addition, it is essential to collect sensing data from heterogeneous sensors and present the risk factors of the manufacturing process, including failures and faults of equipment, through explainable prediction.

Cynthia Changxin Wang ^[4] performed an axis-aligned instance segmentation and proposed a safety warning algorithm to prevent on-the-job accidents of mobile devices. The proposed axis-aligned instance segmentation calculated and bounded the major axis of an object and rotated the slanted object for detection. This method accurately detected an asymmetrical object through the orientation. On the other hand, in the method, the object boundary became increasingly obscure the farther the object was from a recognition axis, and it detected only a single object.

In an industrial site, it is impossible to explain the prediction results, and it is challenging to apply an AI model in which the performance is guaranteed only if features of large sensing data are extracted. In addition, it is important to minimize the false positive and false negative rates of each item in the manufacturing process, and to advance and optimize the Mask R-CNN ^[5]-based occlusion area detection and anomaly algorithm. In this way, it is possible to find a particular area of data, extract a feature from that area, and determine if there is any defect.

This study proposes the Mask R-CNN-based occlusion anomaly detection considering the orientation of manufacturing process data. Occlusion anomaly detection refers to a mechanism for performing two tasks, which detects occlusions and abnormal objects presented in the image area. Compared to a conventional algorithm based on Mask R-CNN, the proposed method increases the accuracy of object recognition in detecting an object. Data preprocessing is performed to express images of manufacturing at various angles in the manufacturing process. Subsequently, Mask R-CNN is used to determine if they have an object in pixels, extract the bounding box and masking information, and detect the occlusion and anomaly of an image. By applying Mask R-CNN capable of detecting an area, the proposed method enhances the defect detection ability to decrease the false positive rate and the false negative rate in a manufacturing line by combining with occlusion area detection in image preprocessing.

Chapter 2 describes the related works on object detection-based anomaly detection. Chapter 3 introduces the Mask R-CNN-based occlusion anomaly detection considering the orientation in manufacturing process data. Chapter 4 reports the results and performance evaluation. Chapter 5 describes the conclusions of this study.

2. Related Work

In manufacturing, methods for detecting defective products using camera image data are being studied. Mohr et al. classify them into CNNs by learning normal and abnormal image data ^[6]. Suh et al. proposed a method to locate defective products using infrared cameras ^[7]. On the other hand, the above studies detected defective products with models learned using single object image data without considering occluded objects generated in a real manufacturing process. In addition, the process is limited by the expensive infrared cameras. Therefore, this study presents a model that uses data augmentation techniques to increase the anomaly detection accuracy among occluded objects at low cost.

2.1 Object Detection

Object detection classifies an instance through a bounding box and finds the location and classes of various objects. Object detection is made possible through the classification, localization, and detection of image data. In other words, the location of an object in an image is found through localization, and the object is detected and classified. Deep learning-based object detection consists of one-stage detector and two-stage detector ^[8]. As a typical method of a one-stage detector, there is YOLO ^[9]. Because it performs both classification and localization, it features a fast speed ^[10]. YOLO finds anchors saved after an input image is segmented by a grid and predicts an anchor box. The anchor box is a box in which an object is possibly located in the input image. It obtains a confidence score according to which an object is classified. Mask R-CNN is a typical algorithm of a two-stage detector. Fast R-CNN ^[11] and Faster R-CNN are models for object detection ^[12]. These object detection methods use a bounding box to classify an instance, identify multiple objects, and detect the location and class of each object. Unlike these methods, Mask R-CNN utilizes instance segmentation that has the combined concept of object detection with semantic segmentation. For the same class, conventional semantic segmentation makes segments into the same area without classification, whereas instance segmentation classifies an instance differently, even in the same class. Fig. 1 shows the object detection process in Mask R-CNN.

In Fig. 1, all three are forks in the same class but are recognized as instances where they are segmented into different colors. In the manufacturing process, there are multiple objects whose shape is equal. Therefore, Mask R-CNN-based model using instance segmentation, rather than a conventional model, can improve the detection performance ^[13].

Fig. 1. Object Detection Process in Mask R-CNN.

3. Mask R-CNN-based Occlusion Anomaly Detection Considering Orientation in Manufacturing Process

In a manufacturing process, one of the production processes may not be performed, or a mechanical failure can occur, resulting in sub-normal production ^[18]. For example, in manufacturing forks, a fork without a blade or fork-head part is sometimes produced. People can detect such products, but there are high associated labor costs. Moreover, it is difficult to detect such products if they are occluded. This study introduces the Mask R-CNN-based occlusion anomaly detection considering the orientation in manufacturing process data to deal with the problem. The proposed method proceeds in two steps. Fig. 2 shows the Mask R-CNN-based occlusion anomaly detection process considering the orientation in manufacturing process data.

In Fig. 2, the first is the data preprocessing step in which the number of cases where defective forks are mixed in the normal forks in manufacturing forks. The second is occlusion anomaly detection using Mask R-CNN, in which defective forks occluded in normal forks are detected by Mask R-CNN.

Fig. 2. Occlusion Anomaly Detection Process.

3.1 Data Preprocessing Considering Orientation

In this study, the manufacturing process data is fork class data of MS COCO (Microsoft Common Objects in Context) ^[19]. The MS COCO dataset is an extensive dataset of object detection, segmentation, and captioning. In manufacturing tableware, forks are not evenly aligned but are occluded or produced in diverse directions. In addition, there is a lack of occlusion data compared to the image dataset of non-occluded products. The OpenCV library was applied for image preprocessing to solve the problem. OpenCV is an open-source library to process images and videos in real time. It is used for the preprocessing work for object detection. The accuracy of object recognition is increased using image augmentation. This study utilizes the image augmentation method with orientation, compositing, and pasting preprocessing, among various image augmentation methods using OpenCV.

First, orientation is applied to the image dataset ^[20]. There are transition and rotation functions usable for the orientation. The transition function moves an image left and right, and up and down so that it is impossible to consider angles. Because the rotation function considers the rotation axis and rotation angle of an image, it is possible to consider angles and directions. Accordingly, by rotating the image data of a fork at 15$^{\circ}$ intervals, it is possible to generate new images, express the product images with diverse angles generated in the manufacturing process, and produce occlusion images. Fig. 3 presents an image for the equation. Eq. (1) is the formula for image orientation.

(1)

$ new_{x}=\left(H_{p}-y\right)\times \sin \left(\theta \right)+\left(W_{p}-x\right)\times \cos \left(\theta \right)+x \\ new_{y}=\left(H_{p}-y\right)\times \cos \left(\theta \right)-\left(W_{p}-x\right)\times \sin \left(\theta \right)+y $

In Eq. (1), $H_{p}$ is an increasing pixel height; $W_{p}$ is an increasing pixel width; $\theta $ is the rotating angle. In the way of changing the coordinates of image pixels based on the reference coordinates $x_{1}$, $y_{1}$ of the image, the coordinates $new_{x}$, $new_{y}$ of the rotated pixels are calculated using Eq. (1).

Fig. 3. Image Orientation.

3.2 Occlusion Anomaly Detection using Mask R-CNN

This study proposes Mask R-CNN-based occlusion detection method considering the orientation of manufacturing process data. The model was implemented based on FPN (Feature Pyramid Network) ^[21] and ResNet101 backbone. Mask R-CNN produces a bounding box and a segmentation mask for each instance of the object that appears in an image. Unlike other models for object detection, it generates a segmentation mask to identify different instances in the same class ^[22]. Fig. 4 shows the occlusion and anomaly detection process using Mask R-CNN.

Fig. 4. Occlusion and Anomaly Detection Process using Mask R-CNN.

The Mask R-CNN-based occlusion anomaly detection process is the same as the combination of object detection with semantic segmentation. First, the image generated through preprocessing is input into the model, and an anchor box is marked in the image position in which an object can be found for object detection. In the later step, the most optimal anchor box detected is marked. The model generates a mask for each instance and adjusts the size for semantic segmentation ^[23]. The occlusion and anomaly areas of the image can be detected if the box and mask obtained in the two steps are positioned correctly in the image. Mask R-CNN generates three mask branches. That is the distinction of Mask R-CNN from conventional object detection. The classification branch plays a role in class prediction. The bbox regression branch changes the box coordinates. FCN-based mask branch plays a role in generating a binary mask. The feature map produced by three branches is rescaled, and the mask segment to compare with an actual mask is generated. Mask R-CNN solved the problem of RoI Pooling in which a decimal point is ignored when coordinates are reduced and is capable of predicting a class in the unit of the pixel through RoIAlign.

4. Performance Evaluation

For performance evaluation, based on the weight and preprocessed data of the model pre-trained COCO dataset, the Mask R-CNN-based model proposed in this study was compared with YOLOv2 ^[24] and YOLOv3 ^[25]. In particular, the Mask R-CNN and YOLO object detection algorithms were compared in terms of whether to detect occlusion and anomaly considering the orientation and precision, recall, and accuracy of object detection. The object detection performance was evaluated with IoU (Intersection over Union) ^[26]. The IoU represents the union-intersection ratio of the ground truth bounding box and predicted bounding box. The closer the IoU is to 1, the higher the object detection accuracy. Eq. (2) presents the formula to calculate IoU.

In Eq. (2), $F_{gt}$ is the ground truth of the actual bounding box; $F_{P}$ is the predicted value of the bounding box. A test was conducted assuming that the fork is a normal object and the defective fork is an anomaly. First, the results of the raw image and the image considering the orientation were compared.

Fig. 5 shows the case where object recognition is performed through Mask R-CNN on the original image without considering the orientation. Fig. 6 presents the case where an object is recognized by Mask R-CNN after the original image was changed by preprocessing considering the orientation. In Fig. 5, Mask R-CNN recognized eight out of nine normal objects. In addition, its recognition rate of object detection was approximately 75.83%. Fig. 6 reveals a better recognition rate (83.41%) than the image without considering the orientation and recognition rate. These results show that Mask R-CNN detects more occluded objects in the image considering the orientation. In the case of anomaly detection, Mask R-CNN did not detect anomaly objects as normal objects in the images considering orientation. Hence, Mask R-CNN performs well in anomaly detection regardless of the orientation. Table 1 presents the results of the IoU performance evaluation.

As shown in Table 1, Mask R-CNN showed better performance than YOLO for the original image without consideration of the orientation. Hence, Mask R-CNN detected more objects in the image of an occluded object. Nevertheless, occlusion anomaly detection in consideration of the orientation was performed because each occluded object failed to be detected correctly. In the performance evaluation, Mask R-CNN showed superior IoU performance than YOLO. This is because Mask R-CNN is better for recognizing objects and detecting occluded objects than YOLO. Therefore, Mask R-CNN shows the best performance in the anomaly detection of occluded objects in a manufacturing process.

Precision and Recall were performed as performance indicators for anomaly detection. Precision represents the degree to which only related objects are detected among objects in image data. Recall indicates the degree to which all related objects have been detected correctly in the input image data. For anomaly detection, 200 images of a normal fork and defective fork images were validated using considering orientation image data. For defective forks, they should not be classified as fork-object classes. In other words, they are not classified into any class. Table 2 lists the Confusion Matrix of defective fork detection.

According to Table 2 and the performance evaluation index, the precision was 172/(172+25) = 0.87, and the recall was 172/(172+28) = 0.86. This result was compared with the YOLOv3 and YOLOv2 models. The environmental conditions for the performance evaluation were the same as the environment of Mask R-CNN. Table 3 lists the performance evaluation results of object feature detection.

The Mask R-CNN-based method proposed in this study showed superior results in all performance evaluation indicators of precision and recall than YOLOv3 and YOLOv2.

Fig. 5. Result of the raw image.

Fig. 6. Result of the image considering the orientation.

5. Conclusion

This study proposed the Mask R-CNN-based occlusion anomaly detection considering the orientation in the manufacturing process data. As the method for the anomaly detection of occluded objects in the manufacturing process, Mask R-CNN was used to detect abnormal objects hidden in normal objects from the data considering the orientation rather than alignment data. The proposed method proceeds in two steps. First, the alignment data was changed to the data considering the orientation. In other words, the rotation function of the OpenCV library was used to rotate images of the dataset and generate new images considering their orientation. In the second step, Mask R-CNN was applied for occlusion anomaly detection. In this step, the box and mask obtained by Mask R-CNN were used to detect an occlusion from the data, including occluded objects. Accordingly, the abnormal object ‘defective fork’ occluded in the normal object ‘fork’ was detected. For the performance evaluation of detection accuracy, Mask R-CNN was compared with YOLOv2 and YOLOv3 using IoU. When anomaly detection was performed in the original data without consideration of the orientation, Mask R-CNN had better object detection accuracy. For the performance evaluation in consideration of the orientation, Mask R-CNN had better object detection accuracy than YOLO. In addition, in the performance evaluation of anomaly detection, Mask R-CNN shows the best performance in precision and recall. Accordingly, for occlusion anomaly detection in the manufacturing process, it is good to apply Mask R-CNN considering the orientation. In the case of occluded objects that occur in a real manufacturing process, recognition accuracy is poor due to problems, such as misrecognition due to texture bias. To solve this problem, it is possible to increase the recognition performance of the occlusion object by Mask R-CNN-based occlusion anomaly detection considering the orientation. Furthermore, it is possible to prevent issues that may occur in the distribution process by finding defective products that are occluded and difficult to identify in the anomaly detection process in real manufacturing.

Future research will evaluate methods for anomaly detection in the data, including more occluded objects in diverse manufacturing situations.