1. Introduction

The limited available spectrum for radio frequency (RF) communications will not be able to meet the exponentially increasing demand for wireless Internet access in the near future, leading to a major spectrum crisis. Owing to this undeniable situation, the introduction of new spectrum for better energy efficiency, greater connection bandwidth, and lower usage costs for 6$^{\mathrm{th}}$ generation (6G) communications and future Internet connections becomes very important. Optical wireless communications (OWC) ^[1,2] has recently been explored as a complementary alternative to RF links for development of heterogeneous networks that enable peer-to-peer connectivity in a cost-effective and reliable way. OWC uses a very wide spectrum of optical domains, including infrared, visible light, and ultraviolet domains, to establish communication performance ^[2].

Recently, a variety of research into D2C communications has been conducted. Wang et al. ^[7,8] proposed InFrame and InFrame++, which provide full-frame communication with imperceptible video artifacts. For this, complementary frame composition and a hierarchical frame structure were designed. HiLight ^[9] uses the orthogonal transparency (alpha) channel to transmit data without the need for coded images. DisCo ^[10], based on a rolling-shutter camera, executes decoding by translating a temporal sequence into a spatial pattern. Secure barcode-based visible light communication (SBVLC) ^[11] considers physical security issues and the design principles of 2D barcodes to add security features. RainBar ^[12] and RainBar+ ^[13] were designed with a high-capacity barcode layout to enable flexible frame synchronization and code extraction in VLC systems. In ^[14] and ^[15], the SoftLight scheme, presented over screen-camera links, uses color modulation schemes with channel coding and a $\textit{soft hint}$ for data decoding through the barcode layout. In ^[16], complementary color barcode-based optical camera communications (CCB-OCC) was proposed, where symbols are sent with carefully designed complementary color pairs that are perceived by the human eye as a white bar, but the color pattern is detectable by a camera. Although the CCB-OCC method was first presented in ^[16], it is necessary to improve the detection performance of color barcodes for practical use.

In this paper, we propose a new CCB-OCC system to improve data rate performance using deep neural network (DNN)-based barcode detection and adaptive color-value extraction. The contributions of this study are twofold. First, a novel DNN network is designed for detecting robust and seamless barcode image regions in D2C links. Second, value extraction from color histograms was refined to mitigate the effect of D2C noise and synchronization jitter. In the conventional CCB-OCC technique, an image-processing-based approach was used to detect the color barcode region, but this could result in barcode detection failure or loss of information from the barcode. In addition, fixed-peak, position-based signal extraction used in the conventional technique cannot solve the problem caused by synchronization jitter between display and camera. However, the proposed method uses a DNN-based model with real-time barcode detection and adaptive color-value extraction, which can provide a robust D2C link for data transmission. Experimental results validated the proposed CCB-OCC system’s significant improvement in data rate performance, and the proposed scheme can be regarded as a potential candidate for next-generation short-distance machine-to-machine (M2M) communications.

2. System Model of CCB-OCC

In the CCB-OCC scheme, the transmitting side encodes data into color barcode sequences and displays a consecutive packet of color barcodes on its electronic display. When the refresh rate of the display is greater than 120Hz, these barcode images are perceived by the human eye as a white bar. When captured with a camera, however, they are perceived as a color barcode owing to the rolling shutter mechanism of a CMOS sensor in a camera device. The receiving side uses a camera to capture the display and acquires consecutive images, including the color barcodes. This visible spatial pattern in the received image represents the data. After detecting a color barcode, packet synchronization is performed by checking the location of pilot symbols. After that, channel estimation obtains a D2C channel based on the color space, and the transmitted data are decoded by using the obtained channel information. The CCB-OCC system model is presented in Fig. 1.

Fig. 1. The system model for complementary color barcode-based OCC.

For the D2C communications scenario, the main objective of the transmitter is to encode the transmit data into color barcodes in complementary pairs and to display them on the monitor without noticeable artifacts visible to the human eye. For this, the display monitor’s refresh rate should be 120Hz or higher. In addition, a bit-to-color-mapping process converts the input bit stream into a specific color. Here, binary bits are mapped to specific colors according to well-designed symbolic constellation structures. Each color-mapped symbol consists of a pair of complementary colors, constructing pilot and data symbols. Images captured by the camera of the receiving device are sequentially stored in a memory buffer to detect continuous bit signals. To extract the signal from the received image, it is necessary to detect the pure color barcode area, excluding the background. Packets are composed of pilot and data symbols in red-green-blue (RGB) colors, and the pilot symbols are retrieved through color barcode pixel value extraction for packet synchronization and channel estimation. The wireless optical channel between the electronic display and the camera is estimated by obtaining each component through histogram analysis of the RGB channel, and the remaining data symbols are decoded using the predicted channel information.

CCB-OCC uses a unique color barcode design that makes the continuous information transmitted from the display invisible to the human eye but detectable by devices equipped with a rolling-shutter-based camera. This is done by sequentially presenting on the display device the complementary colors opposite each other in the hue circle structure ^[16]. The packet structure of the CCB-OCC scheme is defined with pilot color symbols and data symbols with complementary color pairs. These transmit symbols are encoded within the color barcode area of each image frame that appears through the display device. As shown in Fig. 2, the pilot symbol consists of six consecutive frames in the RR$^{\prime}$GG$^{\prime}$BB$^{\prime}$ color pattern sequence. The remaining data symbols are transmitted as packets consisting of complementary color pairs of data. The image frames containing this encoded data are displayed on the screen based on the refresh rate, and the data can be decoded by capturing successive images with the camera or receiving device. When a pair of complementary colors appearing in a continuous image is rapidly scanned, the human eye does not recognize the original color but perceives it as a white bar representing the sum of the complementary colors. Using these complementary colors, the proposed color barcode pattern is visually unobtrusive and does not interfere with the overall quality of the display content. To minimize the effect of bit errors when converting from decoded symbols to binary numbers, we use a gray-coding-based code constellation.

Fig. 2. Packet structure of CCB-OCC.

Fig. 3 shows the complementary color pairs in a series of images from the display. When a color barcode on a monitor with a refresh rate of 60 Hz is received by a CMOS camera sensor at 30$\textit{fps}$, at least two colors appear in every frame. As seen in the figure, the RR'GG'BB' color pattern appears in which R, G, and B indicate the primary red, blue, and green, with R$^{\prime}$, G$^{\prime}$, and B$^{\prime}$ indicating their complementary colors. Theoretically, the number of colors observed in a color barcode area of one received image is the ratio of the camera's incoming capture rate to the transmission rate, i.e. the ratio of the display refresh rate to the camera capture rate. As shown in Fig. 3, six RR'GG'BB' colors are obtained along three consecutive image frames received by the camera, since the camera’s receive rate is half the display transmission rate. However, if the camera capture rate and refresh rate are out of sync, three colors may appear simultaneously in the captured image. Considering both of these cases, it is necessary to estimate the channel and detect the signal.

Fig. 3. An example of received images when the display’s refresh rate is 60$\textit{Hz}$ and the camera’s capture rate is 30$\textit{fps.}$

3. DL-based Color Barcode Detection

D2C communications requires accurate barcode detection to ensure successful decoding from consecutive captured images. Since multimedia information other than the color barcode is included in the captured image, along with the external part of the display device, it is important to accurately detect only the barcode area for accurate signal detection. In this section, we present an approach that can reliably detect barcode regions using deep learning technology.

3.1 Barcode Detection

There are many methods available to detect an object with bounding boxes. Among those methods, for real-time applications, the faster region-based convolutional neural network (Faster R-CNN) ^[17] and You Only Look Once (YOLO) ^[18] are the most popular in various research fields. Faster R-CNN is a CNN that considers the entire input image as one candidate region. Features are extracted by pooling one feature map generated through the learned CNN. As a method for generating a candidate region, an object is recognized after estimating the extracted feature map as a candidate region by applying a separate region proposal network instead of the selective search technique.

YOLO is an algorithm that can predict an object existing in an image (and the location of the object) by looking at the image only once. Instead of detecting it as an object to be classified, it approaches it as a single regression problem by dividing the bounding box in a multidimensional manner and applying class probability. The input image is divided into a grid through the CNN, and an object in the corresponding area is recognized by generating a bounding box and a class probability according to each section. Because YOLO does not apply a separate network for extracting candidate regions, it provides performance in terms of processing time that is superior to a Faster R-CNN. Based on this, we used a YOLO v3 model as a function to detect color barcode regions from received images. In order to train the YOLO model, a data set including labeling for the color barcode area was built using software called YOLO Mark, which is shown in Fig. 4.

Fig. 4. Color barcode labeling in the dataset used to train the YOLO model.

To extract the signal contained in the color barcode detected using YOLO, an additional post-processing step is required. First, an image-difference operation must be performed in order to identify the pure barcode area. The area within the bounding box detected through YOLO contains not only color barcodes but also a background that includes the display’s external parts. If this background is not removed, noise will be created when color values are extracted through histogram analysis from each color channel at a later stage. If only the image-difference operation is performed, the image-difference value appears large in areas other than the color barcode due to jitter in the camera capture process or relative movement between the display and the camera. Then, by using the binary image as a mask filter, only the color channel values within the barcode image area are obtained from the original image to be analyzed as a histogram. This series of processes is shown in Fig. 5.

Fig. 5. Processing detected color barcodes into histograms of color channels.

Fig. 6. Varying the peak position when obtaining a representative value of the color barcode.

4. Experimental Result

In this section, we discuss the extensive experiments with YOLO-based barcode detection. The representative-value position-correction technique proposed in this study can more accurately detect a color barcode in the captured image. Note that a distance of more than 110cm will result in distortion and the blur effect, making it more difficult to extract the precise point of channel estimation and decoding from the RGB histogram.

As seen in Fig. 7, we investigated the impact of view angle on the achievable data rate (ADR) of the proposed CCB-OCC scheme. Here, the distance parameter was set to 90cm. Table 1 presents the ADR based on the yaw angle. As can be seen, when the yaw angle between the display and the camera lies between -20 and 20 degrees, the proposed scheme presents no significant difference in the data rate. This is the opposite result from the existing technique’s performance, which differs depending on the angle. Because the conventional technique extracts a color barcode using the image processing method, if the yaw angle is large, the barcode image region becomes too thin, causing color information loss. But the proposed YOLO-based technique reliably detects the color barcode region through bounding box detection even at a large yaw angle, so it can extract representative color values more accurately. Therefore, the performance of the proposed method is not significantly different, depending on the view angle, but presents a better data rate than the existing method.

Fig. 7. Experimental environment with varying distances and angle values.

Fig. 8. Performance from the achievable data rate according to D2C distance.

We evaluated the performance of the CCB-OCC scheme using YOLO-based barcode detection. Our receiving device was a Samsung Galaxy S9, which is a common Android smartphone equipped with a standard camera featuring 1920${\times}$1080 resolution at a 30fps video capture rate. At the transmitter, the resolution of the electronic display was 1920${\times}$1080@60Hz, and experiments were conducted indoors under normal lighting conditions. In the experimental environment, as shown in Fig. 8, the performance can be measured by changing the distance between the display and the camera (to 110cm from 90cm) and the view angle (from -20 to 20 degrees). To verify the performance of the proposed method, it was compared with results from the conventional CCB-OCC scheme ^[16].

Fig. 8 shows the achievable data rate from the proposed CCB-OCC scheme based on the distance from display to camera. As the distance increases, the resolution of the captured color barcode image decreases, the noise effect increases, and the channel estimation and data decoding accuracy decrease. As can be seen, the proposed scheme outperformed the conventional scheme when the distance was between 90cm and 110cm. By automatically extracting the color bar code area through the YOLO model and with image post-processing, it is possible to acquire pilot and data signals more precisely than with manual barcode detection used in the existing technique. The proposed scheme provided a maximum data rate of 79.7bps and a minimum rate of 76.2bps in the 90-110cm range.

In Table 1, we can see that the proposed CCB-OCC scheme showed better achievable data rates than the conventional scheme at various angles. Even if the yaw angle between the camera and display was different by 20 degrees, the proposed scheme provided a data rate of 80.7bps or more. The YOLO model, which guarantees real-time object detection, can guarantee robust data rate performance by stably detecting color barcode images taken from various angles.

5. Conclusion

In this paper, we designed and implemented a new CCB-OCC system for data rate improvement using DNN-based barcode detection and adaptive color extraction. We introduced a DNN-based barcode detection concept where YOLO v3 was used to detect a color barcode without losing information from within the barcode region. In addition, a color-value extraction scheme obtained symbol values from color histograms in an adaptive manner to compensate for synchronization jitter. The experimental results proved that the proposed CCB-OCC scheme outperforms the existing CCB-OCC scheme for various distances and angles in D2C communications links.