2.1 Background

When light is incident on an object, the characteristics of the reflected light are affected by differences in surface reflection, such as diffusion or blocking of light, which result in shading. The details of shading are described in the Cook-Torrence reflection model ^[10], which represents the reflection of light appearing in real objects in computer graphics. The Cook-Torrence reflection model is based on Physically Based Rendering (PBR) ^[11]. The model is a shader that uses computer graphics to regenerate the surface of an object with microfacets that cannot be observed by the naked eye. The shader then simulates the reflection.

The reflection conditions determined by micro facets can be expressed by Eqs. (1) and (2):

where $\textit{I}$$_{r}$ is the intensity of the reflected light, $\textit{I}$$_{ia}$ is the intensity of the incident ambient light, R$_{\mathrm{amb}}$ is the ambient reflectance, and $\textit{I}$$_{i}$ is the average intensity of the incident light. $\textit{N}$ is the unit surface normal, $\textit{L}$ is the unit vector in the direction of the light, and $\textit{d}$ is the fraction of reflectance that is diffuse. $\textit{d}{\omega}$$_{i}$ is the solid angle of the beam of the incident light, s is the fraction of reflectance that is specular, $\textit{R}$$_{amb}$ is the ambient reflectance, and $\textit{R}$$_{d}$ is the diffuse bidirectional reflectance. $\textit{D}$ is the facet slope distribution function, $\textit{G}$ is the geometrical attenuation factor, $\textit{F}$ is the reflectance of a perfectly smooth surface, and $\textit{R}$$_{s}$ is the specular bidirectional reflectance. Fig. 2 compares the surface reflection with the variation of G, which is visually distinct. The image pixel standard deviations (${\sigma}$) and mean values (${\mu}$) of the images vary, as shown in Table 1.

Fig. 2. Comparison of surface reflection by G value change.

Fig. 3. Mitutoyo stylus SJ-210.

Fig. 4. Surface roughness measurement using a stylus.

2.2 Surface Roughness Measurement and Sample Classifications

Arithmetic mean deviation (Ra) values were measured using a Mitutoyo SJ-210 surface roughness tester following ISO 1997 standards ^[12]. Fig. 3 shows the Mitutoyo SJ-210 stylus with 0.75mN60$^{\circ}$, 2${\mu}$mR, which was used to measure the surface roughness of paper samples.

The stylus of the roughness tester is mechanically drawn across the surface across the sample length and provides various roughness parameters, as shown in Fig. 4. Rmax is the largest difference between peaks and valleys in an individual sample length, Ra is the average roughness over the sampling length, and Rt is the distance between the highest peak and lowest valley over the sampling length. The Ra values for the sample paper surface ranged from 0.325 ${\mu}$m to 24.709 ${\mu}$m.

Labels were divided into three categories following the ISO 1302:2002 standard ^[13], as shown in Table 2. The precision machining category of machining was excluded from this study because of insufficient samples. All the categorical features in the training data were encoded as a one-hot numeric array using the function $\texttt{sklearn. preprocessing.OneHotEncoder()}$. The number of images in each class is shown in Table 3.

2.3 Image Sampling and Image Color Space Conversion

Images of the paper samples were taken using a ViTiny UM12 Microscope camera with a magnification of 18X in fixed focus environments. The images were taken and stored at an image resolution of 640x480 pixels. The detailed specifications of the ViTiny UM12 microscope camera are shown in Table 4.

This study focuses on using paper as roughness samples for deep learning as they relatively easy to acquire and also have various surface qualities, textures, and materials. The paper samples were from the Samwon Paper Ltd. and Doosung Paper Co., Ltd. A total of 305 paper samples were selected for extracting adequate feature information for deep neural network (DNN) training.

To prevent interference from ambient light, a custom darkroom (420mm ${\times}$ 297mm ${\times}$ 350mm) was built for taking images from the microscopic camera, as shown in Fig. 5. To collect shading information based on the surface roughness of the sample, a light source was installed normal to the surface of the object. A digital microscope camera for photographing the sample surface was placed within the field of view (FOV) adjacent to the light source. Images taken in these environments were sent to the learning data acquisition system via OpenCV, a popular CV library ^[14].

The default format of the images taken by the camera was the JPG format. JPG images are stored in RGB color space based on the sensitivity of color detection cells in the human visual system. However, YCbCr color space is often used in digital image processing to take advantage of the lower resolution capacity of the human visual system for color luminosity. Thus, conversion from RGB to YCbCr color space is widely used in image processing.

Each pixel of an image in RGB format varies in intensity from 0 to 255, with 0 representing black and 255 representing white. RGB color space can be converted to YCbCr using Eq. (3), which can be implemented using the function $\texttt{cv2.cvtColor(input\_image, cv2.COLOR\_}$ $\texttt{BGR2YCrCb}\texttt{)}$ in OpenCV.

In YCbCr color space, Y channels contain the luminance (intensity) component, and Cb and Cr have the color information. As human and neural networks are more sensitive to luminance changes, Y channel information will suffice for deep learning, and Cb and Cr channel information is generally not required. When comparing this to the average Roughness (Ra) measured by the stylus, a distinction can be seen. In Fig. 7, the lower groups have a rougher surface than the upper group, which is indicated by data and is also visually noticeable. A histogram of these results is shown in Fig. 7(b).

The shading of the surface when illuminating the sample with the light source shows a significant relation to the surface roughness of the sample. Hence, quantitative indicators such as standard deviation, mean value, and histograms were extracted from the Y channel of the images. The detailed procedure is explained in the next section.

Fig. 5. A custom-built darkroom for imaging in a controlled environment.

Fig. 6. Representative sample images and histograms for training deep learning model (a) Comparison of roughness samples in YCbCr image, (b) Comparison of roughness samples in Histogram image, (c) Comparison of roughness samples in Original image.

Fig. 7. Classification based on KNN classifier (a) Original data distribution, (b) Fail case of KNN cluster 1, (c) Fail case of KNN cluster 2.

Fig. 8. A typical LSTM cell.

2.4 Dataset

An image is essentially an array containing pixel data. NumPy ^[15] was used to manipulate the image. NumPy (np) is a fundamental package for scientific computing in the Python programming environment and contains a large number of functions for mathematical operations. Thus, the Y channel, standard deviation, mean, and roughness class based on Ra values were taken into consideration while creating a deep learning model to evaluate the surface roughness of the sample.

The standard deviation and mean values were calculated using the functions np.std() and np.mean() of NumPy for images composed of only Y channel information. For ease of training, the image was resized to 480 x 480 pixels. LabelEncoder() and OneHotEncoder() functions from scikit-learn were used to encode the class label and the numerical data and saved in *.npy format, which is the standard binary file format in NumPy.

2.5 Computer Software and Hardware

The deep learning model training was performed on a dedicated workstation with 64 GB of RAM, GeForce GTX 2080Ti FTW3 ${\times}$ 2 GPUs, and AMD Ryzen 3960X processor. The workstation runs with Ubuntu 18.04 LTS and Python 3.7.

2.6 Model Design

Provided the roughness of the surface, the K-Nearest Neighbor Algorithm (KNN) ^[16] can be applied using $\texttt{sklearn.neighbors.KNeighborsClassifier()}$ for the classification of surface roughness. Similarly, histogram comparison using OpenCV function $\texttt{cv2.compareHist}$() can also be used for a similarity check when the color histogram is provided. However, both methods have limitations. KNN needs the measured roughness values for classification, which is time consuming, expensive, and considered to be a bottleneck operation in categorizing the roughness of the surface. Also, if two neighbors k+1 and k have identical roughness values but different labels, the result will depend on the ordering of the training data, making it very difficult to correctly predict the class. This is shown in Fig. 8. Histogram comparison uses various metrics for comparison. However, the values might be different under other lighting conditions.

To overcome these limitations, the Keras functional API was implemented to build a composite deep learning model. The core of deep learning is Artificial Neural Networks (ANNs), which are versatile, powerful, and scalable. This makes them ideal for tackling large and highly complex machine learning tasks such as image classification, speech recognition, machine translation, and recommender systems ^[17]. The functional Keras API was used to develop complex models with multiple inputs and modalities that offer ways to create deep learning models that have much more flexibility and complexity. This work implements a composite model including a Convolution Neural Network (CNN) and bidirectional Long-Short Term Memory (LSTM) for roughness classification. The inputs for the deep learning model are the image histogram, standard deviation, and mean values of the luminance of the image.

A CNN is a type of ANN developed from the idea that human visual nerves have partial connective structures. It consists of a convolution layer and a pooling layer, not a full-connected layer, making it different from a conventional ANN ^[18]. The convolution and pooling layers each have a role in extracting and compressing features from the input data, and this structure enhances the training of the model as it goes through the CNN layers.

Eq. (4) gives the output of a given neuron in a convolution layer:

where $\textit{z}$$_{i,j,k}$ is the output of the neuron located in row $\textit{i}$ and column $\textit{j}$ in feature map $\textit{k}$ of the convolution layer (layer $\textit{l}$). $\textit{s}$$_{h}$ and $\textit{s}$$_{w}$ are the vertical and horizontal strides, $\textit{f}$$_{h}$ and $\textit{f}$$_{w}$ are the height and width of the receptive field, and $\textit{f}$$_{n` }$ is the number of feature maps in the previous layer ($\textit{l}$-1). $\textit{x}$$_{i` ,j` ,k` }$ is the output of the neuron located in layer $\textit{l-1}$, row $\textit{i’}$, column $\textit{j’}$, feature map k’, $\textit{b}$$_{k}$ is the bias term, and $\textit{w}$$_{u,v,k` ,k}$ is the connection weight between any neuron in feature map $\textit{k}$ of the layer $\textit{l}$.

LSTM is a model that seeks to address the long-term dependency problems of existing Recurrent Neural Networks (RNNs) by introducing input, output, and forget gates, which change training structures ^[19]. Each gate determines specific operations, thereby finding the core of the learning data and allowing the model to remember its content longer. Fig. 9 shows a typical LSTM cell, and Eq. (5) summarizes its output at each time step.

Fig. 9. The detailed architecture of the composite CNN and bidirectional LSTM used in the study.

c$_{\mathrm{(t)}}$ and c$_{\mathrm{(t-1)}}$ are the long-term state at frame t and t-1, h$_{\mathrm{(t)}}$ and h$_{\mathrm{(t-1)}}$ are the short-term state at time t and (t-1), and x$_{\mathrm{(t)}}$ is the current input vector. Similarly, W$_{\mathrm{xi}}$, W$_{\mathrm{xf}}$, W$_{\mathrm{xo}}$, W$_{\mathrm{xg}}$, W$_{\mathrm{hi}}$, W$_{\mathrm{hf}}$, Who, and W$_{\mathrm{hg}}$ are the weight matrices of each of the four layers for their connection to the input vector x$_{\mathrm{(t)}}$ and previous short-term state g$_{\mathrm{(t-1)}}$ respectively. B$_{\mathrm{i}}$, b$_{\mathrm{f}}$, b$_{\mathrm{o}}$, and b$_{\mathrm{g}}$ are the bias terms for each of the four layers.

A wrapper layer known as $\texttt{Bidirectional()}$ was used in the model. Bidirectional LSTMs (or Bi-LSTMs) focus on the problem of obtaining the most out of the input sequence by stepping through input time steps in both the forward and backward directions ^[20]. Bi-LSTMs train two LSTMs instead of one LSTM on the input sequence.

In machine learning, many parameters control the learning process, which are known as hyperparameters ^[21]. Although there are numerous hyperparameters, only five were optimized in the proposed deep learning model. Before performing hyperparameter tuning, the search space was defined. The details of the parameters (name, minimum, maximum, steps, and default values) are presented in Table 5.

Hyperparameter tuning was carried out using the $\texttt{RandomSearch()}$ function in the Keras Tuner ^[22] library. The process for finding the best hyperparameters depends on the computing time and resources. In the current study, the search time was about nine hours on 2 ${\times}$ GeForce GTX 2080Ti FTW3 GPUs. The best accuracy achieved from the $\texttt{get\_best\_models()}$ function was 84.6%.

The CNN portion of the model used the $\texttt{Conv2D()}$ function for convolution operations, $\texttt{MaxPooling()}$ for dimensionality reduction, and $\texttt{Dropout()}$ to prevent the model from overfitting. The LSTM portion consists of a multilayered bi-directional LSTM cell and a $\texttt{Dropout()}$ to prevent overfitting. After concatenating CNN and LSTM, the $\texttt{softmax()}$ function was used in the last layer, and the $\texttt{categorical\_crossentropy}$ loss function was used because the study involves multiclass classification tasks. The detailed steps of model building and a comprehensive flow chart are shown in Fig. 10.

Although various numbers of epochs were tried, based on the minimum validation and training error, 30 epochs were used before saving the model in .h5 format. Overfitting occurred when training the model with over 35 epochs. The batch size was 4, and the RMSprop optimizer was used with an input image size of 480${\times}$480 pixels.