2.1 Twin Network

A twin network is a neural network that contains two or more sub-networks ^[11]. The network inputs multiple images and extracts feature vectors by processing images separately through each sub-network. Then, it computes the distance of extracted feature vectors to measure similarity between the input images ^[11]. The main advantage of a twin network is sharing weights between sub-networks ^[11]. By sharing the network weights, a twin network can identify whether similar features exist, so it can measure similarity between multiple images. Recently, a twin network composed of a convolutional encoder-decoder structure has been applied for various applications, such as video object segmentation ^[20] and change detection ^[21-23]. These methods also share the weights between encoders ^[20-23].

As shown in Fig. 2, the proposed network consists of two encoders and one decoder, and the weights in the two encoders are shared. This network takes the golden and test die images, extracts the feature maps from sub-encoders, and concatenates extracted feature maps. The merged feature maps are used as the decoder input, and the network finally outputs the different areas between the golden and test die images as defects. To detect small defects accurately, we also apply a difference skip-connection ^[24], which computes the absolute difference of the feature maps between encoder-convolution layers and then transfers the difference values to the decoder, as shown in Fig. 2.

2.2 Bayesian Twin Network

A Bayesian neural network models network weights as random variables instead of fixed values and computes the posterior distribution of network weights given training data. Using a posterior distribution, the network may predict a target value of test data by marginalizing over network weights $\mathbf{W}$ given training data $\mathbf{X}$ and its corresponding label set $\mathbf{Y}$, which is defined as follows ^[15]:

Fig. 2. Twin network-based wafer die-to-die inspection system.

where $\boldsymbol{x}^{\boldsymbol{*}}$ is test data, $\boldsymbol{y}^{\boldsymbol{*}}$ is the prediction for the test data, and $p\left(\mathbf{W}|\mathbf{X},\mathbf{Y}\right)$ represents the posterior distribution of network weights. The network weights $\mathbf{W}$ are composed of $L$ layers $\left\{\mathbf{W}_{i}\right\}_{i=1}^{L}$, where $\mathbf{W}_{i}$ is a matrix of dimensions $K_{i}\times K_{i-1}$, and $K_{i}$ is the number of units for each layer $i$.

In the prediction stage, the integral above is not tractable, so it is usually approximated by Monte Carlo integration using samples drawn from the posterior distribution. Also, it is impractical to compute the posterior distribution exactly, so one needs to approximate the posterior distribution by applying variational inference techniques ^[16]. These techniques approximate the posterior distribution to some variational distribution $q\left(\mathbf{W}\right)$, from which samples can be easily drawn. This is done by minimizing the Kullback-Leibler (KL) divergence between $q\left(\mathbf{W}\right)$ and the posterior distribution. The KL divergence is defined as follows ^[15]:

A previous study showed that a model trained with dropout is an approximate Bayesian network by defining the variational distribution $q\left(\mathbf{W}_{i}\right)$ for every layer $i$ with units $j$ as follows ^[15]:

where $\mathrm{b}_{i,j}$ is a Bernoulli-distributed random variable with probability $p_{i}$, and $\mathbf{M}_{i}$ is a variational parameter to be optimized. The integral in Eq. (2) can be approximated by summing over Monte-Carlo samples drawn from the variational distribution $q\left(\mathbf{W}\right)$ as follows ^[15]:

where $N$ is the number of data, $T$ is the number of Monte Carlo samples, and $~ \hat{\mathbf{W}}_{n,t}$ is a Monte Carlo sample from the variational distribution $q\left(\mathbf{W}\right).$

The first term approximated with a single Monte Carlo sample is identical to the cross-entropy loss of a model trained with dropout ^[15]. In addition, the second term can be approximated as $L2$ regularization ^[15]. Therefore, minimizing a loss function composed of the cross-entropy loss and $L2$ regularization is approximately equivalent to minimizing KL divergence ^[15].

We trained the twin network with dropout for the Bayesian twin network. After training, we sampled the network weights 50 times from the approximate posterior distribution using dropout. We used the mean of these samples as our prediction and used variance to measure model uncertainty.

4.1 Training and Inference

The twin network had two encoders with shared weights and one decoder, as shown in Fig. 2, while the conventional method had one encoder and a corresponding decoder. The encoder networks of all methods had four convolutional layers and max-pooling layers. The decoder networks of all methods had four transposed convolutional layers and five convolutional layers.

We used batch normalization and activation layers after every convolutional layer and transposed convolutional layer. All models used the same number of filters and filter sizes. The numbers of filters in each encoder-convolutional layer were 16, 32, 64, and 128, and those in each decoder-convolutional layer were 128, 64, 32, 16, and 2. The filter size was $3\times 3$.

We implemented all methods using the PyTorch library ^[25] and an NVIDIA GeForce GTX 1070 GPU (Nvidia Corporation, USA). In addition, all models were trained using the Adam optimizer with an initial learning rate of $1\times 10^{-3}$, a batch size of 64, and a weight decay of $1\times 10^{-5}.$ We also randomly selected 20$\% $ of the training data as validation data in every epoch to monitor the performance of the models and used the validation data for early stopping. For the Bayesian twin network, we inserted dropout after every max-pooling layer of the encoders except for the first layer and after every convolutional layer of the decoder except for the last two layers. We applied dropout with a rate of 0.1 and obtained 50 Monte Carlo samples at the test stage.

4.2 Performance Measurement

We used precision and recall to measure the performance of the models and computed them in terms of the number of connected objects in the prediction images obtained by thresholding softmax scores. We define precision as follows:

where true positive ($TP$) indicates the number of connected objects correctly inspected as defects by comparing the input sub-die images. $FP$ is the number of connected objects incorrectly inspected as defects (false positives). We define recall as follows:

where $FN$ denotes the number of actual defects not detected (false negatives). Higher precision and recall indicate better performance.

4.3 Experimental Results

We evaluated the performance of the conventional method and the proposed method using the patterned wafer data. To ensure a fair comparison, we set the threshold individually for each method. The recall and precision of each network are reported in Tables 1 and 2.

Table 1 summarizes the precision of each method when the recall of each method is the same. The twin network shows better performance than the conventional method in the sense that it exhibited higher precision when the recall is the same. The precision of the twin network was about 0.39 percentage points higher than that of the conventional method.

We also compared the performance of the Bayesian twin network and the twin network to confirm whether Bayesian learning could improve the performance of the twin network. The Bayesian twin network achieved the best performance, and its precision was about 1.85 percentage points higher than that of the twin network.

Table 2 summarizes the recall of each method when the precision of each method is very similar. As shown in the table, the recall of the twin network was about 0.47 percentage points higher than that of the conventional method. In addition, the recall of the Bayesian twin network was about 0.33 percentage points higher than that of the twin network.

Tables 1 and 2 confirm that the twin network shows better performance than the conventional method. We think that it is because the twin network focuses more on image comparison than the conventional method. The conventional method merges the two sub-die images and treats them as different color channels. On the other hand, the twin network processes the two sub-die images separately through each encoder with shared weights and then merges the two branches. We think that the shared weights allow the twin network to more effectively identify whether similar features exist in the two input images.

We also confirmed that Bayesian learning could improve the performance of the twin network. We suspect that the reason is that Bayesian learning may prevent overfitting through marginalization over the network weights. Moreover, Bayesian learning may measure uncertainty of a trained model, as shown in Fig. 6. We think that providing the uncertainty information of the model predictions can be helpful to improve the trust of the twin network-based wafer die-to-die inspection system.

Fig. 6. Bayesian twin network prediction results

We also evaluated the performance of all methods using identical pair-set images to further verify that the structure of the twin network focuses more on image comparison. Table 3 shows the test results of each network using identical pair-set images. Identical pair-set images are composed of the same sub-die images, so comparison networks should detect no defects.

As shown in Table 3, the twin network and the Bayesian twin network correctly inspected all identical pair-set images. On the other hand, the number of false positives of the conventional method was 1,722. Fig. 7 shows the prediction results of each method using identical pair-set images. Table 3 and Fig. 7 confirm that the structure of the twin network is more suitable for die-to-die inspection.

Fig. 7. Prediction results of the conventional method, twin network, and Bayesian twin network using identical pair-set images.

Table 4 shows the inference time of each method. This was computed using the same hardware described in Section 4.1. The inference time of the conventional method was about 8.04 milliseconds shorter than that of the twin network. We think it is because the structure of the twin network is composed of two encoders, whereas the network structure of the conventional method is composed of one encoder. As expected, the Bayesian twin network was the slowest because it requires multiple inferences at test time. Although the Bayesian twin network requires more computation than other methods, we think it is useful because it could improve the performance of the twin network and measure model uncertainty. One may conceive of a method to reduce the amount of computation while maintaining the advantages of Bayesian learning, which is a subject for future study.