3.1 DenseNet-BC

DenseNet connects a layer directly to all subsequent layers, as shown in Fig. 1. The input of a layer in DenseNet is the concatenated output of all preceding layers in a dense block. For example, the n$^{\mathrm{th}}$ layer input in an m-layer dense block can be defined as follows. The Haskell operator symbol ``++'' was used for concatenation to simplify the notations.

where $(x_{0}++x_{1}++x_{2}++\ldots ++x_{n-1})$ is the concatenated output of the preceding layers, and F is a composite residual function ^[4]. Further information on Denseblock is reported elsewhere ^[4].

Fig. 1. Dense block as described in [4]. A layer in DenseNet receives input from all preceding layers in the network and concatenates its output with the input.

3.2 Dense Skip Connection

This paper proposes the use of a Dense skip connection to bring the information in the decoder path close to the input of the encoder path. A straightforward way is to connect a concatenated output of the encoder layer to the decoder layer, i.e., a skip connection between high-resolution dense blocks and low-resolution dense blocks. On the other hand, these skip connections require considerable computation and memory. The dense block output is the concatenated output of all the layers in the dense block, including the input of the dense block. Suppose the dense block output is used as an input to another dense block that converts the two dense blocks into a single dense block with an increased number of layers, resulting in higher memory and computation requirements. The output of a dense block that has a skip connection with another dense block can be obtained as follows:

where Bi is the output of block i; $x_{0}++x_{1}++\ldots ++x_{n}$is the concatenated output of block i with n number of layers. B$_{i-1}$ is the concatenated output of the previous dense block.

Fig. 2 shows how a skip connection between two dense blocks allows them to act like a single dense block, which requires more computation and memory resources. Fig. 2 (left) shows two dense blocks with two layers. The output of a dense block is the concatenated output of all the layers (L1, L2) in the dense block and is denoted as y. This output is fed to the other dense block, and every layer (S 1, S 2) in the dense block will receive y as an input. Y in Fig. 2 (left) is shown as the output of the dense block. Intuitively, the graphs in Fig. 2 (left) can be constructed as Fig. 2 (right), where Xo = x and the output Yo of the network will be the same as Y. Moreover, skip connections between the dense blocks are more complex than the skip connections between layers. The L1 norm of the trained weights (normalized by the input connections) of FC-DenseNet-103 was plotted to study the effect of skip connections between dense blocks (Fig. 3). The effects of the transition-up layer on every layer of the subsequent dense block were also plotted (Fig. 3, first row of the plot).

Fig. 2. Illustration showing how a concatenated input converts two disjointed dense blocks into one big dense block. x and y are the input and output of dense block L1, respectively, which has two convolutional layers. The output is again inputted to dense block L2, which has two convolutional layers. Xo is the input of a single dense block with four convolutional layers, and Yo is the output of this dense block. The output of two disjointed dense blocks with skip connections will be identical to that of a single dense block.

Fig. 3 shows the average heatmaps of the first (left) and last (right) dense block weights in the decoder path. The first dense block has 12 layers, while the skip connection carries the output of 12 layers and the block input. Each plot in Fig. 3 contains the average weights of the transition-up layer (first row only), skip connection and current dense block. A red pixel in Fig. 3 indicates that the target layer uses (on average) the source layer output. Fig. 3 (left) shows that the middle dense block is not used by the first dense block in the decoder path and does not show any change. Therefore, the FC-DenseNet architecture depends mostly on the feature maps of the corresponding encoder (the skip connection rows in Fig. 3 (left) show the weight changes). The first row in Fig. 3 (right) shows the dependence of the dense block on the transition-up layer as the weight change can be observed in the first row. On the other hand, the overall (average) weight of the layers of skip connection is much higher than that of the transition-up layer. Furthermore, the first column of the plots in Fig. 3 shows that the first layer of the dense block of the skip connection has most of the weight for this dense block. The other layers of this dense block do not receive a larger share in the overall weight, which proves the correlation of low-level features in the decoder path. The number of weights for the skip connection is much higher than the transition-up layer in the decoder path.

Fig. 3. L1 norm of the weights of a dense block with its skip connections. The rows of the plot contain a deconvolutional layer (first row), a skip connection (the encoder’s dense block), and the current dense block (the decoder’s dense block).

A bottleneck-dense skip connection was proposed to address the aforementioned issue. The last layer of the dense block of the encoder processes the feature maps of all preceding layers, and the output of the last layer of the dense block of the encoder can be used as a bottleneck skip connection. The outputs of the deconvolutional feature were concatenated with bottleneck skip connection features. The bottleneck skip connection will carry a k number of features where k is the growth rate of a dense block.

3.3 DenseNet to SenseNet

This paper proposes a new semantic image segmentation architecture that combines all the building blocks explained in the previous subsections (Fig. 4). The SenseNet architecture consists of the following: 1) the encoder path that learns the latent features using successive convolution dense blocks and transitions down layers, 2) the decoder path that has learnable dense blocks and transition up layers, and 3) bottleneck dense skip connections that carry context information from the encoder (high-resolution dense blocks) to the decoder (low-resolution dense blocks) and limits the network to expand the parameters. The transition down layer has a 1 ${\times}$ 1 convolutional layer followed by a max pooling layer so that the dimension of the input feature maps is reduced (Table 1). Furthermore, 1 ${\times}$ 1 is believed to be similar to a fully connected single neuron. It connects to all values in the input.

Fig. 4. Proposed SenseNet model with two dense blocks that takes an RGB image (three channels) as input and performs a convolutional operation using successive dense blocks. The encoder path of the proposed model has two blocks (both labeled as DB). There is a dense block between the encoder and the decoder. The last two blocks (both labeled as DB) constitute the decoder path. The decoder path first up-samples the feature maps using deconvolution operation and then concatenates the feature maps with the up-sampled output through skip connections. The bottleneck skip connections are shown by a dotted line.

After weighting, sum, and activation, it ends up with a value of one. In the SenseNet case, it will pool features of the feature maps of all dense skip connections and generate a new output, which is a combination of the inputs of all layers. The transition up layer uses the deconvolutional layer that gradually reverses the effects of convolution. The bottleneck skip connection output is concatenated with the transition-up layer output to obtain various input features for the subsequent dense block and to pass the context information from the encoder (high-resolution layers) to the decoder (low-resolution layers). The output of a deconvolutional dense block is obtained as follows:

where B$_{i}$ is the output of the i$^{\mathrm{th}}$ dense block and x$_{0}$, x$_{1}$, and x$_{n-1}$ are the outputs of the n layers in the i$^{\mathrm{th}}$ dense block. Note that B$_{i-1}$ is not concatenated with the output of the dense block, although every layer in the i$^{th}$ block will receive the output of the i${-$1}$^{th}$ block (B$_{i-1}$) as an input. Table 1 presents the architecture of the transition-down and transition-up layers. Table 2 lists the building blocks of SenseNet and shows the architecture of the bottleneck layer and composite function.

4.1 Experimental Setting

Dataset: CamVid ^[15] is a segmented video dataset for understanding urban scenes. The extracted frames from the CamVid dataset were used ^[13]. The frame was divided as follows: 367 frames for training, 101 frames for validation, and 233 frames for testing. Each frame was 360 ${\times}$ 480 in size, and its pixels were categorized into 11 semantic classes. SenseNet was trained on frames from CamVid, which were cropped randomly to 224 ${\times}$ 224. The images were normalized to the mean and standard deviation of the data. Data augmentation was used to generate a variety of inputs on which SenseNet was trained. It also reduced the need for regularization in the models. The images with a probability of 0.5 were flipped horizontally.

4.2 Training and Inference Details

As previously explained, SenseNet architecture was proposed for image segmentation. The architecture was implemented in Tensorflow with one GPU. The hyperparameters of the architecture are given as follows. SenseNet was trained using the RMSProp optimizer. The learning rate of RMSProp was set to 0.001 and reduced as follows:

where initial_lr$_{\mathrm{i}}$ is the initial learning rate, x is the current epoch, and z is the number of epochs after the learning rate is reduced. The learning rate will be reduced based on z. i and z were set to 1$^{\mathrm{e-3}}$ and 3, respectively. A dropout rate of 0.2 was used. The l2 norm and a weight decay of 1${\times}$10$^{-4}$ were used to regularize the model. A batch size of 2 was used. Batch normalization with a moving mean and variance was applied. Standard geometric transformations such as image flipping, cropping, and scaling were used as image augmentation for training the model.

4.3 Results and Analysis

Table 3 lists the main results of the SenseNet. Different configurations of layers can be applied for other tasks.

The evaluation results show that SenseNet outperformed the conventional models in terms of IoU. Table 3 lists the IoU and global accuracy scores of SenseNet. The performance of the SenseNet increased as the number of parameters increased. For the evaluation, SenseNet was trained and tested on images cropped randomly to 224 ${\times}$ 224 and on overlapping images cropped to 224 ${\times}$ 224. This section reports the results of three configurations of the SenseNet model: 1) SenseNet-78 (a growth rate of 12), 2) SenseNet-108 (a growth rate of 12), and SenseNet-abc (a growth rate of 16). SenseNet-78 and SenseNet-108 have four dense blocks, while SenseNet-abc has five dense blocks. A maximum of four images of 224 ${\times}$ 224 were cropped from one image from the dataset, and the SenseNet model was fine-tuned on these cropped images. The accuracy of SenseNet increases with the size of the dataset (Table 3).

Fig. 5 shows the error rate reduction and increase in accuracy of the multiple configurations of SenseNet. The decrease in the error rate demonstrates the high performance of SenseNet. The results on the CamVid dataset clearly show that SenseNet achieves state-of-the-art performance in terms of IoU. SenseNet does not require pre-training. Many of the semantic segmentation models listed in Table 3 were pretrained on larger datasets, such as ImageNet ^[22], to perform segmentation. The segmentation results on unrepresented classes in the dataset can be improved by addressing the class imbalance issue. SenseNet uses fewer parameters (Table 3) and outperforms FCDenseNet. Bottleneck skip connections reduce memory requirements and computational complexity. SenseNet outperforms FC-DenseNet in terms of IoU and achieves a similar accuracy to FC-DenseNet; SenseNet used only 2.3M (million) parameters, while FC-DenseNet-103 used 9.4M (million) parameters.

Fig. 5. Changes in the validation loss and validation accuracy of two SenseNet models (SenseNet-78 and SenseNet-108). The accuracy is indicated by a solid line, while the loss is indicated by a dotted line.

Multipath-DenseNet ^[16] showed that DenseNet makes several shorter paths with the dense block input in very deep neural networks because of the higher number of low-level feature maps. Longer skip connections make it more difficult for a network to learn in the decoder path. Moreover, the classes in the dataset are very imbalanced. Addressing the class imbalance of the data in CamVid would improve the performance of SenseNet. The results of the implementation of FC-DenseNet are presented. The results of FCDenseNet 56, large configurations of FC-DenseNet (67 and 103) were not implemented due to computational resource constraints. The results in Table 3 of FC-Densenet show that the classes where the number of instances and pixels were higher have very high accuracy, and lower accuracy is observed in unrepresentative classes. This study did not find information about handling the class imbalance reported elsewhere ^[14].