2.1. Semantic Mask Generation

For semantic mask generation, we utilized DDPM. DDPM generates images through a forward process and a reverse process. The forward process involves adding Gaussian noise to the original image $x_0$ at each timestep $t$ based on a variance schedule ${\beta}_t$. The probability distribution at each timestep is defined as follows:

Here, $x_t$ is the image at $t$, $\mathcal{N}$ denotes the Gaussian distribution, and $I$ represents the identity matrix. Using the notation ${\alpha}_t{}:=\prod^t_{s=1}{{}(1-{\beta}_s{})}$, we can express the marginal distribution as follows:

The reverse process is the opposite of the forward process, gradually restoring the original image from the noise-added image according to each timestep. The probability distribution $p_{\theta} (x_{t-1}\mid x_t)$ at $t$, based on the mean vector ${\mu}_{\theta}(x_t,\ t)$ and the covariance matrix ${{\Sigma}}_{\theta}(x_t,\ t)$, is defined as follows:

DDPM's unconditional generation creates various semantic masks from random noise, enabling the generation of diverse objects and scenes without limitation. This allows the SDM to generate new scenes not present in the existing dataset. Generating a wide range of scenes is crucial, as it directly improves the performance of semantic segmentation models.

For unconditional generation, semantic masks are extracted from the target dataset and used to train the DDPM. Since training with grayscale semantic masks limits the model's expressive range, the semantic mask training is conducted in the RGB domain. The use of RGB representation for semantic masks is important because a broader range of representation helps distinguish between classes more clearly during the subsequent color clustering process. This is also related to the pixel value inconsistency problem in generative models, as the semantic masks generated by the model may have non-uniform pixel values, unlike real semantic masks. Therefore, a color clustering process is necessary to unify pixel values for each class. Only after color clustering is completed can the semantic mask be ready for use as a condition in the SDM, and later be combined with synthetic images to form a synthetic dataset.

2.2. Synthetic Image Generation

SDM generates a synthetic image $y_0$ with the semantic mask generated by DDPM as the condition $x$. To achieve this, the SDM is trained using both real images and semantic masks from the target dataset. SDM follows the forward and reverse processes based on the DDPM framework. Here, $y_t$ represents the image at $t$, and the forward process is the same as in DDPM. The reverse process $p_{\theta}(y_{0:T}\mid x)$ proceeds through a Markov chain and is defined as follows:

Here, ${\mu}_{\theta}(y_t,x,t)$ and ${{\Sigma}}_{\theta}(y_t,x,t)$ represent the mean vector and covariance matrix at timestep $t$ conditioned on $x$, which are learned through the SDM, similar to DDPM. Note that SDM is a U-Net-based network that predicts noise from a noisy input image. The noisy input image $y_t$ is processed by the encoder, while the semantic mask $x$ is processed by the decoder, maximizing the characteristics of each input. Multi-layer spatially-adaptive normalization operators are employed to further improve image generation quality. Additionally, classifier-free guidance ^[17] is used during the sampling process to ensure high consistency with the semantic label map.

Therefore, the synthetic images generated by SDM achieve perfect consistency with the semantic masks generated by DDPM. This eliminates the need for manual annotation and allows the creation of high-quality images that do not exist in the target dataset. As a result, this can contribute to improving the performance of semantic segmentation.

3.1. Datasets

We constructed an LWIR dataset for the experiment by collecting indoor video data at a university and a karaoke room with permission, using LWIR imaging equipment. The videos were captured at a resolution of 960x600 in grayscale. Noisy videos that made it difficult to identify the indoor structure or objects were excluded, and manual annotation was performed on the remaining videos, resulting in a dataset of 2,650 images. Since our goal is to detect exits in smoke-filled indoor environments due to fires, we annotated only three categories: door, floor, and background. In the semantic mask, the door is represented in light blue and the floor in yellow. Of the 2,650 images, 2,400 were used for training and 250 for validation.

The LWIR dataset, with only three classes, is relatively easier to generate. To further validate our approach, we tested on RGB domain datasets with more classes, using Cityscapes ^[15] and ADE20K ^[14]. Like the LWIR dataset, we assumed limited data availability, using only fine annotations from Cityscapes, which includes 2,975 training images, 500 validation images, and 19 classes. The ADE20K-Street dataset, a subset focused on street scenes, has 2,038 training images, 203 validation images, and 64 classes.

3.2. Qualitative Results

Fig. 2 presents the qualitative results of Noise-to-Dataset. Looking at Fig. 2(a), as seen in the first and second rows, when only one class is included in the generated mask, synthetic images that effectively represent the texture and color characteristics of LWIR images are well generated. Even in the third row, which shows the results of a more complex generated mask, it can be observed that not only the door and floor but also the background areas are well generated. However, there are also bad cases, which mostly occur due to incorrectly generated masks. Inappropriate shapes, positions, and sizes of the door/floor classes in the generated mask lead the SDM to produce unrealistic synthetic images. Moreover, in the case of the LWIR dataset, since there are only three classes, the number of possible masks that can be generated is limited, leading to a lack of diversity in the generated mask patterns.

In the case of Cityscapes and ADE20K-Street, since DDPM learns 19 and 64 classes respectively, the generated masks have far more variations compared to the LWIR dataset. Note that as the number of classes increases, the likelihood of generating masks not present in the original dataset also increases. Upon analysis, we confirmed that most generated masks, such as those in Figs. 2(b) and 2(c), do not exist in the original dataset, resulting in the creation of synthetic images from a wide range of scenes.

Fig. 2. Qualitative results of Noise-to-Dataset: (a) Synthetic LWIR dataset. (b) Synthetic Cityscapes dataset. (c) Synthetic ADE20K-Street dataset.

3.3. Quantitative Results

To quantitatively evaluate the effectiveness of the synthetic dataset, we adopt mean intersection-over-union (mIoU), a metric commonly used for semantic segmentation performance evaluation. For this experiment, we use the SegFormer model ^[18], the Segmenter model ^[19] and the SegNeXt model ^[20]. First, we observe the performance changes according to the size of the synthetic dataset added to the training dataset (see Table 1). In the case of the SegFormer model with a MiT-B4 backbone (SegFormer-B4), training with only 2,400 real data achieves a performance of 78.14%, while training with a total of 5,400 data, including an additional 3,000 synthetic data, shows a performance of 79.63%, resulting in a gain of $+1.49%$. Meanwhile, SegFormer-B2 shows a relatively smaller gain of $+0.90%$, indicating that the larger the backbone, the greater the improvement. Furthermore, in experiments using the Segmenter model with a ViT-S backbone (Segmenter-S), a maximum gain of $+2.05%$was obtained.

On the other hand, in the case of SegFormer-B2, when comparing the addition of 2,000 synthetic data with 3,000 synthetic data, we observe negligible mIoU improvement, i.e., 77.62% vs. 77.64% (+0.02%). A similar phenomenon was observed with the Segmenter-T model. While the size of the backbone may have some influence, this is mainly attributed to the characteristics of our LWIR dataset, which consists of only three classes. In fact, qualitative results show that due to the limited variety of classes, the diversity of the generated masks is also insufficient. As the number of synthetic datasets increases, there is a saturation of scene patterns, which does not contribute significantly to performance improvement.

Therefore, we conducted experiments on RGB-scale datasets with a larger number of classes. Tables 2 and 3 verify the effectiveness of the synthetic dataset on the Cityscapes dataset and the ADE20K-Street dataset, respectively. Comparing the performance before and after including the synthetic dataset, we observed a maximum gain of +0.93% with the Segmenter-S model on the Cityscapes dataset, and a maximum gain of +1.09% with the Segmenter-S model on the ADE20K-Street dataset. This demonstrates that Noise-to-Dataset is also effective on standard RGB datasets.

3.4. Ablation Study

Table 4 compares the effectiveness of using real masks from an existing dataset versus generated masks when synthesizing images. For this experiment, the Segmenter-S model was used. Additionally, 2,000 synthetic images generated from real masks and 2,000 synthetic images generated using the proposed mask generation method were used for training, ensuring a fair comparison by maintaining the same data quantity. The experimental results show that training with synthetic images generated from real masks achieved a performance of 79.13%, whereas training with synthetic images generated using the proposed mask generation method achieved 80.14%, resulting in a performance improvement of +1.01%. These results highlight that semantic mask generation not only serves to expand the dataset but also enhances the segmentation model's performance by introducing new variations that do not exist in the original dataset. Furthermore, it demonstrates that the generated masks contain structurally meaningful information beneficial to the segmentation model.

3.5. Experimental Details

We evaluated the performance of semantic segmentation using three models from the MMSegmentation framework: SegFormer, Segmenter, and SegNeXt. The experiments were conducted in an environment with Python 3.8 and PyTorch 1.11.0, utilizing CUDA Toolkit 11.3 for GPU acceleration. The implementation was based on MMSegmentation, with dependencies including MMEngine 0.7.3 and MMCV 2.0.1. For computational resources, we used an NVIDIA RTX A6000 GPU and an AMD EPYC 7413 CPU. This setup provided a stable environment for training and evaluating the models while ensuring compatibility with the MMSegmentation framework.