Semantic segmentation mask-guided diffusion models: A pathway to enriched datasets in autonomous systems

Abstract

In the autonomous vehicle industry, deep learning models are critically dependent on the balance and variety of training data. Achieving this balance is particularly challenging due to the scarcity of data in rare scenarios, such as unique weather conditions or specific traffic configurations. Deep learning-based methods, particularly those within the emerging field of generative artificial intelligence (AI), hold potential for advanced solutions. A key development in this domain is the diffusion-based approach, capable of generating images from a random noise distribution. Predominantly, these models utilize a 'text2image' methodology, enabling the generation of images with text prompts. However, despite their advanced capabilities, these models do not yet provide complete explicit control over the generated content, particularly in terms of the relative positioning of objects within images. This research explores the use of a semantic segmentation-based control mechanism within a generative diffusion model, focusing on its application to the automotive domain. With the integration of this mechanism, the model facilitates the creation of diverse and contextually relevant self-driving scene setups, thus enriching the datasets used for comprehensive training in autonomous vehicles. In addition to assessing the quality of generation, the impact of these enriched datasets was also evaluated using a semantic segmentation network, which is essential for Advanced Driver-Assistance Systems (ADAS). The study compares the network's performance when trained on the original dataset versus an augmented one that includes model-generated images. The evaluation highlights the practical benefits of applying semantic segmentation guidance in this specific domain.