关键词: 文本生成图像, 生成对抗网络, 多模态任务

Abstract: Text-to-image generation aims to generate realistic images from natural language descriptions, and is a cross-modal analysis task involving text and images. In view of the fact that the generative confrontation network has the advantages of realistic image generation and high efficiency, it has become the mainstream model for text generation image tasks. However, the current methods often divide text features into word-level and sentence-level training separately, and the text information is not fully utilized, which easily leads to the problem that the generated image does not match the text. In response to this problem, this paper proposes an image confrontation cascade generation model (Union-GAN) that couples word-level and sentence-level text features, and introduces a text-image joint perception module (Union-Block) in each image generation stage.  By combining channel affine transformation and cross-modal attention, it fully utilizes the word-level semantic and overall semantic information of the text to generate images that not only match the text semantic description but also maintain clear structures. Meanwhile, jointly optimizing the discriminator and adding spatial attention to the corresponding discriminator allows the supervisory signal from the text to prompt the generator to generate more relevant images. Compared with multiple current representative networks such as AttnGAN on the CUB-200-2011 dataset, experimental results show that the FID score of our Union-GAN is 13.67, an increase of 42.9% compared to AttnGAN, and the IS score is 4.52, an increase of 0.16.

Key words: text-to-image generation, generative adversarial network(GAN), multimodal task