Robustness of Generative Adversarial CLIPs Against Single-Character Adversarial Attacks in Text-to-Image Generation

Abstract

Generative Adversarial Networks (GANs) have emerged as a powerful type of generative model, particularly effective at creating images from textual descriptions. Similar to diffusion models, GANs rely on text encoders to extract embeddings from these descriptions. However, this reliance introduces specific vulnerabilities to adversarial attacks. A notable example is a single-character adversarial attack, where altering a single character in the text description can lead to significant performance degradation in the generated image quality and model’s performance. In this study, we systematically evaluate the susceptibility of GANs to such attacks using Generative Adversarial CLIP (GALIP), a single-stage architecture that leverages a pre-trained Contrastive Language-Image Pre-training (CLIP) text encoder for text embeddings. We meticulously selected captions with single-character modifications that exhibit maximum and median-distance embeddings for the attack. Experimental results show up to 310.5% degradation in Fréchet Inception Distance (FID) scores, underscoring the importance of developing improved defenses in text-to-image synthesis.

Keywords

• Single-character attack
• GALIP
• GAN
• CLIP text encoder
• Text-to-image generation