Latent Space Expanded Variational Autoencoder for Sentence Generation

Abstract

Sentence generation is a key task in many natural language processing systems. Models based on a variational autoencoder (VAE) can generate plausible sentences from a continuous latent space. However, the VAE forces the latent distribution of each input sentence to match the same prior, which results in a large overlap among the latent subspaces of different sentences and a limited informative latent space. Therefore, the sentences generated by sampling from a subspace may have little correlation with the corresponding input, and the latent space cannot capture rich useful information from the input sentences, which leads to the failure of the model to generate diverse sentences from the latent space. Additionally, the Kullback-Leibler (KL) divergence collapse problem makes the VAE notoriously difficult to train. In this paper, a latent space expanded VAE (LSE-VAE) model is presented for sentence generation. The model maps each sentence to a continuous latent subspace under the constraint of its own prior distribution, and constrains nearby sentences to map to nearby subspaces. Sentences are dispersed to a large continuous latent space according to sentence similarity, where the latent subspaces of different sentences may be relatively far away from each other and arranged in an orderly manner. The experimental results show that the LSE-VAE improves the reconstruction ability of the VAE, generates plausible and more diverse sentences, and learns a larger informative latent space than the VAE with the properties of continuity and smoothness. The LSE-VAE does not suffer from the KL collapse problem, and it is robust to hyperparameters and much easier to train.

Keywords

• Latent space
• natural language processing
• sentence generation
• variational autoencoder