Portrait Reification with Generative Diffusion Models

Abstract

An application of Generative Diffusion Techniques for the reification of human portraits in artistic paintings is presented. By reification we intend the transformation of the painter’s figurative abstraction into a real human face. The application exploits a recent embedding technique for Denoising Diffusion Implicit Models (DDIM), inverting the generative process and mapping the visible image into its latent representation. In this way, we can first embed the portrait into the latent space, and then use the reverse diffusion model, trained to generate real human faces, to produce the most likely real approximation of the portrait. The actual deployment of the application involves several additional techniques, mostly aimed to automatically identify, align, and crop the relevant portion of the face, and to postprocess the generated reification in order to enhance its quality and to allow a smooth reinsertion in the original painting.

Keywords

• diffusion models
• image generation
• embedding
• reification
• denoising