Direct Video Frame Interpolation With Multiple Latent Encoders

Abstract

We present a simple but effective video interpolation framework that can be applied to various types of videos including conventional videos and 360° videos. Our main idea is to predict the latent feature of an intermediate frame, through the latent feature encoders between encoder and decoder networks, without explicitly computing optical flow or depth maps. The latent feature encoders take latent features of input images and then predict the latent feature of a target image, i.e. an intermediate frame. Afterward, the decoder network reconstructs the target image from the latent feature. The proposed framework consists of fully convolutional networks, and it is therefore end-to-end trainable from scratch without requiring additional information except for consecutive frames. We experimentally verify the superiority of proposed method by comparing it to state-of-the-art methods with various types of datasets. Moreover, an ablation study is carried out to analyze the key components of the proposed method. Our proposed method performs interpolation in latent domain, it is advantageous to apply various video interpolation (e. g. NIR and depth videos) without limiting the type of input data.

Keywords

• Video interpolation
• 360° video interpolation
• latent space learning
• convolutional neural network