Generating visual explanations with natural language

Abstract

Abstract We generate natural language explanations for a fine‐grained visual recognition task. Our explanations fulfill two criteria. First, explanations are class discriminative, meaning they mention attributes in an image which are important to identify a class. Second, explanations are image relevant, meaning they reflect the actual content of an image. Our system, composed of an explanation sampler and phrase‐critic model, generates class discriminative and image relevant explanations. In addition, we demonstrate that our explanations can help humans decide whether to accept or reject an AI decision.

Keywords

• computer vision
• explanations
• fine‐grained classification
• natural language