Automatic Chart Understanding: A Review

Abstract

Automated chart analysis has vast potential to improve the accessibility of charts for a wider audience, e.g., people with visual impairments or other disabilities, by generating captions for chart images that can quickly convey the information being represented. Additionally, it can improve the performance of automatic document analysis systems, by enabling them to extract valuable information from the documents with graphical/visual scientific content. Although recent advancements in modality translation and multi-modal learning have led to the development of more or less successful image captioning and visual question answering methods, but most of them have been designed for general images, and cannot be successfully applied to specific areas such as medical images or scientific charts and graphs. Therefore, further research is necessary to develop automated chart analysis methods that can be effectively applied to these specific areas. In this paper, a comprehensive review of chart analysis methods is presented. The review covers a wide range of chart types, including line charts, bar charts, scatter plots, and includes an in-depth analysis of each method. Additionally, this paper provides a more extensive coverage of chart analysis methods compared to previous studies, making it a valuable resource for researchers and practitioners in the field. Various techniques can be categorized from different aspects, such as chart type, model architecture, learning algorithm, visual feature space, and language modeling. In this paper, different methods are classified from a more technical viewpoint, by considering the approach used for modeling the problem. A taxonomy is proposed which divides the methods into three major categories: rule-based, chart captioning, and chart question- answering approaches. The rule-based approach uses the classical knowledge representation methods for reasoning, which has been diminished by the emergence of deep learning models. Chart captioning provides a general summary of the information conveyed by a chart through recent modern learning methods but may miss some detailed information which may be of special interest. On the other hand, the question answering allows for a direct response to a more specific user question by combining image analysis and text understanding techniques. Finally, the existing challenges and the potential research directions of the interesting chart understanding problem are discussed.

Keywords

• Automated chart analysis
• image captioning
• visual question answering
• modality translation
• multi-modal learning