A Novel Attention-Based Multi-Modal Modeling Technique on Mixed Type Data for Improving TFT-LCD Repair Process

Abstract

In Thin-Film Transistor Liquid-Crystal Display (TFT-LCD) manufacturing, conducting a machine learning based system with multiple data types has become actively desired to solve complicated problems. This paper proposes a multi-modal learning approach: TabVisionNet, which is modeled by utilizing the information from both tabular data and image data. A novel attention mechanism called Sequential Decision Attention was integrated into the multi-modal modeling framework that improves the comprehension of the information from two modalities. This cross-modal attention mechanism can capture the complex relationship between modalities then gain better generalization and faster convergence in the training process. Conducting an experiment, the performance of our novel approach was significantly better than single-modal and other multi-modal learning approaches in our real case scenario.

Keywords

• Smart manufacturing
• TFT-LCD
• deep learning
• multi-modality machine learning
• tabular data
• image data and attention mechanism