Developing an Algorithm for Fast Performance Estimation of Recurrent Memory Cells

Abstract

We propose a novel graph-oriented machine learning algorithm which we use for estimating the performance of a recurrent memory cell on a given task. Recurrent neural networks have been successfully used for solving numerous tasks and usually, for each new problem, generic architectures are used. Adapting the architecture could provide superior results, but would be time-consuming if it would not be automated. Neural architecture search algorithms aim at optimizing the architectures for each specific task, but without a fast performance estimation strategy it is difficult to discover high-quality architectures, as evaluating each candidate takes a long period of time. As a case study, we selected the task of sentiment analysis on tweets. Analyzing the sentiments expressed in posts on social networks offers important insights into what are the opinions on different topics and this has applications in numerous domains. We present the architecture of the estimation algorithm, discussing each component. Using this algorithm, we were able to evaluate one million recurrent memory cell architectures and we discovered novel designs that obtain good performances on sentiment analysis. We describe the discovered design that obtains the best performances. We also describe the methodology that we designed, such that it can be applied to other tasks.

Keywords

• Neural architecture search
• graph neural network
• performance estimation strategy
• recurrent neural network
• sentiment analysis
• tweet