A Hybrid Deep Implicit Neural Model for Sentiment Analysis via Transfer Learning

Abstract

We present a neural model for sentiment analysis of social network texts with a special focus on cryptocurrency-related content using deep transfer learning. A challenge of deep learning is its need for abundant data. Therefore, we use relation-based transfer learning to analyze low-volume sentiment on SemEval public data. We use the pre-trained BERT-Base model to extract features from the source domain dataset, which is very similar to the extracted tweets of cryptocurrencies. We extract the tweets of expert influencers in cryptocurrency, referred to as the target domain. The extracted features are then injected into an implicit neural network in the target domain. The implicit neural network (INN) was recently designed to work on continuous data such as photos and videos and has not yet been used on text data. Our results show that using implicit neural networks in the text has synergistic effects due to its ability to process continuous and intermittent data. In addition, using rich data in the source domain has caused our proposed model to achieve accuracy of 88% and a loss rate of about 3% on the SemEval data, which is 3% improvement compared to the state-of-the-art. We use the time decay method to slowly change the learning rate in the Adaptive Moment Estimation weighted optimization algorithm. In our proposed model, this technique has helped to reduce the number of neural network cycles by at least 10% to reach convergence.

Keywords

• Sentiment analysis
• transfer learning
• pre-trained neural networks
• implicit neural network