Performance evaluation of DNN with other machine learning techniques in a cluster using Apache Spark and MLlib

Abstract

Sentiment analysis on large data has become challenging due to the diversity, and nature of data. Advancements in the internet, along with large data availability have obviated the traditional limitations to distributed computing. The objective of this work is to carry out sentiment analysis on Apache Spark distributed Framework to speed up computations and enhance machine performance in diverse environments. The analysis, such as polarity identification, subjective analysis and email spam etc., are carried on various text datasets. After pre-processing, Term Frequency-Inverse Document Frequency (TF-IDF) and unsupervised Spark-Latent Dirichlet Allocation (LDA) clustering algorithms are used for feature extraction and selection to improve the accuracy. Deep Neural Networks (DNN), Support Vector Machines (SVM), Tree ensemble classifiers are used to evaluate the performance of the framework on single node and cluster environments. Finally, the proposed work aims at building an approach for enhancing machine performance, more in terms of runtime over accuracy.

Keywords

• Sentiment analysis
• Classification
• Text mining
• Big data
• Distributed computing
• Apache Spark framework