An efficient predictive analytics system for high dimensional big data

Abstract

The excessive growth of high dimensional big data has resulted in a greater challenge for data scientists to efficiently obtain valuable knowledge from these data. Traditional data mining techniques are not fit to process big data. Predictive analytics has grown in prominence alongside the emergence of big data. In this paper, an efficient predictive analytics system for high dimensional big data is proposed by enhancing scalable random forest (SRF) algorithm on the Apache Spark platform. SRF is enhanced by optimizing the hyperparameters and prediction performance is improved by reducing the dimensions. The effectiveness of the proposed system is examined on five real-world datasets. Experimental results demonstrated that the proposed system achieves the highly competitive performance compared with RF algorithm implemented by Spark MLlib.

Keywords

• Big data
• High dimensionality
• Predictive analytics
• Scalable random forest
• Dimension reduction
• Parameter optimization