Performance Analysis of the Distributed Support Vector Machine Algorithm Using Spark for Predicting Flight Delays

Abstract

In big data analysis requires powerful machine learning frameworks, strategies, and environments to analyze data at scale. Therefore, Apache Spark is used as a cluster computing framework to process big data in parallel and can run on multiple clusters. In this study, the Support Vector Machine (SVM) algorithm is used as a classification method to predict whether a flight will experience a delayed arrival. This study also aims to analyze the performance of the distributed SVM algorithm using the Apache Spark framework in classifying delayed flight arrivals. Running time evaluation is important in proving how fast the data processing is done using Apache Spark. In addition, there is a test to prove the effect of using the SVM algorithm with a distributed system on the results of the classification accuracy of delayed flight arrivals. Distributed SVM performance testing is carried out using variations in data size and the number of worker nodes in the built cluster. From the test results, it was found that the most effective number of worker nodes used in the flight delay classification process was 4 worker nodes with the lowest running time results from the experiment of 4 variations in the number of worker nodes. In terms of accuracy, adding the number of worker nodes does not affect the accuracy of the program. The difference in the accuracy of results is caused by the random oversampling process on the data performed on each system test. Using the SVM algorithm with Spark is sufficient to provide good performance in the classification process with the highest accuracy result in the test being 93.98 %.