大数据 (Jan 2017)
Design scheme of massive traffic data real-time access based on HBase and ElasticSearch
Abstract
Traffic data has the characteristics of massive and real-time,and its massive data acquisition,storage and retrieval has become a key issue in the vehicle remote monitoring platform.According to the study of these problems,the cluster technology of LVS was used to solve the data acquisition load balance,the queue cache model was used to solve I/O delay,and HBase distributed data storage scheme was used to solve the massive data storage.HBase integration ElasticSearch,which was aimed to solve the real-time online data processing problems of Hadoop,was designed to build a hierarchical index.Through the design and implementation of the key technologies,the number of vehicle monitoring had been promoted from 400 to 1 million,online query speed increased about 10 to 20 times based on PB level data.The results verified the efficiency of the scheme.
