A Design of ETL for the Construction of Traffic Network Based on Big Data

Abstract

Intelligent traffic management system based on big data is the trend of the development of the transportation system. It integrates information technology, wireless communication technology, computer technology, sensor technology and other advanced technologies to form a comprehensive and efficient integrated traffic management system. Traffic system based on big data needs to deal with a large number of unstructured data and semi-structured data. Also, the capacity of the data becomes larger, the data grows faster, and the format of the data becomes more complex. Traditional ETL technology has been unable to meet the needs of the construction of intelligent traffic network which is based on big data. According to the characteristics of the traffic network based on big data, this paper designs a kind of ETL system with high universality and high data processing efficiency. First, in order to improve the efficiency of data processing, we optimize the workflow of ETL. In order to make ETL suitable for big data traffic network environment, we redesign the ETL data processing rules by identifying and merging. And then we optimize the extracting, transforming and loading of the ETL system. Finally, the experimental results show that the redesigned ETL system can effectively serve the traffic network system based on big data. This method has a high efficiency in processing complex data structure and large data capacity of big data.