Architecture design of coal mine data center based on Hadoop

Abstract

According to business requirements of coal mine safety production and development requirements of smart mine, the new coal mine data center needs to meet efficient processing, caching, calculation, storage and release of massive and multivariate data of coal mine under the same space-time coordinate system. In view of problems of discrete data storage and difficulties in data integration, business application and data analysis of traditional coal mine data center, an architecture of coal mine data center based on Hadoop was designed. Real-time data stream engine storm is adopted for real-time data calculation, and MapReduce and Spark are applied to implement batch processing and memory calculation to solve high-frequency time-series data storage and massive data calculation problems. Hadoop distributed file system (HDFS) is used for files reliable storage, and distributed storage database HBase is used for persistent storage of historical data, which can meet data analysis needs of the same space-time coordinate system by its modeless sparse design. Redis is used as a real-time database storage data snapshot for the latest data retrieval. Coal mine data center based on Hadoop makes full use of high-speed data access and compression performance of big data technology, complexity of data application in coal enterprises and daily operation and maintenance difficulty of data center are effectively reduced through unified cloud computing data center, which can lay a data foundation for the construction of smart mine.

Keywords

• smart mine
• coal mine data center
• hadoop
• big data
• cloud computing