应用气象学报 (Jan 2024)
Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform
Abstract
With the advancement of meteorological services, various product processing systems and supporting data management systems have been developed for different business systems. However, this has led to the problem of system non-intensification. The lack of intensification in meteorological services can result in inconsistent data standards and make the operation and maintenance more challenging, which can lead to significant waste in investment due to duplicate data storage and inconsistent data caused by untimely synchronization. Moreover, the lack of information and technology hinders the integration of upstream and downstream businesses.The intensive development of meteorological business systems and the reform of "cloud+end" technology system are important measures to achieve high-quality development of meteorological business. China Meteorological Administration proposed to build a "cloud+end" technology system in 2020. CMA Big Data and Cloud Platform serves as the cloud, while the meteorological business system is the end, clarifying the positioning of CMA Big Data and Cloud Platform as a key foundational technology platform. The data processing line (DPL) is an intensive environment for meteorological algorithms. It addresses business needs such as efficient and stable processing of data products, data sharing and collaboration among business systems, and efficient and intensive business applications. To achieve this goal, algorithm libraries and task control functions are established by utilizing technologies such as integrated digital and computing, efficient task scheduling, visual process arrangement, and containers. The algorithm library facilitates the standardization, unified management and sharing of algorithms. Task control supports multiple scheduling strategies with high reliability and fault tolerance, enabling efficient and stable scheduling operations for algorithms. All functions mentioned above are in the form of interfaces for the application frontend. At the same time, based on the meteorological business comprehensive monitoring system (referred to as Tianjing) enables automatic collection of algorithm operation status and detection of abnormal alarms. Since its operation in 2021, the processing assembly line has facilitated the real-time operation of 202 business systems nationwide, resulting in a performance improvement of 1-10 times and a significant increase in efficiency. It plays an important supporting role in improving the operational efficiency of business systems, enhancing their collaboration, accelerating the process of "cloud+end" business technology system reform, and promoting the intensive development of meteorological business.
Keywords
