大数据 (Jul 2020)
Sunway parallel storage system for big data heterogeneous system
Abstract
With the integration of big data applications and traditional high-performance computing applications and the introduction of heterogeneous computing,the traditional parallel storage system for high-performance computing faces the problems of poor I/O support,performance interference,and low efficiency.By introducing multi-level storage architecture into the system architecture,the cache mapping mechanism was designed to reduce the I/O load.The I/O forwarding strategy was adjusted in the forwarding service layer to balance the I/O load.In the back-end storage layer,the high availability function of the system was adjusted to solve the conflict between the big data I/O access mode and the original high availability functions.After optimized design and improvement,the parallel storage system can better adapt to the heterogeneous multi-core architecture,making some applications get more than 10 times of I/O performance improvement.
