A Reliable and Efficient Data Migration Method Based On GlusterFS

Abstract

Big data storage and high speed data access have become the main performance bottleneck for many big data applications. The higher speed data access and lower cost for storage must be required than some applications having small-scale data. Distributed hierarchical storage provides a good storage way to speed data access and lower cost. But it a data migration method which you choose decide the performance of distributed hierarchical storage system because data migration occurs frequently in hierarchical storage systems. There are many data migration methods, which most of those cannot ensure data utterly integrity after data migration. In this paper, we invent a reliable and efficient data migration method to ensure the utterly integrity of migrated data by MD5 checksum and improve performance of data migration by the pipeline technology. Through adjusting parameters, we get the best performance of data migration by using pipeline in our storage system which is a hierarchical storage system based on GlusterFS.