Improving the efficiency of distributed data warehouses

Abstract

The article considers the problem of optimal processing and storage of big data. It is proposed to prepare a repository based on several data warehouse replicas, which is a combination of several different types of data repositories in one and is adaptable to user tasks. A set of programs has been developed, thanks to which it is possible to choose the method of entering large data, select the algorithm for changing the internal structure of the repository, perform the conversion algorithm, obtain the results of data queries. A unit responsible for putting the conversion algorithms into operation has been added to the system for converting the internal structure of the data warehouse. The metrics of the amount of used memory for backups and the speed of execution of data queries were used to estimate the performance. Practical significance: development of software that uses existing repository replicas (created for backup) to increase the performance of the repository as a whole. The advantage of the proposed solution is that there is no need for additional space for data storage, and only the storage control module is added. Ref. 10, pic. 2., tabl. 1.

Keywords

• big data
• data warehouse
• optimal allocation of resources
• distributed data processing systems
• cloud computing