Performance Enhancement of Distributed Processing Systems Using Novel Hybrid Shard Selection Algorithm

Abstract

Distributed processing systems play a crucial role in query search operations, where large-scale data are partitioned across multiple nodes using shard selection algorithms. However, the existing shard selection algorithms pose significant challenges, such as shard ranking, shard cut-off estimation, high latency, low throughput, and high processing costs. These limitations become more pronounced as the data size increases, affecting the efficiency and effectiveness of search operations. To address these challenges, the novel Hybrid Shard Selection Algorithm (HSSA) is proposed as a solution in this paper, designed specifically to enhance the effectiveness and efficiency of search operations within distributed processing systems. HSSA employs an advanced sharding approach that adeptly navigates and targets pertinent shards based on specific queries. This not only curtails search-related overhead but also enhances operational efficiency. Through rigorous testing using the Gov2 dataset, the HSSA algorithm has proven its merits. When set against well-established algorithms like CORI, Rank-S, and SHiRE, HSSA stands out, registering remarkable gains in average throughput by 21%, 16%, and 12%, while also slashing latency by 14.2%, 9.4%, and 8.2%, respectively. The insights gained from this research underscore HSSA's capability to effectively bridge the gaps inherent in traditional shard selection strategies. Furthermore, its exemplary efficacy with datasets of varied sizes amplifies its relevance for practical integration within distributed processing landscapes.

Keywords

• sharding
• cluster
• indexing
• partitioning
• allocation