Beyond Precision and Recall: Measuring Search Engine Consistency Using Rank Stability

Abstract

Traditional information retrieval metrics assess document relevance but neglect ranking stability—a critical factor in today’s dynamic web environments where search algorithms constantly evolve. We present Rmeasure, a comprehensive framework that quantifies search engine consistency by analyzing both overlapping and non-overlapping results through the lens of psychophysical principles, particularly the Weber-Fechner Law. Our three-component approach (Roverlap, Rnon-overlap, and Rcomprehensive) provides multidimensional insights into ranking variations. Experimental evaluation across Google and Bing using diverse query types reveals significant performance differences: Bing demonstrates superior ranking stability (minimum Roverlap of 0.0527), while Google exhibits considerable fluctuation (Rnon-overlap reaching 0.3653), especially for high-volume queries. Statistical measures confirm Bing’s consistency advantage, with Google showing rank differences averaging up to 21.622 positions. Beyond technical contributions, Rmeasure advances UN Sustainable Development Goals by enhancing information access equity (SDG 4), fostering innovation in search technology evaluation (SDG 9), and promoting consistent retrieval experiences across diverse user populations (SDG 10). By integrating perceptual modeling with rank-based analytics, Rmeasure extends search engine evaluation beyond traditional relevance metrics, offering a robust methodology for assessing real-world search performance.

Keywords

• Search engine evaluation
• rank correlation
• rank stability
• repeated queries
• ranking consistency
• overlapping results