Sequential Confidence Intervals for Comparing Two Proportions with Applications in A/B Testing

Abstract

This article addresses the use of fixed-width confidence intervals (FWCIs) for comparing two independent Bernoulli populations in A/B testing scenarios. Two sequential estimation procedures are proposed: one for estimating the difference in log probabilities of success and the other for log odds ratios. Both methods showcase great efficiency, as established via theoretical analysis and Monte Carlo simulations. The practical utility of these methods is demonstrated through two real-world applications: analyzing retention rates in mobile game Cookie Cats and evaluating the effectiveness of online advertising.

Keywords

• A/B testing
• Bernoulli proportions
• fixed-width confidence interval
• mobile gaming
• online advertising
• sequential estimation