Identifying Key Success Factors in Stopping Flaky Tests in Automated REST Service Testing

Abstract

A flaky test is a test which could fail or pass for the same version of a certain software code. In continuous software development environments, flaky tests represent a problem. It is difficult to get an effective and reliable testing pipeline with a set of flaky tests. Also, according to many practitioners, despite the persistence of flaky tests in software development, they have not drawn much attention from the research community. In this paper, we describe how a company faced this issue, and implemented solutions to solve flaky tests for REST web services. The paper concludes proposing a set of key success factors for stopping flaky tests in this type of testing.

Keywords

• flaky tests
• continuous integration
• continuous deployment
• continuous delivery
• web service testing