A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing

Abstract

Seed scheduling, which determines which seed is input to the fuzzer first and the number of mutated test cases that are generated for the input seed, significantly influences crash detection performance in fuzz testing. Even for the same fuzzer, the performance in terms of detecting crashes that cause program failure varies considerably depending on the seed-scheduling method used. Most existing coverage-guided fuzzers use a heuristic seed-scheduling method. These heuristic methods can’t properly determine the seed with a high potential to cause the crash; thus, the fuzzer detects the crash inefficiently. Moreover, the fuzzer’s crash detection performance is affected by the characteristics of target programs. To address this problem, we propose a general-purpose reinforced seed-scheduling method that not only improves the crash detection performance of fuzz testing but also remains unaffected by the characteristics of the target program. The fuzzer with the proposed method detected the most crashes in all but one of the target programs in which crashes were detected in the experimental results conducted on various programs, and showed better crash detection efficiency than the comparison targets overall.

Keywords

• Coverage-guided fuzzing
• power schedule
• reinforcement learning
• seed scheduling
• seed selection