Random Testing: Theoretical Results and Practical Implications.
IEEE Transactions on Software Engineering 38(2):258 - 277
A substantial amount of work has shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications appear in the literature. Although it is not going to solve all possible testing problems, random testing is an essential tool in the hands of software testers. In this paper, we review and analyze the debate about random testing. Its benefits and drawbacks are discussed. Recent developments of more sophisticated random testing techniques are reviewed as well. Novel results addressing general questions about random testing are also presented, such as how long random testing needs on average to achieve testing targets (e.g., coverage), how does it scale and how likely is it to yield similar results if we re-run random testing on the same testing problem. Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Our theoretical results are backed up by simulations and we show how they can be applied to most types of software and testing criteria. In light of our results, we then assess the validity of empirical analyses reported in the literature. Results show that random testing is more effective and predictable than previously thought.