As hardware failures are no longer rare in the era of cloud computing, cloud software systems must ''prevail'' against multiple, diverse failures that are likely to occur. Testing software against multiple failures poses the problem of combinatorial explosion of multiple failures. To address this, a tester can write diverse policies that prune down the space of multiple failures while meeting her testing objective. In this paper, we present PreFail, a programmable failure-injection framework that enables testers to write a wide range of pruning policies. Using the principle of separation of mechanism and policy, we decouple a failure-injection framework into two components: failure-injection engine and driver. The policies written in the driver decide which failures should be injected by the engine. We define clear abstractions on which the two components interact. We integrate PreFail to three cloud software systems, show a wide variety of pruning policies that we can write for them and the speed-ups that we obtain with those policies.
PreFail: A Programmable Failure-Injection Framework
Researchers may make free and open use of the UC Berkeley Library’s digitized public domain materials. However, some materials in our online collections may be protected by U.S. copyright law (Title 17, U.S.C.). Use or reproduction of materials protected by copyright beyond that allowed by fair use (Title 17, U.S.C. § 107) requires permission from the copyright owners. The use or reproduction of some materials may also be restricted by terms of University of California gift or purchase agreements, privacy and publicity rights, or trademark law. Responsibility for determining rights status and permissibility of any use or reproduction rests exclusively with the researcher. To learn more or make inquiries, please see our permissions policies (https://www.lib.berkeley.edu/about/permissions-policies).