Description
Recent data-loss incidents have shown that existing large distributed systems are still vulnerable to failures. To improve the situation, we propose two new testing approaches: failure testing service (FTS) and declarative testing specification (DTS). FTS enables us to systematically push a system into thousands of failure scenarios, leading us to many critical recovery bugs. With DTS, we introduce "micro-specifications", clear and concise specifications written in Datalog style, which enables developers to easily write, refine, and manage potentially hundreds of specifications.