Current machine learning models suffer from evasion at- tacks such as adversarial examples. This introduces security and safety concerns that lack any clear solution. Recently, the use of random transformations (RT) has emerged as a promising defense against adversarial examples. However, it has not been rigorously evaluated, and its effectiveness is not well-understood. In this paper, we attempt to construct the strongest possible RT defense through the informed selection of transformations and the use of Bayesian optimization to tune their parameters. Furthermore, we attempt to identify the strongest possible attack to evaluate our RT defense. Our new attack vastly outperforms the naive attack, reducing the accuracy of our model by an additional 30%. In the process of formulating our defense and attack, we perform several ablation studies for both problems, drawing insights that we hope will broadly benefit scientific communities that study stochastic neural networks and robustness properties.