Current mobile platforms provide privacy management interfaces to regulate how applications access sensitive data. Prior research has shown how these interfaces are insufficient from a usability standpoint: they do not allow users to make contextual decisions (i.e., different decisions for a given application based on what the user was actually doing with that application). Prior work has demonstrated that classifiers can be built to automatically make privacy decisions that are more in line with users' preferences. However, if certain privacy decisions are automatically made---without immediate user consent---feedback mechanisms are needed to allow users to both audit those decisions and correct errors. In this paper, we describe our user-centered approach to designing such an interface. In addition to implementing this interface in Android, we created an interactive HTML5 simulation that we used to perform two large-scale user studies. Our final 580-person validation study showed that as compared to the default Android settings interface, users of our new interface were significantly more likely to understand and control the circumstances under which applications could access sensitive data.