Simultaneous Localization and Mapping (SLAM) algorithms perform visual-inertial estimation via filtering or batch optimization methods. Empirical evidence suggests that filtering algorithms are computationally more efficient, while optimization methods are more accurate. This paper presents an optimization-based framework that unifies these approaches, and allows the flexible implementation of different design choices, e.g., selecting the number and types of variables maintained in the algorithm at each time. We mathematically prove that filtering methods, e.g., EKF SLAM and MSCKF, correspond to specific design choices in our generalized framework. Finally, we reformulate the MSCKF using our framework, implement the reformulation on challenging image sequences in a baseline SLAM dataset in simulation, and use the proposed re-interpretation to contrast the performance characteristics of the two classes of algorithms. Finally, we describe future extensions of our work to the dynamic SLAM problem and multi-agent planning problems.