Maintaining an in-focus image over long time scales is an essential and non-trivial task for a variety of microscopy applications. Here, we describe a fast and robust auto-focusing method that is compatible with a wide range of existing microscopes. It requires only the addition of one or a few off-axis illumination sources (e.g. LEDs), and can predict the focus correction from a single image with this illumination. We designed a neural network architecture, the fully connected Fourier neural network (FCFNN), that exploits an understanding of the physics of the illumination in order to make accurate predictions with 2-3 orders of magnitude fewer learned parameters and less memory usage than existing state-of-the-art architectures, allowing it to be trained without any specialized hardware. We provide an open-source implementation of our method, in order to enable fast and inexpensive autofocus compatible with a variety of microscopes.