This thesis presents methods for training intelligent robotic systems to navigate challenging, diverse environments. In particular it focuses on legged locomotion for hexapods rather than well-studied bipedal or quadrupedal robots. We employ hierarchical reinforcement learning to integrate proprioception with a low-level gait controller, and train on-policy algorithms entirely in simulation before transferring to a real robot. We develop command-conditioned policies in PyBullet that learn to walk at commanded target velocities, and switch to Isaac Gym to train policies with multi-objective rewards both in rough and flat terrains. A comparison is established between gaits with motion priors and prior-free gaits. Both methods successfully surmount joist obstacles typically seen in attics in the real world. We propose novel approaches to the sim-to-real problem designed to address limitations of affordable hardware. We demonstrate methods in the Isaac Gym simulation for integrating vision.