Activity based travel demand models are becoming essential tools used in transportation planning and regional development scenario evaluation. They describe travel itineraries of individual travelers, namely what activities they are participating in, when they perform these activities, and how they choose to travel to the activity locales. However, data collection for activity based models is performed through travel surveys that are infrequent, expensive, and reflect the changes in transportation with significant delays. Thanks to the ubiquitous cell phone data, we see an opportunity to substantially complement these surveys with data extracted from network carrier mobile phone usage logs, such as call detail records (CDRs). In this paper, we develop Input-Output Hidden Markov Models (IOHMMs) to infer travelers' activity patterns from CDRs. We apply the model to the data collected by a major network carrier serving millions of users in the San Francisco Bay Area. Our approach delivers an end-to-end actionable solution to the practitioners in the form of a modular and interpretable activity-based travel demand model. It is experimentally validated with three independent data sources: aggregated statistics from travel surveys, a set of collected ground truth activities, and the results of a traffic micro-simulation informed with the travel plans synthesized from the developed generative model.