In our previous studies into web design, we found that pens, paper, walls, and tables were often used for explaining, developing, and communicating ideas during the early phases of design. One practice was to produce information architectures using paper on walls. This practice inspired us to work towards The Designer's Outpost, a tangible user interface that combines the affordances of paper and large physical workspaces with the advantages of electronic media to support information design. In this paper, we describe a series of participatory design studies that explored the combination of physical and electronic media in depth. These three studies employed both low- and high-fidelity prototypes. The design teams in our studies encouraged us to support free ink electronic annotations to sitemap pages, versioning of design artifacts, fluid transitions to other tools, and opportunities for collocated and remote collaboration. In parallel, we built a set of prototypes for the underlying vision system. These prototypes led us to difference image-based recognition algorithms and a two-camera infrastructure: a rear-mounted video camera for capturing movement, and a front-mounted high-resolution camera for capturing ink.