Modern Internet applications are both powerful and highly interactive, but creating and operating them requires new approaches to the software development, deployment, and debugging processes. Applications like Google Maps, Facebook, and Map/Reduce must run continuously, handle millions of concurrent requests, and scale by taking advantage of the massive amounts of parallelism available in large-scale Internet datacenters. When a failure occurs, discovering the exact set of machines and resources responsible, as well as the location among those resources where the failure occurred, is a daunting task. The lack of visibility into these distributed systems prevents their reliable operation.
To improve distributed system visibility, we have developed an integrated tracing framework called X-Trace. A user or operator invokes X-Trace when initiating an application task (e.g., a web request), by inserting X-Trace metadata with a task identifier in the resulting request. This metadata is then propagated down to lower layers through protocol interfaces (which may need to be modified to carry X-Trace metadata), and also along all recursive requests that result from the original task (by modified software stacks). The X-Trace infrastructure makes use of this metadata to build a task graph, which represents a trace of the execution of the distributed application. Using these recovered task graphs, we have been able to identify correctness and performance bugs in a wide variety of distributed applications, from web and overlay applications, to the Hadoop Map/Reduce system. In this work, we present the design and implementation of X-Trace, show its application to the 802.1X network authentication protocol, and then present an API and software tool for manipulating large-scale network traces in a scalable manner.
Title
Improving Distributed Application Reliability with End-to-End Datapath Tracing
Published
2008-05-22
Full Collection Name
Electrical Engineering & Computer Sciences Technical Reports
Other Identifiers
EECS-2008-68
Type
Text
Extent
227 p
Archive
The Engineering Library
Usage Statement
Researchers may make free and open use of the UC Berkeley Library’s digitized public domain materials. However, some materials in our online collections may be protected by U.S. copyright law (Title 17, U.S.C.). Use or reproduction of materials protected by copyright beyond that allowed by fair use (Title 17, U.S.C. § 107) requires permission from the copyright owners. The use or reproduction of some materials may also be restricted by terms of University of California gift or purchase agreements, privacy and publicity rights, or trademark law. Responsibility for determining rights status and permissibility of any use or reproduction rests exclusively with the researcher. To learn more or make inquiries, please see our permissions policies (https://www.lib.berkeley.edu/about/permissions-policies).