What is Distributed Tracing?
Distributed tracing, also called distributed request tracing, is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.
Who Uses Distributed Tracing?
IT and DevOps teams can use distributed tracing to monitor applications. Distributed tracing is particularly well-suited to debugging and monitoring modern distributed software architectures, such as microservices.
Developers can use distributed tracing to help debug and optimize their code.
What is OpenTracing?
It is probably easier to start with what OpenTracing is NOT.
OpenTracing is not a download or a program. Distributed tracing requires that software developers add instrumentation to the code of an application, or to the frameworks used in the application.
OpenTracing is not a standard. The Cloud Native Computing Foundation (CNCF) is not an official standards body. The OpenTracing API project is working towards creating more standardized APIs and instrumentation for distributed tracing.
OpenTracing is comprised of an API specification, frameworks and libraries that have implemented the specification, and documentation for the project. OpenTracing allows developers to add instrumentation to their application code using APIs that do not lock them into any one particular product or vendor.
For more information about where OpenTracing has already been implemented, see the list of languages and the list of tracers that support the OpenTracing specification.
Concepts and Terminology
All language-specific OpenTracing APIs share some core concepts and terminology. These concepts are so central and important to the project that they have their own repository (github.com/opentracing/specification) and semver scheme.
- The OpenTracing Semantic Specification is a versioned description of the current pan-language OpenTracing standard
- The Semantic Conventions spec describes conventional Span tags and log keys for common semantic scenarios
Both files are versioned and the GitHub repository is tagged according to the rules described by the versioning policy.
Spans
What is a Span?
The “span” is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system.
Each component of the distributed system contributes a span - a named, timed operation representing a piece of the workflow.
Spans can (and generally do) contain “References” to other spans, which allows multiple Spans to be assembled into one complete Trace - a visualization of the life of a request as it moves through a distributed system.
Each span encapsulates the following state according to the OpenTracing specification:
- An operation name
- A start timestamp and finish timestamp
- A set of key:value span Tags
- A set of key:value span Logs
- A SpanContext
Tags
Tags are key:value pairs that enable user-defined annotation of spans in order to query, filter, and comprehend trace data.
Span tags should apply to the whole span. There is a list available at semantic_conventions.md listing conventional span tags for common scenarios. Examples may include tag keys like db.instance to identify a database host, http.status_code to represent the HTTP response code, or error which can be set to True if the operation represented by the Span fails.
Logs
Logs are key:value pairs that are useful for capturing span-specific logging messages and other debugging or informational output from the application itself. Logs may be useful for documenting a specific moment or event within the span (in contrast to tags which should apply to the span as a whole).
SpanContext
The SpanContext carries data across process boundaries. Specifically, it has two major components:
- An implementation-dependent state to refer to the distinct span within a trace
- i.e., the implementing Tracer’s definition of spanID and traceID
- Any Baggage Items
- These are key:value pairs that cross process-boundaries.
- These may be useful to have some data available for access throughout the trace.
Example Span:
t=0 operation name: db_query t=x +-----------------------------------------------------+ | · · · · · · · · · · Span · · · · · · · · · · | +-----------------------------------------------------+ Tags: - db.instance:"jdbc:mysql://127.0.0.1:3306/customers - db.statement: "SELECT * FROM mytable WHERE foo='bar';" Logs: - message:"Can't connect to mysql server on '127.0.0.1'(10061)" SpanContext: - trace_id:"abc123" - span_id:"xyz789" - Baggage Items: - special_id:"vsid1738"