Vostok.Tracing.Abstractions 0.1.8-pre000018

Vostok tracing

Distributed tracing allows to reconstruct the history of the logical operation spanning many applications and machines in time as a tree of smaller actions or events called spans. Spans can represent HTTP requests, database queries or any other significant interactions or events in a distributed system. A single span always describes a local event in a single process: an HTTP request usually produces two spans (client-side and server-side). Each kind of span stores specific information about performed action.

<br/>

• Span structure
• Common annotations
• Kind-specific annotations
  • HTTP requests
    • HTTP client (direct)
    • HTTP client (cluster)
    • HTTP server
  • Custom (not HTTP) requests
    • Custom (not HTTP) client (direct)
    • Custom (not HTTP) client (cluster)
  • Custom operations
  • Database requests
    • Cassandra
    • MongoDB
    • ElasticSearch
    • MS SQL
  • Distributed task queues
    • Queue (producer)
    • Queue (task-lifecycle)
    • Queue (task-lifecycle-event)
    • Queue (consumer)
    • Queue (manager)
    • Custom events
• Queue tracing conventions

<br/>

Span structure

Every span consists of following fields:

• TraceId — unique identifier of the trace containing the span (Guid).

  • Gets assigned on first operation (usually on a front-end application instance).
  • Serves as a correlation identifier between spans

• SpanId — unique identifier of the span itself (Guid).

• ParentSpanId — identifier of the parent span in the tree (Guid).

  • May be absent for root span in the tree.

• BeginTimestamp — beginning timestamp of the event or interaction described by this span (UTC timestamp + timezone offset).

• EndTimestamp — ending timestamp of the event or interaction described by this span (UTC timestamp + timezone offset).

  • Always measured with the same clock as BeginTimestamp. This allows to derive span duration as a difference between EndTimestamp and BeginTimestamp.
  • May be absent for a special kind of 'endless' spans described further.

• Annotations — payload in key-value format (string --> object). Keys are case-sensitive.

<br/>

Common annotations

These are the annotations relevant for any span:

<br/> <br/>

Kind-specific annotations

HTTP requests

Common annotations for all spans related to HTTP requests:

Normalized URL is a short URL without scheme, authority and query parameters. Unique path segments (entity ids, search queries, hex values) are replaced with placeholders. Example before and after normalization: http://vm-app1/users/a534bcbd/ --> users/{id}

<br/>

HTTP client (direct)

Submitting an HTTP request directly to an external URL or a service replica.

<br/>

HTTP client (cluster)

Submitting an HTTP request to a clustered application with several replicas.

<br/>

HTTP server

Handling an HTTP request on server.

<br/> <br/>

Custom (not HTTP) requests

Common annotations for all spans related to custom requests:

<br/>

Custom (not HTTP) client (direct)

Submitting custom request directly to a service replica.

<br/>

Custom (not HTTP) client (cluster)

Submitting custom request to a clustered application with several replicas.

<br/> <br/>

Custom operations

Performing custom server operation.

<br/> <br/>

Database requests

Submitting a request to database.

MS SQL

TODO

Cassandra

TODO

MongoDB

TODO

ElasticSearch

TODO

<br/> <br/>

Distributed task queues

See queue tracing conventions for more context.

Common annotations for all spans related to distributed task queues:

<br/>

Queue (producer)

A span that represents insertion of a task to queue (from the producer standpoint). Note that this span should not be in the same trace with subsequent spans related to inserted task: instead, it's connected with dedicated task trace with queue.taskTraceId annotation.

<br/>

Queue (task-lifecycle)

A span that represents whole lifecycle of the task in queue. It serves as a root span in the task's personal trace.

Spans of this kind are special in two ways:

1. They are non-local: task lifecycle spans do not directly relate with events happening inside any single process.
2. They do not have an ending timestamp: it must be inferred from the rightmost end timestamp of all the child spans.

<br/>

Queue (task-lifecycle-event)

A span that represents an event that somehow changes task state.

Such spans usually have zero duration and are produced by brokers or client libraries.

<br/>

Queue (consumer)

A span that represents execution of a queued task on the consumer.

<br/>

Queue (manager)

A span that represents a management action on the task, but from the client's standpoint, as opposed to queue-task-lifecycle-event spans. Note that this span should not be in the same trace with subsequent spans related to managed task.

<br/> <br/>

Queue tracing conventions

Following conventions apply to tracing of queue tasks:

• Each task should have a dedicated trace with a root span of queue-task-lifecycle kind. This span does not have an explicit end timestamp. It gets "stretched" by child spans instead.

• Each task should contain its personal traceId and root span's spanId embedded in content, available for consumers.

• Root lifecycle spans can have child spans of queue-task-lifecycle-event and queue-consumer kinds. External operations from producers and management clients happen in their own traces.

• A span of queue-task-lifecycle-event kind can have a link to traceId of external operation that triggered the event.

• A span of queue-producer kind should have a link to traceId of the produced task.

If a queue has no broker or there's no access to broker code, its tracing duties have to be performed by client libraries (creation of queue-task-lifecycle and queue-task-lifecycle-event spans).