OpenTelemetry.Instrumentation.Kusto 0.1.0-alpha.3

Kusto Instrumentation for OpenTelemetry

This is an Instrumentation Library, which instruments Azure Data Explorer (Kusto) client libraries and collects telemetry about Kusto operations.

Steps to enable OpenTelemetry.Instrumentation.Kusto

Step 1: Install Package

Add a reference to the OpenTelemetry.Instrumentation.Kusto package. Also, add any other instrumentations & exporters you will need.

dotnet add package OpenTelemetry.Instrumentation.Kusto --prerelease

Step 2: Enable Kusto Instrumentation at application startup

Kusto instrumentation must be enabled at application startup.

Traces

The following example demonstrates adding Kusto traces instrumentation to a console application. This example also sets up the OpenTelemetry Console exporter, which requires adding the package OpenTelemetry.Exporter.Console to the application.

using OpenTelemetry.Trace;

public class Program
{
    public static void Main(string[] args)
    {
        using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .AddKustoInstrumentation()
            .AddConsoleExporter()
            .Build();
    }
}

Metrics

The following example demonstrates adding Kusto metrics instrumentation to a console application. This example also sets up the OpenTelemetry Console exporter, which requires adding the package OpenTelemetry.Exporter.Console to the application.

using OpenTelemetry.Metrics;

public class Program
{
    public static void Main(string[] args)
    {
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddKustoInstrumentation()
            .AddConsoleExporter()
            .Build();
    }
}

List of metrics produced

The instrumentation is implemented based on metrics semantic conventions. Currently, the instrumentation supports the following metric:

[^1]: db.query.summary is only included when RecordQuerySummary is enabled [^2]: db.query.text is only included when RecordQueryText is enabled [^3]: error.type is only included when an error occurs

Advanced configuration

This instrumentation can be configured to change the default behavior by using KustoInstrumentationOptions.

Multiple providers

The Kusto instrumentation uses a single, process-wide trace listener, so all TracerProviders and MeterProviders in a process share one set of options. When more than one provider configures the instrumentation, the most recent AddKustoInstrumentation call wins, so configure the options consistently across providers.

Capturing query text

RecordQueryText and RecordQuerySummary rely on the Kusto client emitting the query text in its trace output, which it only does when the KUSTO_DATA_TRACE_REQUEST_BODY environment variable is set to 1. The client reads this variable once, so it must be set before any Kusto client is created. The instrumentation deliberately does not set it, because doing so would enable query-body tracing process-wide as a side effect; that is left to the host as a deliberate choice. If either option is enabled without the variable set, no query text or summary is recorded and a warning is written to the OpenTelemetry-Instrumentation-Kusto event source. The Kusto client's tracing system is described in Controlling and suppressing Kusto SDK client-side tracing.

For example, set the variable during application startup:

Environment.SetEnvironmentVariable("KUSTO_DATA_TRACE_REQUEST_BODY", "1");

RecordQueryText

This option can be set to instruct the instrumentation to record the sanitized query text as an attribute on the activity. Query text is sanitized to remove literal values and replace them with a placeholder character.

The default value is false and can be changed by the code like below.

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddKustoInstrumentation(
        options => options.RecordQueryText = true)
    .AddConsoleExporter()
    .Build();

RecordQuerySummary

This option can be set to instruct the instrumentation to record a query summary as an attribute on the activity. The query summary is automatically generated from the query text and contains the operation type and relevant object names.

The default value is true and can be changed by the code like below.

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddKustoInstrumentation(
        options => options.RecordQuerySummary = false)
    .AddConsoleExporter()
    .Build();

Enrich

This option can be used to enrich the activity with additional information from the raw TraceRecord object. The Enrich action is called only when activity.IsAllDataRequested is true. It contains the activity itself (which can be enriched) and the actual TraceRecord from the Kusto client library.

The following code snippet shows how to add additional tags using Enrich.

using OpenTelemetry.Instrumentation.Kusto.Implementation;

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddKustoInstrumentation(opt => opt.Enrich = (activity, record) =>
    {
        // Add custom tags based on the TraceRecord
        activity.SetTag("azure.kusto.activity_id", record.Activity.ActivityId);
        activity.SetTag("azure.kusto.activity_type", record.Activity.ActivityType);
    })
    .Build();

Processor, is the general extensibility point to add additional properties to any activity. The Enrich option is specific to this instrumentation, and is provided to get access to the TraceRecord object.

Custom Query Summarization

The Enrich callback can be used to implement custom query summarization logic. For example, you can extract summary information from query comments:

using OpenTelemetry.Instrumentation.Kusto.Implementation;

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddKustoInstrumentation(opt =>
    {
        // Disable automatic summarization
        opt.RecordQuerySummary = false;

        // Extract custom summary from query comments
        opt.Enrich = (activity, record) =>
        {
            const string key = "// otel-custom-summary=";
            var message = record.Message.AsSpan();
            var begin = message.IndexOf(key, StringComparison.Ordinal);

            if (begin < 0)
            {
                return;
            }

            var summary = message.Slice(begin + key.Length);
            var end = summary.IndexOfAny('/r', '/n');
            if (end < 0)
            {
                end = summary.Length;
            }

            summary = summary.Slice(0, end).Trim();
            var summaryString = summary.ToString();

            activity.SetTag("db.query.summary", summaryString);
            activity.DisplayName = summaryString;
        };
    })
    .Build();

With this configuration, a query like:

// otel-custom-summary=Get active users
Users
| where IsActive == true
| take 100

Would result in an activity with the summary and display name set to "Get active users".

References

• OpenTelemetry Project
• Azure Data Explorer (Kusto)
• OpenTelemetry semantic conventions for database spans
• OpenTelemetry semantic conventions for database metrics