Optimus.Bank.Observability.Extensions 1.0.12

Observability Extensions

Standardized observability setup for .NET applications using Serilog for structured logging and OpenTelemetry for distributed tracing, metrics, and log export.

This library provides simple extension methods to configure consistent observability across ASP.NET Core web APIs and Worker Services / Hosted Services with minimal boilerplate.

Features

• One-line setup for Serilog + OpenTelemetry
• Automatic OTLP export (gRPC or HTTP/Protobuf) for traces, metrics, and logs — metrics are always pushed to ExporterUri regardless of whether the Prometheus scrape endpoint is enabled
• Prometheus /metrics endpoint support (web apps, configurable via EnablePrometheusMetricsEndpoint)
• File logging (text + JSON) with daily rolling and sensitive data masking
• Built-in filtering of health/metrics endpoints from traces and logs
• Resource attributes: service name, environment, host name
• Common instrumentation: ASP.NET Core, HttpClient, Runtime, SQL, EF Core, gRPC, Redis, RabbitMQ, etc.
• Configurable via single "ObservabilityOptions" section in appsettings.json
• Health checks (/health) and metrics (/metrics) endpoints can each be individually toggled on/off, and are automatically excluded from traces/logs when enabled
• Supports both ASP.NET Core web applications and Worker Services / Hosted Services
• Sensitive data masking for common properties (e.g., passwords, tokens)
• Supports Microsoft logging levels configuration
• Daily rolling logs with optional text and JSON formats
• Automatic dependency injection for Serilog and OpenTelemetry
• Distributed trace propagation across message brokers (RabbitMQ, etc.) with automatic W3C trace context injection/extraction, exposed via ITraceContextInjector / ITraceContextExtractor for testability

Sensitive Data Masking

The package includes two complementary approaches to protect sensitive information in logs, audit trails, API responses, and user interfaces:

1. Property-level masking (via Serilog sink/enricher)
   Automatically masks values of specified property names (e.g. Password, Token, ApiKey, RefreshToken).
   Configurable via SensitivePropertyNames in appsettings.json. Defaults include common secret field names.

2. Value-level masking
   A set of string extension methods in SensitiveDataExtensions to explicitly mask sensitive values before logging, returning responses, or enriching events.
   Designed for KYC data (BVN, NIN, account, phone) as well as common international PII (email, IBAN, credit card, passport).

Usage Examples

// Structured logging with safe values
logger.Information(
    "User with BVN {Bvn} and phone {Phone} initiated transfer to {AccountNumber}",
    user.Bvn.MaskBvn(),
    user.Phone.MaskPhoneNumber(),
    transfer.ToAccount.MaskAccountNumber()
);

// Masking before returning in API response or audit log
var safeEmail = request.Email.MaskEmail();
var safeIban  = payment.Iban.MaskIban();
logger.Information("Payout initiated to {Email} via IBAN {Iban}", safeEmail, safeIban);

// Chaining in complex scenarios
logger.Warning(
    "Failed verification attempt - NIN {Nin}, Passport {Passport}, IP {Ip}",
    attempt.Nin.MaskNin(),
    attempt.PassportNumber.MaskPassportNumber(),
    attempt.IpAddress  // IP usually not masked unless policy requires
);

Masking Behavior Overview

Safety note
These methods are designed to be non-destructive:

• If the input is empty, null, whitespace → returns empty string
• If format looks invalid (wrong length, no @ in email, too short to mask meaningfully) → returns original value unchanged or applies best-effort masking
• Never throws exceptions during masking (fail-safe behavior)
• For maximum security (e.g. PIN/OTP), call the generic Mask() with visibleFirst: 0, visibleLast: 0 to force full masking.

Advanced / Custom Masking

For non-standard formats or short secrets (PIN, OTP, promo codes, reference IDs), use the generic method directly:

// 4-digit PIN → full mask
string safePin = pin.Mask(visibleFirst: 0, visibleLast: 0);     // "****"

// Show only last 2 of a 6-digit OTP
string safeOtp = otp.Mask(visibleLast: 2);                      // "****12"

// Custom short code with minimum 6 masks
string safeCode = code.Mask(visibleFirst: 2, minMaskedChars: 6); // "AB******"

This combination gives you both convenience (named methods for common Nigerian PII) and flexibility (generic method for everything else).

Installation

dotnet add package Optimus.Bank.Observability.Extensions

Note: This package brings in several OpenTelemetry and Serilog dependencies automatically. You do not need to add them manually.

Important troubleshooting note: If you're having issues (e.g. duplicate Serilog configurations, conflicting sinks, or unexpected logging behavior), remove any direct Serilog-related packages from your consuming project. This package installs and configures its own version of Serilog — manual Serilog packages can cause conflicts.

Quick Start

ASP.NET Core Web Application

var builder = WebApplication.CreateBuilder(args);

// One line to configure Serilog + OpenTelemetry
builder.AddObservability();

var app = builder.Build();

// Add observability middleware (Prometheus + request logging)
app.UseObservabilityMiddleware();

app.MapGet("/", () => "Hello World!");
app.Run();

Worker Service / Hosted Service (No Message Consumers)

var builder = Host.CreateApplicationBuilder(args);

// Same one-liner for workers without custom ActivitySources
builder.AddObservability();

builder.Services.AddHostedService<MyBackgroundWorker>();

var host = builder.Build();
await host.RunAsync();

Worker Service with Message Consumers (RabbitMQ, etc.)

When your worker consumes messages from queues and needs distributed trace propagation:

var builder = Host.CreateApplicationBuilder(args);

// Register ActivitySources for trace propagation
builder.AddObservability(sources: [
    OrderConstants.ActivitySourceName,
    PaymentConstants.ActivitySourceName
]);

builder.Services.AddHostedService<OrderConsumer>();
builder.Services.AddHostedService<PaymentConsumer>();

var host = builder.Build();
await host.RunAsync();

Define ActivitySource names as constants:

public static class OrderConstants
{
    public const string ActivitySourceName = "OrderWorker.Orders";
}

public static class PaymentConstants
{
    public const string ActivitySourceName = "OrderWorker.Payments";
}

Configuration (appsettings.json)

{
  "ObservabilityOptions": {
    "ApplicationName": "OrderService",
    "ExporterUri": "http://otel-collector:4319",
    "ExportProtocol": "Grpc",
    "LogFilePath": "./logs",
    "EnableTextLog": true,
    "EnableJsonLog": false,
    "EnableConsoleLog": false,
    "MicrosoftLogLevel": "Warning",
    "EnablePrometheusMetricsEndpoint": false,
    "EnableHealthCheckEndpoint": true,
    "SensitivePropertyNames": [ "Password", "Token", "ApiKey", "Secret", "CardNumber" ],
    "LoggerLevelOverrides": {
        "System.Net.Http.HttpClient": "Information",
        "Microsoft.EntityFrameworkCore.Database.Command": "Information",
        "Optimus.PaymentService": "Debug"
    }
  }
}

Required fields: ApplicationName and ExporterUri.

Endpoint Exposure

By default, EnablePrometheusMetricsEndpoint is false and EnableHealthCheckEndpoint is true.

/// <summary>
/// Gets or sets a value indicating whether a prometheus endpoint should be enabled.
/// </summary>
public bool EnablePrometheusMetricsEndpoint { get; set; } = false;

/// <summary>
/// Gets or sets a value indicating whether the application should expose a health check endpoint.
/// </summary>
public bool EnableHealthCheckEndpoint { get; set; } = true;

• EnablePrometheusMetricsEndpoint — when false (default), the /metrics scrape endpoint is not mapped by UseObservabilityMiddleware(). Metrics are still exported via OTLP to ExporterUri regardless of this setting — this flag only controls the pull-based Prometheus scrape endpoint, not push-based OTLP export.
• EnableHealthCheckEndpoint — when false, the /health endpoint is not mapped. Defaults to true for backward compatibility.

Security note: Both endpoints were previously always mapped in UseObservabilityMiddleware(). If you're upgrading and were relying on /metrics being publicly reachable directly on the service (e.g. an external Prometheus scraper hitting the pod instead of going through the OTel Collector), you now need to explicitly set EnablePrometheusMetricsEndpoint: true. This change was made to reduce the default attack surface, since /metrics and /health can leak internal service details if left open by default.

Distributed Trace Propagation Across Message Brokers

This package provides automatic W3C trace context propagation across message broker boundaries (RabbitMQ, Kafka, Azure Service Bus, etc.) through ITraceContextInjector and ITraceContextExtractor.

Interfaces required in producers/consumers: Producers and consumers should inject ITraceContextInjector and ITraceContextExtractor respectively — not the concrete TraceContextInjector / TraceContextExtractor classes. Both are registered against their interfaces in AddObservability(), and using the interfaces keeps your code mockable in unit tests (e.g. with AutoMocker) without needing to touch real Activity/ActivitySource state.

How It Works

API Request (TraceId: abc123)
    │
    ├─ HTTP POST /orders
    ├─ RabbitMQ Publish → ITraceContextInjector.Inject(properties)
    │                     └─ Writes traceparent/tracestate headers
    │
    ~~~~ message travels through queue ~~~~
    │
    └─ Worker Consumer → ITraceContextExtractor.StartConsumerActivity(...)
                        └─ Extracts headers, starts linked child span
                        └─ All logs carry same TraceId: abc123

The full request chain appears in Jaeger/Tempo/Grafana as a single distributed trace.

Producer Side (Publishing Messages)

Inject the current trace context into message headers before publishing. Inject ITraceContextInjector, not the concrete type:

public class OrderController : ControllerBase
{
    private readonly IModel _channel;
    private readonly ITraceContextInjector _injector;

    public OrderController(IModel channel, ITraceContextInjector injector)
    {
        _channel = channel;
        _injector = injector;
    }

    [HttpPost]
    public IActionResult PlaceOrder(OrderRequest request)
    {
        var properties = _channel.CreateBasicProperties();
        
        // ✅ Inject W3C trace context into headers (one line)
        _injector.Inject(properties);

        _channel.BasicPublish(
            exchange: "orders",
            routingKey: "order.placed",
            basicProperties: properties,
            body: JsonSerializer.SerializeToUtf8Bytes(request));

        return Accepted();
    }
}

Consumer Side (Processing Messages)

Extract the parent trace context and start a linked child span. Inject ITraceContextExtractor, not the concrete type:

public class OrderConsumer : BackgroundService
{
    private readonly IModel _channel;
    private readonly ITraceContextExtractor _extractor;
    private readonly ILogger<OrderConsumer> _logger;

    public OrderConsumer(
        IModel channel, 
        ITraceContextExtractor extractor,
        ILogger<OrderConsumer> logger)
    {
        _channel = channel;
        _extractor = extractor;
        _logger = logger;
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var consumer = new AsyncEventingBasicConsumer(_channel);

        consumer.Received += async (_, args) =>
        {
            // ✅ Extract parent trace, start linked child span
            using var activity = _extractor.StartConsumerActivity(
                args.BasicProperties,
                activityName: "order.process",
                sourceName: OrderConstants.ActivitySourceName);

            // All logs/spans below automatically carry the TraceId from the API request
            _logger.LogInformation("Processing order {OrderId}", orderId);
            await ProcessOrderAsync(args.Body);
        };

        _channel.BasicConsume("orders", autoAck: true, consumer);
        return Task.CompletedTask;
    }
}

Important: The sourceName argument must match one of the names you registered via AddObservability(sources: [...]) in Program.cs, otherwise the activity will be dropped and the trace will break.

Note on RegisterSource: RegisterSource is intentionally not exposed on ITraceContextExtractor — it's internal startup plumbing invoked automatically by AddObservability(sources: [...]). Producers and consumers never need to call it directly; just pass your ActivitySource names into AddObservability at startup and reference them by name in StartConsumerActivity.

Non-RabbitMQ Transports (Kafka, Azure Service Bus, etc.)

The package provides generic overloads for other message brokers:

// Producer - generic inject
_injector.Inject(kafkaHeaders, (headers, key, value) => headers.Add(key, value));

// Consumer - generic extract
using var activity = _extractor.StartConsumerActivity(
    kafkaHeaders,
    (headers, key) => headers.TryGetValue(key, out var val) ? new[] { val } : Enumerable.Empty<string>(),
    activityName: "payment.process",
    sourceName: PaymentConstants.ActivitySourceName);

What You Get in Your Traces

With trace propagation enabled, a single API request that publishes to RabbitMQ and is processed by a worker appears as one continuous trace in Jaeger/Tempo:

[API Request]  POST /orders  TraceId: abc123
    │
    ├── [API Span]     HTTP POST /orders
    ├── [API Span]     RabbitMQ Publish → orders
    │
    │   ~~~~ message travels through queue ~~~~
    │
    └── [Worker Span]  order.process          ← same TraceId: abc123
            ├── [Worker Span]  EF Core INSERT orders
            └── [Worker Span]  HTTP POST payment-service

No manual correlation needed — the full chain is linked end-to-end with a single TraceId.

Troubleshooting Trace Propagation

Architecture & Data Flow

The architecture exports all telemetry (traces, metrics, logs) to an OpenTelemetry Collector (via OTLP), which forwards them to backends like Elasticsearch + Kibana (logs) and Jaeger (traces).

Viewing Your Data

• Logs → Kibana (Elasticsearch)
• Traces → Jaeger UI / Grafana Tempo
• Metrics → Prometheus / Grafana (or Elastic stack, depending on collector config)

Happy observability! 🚀

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Acknowledgments

Built using:

• Serilog
• OpenTelemetry .NET