MillerByte.Logging.Api 1.2.1

MillerByte.Logging.Api

A .NET logging library for API observability with first-class support for session tracking, distributed tracing, and GDPR compliance.

Inspired by Logging Sucks by Boris Tane.

Why Traditional API Logging Falls Short

Traditional API logging scatters information across multiple log lines, making it difficult to understand what happened during a request. You end up with dozens of log statements per endpoint, and when something goes wrong, you're left piecing together fragments from different sources.

MillerByte.Logging.Api takes a different approach:

Session-Scoped Logging

Track user journeys across multiple requests by automatically maintaining session context.

Wide API Events

Capture complete request/response information in a single structured document with full context.

Built for Production

Resilient background processing with circuit breakers, retry policies, and fallback logging when your database is unavailable.

GDPR by Default

Export and delete user data with streaming APIs designed for compliance requirements.

What Are Wide API Events?

Instead of scattering logs throughout your API:

// Traditional approach
_logger.LogInformation("Request started: {Method} {Path}", method, path);
_logger.LogInformation("User authenticated: {UserId}", userId);
_logger.LogInformation("Fetching user data");
_logger.LogInformation("Processing business logic");
_logger.LogInformation("Request completed with status {Status}", statusCode);

You capture everything in one structured event:

[ApiLogging]
public async Task<IActionResult> GetUser(int id)
{
    var user = await _userRepository.GetByIdAsync(id);
    return Ok(user);
}

This creates a single MongoDB document containing the complete request context: endpoint info, user identity, request/response bodies, timing data, trace IDs, and any custom log messages you add during processing.

Core Features

Thread-Safe - ConcurrentBag for log messages, atomic MongoDB operations

High Performance - Bounded channel with background batch processing

Resilient - Polly retry policies, circuit breaker, fallback logging

Observable - OpenTelemetry integration, metrics, health checks

GDPR Compliant - Streaming export, data deletion, sensitive data filtering

Idempotent - Prevents duplicate logging with idempotency keys

Session Management - Hybrid strategy (JWT/HttpContext/Manual)

Distributed Tracing - W3C TraceContext support, correlation IDs

Configurable - Rate limiting, sampling, data sanitization

Production Ready - Graceful shutdown, proper disposal, error handling

Installation

dotnet add package MillerByte.Logging.Api

Quick Start

1. Configure in Program.cs

Drop-In Configuration (appsettings.json + env vars)

using MillerByte.Logging.Api;

var builder = WebApplication.CreateBuilder(args);

// appsettings.json:
// {
//	"ConnectionStrings": {
//		"ApiLogging": "mongodb://localhost:27017"
//	},
//	"ApiLogging": {
//		"DatabaseName": "ApiLogs"
//	}
// }

builder.Services.AddApiLogging(builder.Configuration);

builder.Services.AddControllers();

var app = builder.Build();

// IMPORTANT: Middleware order matters!
app.UseApiLoggingRequestBuffering();  // NEW v1.2.0 - Enables request body capture (MUST be first)
app.UseApiLoggingResponseCapture();   // Enables response body capture

app.UseRouting();
app.UseAuthentication(); // If using authentication
app.UseAuthorization();
app.MapControllers();

app.Run();

Drop-In With Optional Features

using MillerByte.Logging.Api;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddApiLogging(
	builder.Configuration,
	configureFeatures: flags =>
	{
		flags.EnableHealthChecks = true;
		flags.EnableMetrics = true;
		flags.EnableOpenTelemetry = true;
	});

builder.Services.AddControllers();

var app = builder.Build();

// IMPORTANT: Middleware order matters!
app.UseApiLoggingRequestBuffering();  // NEW v1.2.0 - Must be before UseRouting()
app.UseApiLoggingResponseCapture();

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();

app.Run();

Local Development (No Authentication)

using MillerByte.Logging.Api;

var builder = WebApplication.CreateBuilder(args);

// Add API Logging
builder.Services.AddApiLogging(options =>
{
	options.ConnectionString = "mongodb://localhost:27017";
	options.DatabaseName = "ApiLogs";
	options.EnableOpenTelemetry = true;
	options.EnableDataSanitization = true;
	options.SamplingRate = 1.0; // Log 100% of requests
});

builder.Services.AddControllers();

var app = builder.Build();

// IMPORTANT: Request buffering must be before UseRouting()
app.UseApiLoggingRequestBuffering();  // NEW v1.2.0 - Enables request body capture
app.UseApiLoggingResponseCapture();

app.UseRouting();
app.UseAuthorization();
app.MapControllers();

app.Run();

Custom Session/User Identifier Keys

using MillerByte.Logging.Api;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddApiLogging(options =>
{
	options.ConnectionString = "mongodb://localhost:27017";
	options.DatabaseName = "ApiLogs";
	
	// Configure where user/session IDs come from
	options.UserIdClaimTypes = new List<string> { "sub", "uid" };
	options.UserIdHeaderNames = new List<string> { "X-User-Id", "X-Api-Key" };
	options.UserIdCookieNames = new List<string> { "UserId" };
	
	options.SessionIdClaimTypes = new List<string> { "sid" };
	options.SessionIdHeaderNames = new List<string> { "X-Session-Id" };
	options.SessionIdCookieNames = new List<string> { "SessionId" };
	
	options.AllowHeaderIdentity = true;
	options.AllowCookieIdentity = true;
	options.AllowClientProvidedSessionId = true;
	options.AllowAnonymousSessions = true;
});

builder.Services.AddControllers();

var app = builder.Build();
app.UseApiLoggingResponseCapture();
app.UseRouting();
app.UseAuthorization();
app.MapControllers();

app.Run();

Production (Secure MongoDB with SSL/TLS)

using MillerByte.Logging.Api;

var builder = WebApplication.CreateBuilder(args);

// Add API Logging with secure MongoDB connection
builder.Services.AddApiLogging(options =>
{
	// Secure connection string with authentication and SSL/TLS
	options.ConnectionString = Environment.GetEnvironmentVariable("MONGODB_CONNECTION_STRING")
		?? "mongodb://username:password@your-server:27017/logging?authSource=logging&tls=true";
	options.DatabaseName = "logging";
	options.EnableOpenTelemetry = true;
	options.EnableDataSanitization = true;
	options.SamplingRate = 1.0;
	
	// Production resilience settings
	options.EnableFallbackLogging = true;
	options.FallbackLogPath = "./logs/api-logging-fallback";
	options.RetryAttempts = 3;
});

builder.Services.AddControllers();

var app = builder.Build();

// Add response capture middleware (BEFORE UseRouting)
app.UseApiLoggingResponseCapture();

app.UseRouting();
app.UseAuthorization();
app.MapControllers();

app.Run();

2. Add to Controllers

using MillerByte.Logging.Api.Attributes;

[ApiController]
[Route("api/[controller]")]
[ApiLogging] // ← Add this attribute
public class UsersController : ControllerBase
{
    [HttpGet("{id}")]
    public IActionResult GetUser(int id)
    {
        return Ok(new { id, name = "John Doe" });
    }
    
    [HttpPost]
    [ApiLogging(CheckIdempotency = true)] // ← Enable idempotency checking
    public IActionResult CreateUser([FromBody] UserDto user)
    {
        // Your logic here
        return Created($"/api/users/{user.Id}", user);
    }
}

3. Add Log Messages During Request Processing

using MillerByte.Logging.Api.Services;

public class UsersController : ControllerBase
{
    private readonly IApiLoggingService _loggingService;
    
    public UsersController(IApiLoggingService loggingService)
    {
        _loggingService = loggingService;
    }
    
    [HttpGet("{id}")]
    [ApiLogging]
    public async Task<IActionResult> GetUser(int id)
    {
        var actionId = HttpContext.Items["ApiLoggingActionId"]?.ToString();
        
        await _loggingService.AddLogMessageAsync(actionId, "Fetching user from database");
        var user = await _userRepository.GetByIdAsync(id);
        
        await _loggingService.AddLogMessageAsync(actionId, $"User found: {user.Name}");
        
        return Ok(user);
    }
}

Configuration Options

Local Development

builder.Services.AddApiLogging(options =>
{
	// MongoDB Connection (Local)
	options.ConnectionString = "mongodb://localhost:27017";
	options.DatabaseName = "ApiLogs";
	options.SessionsCollectionName = "LoginSessions";
	options.ActionsCollectionName = "ApiActions";
	
	// Background Processing
	options.BatchSize = 100;
	options.BatchIntervalMs = 5000;
	options.ChannelCapacity = 10000;
	
	// Data Capture
	options.LogRequestBody = true;
	options.LogResponseBody = true;
	options.IncludeGetRequestLogs = false;
	options.MaxBodySizeBytes = 1048576; // 1MB
	
	// Security & Sanitization
	options.EnableDataSanitization = true;
	options.SensitiveFieldNames = new List<string> 
	{ 
		"password", "token", "apiKey", "secret" 
	};
	
	// Performance & Scaling
	options.SamplingRate = 1.0; // 1.0 = 100%, 0.1 = 10%
	options.AlwaysLogErrors = true; // Always log errors regardless of sampling
	options.MaxActionsPerSessionPerMinute = 1000;
	
	// OpenTelemetry
	options.EnableOpenTelemetry = true;
	options.ActivitySourceName = "MillerByte.Logging.Api";
	
	// Resilience
	options.RetryAttempts = 3;
	options.CircuitBreakerFailureThreshold = 5;
	options.EnableFallbackLogging = true;
	options.FallbackLogPath = "./logs/api-logging-fallback";
	
	// MongoDB Advanced (requires replica set)
	options.UseTransactions = false;
});

Production with Secure MongoDB

builder.Services.AddApiLogging(options =>
{
	// MongoDB Connection (Secure with SSL/TLS and Authentication)
	options.ConnectionString = "mongodb://username:password@your-server:27017/logging?authSource=logging&tls=true&tlsAllowInvalidCertificates=true";
	options.DatabaseName = "logging";
	
	// Best practice: Use environment variables or secrets manager
	// options.ConnectionString = Environment.GetEnvironmentVariable("MONGODB_CONNECTION_STRING");
	
	// Production settings
	options.EnableDataSanitization = true;
	options.EnableFallbackLogging = true;
	options.FallbackLogPath = "/var/log/api-logging-fallback";
	options.SamplingRate = 0.1; // 10% sampling in production to reduce volume
	options.AlwaysLogErrors = true; // Always capture errors regardless of sampling
	options.RetryAttempts = 3;
	options.CircuitBreakerFailureThreshold = 5;
});

Connection String Parameters:

• tls=true - Enable SSL/TLS encryption
• tlsAllowInvalidCertificates=true - Required for self-signed certificates
• authSource=logging - Database where user authentication is stored
• retryWrites=true - Automatic retry for write operations
• connectTimeoutMS=10000 - Connection timeout in milliseconds

GDPR Compliance

Delete User Data

await _loggingService.DeleteUserDataAsync(userId, tenantId);

Export User Data (Streaming)

await foreach (var page in _loggingService.ExportUserDataStreamAsync(userId))
{
    if (!page.Success)
    {
        Console.WriteLine($"Error: {page.ErrorMessage}");
        break;
    }
    
    // Process sessions (only in first page)
    if (page.Sessions?.Any() == true)
    {
        foreach (var session in page.Sessions)
        {
            Console.WriteLine($"Session: {session.Id} - {session.LoginTime}");
        }
    }
    
    // Process actions
    foreach (var action in page.Actions)
    {
        Console.WriteLine($"Action: {action.EndpointInfo.Route} - {action.TimeStamp}");
    }
}

Health Checks

app.MapHealthChecks("/health", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("logging")
});

OpenTelemetry Integration

The package automatically creates activities and spans that integrate with your OpenTelemetry setup:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("MillerByte.Logging.Api")
        .AddAspNetCoreInstrumentation()
        .AddJaegerExporter());

Idempotency

Prevent duplicate logging for retried requests:

[HttpPost]
[ApiLogging(CheckIdempotency = true)]
public IActionResult CreateOrder([FromBody] OrderDto order)
{
    // Client sends: Idempotency-Key: 12345-abcde
    // Duplicate requests with same key won't be logged twice
    return Created($"/api/orders/{order.Id}", order);
}

Session Management

The library automatically manages user sessions using a hybrid strategy:

1. JWT Claims (primary) - Extracts user ID from JWT token claims
2. HttpContext (fallback) - Checks headers, cookies, or HttpContext.Items
3. Manual - Explicitly set session identifiers

Data Retrieval (NEW in v1.2.0)

Query logged sessions and actions using the IApiLoggingQueryService interface (CQRS pattern).

Query Sessions

public class LoggingAnalyticsController : ControllerBase
{
    private readonly IApiLoggingQueryService _queryService;
    
    public LoggingAnalyticsController(IApiLoggingQueryService queryService)
    {
        _queryService = queryService;
    }
    
    [HttpGet("sessions")]
    public async Task<IActionResult> GetSessions(
        [FromQuery] int pageIndex = 0,
        [FromQuery] int pageSize = 25,
        [FromQuery] string? userId = null,
        [FromQuery] bool? isActive = null)
    {
        var parameters = new QueryParameters
        {
            PageIndex = pageIndex,
            PageSize = pageSize,
            Filters = new FilterState
            {
                UserId = userId,
                IsActive = isActive,
                DateRange = new DateRange
                {
                    From = DateTime.UtcNow.AddDays(-7), // Last 7 days
                    To = DateTime.UtcNow
                }
            },
            Sorting = new List<SortingParameter>
            {
                new SortingParameter { Id = "LoginTime", Desc = true }
            }
        };
        
        var result = await _queryService.GetSessionsAsync(parameters);
        return Ok(result);
    }
}

Query Actions

[HttpGet("actions")]
public async Task<IActionResult> GetActions(
    [FromQuery] string? sessionId = null,
    [FromQuery] List<int>? statusCodes = null,
    [FromQuery] string? searchText = null)
{
    var parameters = new QueryParameters
    {
        PageIndex = 0,
        PageSize = 50,
        Filters = new FilterState
        {
            SessionId = sessionId,
            StatusCodes = statusCodes, // e.g., [200, 404, 500]
            SearchText = searchText, // Searches controller, action, route, method, etc.
            Methods = new List<string> { "POST", "PUT", "DELETE" } // Optional method filter
        },
        Sorting = new List<SortingParameter>
        {
            new SortingParameter { Id = "TimeStamp", Desc = true }
        }
    };
    
    var result = await _queryService.GetActionsAsync(parameters);
    return Ok(result);
}

Get Single Item by ID

var session = await _queryService.GetSessionByIdAsync(sessionId);
var action = await _queryService.GetActionByIdAsync(actionId);

Available Filters

For Sessions:

• UserId - Exact match
• TenantId - Exact match
• IsActive - true, false, or null (all)
• SearchText - Multi-field search (UserId, TenantId, Id, EnvironmentName)
• DateRange - Filter by LoginTime

For Actions:

• UserId - Exact match
• TenantId - Exact match
• SessionId - Exact match
• StatusCodes - Array of HTTP status codes (e.g., [200, 404, 500])
• Methods - Array of HTTP methods (e.g., ["GET", "POST"])
• SearchText - Multi-field search (UserId, TenantId, SessionId, Controller, Action, Route, Method, CorrelationId)
• DateRange - Filter by TimeStamp

Multi-Database/Collection Support (NEW in v1.2.0)

Route specific endpoints to different databases or collections using attribute properties.

Attribute-Level Routing

// Write to a different database for specific tenants
[ApiLogging(DatabaseName = "logs-tenant-premium", CollectionName = "premium-actions")]
public class PremiumController : ControllerBase
{
    [HttpGet]
    public IActionResult GetPremiumData() => Ok("Premium data");
}

// Separate audit logs for critical operations
[HttpPost]
[ApiLogging(DatabaseName = "audit-logs", CollectionName = "critical-actions")]
public IActionResult CriticalOperation() => Ok();

Query from Different Databases

// Query from default database
var defaultSessions = await _queryService.GetSessionsAsync(parameters);

// Query from a different database
var premiumSessions = await _queryService.GetSessionsAsync(
    parameters,
    databaseName: "logs-tenant-premium",
    collectionName: "premium-sessions");

// Query a specific action from audit database
var auditAction = await _queryService.GetActionByIdAsync(
    actionId,
    databaseName: "audit-logs",
    collectionName: "critical-actions");

Use Cases:

• Multi-tenant applications with separate databases per tenant
• Different retention policies (e.g., audit logs vs. debug logs)
• Read replicas for analytics queries
• Compliance requirements (e.g., PCI data in separate database)

MongoDB Indexes

The package automatically creates optimized indexes on startup:

• Sessions: (UserId, TenantId, IsActive) with unique constraint
• Actions: (SessionId, TimeStamp), (TraceId), (IdempotencyKey) unique
• TTL indexes for automatic cleanup (30 days for sessions, 90 days for actions)

Best Practices

1. Middleware Order is Critical (v1.2.0+):

   app.UseApiLoggingRequestBuffering();  // MUST be first - enables request body capture
   app.UseApiLoggingResponseCapture();   // Then response capture
   app.UseRouting();                     // Then routing
   app.UseAuthentication();              // Then auth
   

2. Enable Sampling in Production - Set SamplingRate to reduce volume while keeping errors

3. Configure Sensitive Fields - Add your domain-specific sensitive field names

4. Use Idempotency Keys - For critical operations to prevent duplicate logs on retries

5. Monitor Health Checks - Track channel depth and MongoDB connectivity

6. Enable Fallback Logging - Ensure logs aren't lost when MongoDB is down

7. Use Transactions (Optional) - Requires MongoDB replica set for atomic writes

8. Separate Read/Write Services (v1.2.0+) - Inject IApiLoggingService for writing, IApiLoggingQueryService for reading

Troubleshooting

Request Body Not Captured (v1.2.0+)

Symptom: Request body shows [Body not seekable - ensure UseApiLoggingRequestBuffering() is called before UseRouting()]

Solution: Ensure middleware is in correct order:

app.UseApiLoggingRequestBuffering();  // Must be BEFORE UseRouting()
app.UseApiLoggingResponseCapture();
app.UseRouting();

High Memory Usage

• Reduce ChannelCapacity
• Decrease BatchIntervalMs for faster processing
• Enable sampling with SamplingRate < 1.0

Slow Performance

• Increase BatchSize for more efficient writes
• Enable MongoDB compression (enabled by default)
• Use transactions only if you have a replica set
• Consider multi-database routing for high-volume tenants

MongoDB Connection Issues

• Check ConnectionString configuration
• Verify network connectivity
• Check fallback logs at FallbackLogPath

License

MIT License - See LICENSE file for details

Support

• GitHub Issues: https://github.com/Connor-Miller/millerbyte-stack/issues
• Documentation: https://github.com/Connor-Miller/millerbyte-stack/wiki