Iron.ClickHouseLogger.Framework 1.0.2

Iron.ClickHouseLogger

Plug-and-play observability for .NET — HTTP request/response logging, structured debug logs, and metrics, persisted directly to ClickHouse.

Two packages, one concept

Both packages share identical concepts, log schemas, and runtime behaviour. The only difference is the integration surface: the ASP.NET Core package wires into the request pipeline via DI and middleware; the Framework package provides an IHttpModule, a Web API DelegatingHandler, and a static factory that works in Global.asax without a DI container.

Shared core. The batching, connection/failover management, fallback, table DDL, models, and helpers live in a single multi-targeted (net48;net8.0) library, Iron.ClickHouseLogger.Core, that both packages depend on. You never install it directly — it arrives transitively with whichever package you choose. It exists so a fix to the core logic ships to both runtimes from one source.

Why ClickHouseLogger?

Most logging libraries target text search. ClickHouseLogger targets analytics: structured data lands directly in a columnar database capable of aggregating hundreds of millions of rows in seconds.

Core features (both packages)

• Single log table — {app}_logs stores HTTP, debug, and metric rows together; Type column keeps them queryable independently
• Six log levels — Trace → Fatal; entries below MinimumLevel are dropped in-process with zero allocation
• Auto-DDL — CREATE TABLE IF NOT EXISTS + ALTER TABLE ADD COLUMN on startup
• IndexedFields — promote arbitrary map keys to real indexed Nullable(String) columns for fast WHERE lookups
• Batching — ConcurrentQueue, flush by size (default 1 000) or time interval (default 5 s)
• Exponential-backoff retry — configurable attempts before spilling to disk
• Disk fallback — .jsonl files written when ClickHouse is unreachable; drained automatically when the connection restores
• Active-Passive multi-node — configure a primary and one or more failover nodes; ConnectionManager polls all nodes on a configurable interval (default 30 s) and automatically returns to the primary when it recovers
• Header masking — sensitive headers replaced with *** before storage
• Wildcard path filtering — include/exclude paths with * glob patterns
• Graceful shutdown — queue fully flushed before process exit

Delivery semantics & limitations

• At-least-once, not exactly-once. Batches are retried and, on failure, spilled to disk and replayed on recovery. A bulk INSERT over the HTTP interface is not transactional, so a partial write followed by a retry/recovery can produce duplicate rows. The table is a plain MergeTree with no dedup. If you need exactly-once, dedup downstream (e.g. ReplacingMergeTree with Id in ORDER BY + FINAL, or an insert_deduplication_token). Treat the data as observability telemetry, where duplicates are tolerable.
• Backpressure drops, not unbounded buffering. When ClickHouse and the disk fallback are both failing, the in-memory queue is capped at Batch.MaxQueueSize and the fallback directory at Fallback.MaxTotalSize; past those, the newest/oldest entries are dropped with a throttled warning to protect memory and disk. Wire AddClickHouseHealthCheck() and watch the dropped-entry counter.
• Fallback files contain cleartext payloads. .jsonl fallback files store request/response bodies and client IPs in plain text. Point Fallback.Directory at an app-private path; the directory is created owner-only by default (Fallback.RestrictDirectoryPermissions).
• Body masking covers JSON, XML and form-urlencoded. Other captured textual types (e.g. text/plain) are not stored when masking is active (fail-closed) — only their size is recorded. Masking matches by field/element name; secrets embedded in values under non-listed keys are not detected.

Quick install

.NET 8+ (ASP.NET Core)

dotnet add package Iron.ClickHouseLogger

// Program.cs
builder.Services.AddClickHouseLogger(options =>
{
    // Single-node (dev / test)
    options.Host            = "clickhouse-server";
    options.Port            = 8123;  // ClickHouse HTTP port (8443 for TLS) — NOT native TCP 9000
    options.Database        = "logs";
    options.ApplicationName = "PaymentService";
    options.Environment     = builder.Environment.EnvironmentName;

    // Active-Passive multi-node (prod) — set Nodes instead of Host/Port
    // options.Nodes = new List<ClickHouseNodeOptions>
    // {
    //     new() { Host = "clickhouse-primary",   Port = 8123 },  // index 0 = primary
    //     new() { Host = "clickhouse-secondary",  Port = 8123 },  // index 1 = failover
    // };

    options.Log.ExcludePaths  = ["/health", "/swagger*"];
    options.Log.IndexedFields = ["CustomerId", "TenantId"];

    options.Fallback.Directory = "/var/log/clickhouse-fallback/";
});

// ...
app.UseClickHouseRequestResponseLogging();

.NET Framework 4.8 (Web API 2)

Install-Package Iron.ClickHouseLogger.Framework

// Global.asax.cs  Application_Start
ClickHouseLoggerFactory.Configure(options =>
{
    // Single-node (dev / test)
    options.Host            = "clickhouse-server";
    options.Port            = 8123;
    options.Database        = "logs";
    options.ApplicationName = "LegacyApp";
    options.Environment     = "Production";

    // Active-Passive multi-node (prod) — set Nodes instead of Host/Port
    // options.Nodes = new List<ClickHouseNodeOptions>
    // {
    //     new ClickHouseNodeOptions { Host = "clickhouse-primary",   Port = 8123 },
    //     new ClickHouseNodeOptions { Host = "clickhouse-secondary",  Port = 8123 },
    // };

    options.Log.ExcludePaths  = new[] { "/health" };
    options.Log.IndexedFields = new[] { "CustomerId", "TenantId" };

    options.Fallback.Directory = Server.MapPath("~/App_Data/clickhouse-fallback");
});
ClickHouseLoggerFactory.Initialize();

// Web API DelegatingHandler
GlobalConfiguration.Configuration.MessageHandlers.Add(
    ClickHouseLoggerFactory.CreateLoggingHandler());

// Application_End
protected void Application_End(object sender, EventArgs e)
{
    ClickHouseLoggerFactory.Shutdown();
}

Logging API (identical in both packages)

logger.Trace("Entering method");
logger.Debug("Cache miss", new { Key = cacheKey });
logger.Info("Order created",  new { OrderId = id, CustomerId = cid });
logger.Warning("Rate limit approaching", new { Remaining = 10 });
logger.Error("Payment failed", ex, new { Provider = "Stripe" });
logger.Fatal("Unhandled exception", ex);

logger.Metric("payment_duration_ms", sw.ElapsedMilliseconds,
    new { Provider = "Stripe", Success = "true" }, unit: "ms");

In the Core package, IClickHouseLogger is resolved via DI (IServiceCollection).
In the Framework package, it is accessed via ClickHouseLoggerFactory.Logger (static singleton set during Initialize()).

Security & data masking

HTTP request/response bodies are captured and stored (textual content types only, capped at Log.BodySizeLimit). Sensitive data is redacted before storage on three fronts:

• Headers — Log.MaskedHeaders (default: Authorization, Cookie, Set-Cookie, X-Api-Key).
• Body / query / extracted fields — Log.MaskedBodyFields redacts matching field names (case-insensitive) from JSON bodies (recursively), form-urlencoded bodies, query strings, and any extracted indexed/Extra fields. Defaults cover common credential/secret names (password, token, secret, apiKey, cardNumber, cvv, pin, …).

options.Log.MaskedBodyFields = new[] { "password", "token", "ssn", "iban" }; // override/extend
// options.Log.MaskedBodyFields = Array.Empty<string>();                     // disable body masking

⚠️ Limitations. Masking applies to JSON and form-urlencoded payloads. Non-textual bodies and bodies truncated past BodySizeLimit are not field-masked (truncated JSON can't be parsed). For endpoints handling regulated data (PII/PCI), exclude them via Log.ExcludePaths or disable body capture, and treat the masked-field list as a denylist you are responsible for keeping current.

Graceful shutdown

On shutdown the queued entries are flushed within GracefulShutdownTimeout (default 30s). On ASP.NET Core this runs inside an IHostedService.StopAsync, so the host's own shutdown timeout bounds it — ensure it is at least as large as GracefulShutdownTimeout:

builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(30));

On .NET Framework, call ClickHouseLoggerFactory.Shutdown() from Application_End.

Architecture overview

┌───────────────────────────────────────────────────┐
│  Application layer                                │
│  Middleware / HttpModule / DelegatingHandler      │
│  IClickHouseLogger (log + metric)                 │
└──────────────────────┬────────────────────────────┘
                       │  LogEntry { Type = "Http" | "Log" | "Metric" }
        ┌──────────────▼──────────────┐
        │  BatchProcessor<LogEntry>   │
        │  ConcurrentQueue            │
        │  flush-by-size | by-time    │
        │  Exponential-backoff retry  │
        └──────┬───────────┬──────────┘
               │           │
       ┌───────▼──────┐  ┌─────▼───────────┐
       │ClickHouse    │  │FallbackFileWriter│
       │  Writer      │  │  (.jsonl files)  │
       │  (multi-node)│  └─────────────────┘
       └───────┬──────┘
               │
    ┌──────────▼──────────────────────────────┐
    │  ConnectionManager                      │
    │  • polls all nodes (default every 30 s) │
    │  • prefers primary (index 0) always     │
    │  • auto-fails over to next healthy node │
    │  • auto-returns to primary on recovery  │
    │  TableManager — auto-DDL on startup     │
    └─────────────────────────────────────────┘

  Everything in this diagram below the application layer — BatchProcessor,
  ClickHouseWriter, ConnectionManager, FallbackFileWriter, TableManager,
  models, options, helpers — lives in the shared Iron.ClickHouseLogger.Core
  library (multi-targeted net48;net8.0). Only the application-layer integration
  (middleware / HttpModule / DI / factory) is package-specific.

Single-node (default): Host + Port — identical behaviour to v1.
Active-Passive multi-node: set options.Nodes with index 0 as primary; subsequent entries are failover candidates. ConnectionManager switches writes automatically — no application code changes required.

The .NET Framework variant replaces IHostedService lifecycle hooks with ClickHouseLoggerLifecycle.Initialize() / Shutdown() called from Global.asax.

Detailed documentation

Sample projects

ClickHouse table schema (reference)

One table per application. All log types share the same table; the Type column discriminates rows.

Table uses MergeTree, partitioned by toYYYYMM(Timestamp), ordered by (ApplicationName, Type, Timestamp). Full DDL in the Core docs.

License

MIT © Iron.LogHouse