ConduitSharp.Plugin.BodyCapture 1.0.0

ConduitSharp BodyCapture Plugin

Plugins for the ConduitSharp API gateway that capture incoming HTTP request bodies and export them via structured OpenTelemetry logging — so you can ingest and search request bodies in downstream stacks like Grafana Loki.

Two variants ship here. Prefer body-capture-streaming.

Installation

Drop the compiled plugin DLL into your gateway's configured PluginsPath (e.g. ./gateway/plugins).

body-capture-streaming (recommended)

Declares ReadsRequestBody = false, so the route stays on the gateway's zero-copy streaming path — no memory buffer, no temp-file spill. Under the hood it runs ASP.NET Core's own HttpLogging middleware, whose RequestBufferingStream tees the first maxSize bytes into pooled memory as YARP streams the body upstream and drops the rest.

Heap cost is the captured prefix, never the body: a 10 MB upload and a 1 KB one allocate the same ~26 KB per request.

Example routes.json

{
  "id": "my-service",
  "route": {
    "match": { "path": "/api/myservice", "methods": ["POST"] }
  },
  "plugins": [
    {
      "name": "custom",
      "variant": "body-capture-streaming",
      "order": 10,
      "enabled": true,
      "config": {
        "maxSize": 4096
      }
    },
    {
      "name": "http-proxy",
      "order": 99,
      "enabled": true
    }
  ]
}

Options

Why maxSize is capped at 32 KiB

Capture memory rides the streaming path, which never reserves against Gateway:RequestLimits:MaxTotalBufferedBodyBytes — so unlike a buffered body, nothing downstream sheds load if it grows. 32 KiB keeps each captured prefix well under the 85 KiB large object heap threshold (and matches HttpLogging's own RequestBodyLogLimit default), so capture stays on pooled, gen-0-sized buffers no matter how many requests are in flight.

If you need whole bodies, send them to a dedicated audit sink — don't raise this to route full payloads through your log pipeline.

Behaviour inherited from HttpLogging

• Only recognized media types are logged. application/json, text/*, application/xml, and application/*+json are captured; application/octet-stream and other binary types stream through untouched and unlogged. Configure via HttpLogging's MediaTypeOptions.
• Truncation is explicit. An over-long body logs the prefix plus RequestBodyStatus: [Truncated by RequestBodyLogLimit].
• Requests with no Content-Type are not logged (a Debug-level No Content-Type header for request body is emitted instead).

body-capture (buffering)

Declares ReadsRequestBody = true, which forces the gateway to buffer the whole body into a rewindable stream — memory up to Gateway:RequestLimits:MemoryBufferThresholdBytes, then a temp-file spill — before the plugin runs. The gateway rejects a streamOnly route carrying this plugin at startup.

Enable with "name": "body-capture" (or "name": "custom", "variant": "body-capture"). Same maxSize option, uncapped, truncating with ... (truncated).

Reach for this only when a bounded prefix genuinely isn't enough. It puts every request body through the gateway's buffering budget, and a burst of large bodies sheds load with a 503.

Observability

Both variants log at Information through the host's ILoggerFactory, so records flow out through whatever it has wired up — including the gateway's OpenTelemetry logger provider (→ OTLP → Loki), enriched with the ambient trace/route attributes.

body-capture-streaming deliberately re-homes HttpLogging's loggers under its own category. HttpLogging natively logs under Microsoft.AspNetCore.HttpLogging.*, and every stock ASP.NET Core log config filters Microsoft.AspNetCore to Warning — which would drop every captured body silently while the plugin still looked healthy. Renaming means capture obeys the plugin's own log level, and the records land in Loki tagged as body-capture rather than buried in framework noise.

To turn capture down or off without touching routes.json:

"Logging": {
  "LogLevel": {
    "ConduitSharp.Plugin.BodyCapture": "Warning"
  }
}

Size the OTLP batch against your sink's message limit

Body capture makes every log record carry a multi-KB payload, so a batching exporter reaches its sink's maximum message size far sooner than ordinary logging does. Get this wrong and capture fails silently: the collector keeps accepting records, the gateway looks healthy, and nothing reaches the log backend.

Do the arithmetic before enabling capture in production:

bytes per push  ≈  maxSize  ×  records per batch

With maxSize: 4096 and an OpenTelemetry Collector batch processor left on its defaults (timeout only, no size cap), a busy route accumulates a whole flush interval of records into one push — measured on this repo's benchmark rig at ~15.9 MB. Grafana Loki's gRPC receiver caps a message at 4 MiB, so every push was rejected:

HTTP 503, Message=rpc error: code = ResourceExhausted
desc = grpc: received message larger than max (15894276 vs. 4194304)

The exporter then retries the same oversized batch forever. It never shrinks, so it never succeeds — only 3.4% of captured bodies reached Loki, and the only trace of the problem was in the collector's own log. Cap the batch on the sender, and raise the ceiling on the sink:

# otel-collector.yaml — bound records per push, not just the flush interval
processors:
  batch:
    timeout: 5s
    send_batch_size: 500
    send_batch_max_size: 1000

# loki.yaml — headroom so a burst of larger bodies degrades instead of deadlocking
server:
  grpc_server_max_recv_msg_size: 16777216   # 16 MiB
  grpc_server_max_send_msg_size: 16777216

Both sides matter: the cap bounds the common case, the raised limit bounds the tail.

To detect it, compare what your sink received against what it stored — they diverge under this failure. Watch for retries in the collector log (Exporting failed. Will retry); a steadily climbing received-count with a flat stored-count is the same oversized batch being re-sent, not traffic. Other sinks have their own ceiling (Kafka message.max.bytes, gRPC's own 4 MiB default); the arithmetic above is what changes, not the failure mode.

Security

Request bodies routinely carry credentials, tokens, and personal data. Both plugins log them verbatim — there is no redaction. Treat any route with body capture enabled as exporting its payloads to your log backend, and scope retention and access accordingly.