ByTech.EmbeddedCommitLog 1.0.0

Persistent Embedded Commit Log (PECL)

PECL — Persisted Embedded Commit Log. A Kafka-inspired, crash-safe, append-only log that runs entirely in-process. No broker. No network. No ops overhead.

Targets 500K+ records/sec on NVMe with batched durability on .NET 9.

Why PECL?

Use PECL when you need durable, replayable, fan-out messaging inside a single .NET process — without standing up a broker.

Installation

dotnet add package ByTech.EmbeddedCommitLog

Requires .NET 9.0 or later.

Quick Start

Write records

using ByTech.EmbeddedCommitLog.Pipeline;

var config = new PipelineConfiguration
{
    RootDirectory = "/var/data/myapp-log",
};

using var pipeline = new Pipeline(config);
pipeline.Start();

// Append returns the globally monotonic sequence number assigned to the record.
ulong seqNo = pipeline.Append("hello world"u8);

pipeline.Flush(); // optional — ensures data is durable before proceeding
pipeline.Stop();

Push-mode consumer (sink)

Implement ISink — PECL delivers batches to your sink on a background task:

using ByTech.EmbeddedCommitLog.Consumer;
using ByTech.EmbeddedCommitLog.Pipeline;
using ByTech.EmbeddedCommitLog.Sinks;

public sealed class ConsoleSink : ISink
{
    public Task WriteAsync(IReadOnlyList<LogRecord> batch, CancellationToken ct)
    {
        foreach (LogRecord record in batch)
            Console.WriteLine($"[{record.Header.SeqNo}] {record.Payload.Length} bytes");
        return Task.CompletedTask;
    }
}

// Register before Start().
pipeline.RegisterConsumer("console-consumer");
pipeline.RegisterSink("console-consumer", "console-sink", new ConsoleSink());
pipeline.Start();

// ... append records ...

pipeline.Stop(); // drains all pending records before returning

Pull-mode consumer

Pull records on demand — your loop, your pace:

pipeline.RegisterConsumer("audit-consumer");
pipeline.Start();

// ... append records + Flush() ...

while (true)
{
    var result = pipeline.TryRead("audit-consumer");
    if (!result.IsSuccess) break; // no more records currently available

    LogRecord record = result.Value;
    Console.WriteLine($"[{record.Header.SeqNo}] {record.Payload.Length} bytes");
}

Configuration

All options are set on PipelineConfiguration (a sealed record — all properties are init-only). Only RootDirectory is required; every other property has a production-ready default.

Writer

Durability

Backpressure

Retention

GC

Observability

API Overview

Pipeline

The single entry point for all log operations.

// Lifecycle
void Start();
void Stop();       // graceful — drains push-mode consumers
void ForceStop();  // immediate — does not wait for consumer drain
void Dispose();

// Write
ulong Append(ReadOnlySpan<byte> payload,
             ContentType contentType = ContentType.Unknown,
             uint schemaId = 0);
void Flush();      // fsync the active segment

// Consumer registration (call before Start)
void RegisterConsumer(string consumerName);
void RegisterSink(string consumerName, string sinkName, ISink sink);

// Pull-mode read (call after Start)
Result<LogRecord, PeclError> TryRead(string consumerName);

// State
PipelineState State { get; }
PipelineMetrics Metrics { get; }

ISink — push-mode consumer

public interface ISink
{
    Task WriteAsync(IReadOnlyList<LogRecord> batch, CancellationToken ct);
}

PECL calls WriteAsync with a batch of one or more records. The method must not return until the batch is durably committed (throw on failure). Exceptions are captured and surfaced in the AggregateException thrown by Stop().

LogRecord

Immutable read-side envelope delivered to sinks and returned by TryRead:

public sealed record LogRecord(RecordHeader Header, byte[] Payload);

Record metadata is accessed via Header:

ulong seqNo        = record.Header.SeqNo;        // globally monotonic sequence number
ContentType ct     = record.Header.ContentType;  // advisory encoding hint
RecordFlags flags  = record.Header.Flags;
uint schemaId      = record.Header.SchemaId;

Metrics (System.Diagnostics.Metrics)

PECL registers instruments under the meter named by MeterName (default "pecl"). Observable via OpenTelemetry, dotnet-monitor, or any MeterListener.

Performance

Benchmarks: Intel Core i9-14900K, 64 GB RAM, NVMe 2 TB, Windows 11, .NET 9.0.14, BenchmarkDotNet v0.14.0, Release build.

The 500K/sec target is met under Batched durability on NVMe.

On-Disk Layout

{RootDirectory}/
  segments/
    log-000000.seg    ← sealed segments
    log-000001.seg
    log-000002.seg    ← active segment (open for writing)
  cursors/
    my-consumer.cur   ← one file per registered consumer
  checkpoint.dat      ← crash-recovery snapshot

Checkpoint and cursor files use a write-temp + fsync + rename strategy, guaranteeing crash safety even on unclean shutdown.

License

Apache 2.0 © 2026 Branimir Bajt