CodecKit 1.0.0

CodecKit

Typed, bidirectional binary codec library for .NET. Define how a type is encoded and decoded once — CodecKit handles the rest.

// Define a codec
var pointCodec = Codec.Record<Point>()
    .Field("x", p => p.X, Codec.Float32LE)
    .Field("y", p => p.Y, Codec.Float32LE)
    .Build(fields => new Point((float)fields["x"], (float)fields["y"]));

// Encode
byte[] bytes = Codec.EncodeToArray(pointCodec, new Point(1.0f, 2.0f));

// Decode
Point p = Codec.Decode(pointCodec, bytes);

Contents

• Installation
• Overview
• Primitives
• Records
• Collections
• Choice (discriminated unions)
• Framing
• Checksum & integrity
• Compression
• Versioning
• Transforms & combinators
• Error handling
• Recovery

Installation

dotnet add package CodecKit

Targets netstandard2.1, net8.0, and net10.0.

Overview

All codecs implement ICodec<T>, which has two methods:

void Encode(T value, IBufferWriter<byte> writer, CodecContext ctx);
T    Decode(ref SequenceReader<byte> reader, CodecContext ctx);

You don't implement ICodec<T> directly — you compose codecs using the Codec static class.

Encode

// Returns a new byte array
byte[] bytes = Codec.EncodeToArray(codec, value);

// Writes into an existing IBufferWriter<byte>
Codec.Encode(codec, value, writer);

// Atomic: stages to a scratch buffer, only commits to the writer on success
Codec.EncodeAtomic(codec, value, writer);

Decode

// Throws CodecException on failure
MyType value = Codec.Decode(codec, bytes);

// Returns CodecResult<T> — no exceptions
CodecResult<MyType> result = Codec.TryDecode(codec, bytes);
if (result.IsSuccess)
    Console.WriteLine(result.Value);
else
    Console.WriteLine(result.Failure.ErrorCode);

Primitives

Fixed-width integers

Codec.UInt8           // byte
Codec.UInt16LE / BE   // ushort, little/big endian
Codec.UInt32LE / BE   // uint
Codec.UInt64LE / BE   // ulong
Codec.Int8            // sbyte
Codec.Int16LE / BE    // short
Codec.Int32LE / BE    // int
Codec.Int64LE / BE    // long

Variable-length integers

LEB128 encoding — compact for small values.

Codec.VarUInt32   // uint, 1–5 bytes
Codec.VarUInt64   // ulong, 1–10 bytes
Codec.VarInt32    // int, zig-zag encoded
Codec.VarInt64    // long, zig-zag encoded

Floats

Codec.Float32LE / BE   // float (IEEE 754)
Codec.Float64LE / BE   // double (IEEE 754)

Booleans

Strict: only 0x00 (false) and 0x01 (true) are accepted. Anything else throws.

Codec.Bool

Strings

Codec.Utf8String(length)        // fixed-length UTF-8 (pads with nulls on encode)
Codec.Utf8StringRemaining       // reads to end of available data

Bytes

Codec.BytesOwned(length)        // fixed-length blob, returns byte[]
Codec.BytesOwnedRemaining       // reads all remaining bytes
Codec.BytesBorrowed(length)     // returns ReadOnlySpan<byte> (zero-copy, only valid during decode)
Codec.BytesBorrowedRemaining    // same, reads all remaining bytes

GUIDs

Codec.GuidRfc4122    // network byte order (big endian)
Codec.GuidDotNet     // .NET runtime byte order

Magic & padding

// Encodes/decodes a fixed byte sequence. Throws MagicMismatch if bytes don't match.
var magic = Codec.Magic(0x89, 0x50, 0x4E, 0x47); // PNG header

// Writes/verifies a fixed-value filler block
var pad = Codec.Padding(4, 0x00); // 4 zero bytes

Skip

// Advances the reader without reading a value
var skip = Codec.Skip(8);

Records

Compose multiple fields into a single codec for a C# type.

record FileHeader(byte Version, uint Flags);

var codec = Codec.Record<FileHeader>()
    .Field("version", h => h.Version, Codec.UInt8)
    .Field("flags",   h => h.Flags,   Codec.UInt32LE)
    .Build(f => new FileHeader((byte)f["version"], (uint)f["flags"]));

Fields are encoded and decoded in the order they are registered.

Dependent fields

A field whose codec depends on the value of an earlier field:

var codec = Codec.Record<Payload>()
    .Field("length",  p => p.Length, Codec.UInt32LE)
    .Field("data",    p => p.Data,
        Codec.From<uint, byte[]>("length", len => Codec.BytesOwned((int)len)))
    .Build(f => new Payload((uint)f["length"], (byte[])f["data"]));

Collections

Fixed count

// Decode exactly 3 ints
ICodec<IReadOnlyList<int>> threeInts = Codec.Int32LE.Repeat(3);

Count-prefixed list

The count is written before the elements.

// [count: uint32][element][element]...
ICodec<IReadOnlyList<string>> list =
    Codec.Utf8String(16).RepeatPrefixed(Codec.UInt32LE);

Optional

A boolean flag indicates whether a value follows.

// [0x01][value] or [0x00]
ICodec<string?> opt = Codec.Utf8String(32).Optional(Codec.Bool);

Choice (discriminated unions)

A tag field selects which codec to use.

abstract class Shape { }
record Circle(float Radius) : Shape;
record Rectangle(float Width, float Height) : Shape;

var shapeCodec = Codec.Choice<Shape, byte>(
    Codec.UInt8,
    Codec.Case<Shape, Circle>    ((byte)0x01, "Circle",
        Codec.Record<Circle>().Field("r", c => c.Radius, Codec.Float32LE)
            .Build(f => new Circle((float)f["r"]))),
    Codec.Case<Shape, Rectangle> ((byte)0x02, "Rectangle",
        Codec.Record<Rectangle>()
            .Field("w", r => r.Width,  Codec.Float32LE)
            .Field("h", r => r.Height, Codec.Float32LE)
            .Build(f => new Rectangle((float)f["w"], (float)f["h"])))
);

Note: Tag literals must be explicitly cast to match the tag type (e.g. (byte)0x01), otherwise the runtime will throw on mismatched boxing.

Framing

Length-prefixed

Writes the body byte count before the body: [length][body].

// [uint32 length][body]
var framed = Codec.LengthPrefixed(Codec.UInt32LE, innerCodec);

Accepted length codec types: ICodec<int>, ICodec<uint>, ICodec<long>.

Fixed frame

Writes the body in a fixed-size slot, padding the remainder: [body][padding].

// Body is written, zero-padded to 64 bytes total
var fixed64 = Codec.FixedFrame(64, innerCodec);

Checksum & integrity

Wraps any codec with automatic checksum computation and verification.

// [data][crc32] — checksum appended after body
var safe = myCodec.WithChecksum(ChecksumAlgorithms.Crc32, ChecksumPlacement.Trailer);

// [xxhash64][data] — checksum prepended before body
var safe2 = myCodec.WithChecksum(ChecksumAlgorithms.XxHash64, ChecksumPlacement.Header);

On decode, the checksum is verified. A mismatch throws IntegrityException (error code ChecksumMismatch).

Available algorithms: Crc32, XxHash32, XxHash64

Placement options: Trailer, Header

Compression

Wraps any codec with transparent compression. Wire format: [compressed-length][compressed-payload].

var compressed = myCodec.WithCompression(
    CompressionAlgorithms.Zstd,
    CodecCompressionLevel.Optimal);

Available algorithms: Deflate, Brotli, Lz4, Zstd

Compression levels: Fastest, Optimal, SmallestSize

Versioning

Simple versioned union

Reads a version tag and dispatches to the matching codec. Good for in-place version fields.

var codec = Codec.Versioned<IMessage, byte>(
    Codec.UInt8,
    Codec.VersionCase<IMessage, MessageV1>((byte)1, "v1", v1Codec),
    Codec.VersionCase<IMessage, MessageV2>((byte)2, "v2", v2Codec)
);

Version envelope

[version][body-length][body] — unknown future versions are preserved as raw bytes rather than failing.

var codec = Codec.VersionEnvelope<IMessage, byte>(
    Codec.UInt8,
    Codec.VarUInt32,          // body-length codec (int/uint/long)
    (ver, raw) => new UnknownMessage(ver, raw),
    Codec.VersionCase<IMessage, MessageV1>((byte)1, "v1", v1Codec)
);

When a decoder encounters an unknown version, it reads and stores the raw bytes. If the bytes are later re-encoded, they pass through unchanged.

Transforms & combinators

Map

Convert between the wire type and your domain type.

// Encode/decode a Status enum as a single byte
var statusCodec = Codec.UInt8.Map(
    decode: b => (Status)b,
    encode: s => (byte)s);

Validate

Assert a constraint at encode and decode time.

var positive = Codec.Int32LE.Validate(
    predicate: v => v > 0,
    message:   v => $"Expected positive, got {v}");

Then

Sequence two codecs — first is processed, its result is discarded, then the second runs.

// Read the magic header, then decode the payload
var framed = magic.Then(payloadCodec);

Error handling

All codec errors derive from CodecException. Common subclasses:

Every CodecException carries:

• ErrorCode — machine-readable failure kind
• ByteOffset — position in the stream where the failure occurred
• Path — field path (e.g. cells[2].value) for nested codecs

Use TryDecode to handle failures without exceptions:

var result = Codec.TryDecode(myCodec, bytes);

switch (result.Failure.ErrorCode)
{
    case CodecErrorCode.Truncated:       /* short read */ break;
    case CodecErrorCode.ChecksumMismatch: /* corruption */ break;
    case CodecErrorCode.MagicMismatch:   /* wrong format */ break;
}

Recovery

Tools for extracting data from corrupt or partially-written streams.

Scan for magic bytes

Find all positions where a known header pattern occurs:

IReadOnlyList<long> offsets = CodecRecovery.ScanForMagic(
    source:     sequence,
    magicBytes: new byte[] { 0xDB, 0xDB });

Scan frames

Attempt to decode frames one by one. On failure, advance by one byte and try again:

IReadOnlyList<FrameScanResult<MyFrame>> results =
    CodecRecovery.ScanFrames(sequence, frameCodec);

foreach (var r in results)
{
    if (r.IsSuccess)
        Console.WriteLine($"Offset {r.Offset}: {r.Value}");
    else
        Console.WriteLine($"Offset {r.Offset}: {r.Failure.ErrorCode}");
}

Implementing your own codec

Implement ICodec<T> directly when the built-in combinators aren't sufficient:

public sealed class MyCodec : ICodec<MyType>
{
    public void Encode(MyType value, IBufferWriter<byte> writer, CodecContext ctx)
    {
        // Write bytes to writer
    }

    public MyType Decode(ref SequenceReader<byte> reader, CodecContext ctx)
    {
        // Read bytes from reader, throw CodecException on failure
        if (!reader.TryRead(out byte b))
            throw new InsufficientDataException(ctx.ByteOffset, ctx.Path, 1);
        return new MyType(b);
    }
}

Use ctx.Checkpoint and ctx.Rewind for transactional reads — if decoding fails partway through, rewind to leave the reader in a clean state:

var checkpoint = ctx.Checkpoint(ref reader);
try
{
    // ... attempt to decode ...
}
catch
{
    ctx.Rewind(ref reader, checkpoint);
    throw;
}

Testing

The CodecKit.Testing package provides framework-agnostic helpers (works with xUnit, NUnit, MSTest, etc.):

using CodecKit.Testing;

// Assert encode → decode round-trip produces an equal value
CodecAssert.Roundtrip(codec, value);

// Assert that decode → re-encode is byte-for-byte identical (canonical form)
CodecAssert.Canonical(codec, encodedBytes);

// Assert decode succeeds on segmented input (split at every possible boundary)
CodecAssert.DecodeMatchesOnSegmentedInput(codec, bytes);

// Assert decode throws a specific exception type and/or error code
CodecAssert.FailsWith<MyType, IntegrityException>(
    codec, bytes, CodecErrorCode.ChecksumMismatch);