StreamLZ 1.4.0

StreamLZ

High-performance LZ compression library for .NET with streaming support.

Features

• Up to 8.7 GB/s decompress (level 6, silesia), down to 25.5% ratio (level 11, enwik8)
• Simple level scale (1-11) — higher = better ratio, slower
• Streaming — SLZ1 frame format supports files of any size
• Sliding window — cross-block match references for better ratio
• Parallel compress and decompress — automatic multi-threading at L6+ (see Threading Model)
• Async — CompressFileAsync, DecompressFileAsync, IAsyncDisposable on SlzStream
• Validation — TryDecompress (non-throwing), IsValidFrame, content checksums
• Zero allocations on the hot path (pooled scratch buffers)
• Native AOT and trimming compatible
• Targets net8.0 and net10.0

Installation

dotnet add package StreamLZ

Quick Start

using StreamLZ;

// Simplest: compress and decompress byte arrays (SLZ1 framed, self-describing)
byte[] compressed = Slz.CompressFramed(data);
byte[] restored = Slz.DecompressFramed(compressed); // no size tracking needed

// Compress / decompress files
Slz.CompressFile("input.txt", "output.slz");
Slz.DecompressFile("output.slz", "restored.txt");

// Stream-based (any size)
Slz.CompressStream(input, output, level: 6);

// Named compression levels
byte[] fast = Slz.CompressFramed(data, SlzCompressionLevel.Fast);
byte[] max = Slz.CompressFramed(data, SlzCompressionLevel.Maximum);

Compression Levels

See Threading Model below for details on how parallelism works at each level.

API

StreamLZ offers three API tiers. Choose based on your use case:

Framed in-memory (simplest — self-describing round-trip)

Uses the SLZ1 frame format. Output includes size metadata so decompression needs no external information. Best for storing/transmitting compressed blobs.

byte[] compressed = Slz.CompressFramed(data);
byte[] restored = Slz.DecompressFramed(compressed);

// Named levels for readability
byte[] fast = Slz.CompressFramed(data, SlzCompressionLevel.Fast);

Raw in-memory (zero-copy — caller manages buffers)

No framing. Caller must track the original size and provide output buffers (including Slz.SafeSpace extra bytes for decompression). Best for hot paths where you control the buffer lifecycle.

int bound = Slz.GetCompressBound(data.Length);
byte[] dst = new byte[bound];
int compSize = Slz.Compress(data, dst, level: 3);

byte[] output = new byte[originalSize + Slz.SafeSpace];
Slz.Decompress(compressed, output, originalSize);

// Non-throwing variant for untrusted data
if (Slz.TryDecompress(compressed, output, originalSize, out int written))
    // success

Important: Raw and framed formats are not interchangeable. Data compressed with Compress must be decompressed with Decompress (not DecompressFramed), and vice versa.

File and stream (any size, SLZ1 framed)

Uses the SLZ1 frame format with a sliding window for cross-block match references. Supports files of any size with bounded memory usage.

// Sync
Slz.CompressFile("input.txt", "output.slz");
Slz.DecompressFile("output.slz", "restored.txt");
Slz.CompressStream(input, output, level: 6);
Slz.DecompressStream(input, output);

// Async
await Slz.CompressFileAsync("input.txt", "output.slz", cancellationToken: ct);
await Slz.DecompressFileAsync("output.slz", "restored.txt", cancellationToken: ct);

// With content checksum for integrity verification
Slz.CompressFile("input.txt", "output.slz", useContentChecksum: true);

// Limit compression threads (for server workloads)
Slz.CompressFile("input.txt", "output.slz", maxThreads: 4);

SlzStream (GZipStream-style wrapper)

// Compress (supports await using for async disposal)
await using var compressStream = new SlzStream(outputStream, CompressionMode.Compress);
inputStream.CopyTo(compressStream);

// Decompress
await using var decompressStream = new SlzStream(inputStream, CompressionMode.Decompress);
decompressStream.CopyTo(outputStream);

// With options
var options = new SlzStreamOptions
{
    Level = 9,
    UseContentChecksum = true,
    LeaveOpen = true
};
await using var stream = new SlzStream(inner, CompressionMode.Compress, options);

Note: Disposing an SlzStream in compress mode without writing any data produces no output. To get a valid empty SLZ1 stream, write at least one byte, or use CompressFramed(ReadOnlySpan<byte>.Empty).

Validation

bool valid = Slz.IsValidFrame(compressedData);
bool valid = Slz.IsValidFrame(stream); // rewinds if seekable

JIT warmup (optional)

// Called automatically on first use of Slz. Call explicitly at app
// startup to move the ~15ms JIT cost to a predictable point.
Slz.WarmUp();

Comparison vs LZ4, Snappy, Zstd

enwik8 (100 MB text, 3-run median)

silesia (212 MB mixed, 3-run median)

All benchmarks on Intel Arrow Lake-S (Ultra 9 285K), 24-core, .NET 10.

Threading Model

StreamLZ uses different threading strategies depending on the compression level:

• L1-L5 (Fast codec): Single-threaded compress and decompress. The high decompress throughput (5+ GB/s) comes from the simple token format, not parallelism.

• L6-L8 (High codec, self-contained): Fully parallel. Chunks are grouped (4 × 256KB = 1MB per group) and each group is assigned to one thread. Within a group, chunks are compressed/decompressed sequentially with cross-chunk context, giving the match finder a larger search window. Between groups there are no references, preserving full parallelism across all available cores.

• L9-L11 (High codec, sliding window): Compression is single-threaded because chunks reference previous output via a sliding window. Decompression uses a batched two-phase approach that processes chunks in batches of ProcessorCount (e.g. 24 on a 24-core machine). For each batch:

  1. Phase 1 (parallel): ReadLzTable runs on all chunks in the batch simultaneously — this decodes the entropy streams (Huffman/tANS) and unpacks offsets, which is the most CPU-intensive part.
  2. Phase 2 (serial): ProcessLzRuns resolves tokens and copies literals/matches for each chunk in order, since match copies can reference output from earlier chunks.

  Then the next batch starts. This yields ~47% faster decompression than fully serial on a 24-core machine.

Compression thread count can be limited with the maxThreads parameter (e.g. for server workloads). Decompression threading is automatic and cannot be disabled.

License

MIT