ManagedCompressions.Zstd 0.2.0

ManagedCompressions.Zstd

Zstandard (RFC 8878) compression and decompression for .NET, in fully managed C#.

• It contains no native code: no bundled library and no P/Invoke, so it runs wherever .NET runs, and it is compatible with trimming and Native AOT.
• It writes the same frames as libzstd. At every compression level, ManagedCompressions.Zstd writes exactly the bytes that libzstd 1.5.7 writes, with dictionaries, with long-distance matching and in multithreaded compression.
• It is fast. On arm64, decompression is 1.3–1.5× as fast as the libzstd that NativeCompressions and .NET ship, and 1.11–1.21× as fast as libzstd built from source at -O3 on the Silesia corpus. On x64, where libzstd has a hand-written assembly Huffman decoder, it is 1.0–1.2× as fast as NativeCompressions' libzstd. Compression of large inputs is within a few percent of libzstd, except on x64 at levels 1 to 5, which are up to 13 % slower (see Performance).
• A warm compressor or decompressor allocates nothing.

ManagedCompressions.Zstd is an independent implementation of the format. It is not affiliated with the zstd project or with Meta.

Requirements

ManagedCompressions.Zstd targets .NET 11 (net11.0). Its only dependency is System.IO.Hashing, for the XXH64 frame checksums.

dotnet add package ManagedCompressions.Zstd

From source, reference src/ManagedCompressions.Zstd/ManagedCompressions.Zstd.csproj.

Usage

Decompression

using System.Buffers;
using ManagedCompressions.Zstd;

var decompressor = new ZstdDecompressor();   // reuse it: it keeps its tables and buffers between frames

if (ZstdDecompressor.TryGetFrameContentSize(frame, out ulong size))
{
    byte[] content = new byte[size];
    OperationStatus status = decompressor.Decompress(frame, content, out int consumed, out int written);
}

Decompress decodes one whole frame per call:

After NeedMoreData or DestinationTooSmall, nothing has been consumed, so call again with the whole frame or a larger destination. Checksums are verified when a frame has one. A skippable frame decodes to nothing.

Compression

var compressor = new ZstdCompressor(level: 3);   // from ZstdCompressor.MinLevel (-131072) to MaxLevel (22)

byte[] frame = new byte[ZstdCompressor.GetMaxCompressedLength(content.Length)];
compressor.Compress(content, frame, out _, out int length);   // the frame is frame[..length]

Compress writes one whole frame per call, and the frame always declares its content size. Set AppendChecksum = true to end each frame with a checksum of its content. Level 0 selects ZstdCompressor.DefaultLevel, which is 3.

Dictionaries

byte[] dictionary = File.ReadAllBytes("records.dict");   // trained by `zstd --train`, or any raw content

var compressor = new ZstdCompressor(level: 3, dictionary);
var decompressor = new ZstdDecompressor(dictionary);

The dictionary is prepared once, in the constructor, and then used for every frame. ManagedCompressions.Zstd does not train dictionaries, so use zstd --train, or ZstandardDictionary.Train from System.IO.Compression.

A decompressor can also take the dictionary with each frame, which lets one decompressor serve frames of any dictionary:

var decompressor = new ZstdDecompressor();
OperationStatus status = decompressor.Decompress(frame, content, dictionary, out int consumed, out int written);

The dictionary is copied for the call and not kept, but its tables are kept for the next calls that pass the same dictionary. From then on, a frame costs a copy of its dictionary and nothing more, as if the decompressor had been created with it. Another dictionary rebuilds the tables without allocating.

Parallel compression

int length = await compressor.CompressAsync(content, frame, maxDegreeOfParallelism: 8);

CompressAsync cuts the source into jobs and compresses them on the thread pool. It writes the same frame as libzstd with worker threads (ZSTD_c_nbWorkers), so the frame does not depend on the degree of parallelism. Up to 512 KiB, it writes the same frame as Compress.

Threading

Instances are not thread-safe. Use one per thread, or one per concurrent operation.

Compatibility with libzstd

Decompression follows libzstd's decoding and validation, so the two accept and reject the same frames. ManagedCompressions.Zstd also enforces RFC 8878's limit on the size of a block in every case, where libzstd only partly enforces it in one-shot decoding.

Compression reproduces libzstd 1.5.7's ZSTD_compress byte for byte, at every level from -131072 to 22. This covers:

• the nine strategies, from fast to btultra2
• the pre- and post-block splitters
• the long-distance matching that libzstd turns on at level 22 for sources over 64 MiB

With a dictionary, ManagedCompressions.Zstd writes the frames libzstd writes when the dictionary is prepared at the same level (ZSTD_createCDict, then ZSTD_CCtx_refCDict), which is also how the ZstandardDictionary of System.IO.Compression uses it. CompressAsync reproduces libzstd's multithreaded compression (zstdmt), dictionaries and long-distance matching included.

The limitations are these:

• There is no streaming API. A frame is compressed or decompressed in one call, from and into memory.
• Sources and frames must fit in a span, so they are limited to 2 GiB.
• Compression takes a level, a dictionary and the checksum option. libzstd's advanced parameters, such as the window size or the strategy, are not exposed.

Performance

The ranges span compression levels 1 to 19. The reference here is libzstd built from source with Apple clang 21 at -O3, which decompresses some 20 % faster than the libzstd NativeCompressions and .NET ship: against those, decompression is 1.3–1.5× as fast on arm64. On x64 it is 1.0–1.2× as fast as NativeCompressions' libzstd, as the repository's README shows.

The measurements use the data of zstd's own benchmarks, which bench/zstd-corpus.sh downloads:

• silesia.tar is the Silesia corpus, which the benchmarks in zstd's README use, taken as one tar of 211,948,032 bytes compressed into one frame.
• github is the 500 JSON records of GitHub users from zstd's regression tests, 407,963 bytes in all. Each record is its own frame, and all frames go through one compressor or decompressor.
• github + dict is the same records with github.dict, the 110 KiB dictionary zstd's tests use with them.

Three implementations are compared:

• libzstd 1.5.7 compiled at -O3 with Apple clang 21, the reference
• ZstandardEncoder and ZstandardDecoder from .NET 11, which wrap the libzstd the runtime ships
• ManagedCompressions.Zstd, compiled with Native AOT

The machine is an Apple M4 Pro (arm64) running macOS 26.7, on one thread. The three implementations run in one process and take turns, and each figure is the median of its rounds. MB/s means millions of bytes of uncompressed data per second. At every level the three write identical frames, which the benchmark checks, so one compression ratio holds for all of them. Decompression decodes libzstd's frames at each level.

Compression ratio

Compression, MB/s

Compression, speedup over libzstd 1.5.7 (native)

Decompression, MB/s

Decompression, speedup over libzstd 1.5.7 (native)

To reproduce the measurements (macOS, .NET 11 SDK):

tools/native-ref/build.sh && bench/zstd-corpus.sh
dotnet publish bench/ManagedCompressions.Zstd.Perf -c Release -o artifacts/perf
artifacts/perf/ManagedCompressions.Zstd.Perf --corpus silesia,github,github-dict --results corpus.tsv --markdown corpus.md

On Windows x64, from Git Bash, the reference is built with a GNU-compatible clang such as llvm-mingw, because MSVC builds neither zstd's assembly Huffman decoder nor its BMI2 dispatch, which libzstd has wherever a GNU-compatible compiler builds it. --core N keeps the timed thread on one logical processor:

CC=<llvm-mingw>/bin/clang tools/native-ref/build.sh && bench/zstd-corpus.sh
dotnet publish bench/ManagedCompressions.Zstd.Perf -c Release -o artifacts/perf
artifacts/perf/ManagedCompressions.Zstd.Perf --core 2 --corpus silesia,github,github-dict --results corpus.tsv --markdown corpus.md

Testing

Correctness is checked against libzstd itself:

• Differential tests against the libzstd of .NET's System.IO.Compression: ManagedCompressions.Zstd must give back the content libzstd decodes, and write libzstd's compressed frames byte for byte, with and without dictionaries.
• zstd's golden files, for decompression, decompression errors and dictionaries.
• A decodecorpus corpus: random valid frames from zstd's own generator. They exercise the modes an encoder rarely emits, such as RLE and repeated tables, treeless literals and direct Huffman weights.
• Fuzzing, with buffers placed against guard pages.
• Multithreaded compression, checked against libzstd built with threads. These tests are skipped until tools/native-ref/build.sh mt has built that library.

The golden files and the decodecorpus corpus are binary files that are generated rather than kept in the repository. tools/native-ref/testdata.sh writes them from zstd 1.5.7's sources into tests/ManagedCompressions.Zstd.Tests/testdata/. The tests that read them are skipped without them.

Building

tools/native-ref/testdata.sh
dotnet build -c Release
dotnet test

The SDK version is pinned in global.json. testdata.sh downloads the zstd 1.5.7 sources once, checks their SHA-256, builds zstd's decodecorpus with the system C compiler and writes the test data. It runs on macOS and Linux, and on Windows from Git Bash with a GNU-compatible clang as CC. The native reference that tools/native-ref/build.sh builds is only needed for the multithreaded compression tests and for the benchmarks: dylibs on macOS, DLLs on Windows (with the same CC), and shared objects elsewhere.

Repository layout

License

ManagedCompressions.Zstd is licensed under the MIT License. Its compressor is derived from libzstd 1.5.7, and that part remains under zstd's BSD license, as does the test data copied from the zstd repository. See NOTICE, which the package carries.