ManagedCompressions.Lz4 0.2.0

ManagedCompressions

Zstandard and LZ4 written entirely in C#. No native library and no P/Invoke: the code runs wherever .NET runs, trims, and compiles with Native AOT.

• Same bytes as the native libraries. Every frame and block is byte-for-byte what libzstd 1.5.7 and lz4 1.10.0 write, at every level, so the compression ratios are theirs. The tests check this.
• Faster where it counts. Zstandard decompresses 1.3 to 1.5× faster than native libzstd on arm64, and 1.0 to 1.2× on x64, where native libzstd has hand-written assembly. LZ4 decompresses 0 to 4 % faster than native lz4. On small records, compression is up to 1.5× faster with Zstandard and up to 5.7× with LZ4.
• At native speed elsewhere. Zstandard compresses large inputs within about 5 % of libzstd, either way, except on x64 at levels 1 to 5, which are up to 13 % slower. LZ4 compresses 3 to 7 % slower at level 1, and up to 15 % slower at its high-compression levels.
• No allocation per call. A compressor or decompressor is created once and reused.

The speeds are measured against NativeCompressions, Cysharp's binding of the native libraries, and against .NET 11's own System.IO.Compression Zstandard.

At a glance

Apple M4 Pro (arm64) and AMD Ryzen 9 7950X (x64). silesia.tar (211 MB) compressed or decompressed in one call, on one thread, with BenchmarkDotNet. Native is NativeCompressions; JIT and AOT are the managed code under the JIT and under Native AOT. Throughputs are in MB/s of uncompressed data, and × is the managed speed divided by the native speed on the same machine: above 1.00 is faster, and bold is at least 10 % faster.

The levels are the ones in use. For Zstandard: 1, the fastest of the regular levels; 3, the default of zstd, NativeCompressions and .NET; 9; and 19, the highest before zstd's --ultra levels. For LZ4: 1, LZ4_compress_default; 9, lz4's default high-compression level; and 12, its highest.

Zstandard

Apple M4 Pro, arm64

AMD Ryzen 9 7950X, x64

LZ4

Apple M4 Pro, arm64

AMD Ryzen 9 7950X, x64

.NET 11's own Zstandard (ZstandardEncoder, ZstandardDecoder) is the same libzstd. On arm64 it runs at NativeCompressions' speed: 0.98× to 1.02× of it, except for level 19's compression under the JIT (0.93×). That exception is noise. Zstandard's level 19 and LZ4's level 12 take 11 to 45 seconds a call, so they are measured on three calls instead of ten, and at level 19 NativeCompressions' own time varied by 5 % from one job to the other (43.0 s, then 45.5 s) where .NET's and the managed code's varied by 1 to 2 %: read level 19's compression row as on par. On x64, .NET's libzstd is slower than NativeCompressions': 0.82× to 1.06× of it. Level 19 is as noisy there: NativeCompressions took 41.7 s, then 43.3 s, and .NET 43.4 s, then 40.8 s. The full BenchmarkDotNet tables, with Brotli and GZip for scale, are in bench/results/arm64/benchmarkdotnet-silesia.md and bench/results/x64/benchmarkdotnet-silesia.md.

Quick start

The packages target .NET 11:

dotnet add package ManagedCompressions.Zstd
dotnet add package ManagedCompressions.Lz4

Zstandard

using ManagedCompressions.Zstd;

var compressor = new ZstdCompressor(level: 3);        // -131072 to 22; reuse it, it keeps its tables
byte[] frame = new byte[ZstdCompressor.GetMaxCompressedLength(data.Length)];
compressor.Compress(data, frame, out _, out int frameLength);

var decompressor = new ZstdDecompressor();           // reuse it too
ZstdDecompressor.TryGetFrameContentSize(frame, out ulong contentSize);
byte[] content = new byte[contentSize];
decompressor.Decompress(frame.AsSpan(0, frameLength), content, out _, out _);

Dictionaries (new ZstdCompressor(level, dictionary)), checksums (AppendChecksum = true) and parallel compression (CompressAsync, which writes the same frames as libzstd with worker threads) are covered in src/ManagedCompressions.Zstd/README.md.

LZ4

using ManagedCompressions.Lz4;

var compressor = new Lz4Compressor(level: 1);        // 1 = LZ4_compress_default, below 1 = faster, 2-12 = high compression
byte[] block = new byte[Lz4Compressor.GetMaxCompressedLength(data.Length)];
int blockLength = compressor.Compress(data, block);

byte[] content = new byte[data.Length];              // a block does not record its size: store it beside
Lz4Block.Decompress(block.AsSpan(0, blockLength), content);

Lz4Block.TryDecompress reads into a larger destination and returns false on a corrupt block. Lz4Compressor.TryCompress returns false when the destination is too small.

Performance, level by level

Under the JIT, on two machines: an Apple M4 Pro (arm64, macOS) and an AMD Ryzen 9 7950X (x64, Windows). Each cell is the managed speed divided by the native speed (NativeCompressions at its fastest, with its context reused), on that machine. Bold means at least 10 % faster. The workloads are:

• silesia.tar as one 211 MB frame or block;
• the same file cut into 64 KiB pages, each its own frame or block, as a columnar file or a network protocol compresses data;
• 500 JSON records of about 800 bytes each, zstd's own regression data, each its own frame or block;
• zstd only: the same records with their 110 KiB dictionary.

Zstandard

Decompression is faster everywhere: 1.3 to 1.5× on arm64, 1.0 to 1.2× on x64. The native side is stronger on x64. NativeCompressions builds its win-x64 libzstd with GCC, which compiles zstd's hand-written assembly Huffman decoder and its BMI2 code paths. Both exist for x86-64 only, so the arm64 build has no counterpart. Compression of large inputs runs at the native speed, give or take 5 %: the compressor runs libzstd's algorithms step for step to write its exact bytes, so it does the same work. The exception is x64 at levels 1 to 5, the fast, double-fast and greedy strategies, which are 4 to 13 % slower. Small records compress faster because the managed code has a smaller fixed cost per frame.

LZ4

lz4 clears its whole state before every block: 16 KiB for levels 1 and below, 256 KiB for levels 2 to 12. The managed compressor keeps its state from one block to the next, and numbers each block's positions past the last block's, so that whatever earlier blocks left reads exactly as a cleared entry. The bytes are unchanged, and small blocks skip that cost. The gain is smaller on x64, where native lz4 compresses these records up to 1.7× as fast as on arm64, and the managed code at the same speed. On large blocks the high-compression levels are slower, by up to 14 % on arm64 and 9 % on x64: their search through the hash chains is bound by memory latency.

Compression ratios

The managed compressors write the native bytes, so the ratios below hold for both sides, on arm64 and x64 alike.

What it supports

Not supported: streaming (a frame or block is compressed or decompressed in one call, from and into memory), inputs over 2 GiB, the LZ4 frame format, and libzstd's advanced parameters (window size, strategy).

How it was measured

• Machines: an Apple M4 Pro (arm64) under macOS 26.7, and an AMD Ryzen 9 7950X (x64, Zen 4) under Windows 11 (build 26200). Both run .NET 11.0.0-rc.1, on one thread. On Windows the benchmarks run at a high priority, and the level-by-level one keeps its timed thread on one logical processor (--core 2), as macOS keeps it on a performance core.
• Native side: NativeCompressions 1.1.0, which ships zstd 1.5.7 and lz4 1.10.0 built at -O3. On arm64, its libzstd decompresses at the speed of the one .NET ships. A libzstd built from source with Apple clang 21 decompresses some 20 % faster: against that one, Zstandard decompression is 1.11–1.21× as fast, as src/ManagedCompressions.Zstd/README.md shows. On x64, NativeCompressions' libraries are built with GCC 16.2 (MSYS2). That libzstd has zstd's assembly Huffman decoder and its BMI2 code, which only exist for x86-64.
• Level-by-level tables: bench/ManagedCompressions.Bench times the native and the managed side in alternating rounds and reports each one's median, after a warm-up that gives the JIT its final, profiled code. Its first round checks that the managed output matches the native output byte for byte, and that everything decodes back. The raw results are in bench/results/arm64/levels.md and bench/results/x64/levels.md.
• At-a-glance tables: bench/ManagedCompressions.Benchmarks runs BenchmarkDotNet the way NativeCompressions' README does, under the JIT and under Native AOT. Each case first runs the managed codec on the first MiB of silesia.tar, so that the JIT has given it its final, profiled code, as a running service would have it: a single call of 211 MB would otherwise be measured in an earlier tier. Native AOT compiles for the machine's instruction set: native on arm64, x86-64-v4 on x64, where ILC 11.0.0-rc.1 with native compiles a branch on AvxVnni.IsSupported into an infinite loop and BenchmarkDotNet's CPU detection hangs (dotnet/runtime#135603).

To reproduce, start from the data and the native test tools:

bench/zstd-corpus.sh                 # silesia.zip and zstd's JSON records, downloaded and checked
tools/native-ref/testdata.sh         # zstd's golden files, for the tests
tools/native-ref/build.sh mt         # libzstd with threads, the oracle of the parallel compression tests

Then run the benchmarks:

dotnet run -c Release --project bench/ManagedCompressions.Bench -- --codecs zstd,lz4 --seconds 3
dotnet run -c Release --project bench/ManagedCompressions.Benchmarks -- --filter '*'

On Windows, add --core 2 to the first command to keep its timed thread on one logical processor.

Finally, run the tests:

dotnet test -c Release --solution ManagedCompressions.slnx

The tests check the compressors against the native libraries byte for byte, at every level. They also check the decoders against every frame and block the native libraries write, and against zstd's golden files and decodecorpus frames. Fuzzing runs with each buffer placed against an inaccessible memory page, so that a single byte read or written out of bounds faults.

CI builds everything and runs the tests on every pull request, the benchmarks excepted: their figures mean something on a quiet machine only. It also runs the repository's prek hooks, betterleaks for secrets and zizmor for the workflows; prek install runs them before each commit. Each merged pull request updates a draft GitHub release; publishing it runs the same checks on its tag, then pushes the packages to nuget.org.

Repository

License

ManagedCompressions is licensed under the MIT License. The compressors are derived from libzstd (BSD-3-Clause) and lz4 (BSD-2-Clause), and those parts remain under their licenses: see NOTICE, which each package carries. This project is not affiliated with the zstd or lz4 projects, nor with Cysharp.

What the decoders guarantee on untrusted input, and how to report a vulnerability, is in SECURITY.md.