RecursiveAI.OneCompress 0.3.0-preview.1

onearchive-csharp

Compression codecs and archive formats for .NET, written from first principles in managed code. No native dependencies; the compressed bytes are identical on every platform.

Two packages:

• RecursiveAI.OneCompress - codecs and checksums.
• RecursiveAI.OneArchive - archive containers and a simple Archive API, built on OneCompress.

Status: 0.3.0-preview.1. OneCompress has Zstandard (frames, skippable frames, dictionaries and dictionary training, long-distance matching, multithreaded compression in fixed-size jobs, the seekable format), LZ4 (block and frame), Deflate, zlib, gzip and Deflate64, LZMA, LZMA2, xz (with the branch and delta filters) and lzip, bzip2, Brotli (with its built-in dictionary), the LZX decoder, and CRC32, CRC32C, CRC64, Adler-32 and xxHash64. OneArchive reads and writes tar, zip, 7z, cpio and ar, with zip and 7z encryption, reads cab, and handles single compressed streams. Everything is preview: see below.

byte[] packed = Zstd.Compress(data, new ZstdCompressionOptions { Level = 19 });
byte[] again = Zstd.Decompress(packed);

Build

Requires the .NET 10 SDK (see global.json).

build/ci.sh              # build, test (Checked configuration), NativeAOT/trim smoke
build/ci.sh wasm         # browser-wasm build and a headless run under Node (needs the wasm-tools workload)

Build output goes to artifacts/ (ignored by git). Set ONEARCHIVE_ARTIFACTS to put it elsewhere, for example one directory per git worktree.

Preview codecs

Zstandard, LZ4, Deflate, zlib, gzip and the LZMA family (LZMA, LZMA2, xz, lzip) are preview while their speed is below the reference tools' in some rows. Their compression ratio and byte-identical output are already at the target; speed is what is still being tuned. The bzip2 decoder and the lzip decoder meet their targets and stay preview with the rest of this release. Brotli, the bzip2 encoder, Deflate64 and LZX are preview until they have been measured on the benchmark machines. The archive layer is preview while the containers are being added and the formats' interop and fuzz coverage grows. PPMd and BCJ2 are not supported: archive entries that use them are listed, and opening them is a typed "unsupported method" error. Their public types carry [Experimental], so using them is a compiler error until you opt in per codec:

<PropertyGroup>
  <NoWarn>$(NoWarn);ONECOMPRESS001;ONEARCHIVE001</NoWarn>
</PropertyGroup>

or #pragma warning disable ONECOMPRESS001 around the code that uses it. The marker comes off once the codec meets its speed target.

Where they stand

The target for every codec: compressed size within 1% of the reference tool, and compress and decompress speed at least 0.87 of the reference tool's, at every level. Measured memory to memory against zstd 1.5.7, lz4 1.10.0, libdeflate 1.26 (one call), zlib-ng 2.3.3 (streaming), bzip2 1.0.8, xz 5.8.4, lzip 1.26 and 7-Zip 7zz 26.05, single-threaded unless noted, on Silesia and enwik8 and on small game snapshot payloads with trained dictionaries. Speed is ours divided by the reference (1.00 = same speed). x64 is an AMD Ryzen 9 7950X3D on Linux; arm64 is an Apple M2 on macOS. .NET 10.0.402 with the JIT.

Peak memory of the LZMA family's in-memory calls is 0.5-2 GB on these corpora, because they hold the whole input and output; the tools stream (5-675 MB). Use the stream types to bound it.

Output is byte-identical across x64 and arm64, the JIT and NativeAOT, with and without hardware intrinsics, at any thread count, and in the browser (WebAssembly).

Archives

Archive.Extract("data.tar.zst", "out");                         // safe by default: no writes outside "out", limits on
Archive.Create("backup.tar.zst").Add("assets/").Save();          // format and compression from the extension
using var archive = Archive.Open("data.tar");                  // entries, random access when the source can seek

Read and written: tar (ustar, pax, GNU, sparse files); zip (store, Deflate, Deflate64, bzip2 and Zstandard; LZMA and xz entries read; zip64, data descriptors; WinZip AES, and traditional encryption for reading); 7z (Copy, Deflate, Deflate64, BZip2, LZMA, LZMA2, Delta, the branch filters, Zstandard; solid folders, encoded headers, AES-256); cpio (newc, crc, odc) and ar (common, GNU, BSD). Read only: cab (stored, MSZIP, LZX). Single streams (.gz, .zst, .lz4, .bz2, .xz, .lz, and .br, which is recognised by its name) and tar inside them (.tar.gz, .tar.zst, ...) are one archive. AES needs System.Security.Cryptography.Aes, which the browser does not provide. The other containers plug into the same reader and writer interfaces. The model, the safety defaults and limits, and the tar coverage are in docs/archive.md.

Layout

• src/OneCompress, src/OneArchive - the two packages.
• tests/ - xUnit tests.
• fuzz/ - in-process mutational fuzz driver and targets.
• bench/ - benchmarks; the only project that may reference comparison libraries.
• smoke/aot-trim, smoke/wasm - NativeAOT/trim and browser-wasm smoke apps.
• build/ci.sh - the script CI and developers run.

Licence

Licensed under either of MIT or Apache-2.0, at your option. Copyright (c) Recursive AI, LLC.