RecursiveAI.OneCompress 0.7.0-preview.1

onearchive-csharp

Compression codecs and archive formats for .NET, written from first principles in managed code. No native dependencies; the compressed bytes are identical on every platform.

Two packages:

• RecursiveAI.OneCompress - codecs and checksums.
• RecursiveAI.OneArchive - archive containers and a simple Archive API, built on OneCompress.

Status: preview. The latest release on nuget.org is 0.5.0-preview.1; this page describes the code on main, and the measurements below are from 0.5.0-preview.1. OneCompress has Zstandard (frames, skippable frames, dictionaries and dictionary training, long-distance matching, multithreaded compression in fixed-size jobs, the seekable format), LZ4 (block and frame), Deflate, zlib, gzip and Deflate64, LZMA, LZMA2, xz (with the branch and delta filters) and lzip, bzip2, Brotli (with its built-in dictionary), the LZX decoder, and CRC32, CRC32C, CRC64, Adler-32 and xxHash64, plus PPMd (variants H and I). OneArchive reads and writes tar, zip, 7z, cpio, ar and ISO 9660 (with Joliet and Rock Ridge), with zip and 7z encryption, reads cab and RAR 5 (listing, metadata and stored entries), and handles single compressed streams. Everything is preview: see below.

byte[] packed = Zstd.Compress(data, new ZstdCompressionOptions { Level = 19 });
byte[] again = Zstd.Decompress(packed);

Build

Requires the .NET 10 SDK (see global.json).

build/ci.sh              # build, test (Checked configuration), NativeAOT/trim smoke
build/ci.sh wasm         # browser-wasm build and a headless run under Node (needs the wasm-tools workload)

Build output goes to artifacts/ (ignored by git). Set ONEARCHIVE_ARTIFACTS to put it elsewhere, for example one directory per git worktree.

Preview codecs

Zstandard, LZ4, Deflate, zlib, gzip and the LZMA family (LZMA, LZMA2, xz, lzip) are preview while their speed is below the reference tools' in some rows. Their compression ratio and byte-identical output are already at the target; speed is what is still being tuned. Brotli (compress is below the tool's speed at qualities 0 and 5, and quality 11 is 2% larger) and PPMd (decompression at the target on both machines; compression 0.85-0.91 of 7-Zip's speed, same sizes) are preview too. The bzip2 codec, the lzip decoder, Deflate64 and LZX meet their targets on both machines and stay preview with the rest. The archive layer is preview while the formats' interop and fuzz coverage grows. BCJ2 is not supported: 7z entries that use it are listed, and opening them is a typed "unsupported method" error. RAR 5 is read only: compressed entries raise a typed error. Their public types carry [Experimental], so using them is a compiler error until you opt in per codec:

<PropertyGroup>
  <NoWarn>$(NoWarn);ONECOMPRESS001;ONEARCHIVE001</NoWarn>
</PropertyGroup>

or #pragma warning disable ONECOMPRESS001 around the code that uses it. The marker comes off once the codec meets its speed target.

Where they stand

The target for every codec: compressed size within 1% of the reference tool, and compress and decompress speed at least 0.87 of the reference tool's, at every level. Measured memory to memory against zstd 1.5.7, lz4 1.10.0, libdeflate 1.26 (one call), zlib-ng 2.3.3 (streaming), bzip2 1.0.8, xz 5.8.4, lzip 1.26 and 7-Zip 7zz 26.05, single-threaded unless noted, on Silesia and enwik8 and on small game snapshot payloads with trained dictionaries. Speed is ours divided by the reference (1.00 = same speed). x64 is an AMD Ryzen 9 7950X3D on Linux; arm64 is an Apple M2 on macOS. .NET 10.0.402 with the JIT.

The table is the 0.5.0-preview.1 measurement, with the rows for the codecs that changed in 0.5.1-preview.1 remeasured on Silesia: Zstandard levels 1, 3, 5, 9, 12 and 15, LZ4, Deflate in one call, and dictionary training. Those rows say "0.5.1".

Peak memory of the LZMA family's in-memory calls is 0.5-2 GB on these corpora, because they hold the whole input and output; the tools stream (5-675 MB). Use the stream types to bound it.

Output is byte-identical across x64 and arm64, the JIT and NativeAOT, with and without hardware intrinsics, at any thread count, and in the browser (WebAssembly).

Archives

Archive.Extract("data.tar.zst", "out");                         // safe by default: no writes outside "out", limits on
Archive.Create("backup.tar.zst").Add("assets/").Save();          // format and compression from the extension
using var archive = Archive.Open("data.tar");                  // entries, random access when the source can seek

Read and written: tar (ustar, pax, GNU, sparse files); zip (store, Deflate, Deflate64, bzip2, PPMd and Zstandard; LZMA and xz entries read; zip64, data descriptors; WinZip AES, and traditional encryption for reading); 7z (Copy, Deflate, Deflate64, BZip2, LZMA, LZMA2, PPMd, Delta, the branch filters, Zstandard; solid folders, encoded headers, AES-256); cpio (newc, crc, odc), ar (common, GNU, BSD) and ISO 9660 with Joliet and Rock Ridge. Microsoft compound files (.doc, .xls, .ppt, .msg; versions 3 and 4, storages as directories; docs/cfb.md) are read and written. Read only: cab (stored, MSZIP, LZX) and RAR 5 (list and metadata, stored entries, multi-volume sets; compressed entries raise a typed error). Single streams (.gz, .zst, .lz4, .bz2, .xz, .lz, and .br, which is recognised by its name) and tar inside them (.tar.gz, .tar.zst, ...) are one archive. AES needs System.Security.Cryptography.Aes, which the browser does not provide. An app that uses few codecs can trim the rest: set the RecursiveAI.OneArchive.AllCodecs feature switch to false and register what it needs with ArchiveCodecs.AddX() (docs/archive-codecs.md). The other containers plug into the same reader and writer interfaces. The model, the safety defaults and limits, and the tar coverage are in docs/archive.md.

Layout

• src/OneCompress, src/OneArchive - the two packages.
• tests/ - xUnit tests.
• fuzz/ - in-process mutational fuzz driver and targets.
• bench/ - the bar runner, micro-benchmarks and bench/Managed, which compares against the fastest managed .NET libraries; the only projects that may reference comparison libraries.
• tools/ - corpus and reference-tool scripts, table generators, the hash-matrix runner and the per-codec bench drivers.
• docs/ - normative notes on each codec and container (formats, tuning choices, benchmark method).
• smoke/aot-trim, smoke/wasm - NativeAOT/trim and browser-wasm smoke apps.
• build/ci.sh - the script CI and developers run.

Licence

Licensed under either of MIT or Apache-2.0, at your option. Copyright (c) Recursive AI, LLC.