GzipSeekable 0.1.0

GzipSeekable

Random-access (seekable) reading of gzip (DEFLATE) streams in .NET.

gzip streams are normally forward-only: reading byte N means decompressing everything before it. GzipSeekable gives you a standard seekable Stream over gzip data via a zran-style checkpoint index: one sequential decode builds it (persist it; later opens are instant), then any read decodes only from the nearest checkpoint — by default at most ~4 MiB away.

Everything is managed code (a vendored DEFLATE inflater from SharpZipLib with small, guarded resume additions) — no native binaries. Multi-member (concatenated) gzip is handled natively. Targets netstandard2.0 and net8.0.

Quick start

using GzipSeekable;

// Build a checkpoint index once, then seek freely. The build is one sequential decode
// (streaming and resumable if interrupted); the index persists to disk, so later opens
// are instant.
using var compressed = File.OpenRead(@"C:\images\disk.img.gz");
using var index = GzipIndex.LoadOrBuild(compressed, @"C:\images\disk.img.gzsi");
using var stream = new GzipIndexedStream(compressed, index);

stream.Position = 123_456_789;          // decodes from the nearest checkpoint, not from byte 0
var buffer = new byte[4096];
var read = stream.Read(buffer, 0, buffer.Length);

How it works

A DEFLATE checkpoint needs only two things: the last 32 KiB of decompressed history (the sliding window) and the sub-byte bit position where the next deflate block starts — this is exactly zlib's zran.c model. GzipIndex.LoadOrBuild decodes the stream once and drops a checkpoint at the first deflate block boundary past every span (GzipIndexOptions.TargetSpanBytes, default 4 MiB), storing each window DEFLATE-compressed and loading it lazily — only when a read actually resumes there. Runs of a constant byte are recorded as fill spans and served without touching the decoder at all. Each gzip member's CRC32 and length are verified during the build (GzipIndexOptions.VerifyMemberCrc), a sample of checkpoints is byte-verified with a shadow decoder (GzipIndexOptions.VerifiedSampleSpans), sealed records stream to <index>.wip as the build progresses (flat memory; an interrupted build resumes from its last sealed checkpoint), and the file is finalised atomically. The persisted format is "GZSKIDX1"; the index is fingerprinted against the compressed stream and rebuilt automatically if it doesn't match.

Multi-member gzip: every member start becomes a (stateless) checkpoint, and serving continues across member boundaries transparently.

Note for sequential consumers: each Read resumes from a checkpoint, so naive byte-at-a-time forward reading re-decodes spans repeatedly. Read span-sized chunks (see GzipIndexedStream.GetRecommendation) or put a cache in front.

License

MIT. Includes a vendored copy of the DEFLATE inflater from SharpZipLib (MIT). See LICENSE.