SharedFileJournal 0.1.5

SharedFileJournal

A cross-platform .NET library for high-speed concurrent multi-process append to a shared journal file — without per-write file locking.

Key Features

• Multi-process safe: Multiple processes can append concurrently via atomic offset reservation
• No file locks on writes: Uses Interlocked.Add on a memory-mapped metadata region for lock-free space reservation, then RandomAccess.Write at the reserved offset
• Recoverable format: Self-validating records (header with xxHash3 checksum, aligned to 16-byte boundaries) let readers detect and recover from partial/crashed writes
• Cross-platform: Works on Windows, Linux, and macOS with .NET 10+

Architecture

Single-file design

The journal is a single file containing a 4 KB metadata header followed by sequential record data. The first 4096 bytes are memory-mapped for atomic coordination; records start at offset 4096.

Atomic reservation strategy

The metadata file is mapped via MemoryMappedFile.CreateFromFile. A raw pointer to the NextWriteOffset field (at cache-line-aligned offset 64 within the file header) is used with Interlocked.Add for atomic fetch-add semantics. This works across processes because the mapping is backed by the same physical pages, and Interlocked compiles to hardware atomics (lock xadd on x86-64) that are coherent across all sharers.

Record format (16 bytes overhead)

Header (16 bytes): Magic "SFJR" (4B) | PayloadLength (4B) | Checksum (8B)
Payload:           Variable-length byte data
Padding:           0–15 bytes to align total record size to 16-byte boundary

Records are aligned to 16-byte boundaries so that recovery scanning can step by alignment instead of byte-by-byte, eliminating chunk-overlap logic.

Skip markers

When ReadAll encounters a gap (from a crashed writer) and scans forward to find the next valid record, it writes a skip marker at the gap start using an atomic 8-byte CAS via a memory-mapped pointer. Skip markers have the same 16-byte header layout with magic "SFJS" and PayloadLength set to the gap body size. Future readers see the skip marker and jump over the gap in O(1) instead of re-scanning.

Specification

See SPEC.md for the full file format specification, including byte-level header layouts, the read/write algorithms, corruption recovery, and the concurrency model.

Quick Start

using SharedFileJournal;

// Open (or create) a journal — safe for multiple processes
using var journal = new SharedJournal("/path/to/myjournal");

// Append records (thread-safe, process-safe)
journal.Append("hello"u8);
journal.Append(myPayloadBytes);

// Read all valid records
// Note: record.Payload is only valid until the next iteration.
// Copy with .ToArray() if you need to keep it.
foreach (var record in journal.ReadAll())
    Console.WriteLine($"offset={record.Offset} len={record.Payload.Length}");

// Compact: reclaim space from gaps and corrupted records (requires exclusive access)
SharedJournal.Compact("/path/to/myjournal");

API

Durability

Guarantees (V1)

• ✅ No two writers write to the same byte range
• ✅ Readers detect incomplete/corrupt tail records
• ✅ Multiple processes can append concurrently
• ✅ Compaction reclaims space from gaps left by crashed writers
• ✅ Readers automatically fix up gaps with skip markers for faster subsequent reads

Non-guarantees (V1)

• ❌ No stable global commit order beyond reservation order
• ❌ No indexing or deletion of individual records
• ❌ Not a transactional database or queue

Demo

# Run the stress test (4 threads × 10,000 records)
dotnet run --project SharedFileJournal.Demo -- stress

# Other commands
dotnet run --project SharedFileJournal.Demo -- init /tmp/myjournal
dotnet run --project SharedFileJournal.Demo -- write /tmp/myjournal "hello world"
dotnet run --project SharedFileJournal.Demo -- read /tmp/myjournal
dotnet run --project SharedFileJournal.Demo -- compact /tmp/myjournal