MultiplexedFile 0.2.0

MultiplexedFile

A file multiplexer for .NET that lets multiple writers (channels) share a single underlying file stream, with each channel's data organized as an in-file doubly-linked list of blocks.

Useful when you need to interleave several independent logical streams into one physical file without an external index, while still being able to seek and append within each logical stream independently.

• Target framework: .NET 10
• Language: C# (latest)
• License: MIT
• Current version: 0.2.0 (three-layer architecture)

Features

• Multi-channel over one file — A MultiplexedFileStream wraps a single FileStream. Each channel is identified by a 64-bit integer ID and owns an independent logical byte stream.
• In-file doubly-linked list — Each channel's data is a chain of blocks. Block headers carry PrevOffset / NextOffset so you can walk the chain in either direction.
• Per-segment marks — Alloc / WriteStart / End marks (0xAA / 0xBB / 0xCC) are written into block headers per acquire-write-release cycle, and DataLength is recorded in the segment header on close.
• Backtrackable writes — Calling Seek walks the linked list. After backtracking, writes follow the existing chain and only allocate a new block when the current block is full and has no NextOffset.
• Three-layer architecture (v0.2) — BlockStream (low-level, exclusive)
  • ChannelStream (buffered wrapper) + MultiplexedFileStream (orchestrator).
• Buffered writes — ChannelStream accumulates writes in a 4KB in-memory buffer and only opens a BlockStream for actual I/O when the buffer fills or Flush is called. Reduces per-write lock acquisition overhead.
• Virtual cursor — ChannelStream.Seek / Position only update an in-memory cursor; the underlying file is not touched until a real read or write happens.
• Exclusive mode — ChannelStream.EnterExclusive() / ExitExclusive() let you hold a BlockStream open across many writes, bypassing the buffer for high-throughput batches.
• Auto-flush — ChannelStream.EnableAutoFlush(period) starts a timer (default 500ms) that periodically flushes the buffer — low thread load.
• Explicit block allocation — ExtendBlock(channelId, minCapacity) forces a fresh block at the channel tail, even if the current tail still has room.
• Crash-safe layout — Metadata is re-persisted after every block allocation, so the file is always self-consistent on disk.
• Dynamic channel registration — Channels can be added at any time via AddChannel. Metadata is rewritten at the file end on each close.

Install

dotnet add package MultiplexedFile --version 0.2.0

For local builds, pack and push to a local source:

dotnet pack src/MultiplexedFile/MultiplexedFile.csproj -c Debug --output ./nupkgs
./nuget-push.sh                                       # local source
./nuget-push.sh --source nuget.org --api-key <KEY>    # nuget.org

Quick start

using MultiplexedFile;
using System.Text;

string path = "demo.mfx";

// 1) Create a new multiplexed file
using (var mfs = MultiplexedFileStream.Create(path))
{
    mfs.AddChannel(channelId: 1);
    mfs.AddChannel(channelId: 2);

    // Write to channel 1 (buffered; flushed on Dispose)
    using (var cs = mfs.AcquireChannel(1))
    {
        byte[] data = Encoding.UTF8.GetBytes("Hello from channel 1");
        cs.Write(data, 0, data.Length);
    }

    // Write to channel 2 (a separate logical stream in the same file)
    using (var cs = mfs.AcquireChannel(2))
    {
        byte[] data = Encoding.UTF8.GetBytes("Channel 2 has its own blocks");
        cs.Write(data, 0, data.Length);
    }
}

// 2) Reopen later and read each channel independently
using (var mfs = MultiplexedFileStream.Open(path))
{
    foreach (var meta in mfs.GetChannelsSnapshot())
    {
        using var cs = mfs.AcquireChannel((long)meta.ChannelId);
        var buf = new byte[cs.Length];
        cs.Read(buf, 0, buf.Length);
        Console.WriteLine($"channel {meta.ChannelId}: {Encoding.UTF8.GetString(buf)}");
    }
}

Output:

channel 1: Hello from channel 1
channel 2: Channel 2 has its own blocks

Three-layer architecture (v0.2)

Per the design brief, the API is split into three layers:

+-------------------------+
| MultiplexedFileStream   |  Orchestrator. Owns the FileStream,
|                         |  channel registry, on-disk metadata.
+-------------------------+
            |
            |  AcquireChannel        AcquireBlockStream
            v                          v
+-------------------------+  +-------------------------+
| ChannelStream           |  | BlockStream             |
| (buffered wrapper)      |  | (low-level, exclusive)  |
|                         |  |                         |
| 4KB write buffer        |  | Direct file R/W         |
| Virtual cursor          |  | Holds SemaphoreSlim     |
| EnterExclusive() ->     |  | One alive at a time     |
|   holds a BlockStream   |  |                         |
+-------------------------+  +-------------------------+

• ChannelStream is what you usually want. Long-lived, buffered, multiple can be alive simultaneously (no lock held while buffering). Flush / Read / EnterExclusive temporarily acquire the lock to drive a BlockStream.
• BlockStream is the low-level layer. Use it when you need block-precise control or immediate persistence. Only one BlockStream can be alive at a time per MultiplexedFileStream.

Buffered writes and exclusive mode

Default mode is buffered:

using var cs = mfs.AcquireChannel(1);
for (int i = 0; i < 100_000; i++)
{
    cs.Write(BitConverter.GetBytes(i), 0, 4);  // fills 4KB buffer
    // buffer is auto-flushed every 4096 bytes
}
cs.Flush();   // flush any remainder

For high-throughput batch writes, switch to exclusive mode to skip the buffer and write directly through the BlockStream:

using var cs = mfs.AcquireChannel(1);
cs.EnterExclusive();
try
{
    for (int i = 0; i < 1_000_000; i++)
    {
        cs.Write(BitConverter.GetBytes(i), 0, 4);  // direct to file
    }
}
finally
{
    cs.ExitExclusive();  // back to buffered mode, releases BlockStream
}

Auto-flush

Background timer that periodically flushes the write buffer (default 500ms):

using var cs = mfs.AcquireChannel(1);
cs.EnableAutoFlush(TimeSpan.FromMilliseconds(500));
// ... write slowly over time; buffer flushes automatically every 500ms
cs.DisableAutoFlush();

Useful for long-lived writers that trickle data — keeps the on-disk file fresh without forcing a manual Flush on every write.

API reference

MultiplexedFileStream

Implements IDisposable / IAsyncDisposable.

ChannelStream

Inherits from System.IO.Stream. Buffered wrapper around BlockStream. Long-lived, multiple alive per MultiplexedFileStream.

• CanRead / CanWrite / CanSeek — all true until disposed.
• long Length — virtual length: max(base length, buffer end).
• long Position — virtual cursor (in-memory).
• int Read(byte[], int, int) — reads from buffer if hit, else from base.
• void Write(byte[], int, int) — fills buffer; auto-flushes at 4KB.
• long Seek(long, SeekOrigin) — only updates virtual cursor; never blocks.
• void Flush() / Task FlushAsync(CancellationToken) — flush buffer to base.
• void EnterExclusive() / void ExitExclusive() — bypass buffer; hold a BlockStream open for high-throughput direct writes.
• void EnableAutoFlush(TimeSpan?) / void DisableAutoFlush() — periodic background flush (default 500ms).
• int BufferedBytes { get; } / int BufferCapacity { get; } — buffer state.
• bool IsExclusive { get; } / bool IsAutoFlushEnabled { get; } — mode state.
• ulong ChannelId { get; } — owning channel ID.

BlockStream

Inherits from System.IO.Stream. Low-level, exclusive. One alive at a time.

• CanRead / CanWrite / CanSeek — all true until disposed.
• long Length — total bytes across all blocks of the channel.
• long Position — actual cursor in the file.
• int Read(byte[], int, int) — walks the linked list from Position.
• void Write(byte[], int, int) — append or overwrite depending on Position.
• long Seek(long, SeekOrigin) — backtrack by walking the linked list.
• ulong ChannelId { get; } — owning channel ID.
• Dispose releases the exclusive lock and sets EndMark on the segment tail.

BlockHeader / ChannelMetadata / FileHeader

Public readonly struct values for the on-disk layout. Useful when you need to inspect a file directly (e.g. for tooling).

On-disk layout

+------------------+
| FileHeader (8B)  |  ChannelCount, u64 LE
+------------------+
| Data Area        |  Linked blocks; first block at offset 8
+------------------+
| ChannelMetadata[]|  ChannelCount * 24B (rewritten on each close)
+------------------+
| TailCount (8B)   |  ChannelCount, u64 LE (validation)
+------------------+

BlockHeader (48 bytes, 8-byte aligned)

Offset  Size  Field
0       4     Magic "BLK\x01"
4       1     AllocMark       (0xAA when set, else 0)
5       1     WriteStartMark  (0xBB when set, else 0)
6       1     EndMark         (0xCC when set, else 0)
7       1     Reserved
8       8     ChannelId       u64 LE
16      8     PrevOffset      u64 LE  (0 = no previous block)
24      8     NextOffset      u64 LE  (0 = no next block)
32      8     DataLength      u64 LE  bytes actually written in this block
40      8     Capacity        u64 LE  data area capacity of this block

ChannelMetadata (24 bytes)

Offset  Size  Field
0       8     ChannelId    u64 LE
8       8     HeadOffset   u64 LE  first block (0 = channel empty)
16      8     TailOffset   u64 LE  last block (0 = channel empty)

The current cursor position is not persisted (only head/tail pointers are).

Write semantics

Per the design brief:

1. On the first write of an acquire-write-release cycle, if the channel has no blocks yet, a segment-first block is allocated and AllocMark is set.
2. When the first byte is actually written, WriteStartMark is set on the segment-first block.
3. On ChannelStream.Dispose (after flushing buffer) / BlockStream.Dispose, EndMark is set on the last block written in this segment, and its DataLength is flushed back to the header.
4. If Position was set via Seek (backtracking), subsequent writes follow the existing linked list and do not allocate a new block until the current block is full and has no NextOffset.
5. A new block is allocated in exactly two cases:
   • ExtendBlock was called explicitly, or
   • The previous acquire-write-release cycle has finished (i.e. the lock was released) — the next write that needs space past the current tail will allocate a fresh block.

Concurrency model (v0.2)

• A SemaphoreSlim(1, 1) guarantees that at most one BlockStream is alive per MultiplexedFileStream at any time.
• ChannelStream does not hold the lock while buffering — multiple ChannelStream instances can be alive simultaneously.
• ChannelStream.Flush / Read / EnterExclusive temporarily acquire the lock to drive a BlockStream; other ChannelStreams trying to flush at the same time will block.
• IoGuard (internal) serializes all reads / writes to the underlying stream and protects in-memory metadata.
• BlockStream public methods acquire IoGuard for the duration of each call.
• Design brief: "When you acquire, write and release immediately." ChannelStream relaxes this — you can hold it long-term, but each individual BlockStream is still short-lived.

Crash safety

PersistMetadataAndHeaderLocked is invoked after every block allocation and on every BlockStream.Dispose. The file on disk is therefore always a valid multiplexed file: a crash between two writes will leave the file consistent up to the last completed block.

ChannelStream buffer contents are not crash-safe — if the process dies before Flush / Dispose, buffered writes are lost. Use EnterExclusive or EnableAutoFlush for lower-latency persistence.

Testing

25 unit tests cover: create / open, single-block and multi-block reads and writes, multi-channel isolation, multi-segment appends, backtracking with overwrite, ExtendBlock, lock blocking across AcquireBlockStream, persistence across close/reopen, large-data interleaving across five channels, plus v0.2 features: ChannelStream concurrent acquire without blocking, buffer accumulation until threshold, Flush, Dispose-flush, EnterExclusive / ExitExclusive, EnableAutoFlush periodic flush, buffered reads, and virtual cursor Seek not touching base.

dotnet build src/MultiplexedFile/MultiplexedFile.csproj -c Debug
./memorylimit.sh 1024 300 dotnet test tests/MultiplexedFile.Tests/MultiplexedFile.Tests.csproj -c Debug --no-build

Repository layout

multichannel-file-cs/
├── PLAN.md                          Design and task list
├── current-progress.md              Progress log
├── trash.sh                         Safe delete (replaces rm)
├── memorylimit.sh                   Test runner with 1GB / timeout limits
├── nuget-push.sh                    Configurable NuGet push script
├── MultiplexedFile.sln              Solution
├── src/MultiplexedFile/
│   ├── MultiplexedFile.csproj
│   ├── BlockHeader.cs               48B block header + magic + marks
│   ├── ChannelMetadata.cs           24B per-channel head/tail
│   ├── FileHeader.cs                8B channel count
│   ├── MultiplexedFileStream.cs     Orchestrator (Create/Open/Acquire/Extend)
│   ├── BlockStream.cs               v0.2: low-level exclusive block stream
│   └── ChannelStream.cs             v0.2: buffered wrapper (4KB + auto-flush)
└── tests/MultiplexedFile.Tests/
    └── MultiplexedFileTests.cs      25 unit tests

License

MIT — see PackageLicenseExpression in MultiplexedFile.csproj.