Meziantou.Framework.PooledMemoryStream 2.0.7

Meziantou.Framework.PooledMemoryStream

PooledMemoryStream is a MemoryStream whose storage is a chain of pooled byte arrays rented from a process-wide pool shared by all instances. Growing the stream never reallocates and copies a single backing array — new pooled blocks are appended to the chain. Blocks are only ever rented in a small set of discrete sizes (Small / Medium / Large), so the pool stays effective. It also implements IBufferWriter<byte>.

Basic usage

PooledMemoryStream is in the Meziantou.Framework namespace.

using var stream = new PooledMemoryStream();
stream.Write("Hello, "u8);
stream.Write("World!"u8);

stream.Position = 0;
using var reader = new StreamReader(stream);
Assert.Equal("Hello, World!", reader.ReadToEnd());

// The pooled buffers are returned to the shared pool when the stream is disposed.

Use it as an IBufferWriter<byte>

IBufferWriter<T> is in the System.Buffers namespace, and Utf8JsonWriter in System.Text.Json.

using var stream = new PooledMemoryStream();

// PooledMemoryStream implements IBufferWriter<byte> (explicit), so it can be passed where one is expected.
IBufferWriter<byte> writer = stream;
await using (var json = new Utf8JsonWriter(writer))
{
    json.WriteStartObject();
    json.WriteString("message", "hello");
    json.WriteEndObject();
}

var bytes = stream.ToArray();
Assert.Equal("""{"message":"hello"}"""u8.ToArray(), bytes);

Data written through IBufferWriter<byte> is appended at the end of the stream; after Advance, Position is moved to the new end.

Configure the buffer sizes

var options = new PooledMemoryStreamOptions
{
    // Any number of strictly ascending tiers (bytes). Small streams use the smallest size and
    // grow through the larger ones; buffers bigger than the largest tier round up to a multiple of it.
    BufferSizes = [4 * 1024, 128 * 1024, 1024 * 1024, 10 * 1024 * 1024], // 4 KiB / 128 KiB / 1 MiB / 10 MiB
    MaxRetainedBytesPerBucket = 64L * 1024 * 1024,
    ClearOnReturn = false,
};

using var stream = new PooledMemoryStream(options);

PooledMemoryStreamOptions becomes immutable (frozen) the first time it is used to create a stream; after that, setting any property throws InvalidOperationException. The shared PooledMemoryStreamOptions.Default is always frozen. Configure all properties before passing the options to a stream.

Notes

• The byte arrays (including the array returned by GetBuffer() / TryGetBuffer(), and the spans/memory returned by the IBufferWriter<byte> members) are owned by the stream and are returned to the shared pool on Dispose. Do not use them after the stream is disposed.
• The array returned by GetBuffer() / TryGetBuffer() is never returned to the shared pool before Dispose, so another stream cannot rent it while you hold it. Like with MemoryStream, it stays the stream's storage (shrinking with SetLength or Capacity keeps it, so later writes overwrite it) until a later GetBuffer() / TryGetBuffer() call consolidates several blocks into a new array; the old array then stays valid but no longer reflects the stream contents.
• GetBuffer() returns the whole underlying array, not just the first Length bytes. Only [0, Length) is meaningful; the rest is zeroed before the array is handed out, so it never exposes data written by another stream. The spans returned by the IBufferWriter<byte> members are uninitialized, as usual for a buffer writer.
• Because the storage is a chain of blocks rather than a single array, the stream is not bounded by Array.MaxLength: it can hold as many bytes as memory allows. The members that must produce a single array (ToArray(), GetBuffer(), TryGetBuffer()) are still bounded by it and throw an InvalidOperationException on a longer stream. Capacity is an int inherited from MemoryStream, so its getter is clamped to int.MaxValue.
• The instance is not thread-safe (like MemoryStream); the shared pool is.