SharedMemoryStore 2.0.0

SharedMemoryStore

SharedMemoryStore is a monorepo for bounded named shared-memory key-value storage. Its .NET, native C++, and Python distributions store opaque byte keys, optional descriptor bytes, and immutable payload bytes in one versioned memory-mapped protocol so same-host processes can exchange data without copying payloads through a broker process.

Distribution identities:

• NuGet: SharedMemoryStore 2.0.0, targeting net10.0 with .NET BCL runtime dependencies only. The legacy layout remains the default; C# callers opt in to the lock-free layout explicitly.
• CMake: SharedMemoryStore 0.1.0, exposing a C++20 RAII API and fixed-width C ABI 1.0 over the native shared library.
• Python: shared-memory-store 0.1.0, requiring Python 3.10 or newer and using standard-library ctypes with the packaged native library.
• Shared protocol: C# supports mapped layouts 1.2 and 2.0; C++ and Python remain layout-1.2 clients and reject layout 2.0. Resource naming versions are 1 and 2 respectively.
• License: MIT, see the license file.

The managed 1.0.0 line established the original production .NET API contract; the 2.0.0 line adds the explicit lock-free profile while retaining the legacy profile and its layout. The native and Python 0.1.0 lines are independently versioned initial distributions; they do not change or ship inside the NuGet package. Linux and Windows are implementation targets. Same-host Linux Docker containers require shared IPC, owner-liveness, permission, and shared-memory capacity capabilities. See Portability and Compatibility metadata before combining independently released distributions.

What It Provides

The shared lifecycle implemented by the three public APIs supports:

• create or open a named store with explicit capacity limits.
• publish immutable value bytes and optional descriptor bytes under an opaque byte key.
• acquire an owning lease, read descriptor and value views, and release or dispose the lease exactly once.
• remove values and reuse slots after active readers release their leases.
• reserve store-owned payload memory for direct length-delimited frame ingest, advance exact write progress, and commit atomically.
• publish segmented buffered payloads, including .NET ReadOnlySequence<byte>, without a temporary full-payload array.
• abort or explicitly recover incomplete reservations without exposing partial bytes to readers.
• run owner-controlled stale lease recovery when enabled.
• inspect caller-formatted diagnostics snapshots without library console output, including lease recovery results and key-index tombstone health.

The native core implements the mapped protocol and Windows/Linux resource mechanisms once. The C++ API wraps the C ABI with move-only RAII stores, leases, and reservations. The Python package uses ctypes and context-managed objects over that same ABI; it does not maintain a second protocol state machine.

The store does not parse frame headers, own application schemas, provide a cross-host cache, persist data beyond process and mapping lifetime, or turn Docker into distributed storage.

Explicit lock-free C# profile

SharedMemoryStoreOptions.CreateLockFree(...) creates or opens mapped layout 2.0. Existing helpers and manually constructed options remain on the legacy layout unless StoreProfile.LockFree is selected explicitly. Processes using different profiles cannot participate in the same live mapping; incompatible opens fail before payload projection.

The lock-free profile is still a bounded key-value store. An application broker may load-balance work by sending keys to 6-12 workers, while observers and other processes independently acquire the same immutable values. The store does not enqueue keys, choose workers, acknowledge work, or turn a read lease into an exclusive claim. See the runnable LockFreeBrokerKeys sample.

Steady-state layout-2.0 publish, acquire, release, remove, and helping paths do not enter the named cross-process lifecycle lock. Progress is system-wide lock-free, not wait-free: a paused or terminated participant cannot own a store-wide data-path lock, but one caller can still exhaust its bounded retry budget under sustained contention and receive StoreBusy. Reservations belong to one producer; leases are shared exact-generation protections; recovery is an explicit caller-controlled operation rather than a background worker.

The trust boundary is same-host cooperating processes with write access to the mapping. Layout 2.0 is neither cross-host storage nor protection against a malicious mapped writer. Performance depends on payload size, key distribution, contention, lease duration, capacity pressure, process placement, and recovery; qualification reports separate those workloads instead of treating one throughput number as universal.

To migrate under the same public name, drain and close every legacy handle, recreate the mapping as layout 2.0, and republish application-owned values. A side-by-side cutover uses a distinct store name. Rollback recreates layout 1.2; neither direction reinterprets mapped bytes in place.

First Use

Start with Getting started. It separates the NuGet, CMake, and wheel workflows and explains how processes select the same named store. The managed package workflow remains:

dotnet pack src/SharedMemoryStore/SharedMemoryStore.csproj -c Release -o artifacts/package
dotnet new console -f net10.0 -n SharedMemoryStore.Tryout -o artifacts/tryout
dotnet add artifacts/tryout/SharedMemoryStore.Tryout.csproj package SharedMemoryStore --source artifacts/package

Minimal workflow:

using SharedMemoryStore;

var options = new SharedMemoryStoreOptions
{
    Name = $"sms-{Guid.NewGuid():N}",
    OpenMode = OpenMode.CreateOrOpen,
    SlotCount = 2,
    MaxValueBytes = 64,
    MaxDescriptorBytes = 16,
    MaxKeyBytes = 16,
    LeaseRecordCount = 4,
    EnableLeaseRecovery = true,
    TotalBytes = SharedMemoryStoreOptions.CalculateRequiredBytes(2, 64, 16, 16, 4)
};

var open = MemoryStore.TryCreateOrOpen(options, out var store);
if (open != StoreOpenStatus.Success || store is null)
{
    return;
}

using (store)
{
    var status = store.TryPublish([1, 2, 3], [4, 5, 6], [9]);
    status = store.TryAcquire([1, 2, 3], out var lease);
    var firstByte = lease.ValueSpan[0];
    status = lease.Release();
    status = store.TryRemove([1, 2, 3]);

    status = store.TryReserve([4], 3, [1], out var reservation);
    new byte[] { 7, 8, 9 }.CopyTo(reservation.GetSpan());
    status = reservation.Advance(3);
    status = reservation.Commit();
}

Expected operational failures are returned as StoreOpenStatus or StoreStatus values. See Errors and statuses for duplicate keys, missing keys, full stores, oversized values, invalid leases, unsupported platforms, stale leases, cleanup failures, and version mismatches. Current-process lease recovery skips other live owner processes, disposal races return documented statuses or empty token views, and slot lifecycle identity is safe across generation rollover.

Native build and test entry point:

pwsh ./scripts/validate-native.ps1 -Configuration Release

Python wheel build and installed-sample entry point:

python -m pip install build
python -m build --wheel
python -m venv artifacts/python-consumer
artifacts/python-consumer/Scripts/python -m pip install (Get-ChildItem dist/*.whl | Select-Object -First 1)
artifacts/python-consumer/Scripts/python samples/PythonBasicUsage/main.py

On Linux, use artifacts/python-consumer/bin/python. The Python sample must run from an installed wheel because the native loader deliberately searches only the package directory, never the current directory or system library path.

Documentation

• Documentation index: complete table of contents by audience.
• Getting started: NuGet, CMake, wheel, minimal workflow, and interoperability setup.
• Concepts: store, name, key, descriptor, payload, slot, lease, reservation, wait policy, status, diagnostics, recovery, capacity, lifecycle, portability, and package contract vocabulary.
• Byte encoding: canonical key, descriptor, and payload byte layouts with allocation-conscious helper patterns.
• Usage guide: create/open, publish, reserve, segmented publish, acquire, release, remove, reuse, diagnostics, recovery, and dispose.
• Examples: basic values, frame-shaped values, direct reservation ingest, segmented payloads, diagnostics, waits, and error handling.
• Errors and statuses: deterministic status outcomes and troubleshooting.
• Diagnostics: snapshot fields and consumer-owned observability.
• Lifecycle: store ownership, leases, removal, stale recovery, abnormal termination, and cleanup.
• Integration: optional lifecycle, health, hosting, and narrow-interface boundaries outside the core package.
• Performance scope: measured scope and unmeasured claims.
• Portability: distribution versions, Linux, Windows, same-host Docker, layout compatibility, and cross-runtime constraints.
• Samples: ordered runnable sample ladder from minimal usage through frame values, zero-copy ingest, optional hosted integration, and same-host Docker validation.
• Architecture: managed and native dependency direction, source areas, storage, lifecycle, synchronization, recovery, and diagnostics.
• Maintainers: documentation update rules, validation commands, contract boundaries, performance evidence, and release impact.
• Packaging: NuGet, CMake, wheel, compatibility metadata, release notes, and clean consumer validation.
• Release preparation: maintainer checks before publication.

Detailed behavior sources:

• Public API contract
• Error taxonomy contract
• Shared-memory layout contract
• Reservation API contract
• Ingest layout contract
• Reservation diagnostics and errors
• Owner recovery hardening contract
• Disposal and rollover hardening contract
• Index health hardening contract
• Production public API contract
• Contention configuration contract
• Diagnostics integration contract
• Reservation memory contract
• Language-neutral protocol
• Native C ABI contract
• C++ API contract
• Python API contract
• Interoperability contract
• Distribution packaging contract

Runnable samples:

• Basic usage sample
• Frame value sample
• Zero-copy ingest sample
• Hosted service integration sample
• Docker shared-memory sample
• C++ basic usage sample
• Python basic usage sample
• Lock-free broker-key sample

Project Policies

• Contributing: setup, validation, compatibility review, and pull request expectations.
• Code of conduct: project-specific conduct expectations.
• Support: questions, bugs, unsupported scenarios, and best-effort prerelease support.
• Security: private vulnerability reporting guidance.
• Issue templates: bug report, documentation issue, and feature request.
• Pull request template: review checklist for behavior, API, package, validation, documentation, compatibility, security, support, and release-note impact.
• Changelog: reverse-chronological package and documentation history.
• Release notes: release readiness checklist and package notes alignment.

Local Validation

pwsh ./scripts/validate-docs.ps1
dotnet build SharedMemoryStore.slnx -c Release
dotnet run --project samples/BasicUsage/BasicUsage.csproj -c Release
dotnet run --project samples/FrameValue/FrameValue.csproj -c Release
dotnet run --project samples/ZeroCopyIngest/ZeroCopyIngest.csproj -c Release
dotnet run --project samples/HostedServiceIntegration/HostedServiceIntegration.csproj -c Release
dotnet run --project samples/DockerSharedMemory/DockerSharedMemory.csproj -c Release -- all
pwsh ./scripts/validate-package-consumption.ps1
dotnet test SharedMemoryStore.slnx -c Release
dotnet pack src/SharedMemoryStore/SharedMemoryStore.csproj -c Release -o artifacts/package
pwsh ./scripts/validate-cross-platform.ps1 -SkipDocker
pwsh ./scripts/validate-docker-shared-memory.ps1
pwsh ./scripts/validate-native.ps1 -Configuration Release
pwsh ./scripts/validate-python.ps1 -Configuration Release
pwsh ./scripts/validate-interoperability.ps1 -Configuration Release -Stress

Documentation changes must keep package metadata, README content, release notes, support policy, security policy, compatibility metadata, and contract links aligned across managed 2.0.0, native 0.1.0, Python 0.1.0, ABI 1.0, and layouts 1.2 and 2.0.