StoringKit 0.4.0

StoringKit

Object storage for .NET 10 behind one contract. Code written against IObjectStore runs unchanged on Azure Blob Storage, AWS S3, Cloudflare R2 and MinIO, and both providers are held to the same behaviour by one shared test suite run against real emulators.

var put = await store.PutAsync($"invoices/{id}.pdf", pdf, new PutObjectOptions
{
    ContentType = "application/pdf",
    Condition = WriteCondition.IfNotExists,     // create-only: a retried write cannot overwrite
});

if (put.Error is { Code: StoringErrors.AlreadyExists }) { /* someone got there first */ }

var link = await store.GetReadUrlAsync($"invoices/{id}.pdf", TimeSpan.FromMinutes(15));

Packages

dotnet add package StoringKit.Azure     # or StoringKit.S3

A module that only stores and reads files references StoringKit alone and never restores a cloud SDK; the host picks the provider. Azure Blob Storage has no S3-compatible API, which is why it has its own provider rather than going through the AWS SDK or a proxy.

The contract

Results and errors

Expected failures come back as a failed Result carrying a ResultingKit Error. Branch on Error.Code — the codes are the same for every provider — or on Error.Kind.

Transient is the only kind worth retrying. Both SDKs already retry internally, so a transient error here means their retries ran out.

A misconfigured store throws the SDK's exception instead — wrong credentials, a bucket or container that does not exist, a denied permission, a feature the endpoint does not implement (R2 answers 501 NotImplemented). No retry or input change fixes those, so they surface like any other bug. Caller cancellation throws OperationCanceledException.

Conditional writes

WriteCondition.IfNotExists (If-None-Match: *) and WriteCondition.IfMatch(etag) (If-Match) are checked atomically by the service, on every provider: of any number of concurrent create-only writes to one key, exactly one succeeds. They are what idempotent writes are built on — derive the key from the operation's identity, write create-only, and treat storing.already_exists as "already done".

ETags are opaque; pass back exactly what a PutReceipt or StoredObject gave you. Deletes are unconditional: R2 documents no conditional headers on DeleteObject.

JSON documents

GetFromJsonAsync, PutAsJsonAsync and ObjectContent.ReadFromJsonAsync read and write objects as JSON, named and shaped after System.Net.Http.Json. They are extension methods in StoringKit built on PutAsync and GetAsync, so every provider gets them, they return the same errors, and they are traced as ordinary put and get spans with no content recorded.

// Read: the value plus the properties from the same response, including the ETag
Result<StoredDocument<Order>> read = await store.GetFromJsonAsync<Order>($"orders/{id}.json", ct: ct);
if (read.IsFailure) return read.Error;                // storing.not_found, storing.invalid_document, transient, …
var (order, properties) = read.Value;

// Compare-and-swap: write back only if nobody else has since
var put = await store.PutAsJsonAsync($"orders/{id}.json", order! with { Status = "shipped" }, new PutObjectOptions
{
    Condition = WriteCondition.IfMatch(read.Value.ETag),
});

// Create-only
await store.PutAsJsonAsync($"orders/{id}.json", order, new PutObjectOptions { Condition = WriteCondition.IfNotExists });

// Source generation: trim- and AOT-safe
await store.PutAsJsonAsync(key, order, AppJsonContext.Default.Order);
var typed = await store.GetFromJsonAsync(key, AppJsonContext.Default.Order);

// Already holding an open object
await using var content = (await store.GetAsync(key)).Value;
Result<Order?> o = await content.ReadFromJsonAsync<Order>();

• Serializer options default to JsonSerializerDefaults.Web (camelCase, case-insensitive reads), as HttpClient's helpers do. Pass jsonOptions, or a JsonTypeInfo<T> from a JsonSerializerContext. The reflection overloads are marked RequiresUnreferencedCode and RequiresDynamicCode.
• Writes serialize into a seekable in-memory buffer, never a string, so the end-to-end Content-MD5 check still applies and the S3 provider skips its temporary file. The content type is application/json unless PutObjectOptions.ContentType says otherwise. The options you pass are not changed.
• Metadata is not carried over. A put writes exactly the metadata it is given, like PutAsync. To keep it across a read-modify-write, copy properties.Metadata into the new options.
• Reads deserialize straight from the response stream and dispose it. The ETag comes from the same response as the bytes, so it is always the version you read. A connection lost mid-read is a Transient error, like a failed request.
• Bad JSON is a result, not an exception. A stored object that is not valid JSON for the type is storing.invalid_document, with the stream already released, so a journal-style reader can quarantine the record and move on. A value that cannot be serialized (a reference cycle, an unsupported type) is the same error on a put, and nothing is written. The JSON literal null reads as default, so StoredDocument<T>.Value is T?.

There is no built-in update loop. A read-modify-write takes a few lines, and keeping it in your code keeps the retry policy yours:

async Task<Result<PutReceipt>> ShipAsync(IObjectStore store, string key, CancellationToken ct)
{
    for (var attempt = 1; ; attempt++)
    {
        var read = await store.GetFromJsonAsync<Order>(key, ct: ct);
        if (read.IsFailure) return read.Error;

        var put = await store.PutAsJsonAsync(key, read.Value.Value! with { Status = "shipped" },
            new PutObjectOptions { Condition = WriteCondition.IfMatch(read.Value.ETag) }, ct: ct);

        // Someone else wrote first: re-read their version and apply the change again.
        if (put.Error is { Code: StoringErrors.PreconditionFailed } && attempt < 5) continue;
        return put;
    }
}

Checksums

Every put is verified end to end with Content-MD5, which Azure, S3, R2 and MinIO all check before committing. StoringKit computes the MD5 of seekable streams itself; pass PutObjectOptions.ContentMd5 when you already have it (for example from a client upload). A mismatch is storing.checksum_mismatch and nothing is stored.

On Azure, an upload that fits one request (up to 256 MiB by default) is checked against the whole-object MD5. Larger or non-seekable uploads go up in blocks: the SDK sends an MD5 with each block, and the service rejects any block that arrives corrupted. The whole-object MD5 is stored on the blob but the service doesn't check it at commit. The S3 provider always sends one PutObject, so the whole object is always checked.

StoredObject.ContentMd5 reports it back: Azure stores it as the blob's Content-MD5; S3 does not return Content-MD5 after upload, so the S3 provider also keeps it in a storingkit-md5 metadata entry, which is hidden from Metadata. Objects written by other tools — and Azure uploads from non-seekable streams without a supplied MD5 — may have none.

Public and private access

Access belongs to the store, not to individual objects. Azure can make a container publicly readable, but S3 and R2 have no per-object public flag that works everywhere — R2 implements no ACLs at all, and new AWS buckets block public ACLs by default. So StoringKit never changes access settings: make the bucket or container public outside StoringKit, set PublicBaseUrl, and keep public and private objects in separate named stores.

For private objects, hand out GetReadUrlAsync links instead.

Keys and metadata

The rules are the intersection of what every provider stores without rewriting:

• Keys: 1–1024 UTF-8 bytes; / separates segments; no leading, trailing or doubled /; no . or .. segment; no trailing .; no \ (Azure rewrites it to /) and no control characters. Keys are case-sensitive.
• Metadata names: letters, digits and _, not starting with a digit (Azure's rule); stored and returned lower-cased (S3's behaviour).
• Metadata values: non-empty printable ASCII without leading or trailing spaces.
• Size: names and values together at most 2000 bytes (S3 and R2 cap metadata at 2 KB).

Providers

The S3 provider sends each object in a single PutObject, which keeps conditional writes on the one call every S3-compatible store supports them on. Content above 5 GiB fails with storing.too_large.

Cloudflare R2

R2 speaks the S3 API, with differences that matter here:

• Checksums. AWS SDKs released since December 2024 add CRC32 checksums to every upload and validate them on every download (WHEN_SUPPORTED). R2 and other S3-compatible stores reject them. Whenever ServiceUrl is not an amazonaws.com host, StoringKit sets RequestChecksumCalculation and ResponseChecksumValidation to WHEN_REQUIRED, and integrity rests on Content-MD5, which R2 checks.
• Payload signing. R2 does not implement streaming SigV4 (aws-chunked). For non-AWS endpoints StoringKit disables chunked encoding and, over HTTPS, payload signing.
• No ACLs. x-amz-acl and the ACL APIs are not implemented; see public access.
• No conditional deletes documented, no object tagging, no lifecycle on put. None are part of the contract.
• Region is auto.

Registration

Each store has a name. The one called Default is also registered as a plain IObjectStore; the rest are keyed services. Options bind from Storing:{name} and are validated at startup, so a missing bucket name fails the host before it serves a request.

{
  "Storing": {
    "Default": {
      "ConnectionString": "<from a secret store>",
      "ContainerName": "documents"
    },
    "Public": {
      "ConnectionString": "<from a secret store>",
      "ContainerName": "avatars",
      "PublicBaseUrl": "https://cdn.example.com/avatars/"
    }
  }
}

builder.Services.AddAzureBlobStore(builder.Configuration);            // Storing:Default
builder.Services.AddAzureBlobStore(builder.Configuration, "Public");  // Storing:Public

public sealed class AvatarService(
    IObjectStore documents,
    [FromKeyedServices("Public")] IObjectStore avatars) { … }

Azure

Read URLs are signed with the account key when the connection string has one, and with a cached user delegation key when authenticating with Microsoft Entra ID (the identity needs Storage Blob Delegator or equivalent). A SAS connection string cannot sign new URLs; GetReadUrlAsync then throws.

builder.Services.AddAzureBlobStore(o =>
{
    o.ServiceUri = new Uri("https://myaccount.blob.core.windows.net");
    o.Credential = new DefaultAzureCredential();
    o.ContainerName = "documents";
});

S3, R2 and MinIO

Credentials takes any AWSCredentials in code, such as an assumed role.

builder.Services.AddS3Store(o =>
{
    o.BucketName = "documents";
    o.ServiceUrl = new Uri($"https://{accountId}.r2.cloudflarestorage.com");
    o.Region = "auto";
    o.AccessKeyId = builder.Configuration["R2:AccessKeyId"];
    o.SecretAccessKey = builder.Configuration["R2:SecretAccessKey"];
});

Your own provider

Derive from ObjectStore — it validates keys, metadata and arguments, computes MD5s and turns the exceptions you classify into results — and register it with AddObjectStore(name, "myprovider", sp => new MyStore(...)). It is instrumented like the built-in ones.

Observability

StoringKit traces and measures every store operation on its own, the way HttpClient and EF Core do: it uses only ActivitySource and Meter from the runtime, with no OpenTelemetry package. Nothing is recorded until the host listens:

builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddSource(StoringKitInstrumentation.ActivitySourceName))
    .WithMetrics(m => m.AddMeter(StoringKitInstrumentation.MeterName));

Every store registered through AddAzureBlobStore, AddS3Store or AddObjectStore is wrapped automatically.

Keys, content and metadata values are never recorded — keys routinely carry user ids and file names. Exceptions are recorded by type only, since SDK messages can contain the request URL. OpenTelemetry's object-storage attributes are AWS-specific and carry the key, so the attributes use the storingkit. prefix, apart from the general error.type.

Span — {operation} {store} (put Default, get_read_url Public), kind Client, one per call. GetPublicUrl makes no request and has no span.

Metrics — on the StoringKit meter, created through the host's IMeterFactory:

A get is timed until the response headers arrive, not until the caller finishes reading the stream. The histogram advises OpenTelemetry's bucket boundaries for seconds (5 ms to 10 s).

Locks

StoringKit has no lease or lock API, by design. Conditional writes already make a journal safe with any number of writers: create entries with IfNotExists and complete them with IfMatch. When you need mutual exclusion ("only one worker rebuilds this index"), use LockingKit. LockingKit.Azure holds renewable leases on guard blobs, with a token that fires on loss and optional fencing tokens. LockingKit.Postgres does the same with advisory locks.

Testing

StoringKit.IntegrationTests runs one contract suite against Azurite (TestingKit.Azurite) and MinIO (TestingKit.Minio). Use the same fixtures for your own tests: point a store at the fixture's connection string or endpoint, and every behaviour above holds.