Lyo.FileStorage.S3 2.0.0

Lyo.FileStorage.S3

Lyo.FileStorage on S3-compatible endpoints (AWS S3, Backblaze B2, MinIO, and others) through AWSSDK.S3.

Features

• S3 API. One client for AWS and S3-compatible endpoints.
• Multipart uploads. Keyed S3MultipartUploadService is registered with the same key when you call S3FileStorageServiceBuilder.Build (unless already registered). If no IMultipartUploadSessionStore is registered yet, an in-memory store is added (use AddPostgresFileMetadataStoreKeyed(...).Build() before S3 when using PostgreSQL so sessions use the DB). Part size is clamped to the S3 minimum (5 MiB) with an 8 MiB default. Total upload limit aligns with MaxUploadSizeBytes. Server-side copy is used for the final commit (no download+re-upload round trip).
• Streamed PUT spilling. S3UploadStream keeps small payloads in memory and spills to a deletable temp file once it crosses 4 MiB, then uploads via a single PUT under 64 MiB or multipart above that, aborting cleanly on any per-part failure.
• Region. AWS regions are configurable.
• Custom endpoints. Point ServiceUrl at S3-compatible services.
• Key prefixing. Organize objects with key prefixes via the shared CloudObjectKeyBuilder.
• Path organization. Files are organized by GUID prefixes. The suffix is persisted in metadata so reads skip N+1 probes.
• IAM roles. Authentication works with IAM roles.
• Diagnostics. Bucket key listing via IFileStorageDiagnosticsService (prefix-aware; combines KeyPrefix, normalized and traversal-guarded by Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix).
• Server-side copy, move, and direct PUT. CopyFileAsync (CopyObject), MoveFileAsync (CopyObject then delete source, same file id), BeginDirectUploadAsync / CompleteDirectUploadAsync (presigned PUT + finalize). RequiredPutHeaders is populated when SSE or a signed Content-Type applies, via S3UploadServerSideEncryption.BuildRequiredPutHeaders. RenameFileAsync updates display metadata only.
• Presigned GET options. Optional ContentDisposition / ContentType via PreSignedReadUrlOptions (S3 response header overrides). When the caller omits pathPrefix, the metadata-stored prefix is used as the fallback.

Examples

Configuration

using Lyo.FileStorage.S3;
using Lyo.FileStorage.Models;

var options = new S3FileStorageOptions
{
    BucketName = "my-bucket",
    Region = "us-east-1",
    KeyPrefix = "app-files", // Optional global prefix
    AccessKeyId = "your-key", // Optional if using IAM roles
    SecretAccessKey = "your-secret", // Optional if using IAM roles
    // Optional SSE for new uploads, copies, streamed saves, multipart staging, and compatible presigned PUTs:
    ServerSideEncryption = "AES256", // or "aws:kms" / "aws:kms:dsse"
    ServerSideEncryptionAwsKmsKeyId = null // set CMK id/ARN when using KMS
};

var metadataStore = new YourMetadataStore(); // Implement IFileMetadataStore
var service = new S3FileStorageService(options, metadataStore);

Keyed S3FileStorageServiceBuilder registration

services
    .AddS3FileStorageServiceKeyed("client-files")
    .UseFileMetadataStore("postgres-filemetadatastore")
    .UseEncryptionService("two-key-aws")
    .ConfigureS3FileStorage("S3FileStorageOptions")
    .Build(configuration);

Where to read next

Compression and encryption follow FileStorageServiceBase: optional ICompressionResolver (metadata-driven decompress on read; see Lyo.FileStorage, Compression resolver) and ITwoKeyEncryptionService.

S3FileStorageOptions

S3-compatible endpoints

Set ServiceUrl (and ForcePathStyle is usually applied automatically when a custom URL is set):

var options = new S3FileStorageOptions
{
    BucketName = "my-bucket",
    ServiceUrl = "https://s3-compatible.example.com",
    AccessKeyId = "your-key",
    SecretAccessKey = "your-secret"
};

Backblaze B2

Use S3FileStorageBackblazeExtensions.ApplyBackblazeB2Defaults() so ServiceUrl becomes https://s3.{region}.backblazeb2.com when Region is set (e.g. us-west-004). Or call AddS3FileStorageServiceKeyedForBackblaze to bind the BackblazeFileStorage section (see S3FileStorageBackblazeExtensions.BackblazeFileStorageConfigurationSectionName) and register the keyed storage builder.

Other S3-compatible providers

S3FileStorageS3CompatibleExtensions provides endpoint URL builders, ApplyDefaults methods (set ServiceUrl from Region / ProviderAccountId when ServiceUrl is not already set), and AddS3FileStorageServiceKeyedFor helpers with default configuration section names.

Example (MinIO in code, same builder chain as other keyed S3 storage, e.g. UseFileMetadataStore, then Build(configuration)):

services.AddS3FileStorageServiceKeyedForMinio("files", o => {
    o.BucketName = "my-bucket";
    o.ServiceUrl = "localhost:9000"; // or https://minio.example.com — scheme optional for host:port
    o.AccessKeyId = "...";
    o.SecretAccessKey = "...";
})
    .UseFileMetadataStore("your-metadata-store-key")
    .Build(configuration);

S3FileStorageServiceBuilder

AddS3FileStorageServiceKeyed(string keyName) returns a fluent builder that owns the keyed S3FileStorageService + IFileStorageService registration plus any auxiliary services it touches:

Other DI entry points

S3UploadServerSideEncryption

Lyo.FileStorage.S3.S3UploadServerSideEncryption is the shared helper that translates ServerSideEncryption + ServerSideEncryptionAwsKmsKeyId into the right AWS SDK enum, applies headers to PutObjectRequest / multipart InitiateMultipartUploadRequest, and emits RequiredPutHeaders on DirectUploadBeginResult so a browser PUT to the presigned URL carries the same SSE/Content-Type values that were used to sign the URL. Supported values:

Notes

• IAM role-based authentication is supported.
• Dispose() and IAsyncDisposable.DisposeAsync() run when this service owns the IAmazonS3 client.

Error handling

• 404 Not Found. Returns null or empty results rather than throwing.
• Access Denied. Permission issues surface clear error messages.
• Network Errors. Retry logic belongs at the application level.

Object key layout

• Format: {KeyPrefix}/{guid-prefix-2}/{guid-prefix-2}/{guid}.{extension}
• Example: app-files/ab/cd/abcdef1234567890.ag

Health checks

IFileStorageService extends IHealth. Read health from the service: await fileStorage.CheckHealthAsync().

Tests

Lyo.FileStorage.S3.Tests exercises this assembly with isolated, dependency-free unit tests using a DispatchProxy-based IAmazonS3 stub (Support/FakeAmazonS3). Covered: S3UploadServerSideEncryption header/apply logic, S3UploadStream (single PUT + multipart begin→complete + abort + SSE forwarding), S3GetObjectResponseStream disposal, the shared CloudObjectKeyBuilder, and options invariants. Path-prefix traversal coverage lives in Lyo.FileStorage.Tests against the shared Lyo.Exceptions.FileHelpers helper. Deeper end-to-end coverage of presigned signing and live bucket I/O would need LocalStack.

Dependencies

Generated from ProjectReference / PackageReference (same model as docs/Lyo.ProjectGraph.html).

• Lyo.Common.Metadata (direct, lyo)
• Lyo.Compression (direct, lyo)
• Lyo.Configuration (direct, lyo)
• Lyo.Encryption (direct, lyo)
• Lyo.Exceptions (direct, lyo)
• Lyo.FileMetadataStore (direct, lyo)
• Lyo.FileStorage (direct, lyo)
• AWSSDK.Core 4.0.100.4 (direct, third-party)
• AWSSDK.S3 4.0.101 (direct, third-party)
• Microsoft.Extensions.DependencyInjection.Abstractions 10.0.5 (direct, microsoft)
• Lyo.Common.Core (transitive, lyo)
• Lyo.Hashing (transitive, lyo)
• Lyo.Health (transitive, lyo)
• Lyo.IO.Temp (transitive, lyo)
• Lyo.KeyStore (transitive, lyo)
• Lyo.Metrics (transitive, lyo)
• Lyo.Result (transitive, lyo)
• Lyo.Streams (transitive, lyo)
• BouncyCastle.Cryptography 2.6.2 (transitive, third-party, netstandard2.0)
• EasyCompressor 2.1.0 (transitive, third-party)
• Konscious.Security.Cryptography.Argon2 1.3.1 (transitive, third-party)
• Microsoft.Bcl.AsyncInterfaces 10.0.5 (transitive, microsoft, netstandard2.0)
• Microsoft.Extensions.Configuration.Binder 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Hosting.Abstractions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Logging.Abstractions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.ConfigurationExtensions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.DataAnnotations 10.0.5 (transitive, microsoft)
• System.Buffers 4.6.1 (transitive, microsoft, netstandard2.0)
• System.IO.Hashing 10.0.5 (transitive, microsoft, net10.0)
• System.Memory 4.6.3 (transitive, microsoft, netstandard2.0)
• System.Text.Json 10.0.5 (transitive, microsoft, netstandard2.0)
• System.Threading.Tasks.Extensions 4.6.3 (transitive, microsoft, netstandard2.0)