Lyo.FileStorage.S3 1.0.6

Lyo.FileStorage.S3

S3-compatible storage for Lyo.FileStorage (AWS S3, Backblaze B2, MinIO, and others) via AWSSDK.S3.

Features

• S3 API. Same client for AWS and S3-compatible endpoints.
• Multipart uploads. Keyed S3MultipartUploadService is registered with the same key when you call S3FileStorageServiceBuilder.Build (unless already registered). If no IMultipartUploadSessionStore is registered yet, an in-memory store is added (use AddPostgresFileMetadataStoreKeyed(...).Build() before S3 when using PostgreSQL so sessions use the DB). Part size is clamped to the S3 minimum (5 MiB) with an 8 MiB default. Total upload limit aligns with MaxUploadSizeBytes. Server-side copy is used for the final commit (no download+re-upload round trip).
• Staged uploads. Keyed S3StagedFileUploadService is registered with the same key when you call S3FileStorageServiceBuilder.Build (unless already registered). Presigned PUT targets .stage/{stageId}/object. SSE headers flow through S3UploadServerSideEncryption.BuildRequiredPutHeaders like direct upload. Requires IStagedFileUploadStore (Postgres/Sqlite or in-memory fallback).
• Streamed PUT spilling. S3UploadStream keeps small payloads in memory and spills to a deletable temp file once it crosses 4 MiB, then uploads via single PUT under 64 MiB or multipart above that, aborting cleanly on any per-part failure.
• Region. Configurable AWS regions.
• Custom endpoints. S3-compatible services via ServiceUrl.
• Key prefixing. Organized file storage with key prefixes via the shared CloudObjectKeyBuilder.
• Path organization. Files organized by GUID prefixes. Suffix is persisted in metadata so reads skip N+1 probes.
• IAM roles. Works with IAM roles for authentication.
• Diagnostics. Bucket key listing via IFileStorageDiagnosticsService (prefix-aware, combines KeyPrefix, normalized and traversal-guarded by Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix).
• Server-side copy, move, and direct PUT. CopyFileAsync (CopyObject), MoveFileAsync (CopyObject then delete source, same file id), BeginDirectUploadAsync / CompleteDirectUploadAsync (presigned PUT + finalize). RequiredPutHeaders is populated when SSE or a signed Content-Type applies, via S3UploadServerSideEncryption.BuildRequiredPutHeaders. RenameFileAsync updates display metadata only.
• Presigned GET options. Optional ContentDisposition / ContentType via PreSignedReadUrlOptions (S3 response header overrides). When the caller omits pathPrefix, the metadata-stored prefix is used as fallback.

Examples

Configuration

using Lyo.FileStorage.S3;
using Lyo.FileStorage.Models;

var options = new S3FileStorageOptions
{
    BucketName = "my-bucket",
    Region = "us-east-1",
    KeyPrefix = "app-files", // Optional global prefix
    AccessKeyId = "your-key", // Optional if using IAM roles
    SecretAccessKey = "your-secret", // Optional if using IAM roles
    // Optional SSE for new uploads, copies, streamed saves, multipart staging, and compatible presigned PUTs:
    ServerSideEncryption = "AES256", // or "aws:kms" / "aws:kms:dsse"
    ServerSideEncryptionAwsKmsKeyId = null // set CMK id/ARN when using KMS
};

var metadataStore = new YourMetadataStore(); // Implement IFileMetadataStore
var service = new S3FileStorageService(options, metadataStore);

S3FileStorageServiceBuilder keyed registration

services
    .AddS3FileStorageServiceKeyed("client-files")
    .UseFileMetadataStore("postgres-filemetadatastore")
    .UseEncryptionService("two-key-aws")
    .ConfigureS3FileStorage("S3FileStorageOptions")
    .Build(configuration);

Documentation map

Compression and encryption follow FileStorageServiceBase: optional ICompressionResolver (metadata-driven decompress on read; see Lyo.FileStorage, Compression resolver) and ITwoKeyEncryptionService.

S3FileStorageOptions

S3-compatible endpoints

Set ServiceUrl (and usually ForcePathStyle is applied automatically when a custom URL is set):

var options = new S3FileStorageOptions
{
    BucketName = "my-bucket",
    ServiceUrl = "https://s3-compatible.example.com",
    AccessKeyId = "your-key",
    SecretAccessKey = "your-secret"
};

Backblaze B2

Use S3FileStorageBackblazeExtensions.ApplyBackblazeB2Defaults() so ServiceUrl becomes https://s3.{region}.backblazeb2.com when Region is set (e.g. us-west-004). Or call AddS3FileStorageServiceKeyedForBackblaze to bind the BackblazeFileStorage section (see S3FileStorageBackblazeExtensions.BackblazeFileStorageConfigurationSectionName) and register the keyed storage builder.

Other S3-compatible providers

S3FileStorageS3CompatibleExtensions provides endpoint URL builders, ApplyDefaults methods (set ServiceUrl from Region / ProviderAccountId when ServiceUrl is not already set), and AddS3FileStorageServiceKeyedFor helpers with default configuration section names.

Example (MinIO in code, same builder chain as other keyed S3 storage, e.g. UseFileMetadataStore, then Build(configuration)):

services.AddS3FileStorageServiceKeyedForMinio("files", o => {
    o.BucketName = "my-bucket";
    o.ServiceUrl = "localhost:9000"; // or https://minio.example.com — scheme optional for host:port
    o.AccessKeyId = "...";
    o.SecretAccessKey = "...";
})
    .UseFileMetadataStore("your-metadata-store-key")
    .Build(configuration);

S3FileStorageServiceBuilder

AddS3FileStorageServiceKeyed(string keyName) returns a fluent builder that owns the keyed S3FileStorageService + IFileStorageService registration plus any auxiliary services it touches:

Other DI entry points

S3UploadServerSideEncryption

Lyo.FileStorage.S3.S3UploadServerSideEncryption is the shared helper that translates ServerSideEncryption + ServerSideEncryptionAwsKmsKeyId into the right AWS SDK enum, applies headers to PutObjectRequest / multipart InitiateMultipartUploadRequest, and emits RequiredPutHeaders on DirectUploadBeginResult so a browser PUT to the presigned URL carries the same SSE/Content-Type values that were used to sign the URL. Supported values:

Notes

• IAM role-based authentication is supported.
• Dispose() and IAsyncDisposable.DisposeAsync() run when the service owns the IAmazonS3 client.

Error handling

• 404 Not Found. Returns null or empty results instead of throwing.
• Access Denied. Clear error messages for permission issues.
• Network Errors. Retry logic should be handled at the application level.

Object key layout

• Format: {KeyPrefix}/{guid-prefix-2}/{guid-prefix-2}/{guid}.{extension}
• Example: app-files/ab/cd/abcdef1234567890.ag

Health checks

IFileStorageService extends IHealth. Get health directly from the service: await fileStorage.CheckHealthAsync().

Tests

Lyo.FileStorage.S3.Tests exercises this assembly with isolated, dependency-free unit tests using a DispatchProxy-based IAmazonS3 stub (Support/FakeAmazonS3). Covered: S3UploadServerSideEncryption header/apply logic, S3UploadStream (single PUT + multipart begin→complete + abort + SSE forwarding), S3GetObjectResponseStream disposal, the shared CloudObjectKeyBuilder, and options invariants. Path-prefix traversal coverage lives in Lyo.FileStorage.Tests against the shared Lyo.Exceptions.FileHelpers helper. Deeper end-to-end coverage of presigned signing and live bucket I/O would need LocalStack.

Dependencies

Generated from ProjectReference / PackageReference (same model as docs/Lyo.ProjectGraph.html).

• Lyo.Common (direct, lyo)
• Lyo.Compression (direct, lyo)
• Lyo.Encryption (direct, lyo)
• Lyo.Exceptions (direct, lyo)
• Lyo.FileMetadataStore (direct, lyo)
• Lyo.FileStorage (direct, lyo)
• AWSSDK.Core 4.0.100.4 (direct, third-party)
• AWSSDK.S3 4.0.101 (direct, third-party)
• Microsoft.Extensions.Configuration.Binder 10.0.5 (direct, microsoft)
• Microsoft.Extensions.DependencyInjection.Abstractions 10.0.5 (direct, microsoft)
• Lyo.ContentThreatScan (transitive, lyo)
• Lyo.Hashing (transitive, lyo)
• Lyo.Health (transitive, lyo)
• Lyo.IO.Temp (transitive, lyo)
• Lyo.KeyStore (transitive, lyo)
• Lyo.Metrics (transitive, lyo)
• Lyo.Result (transitive, lyo)
• Lyo.Streams (transitive, lyo)
• BouncyCastle.Cryptography 2.6.2 (transitive, third-party, netstandard2.0)
• EasyCompressor 2.1.0 (transitive, third-party)
• Konscious.Security.Cryptography.Argon2 1.3.1 (transitive, third-party)
• Microsoft.Bcl.AsyncInterfaces 10.0.5 (transitive, microsoft, netstandard2.0)
• Microsoft.Extensions.Hosting.Abstractions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Logging.Abstractions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.ConfigurationExtensions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.DataAnnotations 10.0.5 (transitive, microsoft)
• System.Buffers 4.6.1 (transitive, microsoft, netstandard2.0)
• System.IO.Hashing 10.0.5 (transitive, microsoft, net10.0)
• System.Memory 4.6.3 (transitive, microsoft, netstandard2.0)
• System.Text.Json 10.0.5 (transitive, microsoft, netstandard2.0)
• System.Threading.Tasks.Extensions 4.6.3 (transitive, microsoft, netstandard2.0)