Lyo.FileStorage 1.0.7

Lyo.FileStorage

Save, stream-save, read, delete, and metadata for files. Optional compression (Lyo.Compression), two-key encryption (Lyo.Encryption), duplicate hashing, access policies, malware scans, audit hooks, multipart uploads (via IMultipartUploadService), and presigned/direct-upload/copy on cloud-capable backends.

With GenerateDocumentationFile set in Directory.Build.props, IntelliSense shows the same summaries as this README for documented members.

Examples

Local file storage

using Lyo.FileStorage;
using Lyo.FileStorage.Models;

var options = new DiskFileStorageOptions
{
    RootDirectoryPath = "/path/to/storage",
    EnableDuplicateDetection = true,
    DuplicateStrategy = DuplicateHandlingStrategy.ReturnExisting
};

var service = new LocalFileStorageService(options);

// Save a file
var data = File.ReadAllBytes("document.pdf");
var result = await service.SaveFileAsync(
    data,
    originalFileName: "document.pdf",
    compress: true,
    encrypt: true,
    keyId: "my-encryption-key");

// Retrieve a file
var retrievedData = await service.GetFileAsync(result.Id);

// Delete a file
await service.DeleteFileAsync(result.Id);

S3 storage (AWS, Backblaze B2, MinIO, Wasabi, Cloudflare R2)

using Lyo.FileStorage.S3;
using Lyo.FileStorage.Models;

var options = new S3FileStorageOptions
{
    BucketName = "my-bucket",
    Region = "us-east-1",
    AccessKeyId = "your-access-key",
    SecretAccessKey = "your-secret-key"
};

var metadataStore = new YourMetadataStore(); // IFileMetadataStore
var service = new S3FileStorageService(options, metadataStore);

// Same IFileStorageService contract as LocalFileStorageService

Documentation map

For multipart session stores and Postgres metadata, follow references from your host registration (e.g. Lyo.FileMetadataStore.Postgres).

Backend capability matrix

Plaintext direct uploads (BeginDirectUpload) exclude encryption and compression on the edge PUT. Finalized metadata runs through the usual policy, malware, and availability flags.

Staged uploads use a separate staged_file_upload table, not file_metadata until commit. Complete verifies and hashes the staged object. Commit runs the normal compress/encrypt pipeline (API or job worker via UploadCompleted events).

CancellationToken and cloud signing

ct honours cooperative cancellation during most async I/O. Presigned URL generation, some AWS signing, and synchronous SDK paths may complete even after cancellation is requested. See GetPreSignedReadUrlAsync and related remarks on IFileStorageService.

IFileStorageService

• Writes. SaveFileAsync, SaveFromStreamAsync
• Reads. GetFileAsync, GetFileStreamAsync. Compressed payloads are decompressed via ICompressionService.Resolver using stored CompressionAlgorithm metadata. Optional per-call compressionAlgorithmOverride or FileStorageServiceBaseOptions.DecompressionAlgorithmOverride. Bounded by MaxDecompressedFileSize through MaxBytesWriteStream.
• Zip archive. IFileStorageArchiveService.CreateArchiveAsync spools files through Lyo.IO.Temp, zips on disk (nested ZipPath, caller fileName), and returns a stream that deletes the temp session on dispose. Caps: FileStorageArchiveOptions.MaxFileCount / MaxTotalUncompressedBytes. DI: AddFileStorageArchiveServiceFromConfiguration + AddFileStorageArchiveServiceKeyed.
• Metadata / delete. GetMetadataAsync, DeleteFileAsync(Guid, FileDeletionMode, CancellationToken). Deletes the backing object then, depending on FileDeletionMode, either tombstones metadata (RemoveObjectAndTombstoneMetadata, default) or purges it via IFileMetadataStore.PurgeMetadataAsync (RemoveObjectAndPurgeMetadata). Operator/governance flows only. Never accept this mode from end-user input.
• Temporary links. GetPreSignedReadUrlAsync (with optional PreSignedReadUrlOptions overrides). Both overloads accept CancellationToken with a default of default.
• Direct single PUT. BeginDirectUploadAsync, CompleteDirectUploadAsync. S3/Blob populate RequiredPutHeaders for SSE + signed Content-Type parity.
• Copy. CopyFileAsync (new file id)
• Move / rename. MoveFileAsync (same file id, relocate by PathPrefix). RenameFileAsync (metadata OriginalFileName only).
• Key ops. MigrateDeksAsync, RotateDeksAsync. Results split Skipped vs Failed. Missing blobs and short encryption headers fail-fast.
• Events. FileSaved, FileRetrieved, FileDeleted, FileMoved, FileRenamed, FileMetadataRetrieved, FileAuditOccurred. Payloads carry a redacted FileStoreSnapshot (wrapped DEK and KEK salt omitted).
• Health. CheckHealthAsync via IHealth. Depth follows HealthCheckMode.

IFileStorageDiagnosticsService

Optional ListStorageKeysAsync implemented by backends that can enumerate object/path keys (Local, S3, Blob). All implementations normalize the caller-supplied prefix through Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix, the same helper that backs FileStorageServiceBase.ValidatePathPrefix. It strips wrapping slashes and rejects .. segments, doubled separators, and embedded \0 with ArgumentException (HTTP 400 in ASP.NET).

IMultipartUploadService

Large-file client-part uploads. Register multipart services together with keyed IFileStorageService (Extensions.AddLocalMultipartUploadService or S3/Blob equivalents).

IStagedFileUploadService

Two-phase uploads for large or untrusted client payloads: begin, then client PUT to a staging key (…/.stage/{stageId}/object), then complete (verify/hash), then commit (compress/encrypt into canonical storage). Session state lives in IStagedFileUploadStore / staged_file_upload, not file_metadata until commit.

Register AddLocalStagedFileUploadService, AddKeyedS3StagedFileUploadService, or AddAzureBlobStagedFileUploadService. Postgres/Sqlite metadata builders auto-register PostgresStagedFileUploadStore / SqliteStagedFileUploadStore when no store is present. Hook IStagedFileUploadEventHandler (or service events) to enqueue async commit workers after UploadCompleted.

Types in this assembly

Extension points

Multipart, direct upload, DEK operations

• FileStorageDekOperations. Implements MigrateDeksAsync and RotateDeksAsync uniformly across backends, including short-encryption-header detection and per-file failure isolation.
• FileStorageStreamingPipelines. Composes compression / encryption / hash / max-size guards over streamed save and read paths.
• PlainDirectUploadCoordinator. Finalizes plaintext direct-upload PUTs into a normal SaveFileAsync outcome (policy, scan, availability) and runs only when the caller used BeginDirectUploadAsync without compression/encryption hints.
• StagedUploadCoordinator. Shared begin/complete/commit/abort orchestration for IStagedFileUploadService. Backend packages plug in IStagedFilePhysicalIO (presigned PUT, stat, read, delete).

DiskFileStorageOptions

DiskFileStorageOptions adds:

Compression resolver

Duplicate detection

When EnableDuplicateDetection is true, saves hash plaintext and call IFileMetadataStore.FindByHashAsync before persisting transformed bytes.

Reads are unchanged: GetFileAsync / GetFileStreamAsync decode according to stored metadata, not per-request compress/encrypt flags.

Legacy appsettings LocalFileStorage vs DiskFileStorage binder details are documented on DiskFileStorageOptions.LegacySectionName and DiskFileStorageConfigurationBinder. The internal BindDiskFileStorage(IServiceProvider, string? preferredSection) is invoked by the keyed-disk AddFileStorageServiceKeyed overloads that accept a configSectionName. It tries the preferred section, then DiskFileStorageOptions.SectionName (DiskFileStorage), then DiskFileStorageOptions.LegacySectionName (LocalFileStorage) and logs a warning when the legacy section is matched.

Keyed registration overload matrix

AddFileStorageServiceKeyed ships in seven shapes so the same key can wire (a) an existing keyed file store, (b) a fresh LocalFileStorageService from options/section, and either reuse or define the encryption key:

For cloud backends, use the package-specific keyed entry points: AddS3FileStorageServiceKeyed(keyName) in Lyo.FileStorage.S3 (fluent builder) and the non-keyed AddAzureBlobFileStorageService(...) in Lyo.FileStorage.AzureBlob (keyed Blob is an open item. See that README).

DTOs

Dependency injection for disk is usually Extensions.AddFileStorageServiceKeyed overloads keyed with your tenant/service key alongside IFileMetadataStore registration.

Features

• Multiple storage backends. Local disk in this package. Cloud in Lyo.FileStorage.S3 and Lyo.FileStorage.AzureBlob.
• Compression and encryption. Optional ICompressionService (exposes Resolver and ResolveForCompress; policy via AddCompressionPolicySelector), ITwoKeyEncryptionService.
• Metadata. IFileMetadataStore (FileStoreResult).
• Duplicate detection. Configurable hashing strategies (DuplicateHandlingStrategy).
• Streaming. SaveFromStreamAsync, pipeline reads via GetFileStreamAsync.
• Lifetimes. Honour lifetimes on IFileMetadataStore and keystores. Overlapping async calls on one service instance are supported.
• Cleanup. Partial file cleanup helpers on FileStorageServiceBase.
• Metrics and logging. Hooks into IMetrics / ILoggerFactory.

Error handling

Errors cover missing optional services, invalid prefixes, traversal attempts (diagnostics/listing paths), FileNotAvailableException for availability-aware reads, FilePolicyRejectedException for scanners, and FileNotFoundException when configured to throw.

Security

Path prefixes are normalised on cloud/local paths via shared helpers: Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix for both listing prefixes and save/direct-upload entry points, and CloudObjectKeyBuilder for object/blob key shape (Build, FromMetadata from SourceFileName + path prefix, InferTrailingSuffixAfterFileId). Save paths also apply EnsureUnderRoot before writing to disk. HashVerifyingReadStream uses a fixed-time compare and only verifies on EOF. Pre-signed reads fall back to the metadata-recorded PathPrefix so SAS/GET URLs work even when the caller cannot supply the original prefix. DirectUploadReceiveBaseUri trusts the named host. Use only inside controlled Test API topologies.

Concurrency

LocalFileStorageService and FileStorageServiceBase accept overlapping async calls. Honour the lifetimes of IFileMetadataStore and keystores.

Health checks

Use await fileStorage.CheckHealthAsync(ct). Backends choose lightweight vs deeper modes via HealthCheckMode.

Tests

Cloud backends use a DispatchProxy-based lightweight stub for IAmazonS3; deeper end-to-end coverage would need LocalStack/Azurite.

Dependencies

Generated from ProjectReference / PackageReference (same model as docs/Lyo.ProjectGraph.html).

• Lyo.Common (direct, lyo)
• Lyo.Compression (direct, lyo)
• Lyo.ContentThreatScan (direct, lyo)
• Lyo.Encryption (direct, lyo)
• Lyo.Exceptions (direct, lyo)
• Lyo.FileMetadataStore (direct, lyo)
• Lyo.Hashing (direct, lyo)
• Lyo.Health (direct, lyo)
• Lyo.IO.Temp (direct, lyo)
• Lyo.Metrics (direct, lyo)
• Lyo.Streams (direct, lyo)
• Microsoft.Bcl.AsyncInterfaces 10.0.5 (direct, microsoft, netstandard2.0)
• Microsoft.Extensions.Hosting.Abstractions 10.0.5 (direct, microsoft)
• System.Text.Json 10.0.5 (direct, microsoft, netstandard2.0)
• Lyo.KeyStore (transitive, lyo)
• Lyo.Result (transitive, lyo)
• BouncyCastle.Cryptography 2.6.2 (transitive, third-party, netstandard2.0)
• EasyCompressor 2.1.0 (transitive, third-party)
• Konscious.Security.Cryptography.Argon2 1.3.1 (transitive, third-party)
• Microsoft.Extensions.Configuration.Binder 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.DependencyInjection.Abstractions 10.0.5 (transitive, microsoft, net10.0, netstandard2.0)
• Microsoft.Extensions.Logging.Abstractions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.ConfigurationExtensions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.DataAnnotations 10.0.5 (transitive, microsoft)
• System.Buffers 4.6.1 (transitive, microsoft, netstandard2.0)
• System.IO.Hashing 10.0.5 (transitive, microsoft, net10.0)
• System.Memory 4.6.3 (transitive, microsoft, netstandard2.0)
• System.Threading.Tasks.Extensions 4.6.3 (transitive, microsoft, netstandard2.0)