Lyo.FileStorage 1.0.0

Lyo.FileStorage

Production-oriented file storage for .NET: save / stream-save / read / delete / metadata with optional compression ( Lyo.Compression), two-key encryption (Lyo.Encryption), duplicate hashing, access policies, malware scans, audit hooks, multipart uploads ( via IMultipartUploadService), and presigned/direct-upload/copy on cloud-capable backends.

With XML documentation generation enabled in the repo (GenerateDocumentationFile in Directory.Build.props), IntelliSense surfaces the same summaries as this README for documented members.

Examples

Local file storage

using Lyo.FileStorage;
using Lyo.FileStorage.Models;

var options = new DiskFileStorageOptions
{
    RootDirectoryPath = "/path/to/storage",
    EnableDuplicateDetection = true,
    DuplicateStrategy = DuplicateHandlingStrategy.ReturnExisting
};

var service = new LocalFileStorageService(options);

// Save a file
var data = File.ReadAllBytes("document.pdf");
var result = await service.SaveFileAsync(
    data,
    originalFileName: "document.pdf",
    compress: true,
    encrypt: true,
    keyId: "my-encryption-key");

// Retrieve a file
var retrievedData = await service.GetFileAsync(result.Id);

// Delete a file
await service.DeleteFileAsync(result.Id);

S3 storage (AWS, Backblaze B2, MinIO, Wasabi, Cloudflare R2, …)

using Lyo.FileStorage.S3;
using Lyo.FileStorage.Models;

var options = new S3FileStorageOptions
{
    BucketName = "my-bucket",
    Region = "us-east-1",
    AccessKeyId = "your-access-key",
    SecretAccessKey = "your-secret-key"
};

var metadataStore = new YourMetadataStore(); // IFileMetadataStore
var service = new S3FileStorageService(options, metadataStore);

// Same IFileStorageService contract as LocalFileStorageService

Documentation map

For multipart session stores and Postgres metadata, follow references from your host registration (e.g. Lyo.FileMetadataStore.Postgres).

Backend capability matrix

Plaintext direct uploads (BeginDirectUpload) deliberately exclude encryption/compression on the edge PUT; finalized metadata runs through normal policy/malware/availability flags.

Staged uploads use a separate staged_file_upload table (not file_metadata until commit). Complete verifies and hashes the staged object; commit runs the normal compress/encrypt pipeline (API or job worker via UploadCompleted events).

Backend capability matrix — CancellationToken and cloud signing

ct honours cooperative cancellation during most async I/O. Presigned URL generation, some AWS signing, and synchronous SDK paths may complete even after cancellation is requested — see GetPreSignedReadUrlAsync and related remarks on IFileStorageService.

Contracts — IFileStorageService

• Writes — SaveFileAsync, SaveFromStreamAsync
• Reads — GetFileAsync, GetFileStreamAsync (compressed payloads are decompressed via ICompressionService.Resolver using stored CompressionAlgorithm metadata; optional per-call compressionAlgorithmOverride or FileStorageServiceBaseOptions.DecompressionAlgorithmOverride; bounded by MaxDecompressedFileSize through MaxBytesWriteStream)
• Zip archive — IFileStorageArchiveService.CreateArchiveAsync spools files through Lyo.IO.Temp, zips on disk (nested ZipPath, caller fileName), and returns a stream that deletes the temp session on dispose. Caps: FileStorageArchiveOptions.MaxFileCount / MaxTotalUncompressedBytes. DI: AddFileStorageArchiveServiceFromConfiguration + AddFileStorageArchiveServiceKeyed.
• Metadata / delete — GetMetadataAsync, DeleteFileAsync(Guid, FileDeletionMode, CancellationToken) (deletes the backing object then, depending on * *FileDeletionMode**, either tombstones metadata (RemoveObjectAndTombstoneMetadata, default) or purges it via IFileMetadataStore.PurgeMetadataAsync ( RemoveObjectAndPurgeMetadata — operator/governance flows only; never accept this mode from end-user input)
• Temporary links — GetPreSignedReadUrlAsync (with optional PreSignedReadUrlOptions overrides; both overloads accept CancellationToken with a default of default)
• Direct single PUT — BeginDirectUploadAsync, CompleteDirectUploadAsync (S3/Blob populate RequiredPutHeaders for SSE + signed Content-Type parity)
• Copy — CopyFileAsync (new file id)
• Move / rename — MoveFileAsync (same file id, relocate by PathPrefix); RenameFileAsync (metadata OriginalFileName only)
• Key ops — MigrateDeksAsync, RotateDeksAsync (results split Skipped vs Failed; missing blobs and short encryption headers fail-fast)
• Events — FileSaved, FileRetrieved, FileDeleted, FileMoved, FileRenamed, FileMetadataRetrieved, FileAuditOccurred. Payloads carry a redacted FileStoreSnapshot ( wrapped DEK and KEK salt omitted)
• Health — CheckHealthAsync via IHealth (lightweight vs full via HealthCheckMode)

Contracts — IFileStorageDiagnosticsService

Optional ListStorageKeysAsync implemented by backends that can enumerate object/path keys (Local, S3, Blob). All implementations normalize the caller-supplied prefix through Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix — the same helper that backs FileStorageServiceBase.ValidatePathPrefix — which strips wrapping slashes and rejects .. segments, doubled separators, and embedded \0 with ArgumentException (HTTP 400 in ASP.NET).

Contracts — IMultipartUploadService

Large-file client-part uploads; register multipart services together with keyed IFileStorageService (Extensions.AddLocalMultipartUploadService or S3/Blob equivalents).

Contracts — IStagedFileUploadService

Two-phase uploads for large or untrusted client payloads: begin → client PUT to a staging key (…/.stage/{stageId}/object) → complete (verify/hash) → commit ( compress/encrypt into canonical storage). Session state lives in IStagedFileUploadStore / staged_file_upload — not file_metadata until commit.

Register AddLocalStagedFileUploadService, AddKeyedS3StagedFileUploadService, or AddAzureBlobStagedFileUploadService. Postgres/Sqlite metadata builders auto-register * PostgresStagedFileUploadStore* / SqliteStagedFileUploadStore when no store is present. Hook IStagedFileUploadEventHandler (or service events) to enqueue async commit workers after UploadCompleted.

Service matrix (Lyo.FileStorage assembly)

Service matrix (Lyo.FileStorage assembly) — Extension points

Multipart, direct upload, DEK operations

• FileStorageDekOperations — implements MigrateDeksAsync and RotateDeksAsync uniformly across backends, including short-encryption-header detection and per-file failure isolation.
• FileStorageStreamingPipelines — composes compression / encryption / hash / max-size guards over streamed save and read paths.
• PlainDirectUploadCoordinator — finalizes plaintext direct-upload PUTs into a normal SaveFileAsync outcome (policy, scan, availability) and runs only when the caller used BeginDirectUploadAsync without compression/encryption hints.
• StagedUploadCoordinator — shared begin/complete/commit/abort orchestration for IStagedFileUploadService; backend packages plug in IStagedFilePhysicalIO ( presigned PUT, stat, read, delete).

Options — FileStorageServiceBaseOptions / DiskFileStorageOptions

DiskFileStorageOptions adds:

Options — FileStorageServiceBaseOptions / DiskFileStorageOptions — Compression (resolver)

Options — FileStorageServiceBaseOptions / DiskFileStorageOptions — Duplicate detection

When EnableDuplicateDetection is true, saves hash plaintext and call IFileMetadataStore.FindByHashAsync before persisting transformed bytes.

Reads are unchanged: GetFileAsync / GetFileStreamAsync decode according to stored metadata, not per-request compress/encrypt flags.

Legacy appsettings LocalFileStorage vs DiskFileStorage binder details are documented on DiskFileStorageOptions.LegacySectionName and * *DiskFileStorageConfigurationBinder**. The internal BindDiskFileStorage(IServiceProvider, string? preferredSection) is invoked by the keyed-disk *AddFileStorageServiceKeyed ** overloads that accept a configSectionName; it tries the preferred section, then DiskFileStorageOptions.SectionName (DiskFileStorage), then * DiskFileStorageOptions.LegacySectionName (LocalFileStorage) and logs a warning when the legacy section is matched.

Keyed registration overload matrix

AddFileStorageServiceKeyed ships in seven shapes so the same key can wire (a) an existing keyed file store, (b) a fresh LocalFileStorageService from options/section, and either reuse or define the encryption key:

For cloud backends, use the package-specific keyed entry points: AddS3FileStorageServiceKeyed(keyName) in Lyo.FileStorage.S3 (fluent builder) and the non-keyed * AddAzureBlobFileStorageService(...)* in Lyo.FileStorage.AzureBlob (keyed Blob is an open item — see that README).

DTO highlights

Dependency injection for disk is usually Extensions.AddFileStorageServiceKeyed overloads keyed with your tenant/service key alongside IFileMetadataStore registration.

Features (overview)

• Multiple storage backends — Local disk (this package); cloud in Lyo.FileStorage.S3 and Lyo.FileStorage.AzureBlob
• Compression & encryption — Optional ICompressionService (exposes Resolver and ResolveForCompress; policy via AddCompressionPolicySelector), * ITwoKeyEncryptionService*
• Metadata — IFileMetadataStore (FileStoreResult)
• Duplicate detection — Configurable hashing strategies (DuplicateHandlingStrategy)
• Streaming — SaveFromStreamAsync, pipeline reads via GetFileStreamAsync
• Thread safety — Design matches concurrent callers; honour lifetimes on IFileMetadataStore and keystores
• Cleanup — Partial file cleanup helpers on FileStorageServiceBase
• Metrics & logging — Hooks into IMetrics / ILoggerFactory

Production ready

This library has been exercised for library-style production scenarios and includes defensive validation, hashing, auditing hooks, malware scan integration, and observable health checks (CheckHealthAsync).

Error handling

Detailed errors cover missing optional services, invalid prefixes, traversal attempts (diagnostics/listing paths), FileNotAvailableException for availability-aware reads, * FilePolicyRejectedException* for scanners, and FileNotFoundException when configured to throw.

Security

Path prefixes are normalised on cloud/local paths via shared helpers: Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix for both listing prefixes and save/direct-upload entry points, and CloudObjectKeyBuilder for object/blob key shape. Save paths additionally apply EnsureUnderRoot before writing to disk. * HashVerifyingReadStream* uses a fixed-time compare and only verifies on EOF. Pre-signed reads fall back to the metadata-recorded PathPrefix so SAS/GET URLs work even when the caller cannot supply the original prefix. DirectUploadReceiveBaseUri trusts the named host — use only inside controlled (Test API) topologies.

Thread safety

LocalFileStorageService and FileStorageServiceBase are safe for overlapping async calls assuming dependencies are disposed/correct lifetime.

Health checks

Use await fileStorage.CheckHealthAsync(ct); backends choose lightweight vs fuller modes via HealthCheckMode.

Tests

Cloud backends use a DispatchProxy-based lightweight stub for IAmazonS3; deeper end-to-end coverage would need LocalStack/Azurite.

Dependencies

Generated from ProjectReference / PackageReference (same model as docs/Lyo.ProjectGraph.html).

• Lyo.Common — (direct, lyo)
• Lyo.Compression — (direct, lyo)
• Lyo.ContentThreatScan — (direct, lyo)
• Lyo.Encryption — (direct, lyo)
• Lyo.Exceptions — (direct, lyo)
• Lyo.FileMetadataStore — (direct, lyo)
• Lyo.Hashing — (direct, lyo)
• Lyo.Health — (direct, lyo)
• Lyo.IO.Temp — (direct, lyo)
• Lyo.Metrics — (direct, lyo)
• Lyo.Streams — (direct, lyo)
• Microsoft.Bcl.AsyncInterfaces 10.0.5 — (direct, microsoft, netstandard2.0)
• Microsoft.Extensions.Hosting.Abstractions 10.0.5 — (direct, microsoft)
• System.Text.Json 10.0.5 — (direct, microsoft, netstandard2.0)
• Lyo.KeyStore — (transitive, lyo)
• Lyo.Result — (transitive, lyo)
• BouncyCastle.Cryptography 2.6.2 — (transitive, third-party, netstandard2.0)
• EasyCompressor 2.1.0 — (transitive, third-party)
• Konscious.Security.Cryptography.Argon2 1.3.1 — (transitive, third-party)
• Microsoft.Extensions.Configuration.Binder 10.0.5 — (transitive, microsoft)
• Microsoft.Extensions.DependencyInjection.Abstractions 10.0.5 — (transitive, microsoft, net10.0, netstandard2.0)
• Microsoft.Extensions.Logging.Abstractions 10.0.5 — (transitive, microsoft)
• Microsoft.Extensions.Options.ConfigurationExtensions 10.0.5 — (transitive, microsoft)
• Microsoft.Extensions.Options.DataAnnotations 10.0.5 — (transitive, microsoft)
• System.Buffers 4.6.1 — (transitive, microsoft, netstandard2.0)
• System.IO.Hashing 10.0.5 — (transitive, microsoft, net10.0)
• System.Memory 4.6.3 — (transitive, microsoft, netstandard2.0)
• System.Threading.Tasks.Extensions 4.6.3 — (transitive, microsoft, netstandard2.0)