Sa.HybridFileStorage
0.20.0
dotnet add package Sa.HybridFileStorage --version 0.20.0
NuGet\Install-Package Sa.HybridFileStorage -Version 0.20.0
<PackageReference Include="Sa.HybridFileStorage" Version="0.20.0" />
<PackageVersion Include="Sa.HybridFileStorage" Version="0.20.0" />
<PackageReference Include="Sa.HybridFileStorage" />
paket add Sa.HybridFileStorage --version 0.20.0
#r "nuget: Sa.HybridFileStorage, 0.20.0"
#:package Sa.HybridFileStorage@0.20.0
#addin nuget:?package=Sa.HybridFileStorage&version=0.20.0
#tool nuget:?package=Sa.HybridFileStorage&version=0.20.0
Sa.HybridFileStorage
English | Русский
Hybrid file storage abstraction with automatic provider failover. Unifies multiple storage backends (FileSystem, S3, PostgreSQL) under a single resilient API — if one provider becomes unavailable, the system switches to another.
Installation
dotnet add package Sa.HybridFileStorage
Table of Contents
- Installation
- Virtual Folders (Baskets) Concept
- Supported Storage Providers
- Key Features
- File ID Format
- Quick Start
- CRUD Examples
- Cross-Basket Copying
- Batch Operations
- Interceptors
- Read-Only Mode
- Settings Reference
- Domain Types
- Exceptions
Virtual Folders (Baskets) Concept
Sa.HybridFileStorage operates on the idea of virtual folders — called baskets. A basket is a logical, string-named scope that sits above physical storage backends. From your application's perspective, you work with simple folder names like "drafts", "documents", or "archive". Behind each basket name lies a storage provider (or a list of providers) that organise data differently.
How baskets map to storage
Each basket is backed by one or more IFileStorage providers. The same basket name can be served by different physical systems, and you can combine multiple providers for redundancy or tiered storage:
| Basket name | Physical backend | Organisation | Example File ID |
|---|---|---|---|
drafts |
FileSystem (fs://) |
Plain directory tree on disk | fs://drafts/42/notes.txt |
documents |
PostgreSQL (pg://) |
Relational table with date partitioning | pg://documents/42/1751347200/contract.pdf |
archive |
S3 / MinIO (s3://) |
Cloud bucket with flat namespace | s3://archive/42/old-report.zip |
Behind every basket may hide a single storage or a list of stores — the hybrid layer manages failover transparently. You never need to know which physical system holds your file; you only reference it by its File ID.
Configuring baskets to backends
Each provider binds to exactly one basket, set through its own options. Every provider registers
itself with its own Add... method and is picked up by the container automatically:
// Basket "drafts" → local filesystem. Registers its own IFileStorage; the root and the file
// I/O live on a temp folder registered under the same name.
builder.Services.AddSaFileSystemFileStorage("drafts", b => b
.Options(ob => ob.Configure(x => x.Basket = "drafts"))
.TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
folder.RootPath = @"C:\data\drafts"))));
// Basket "documents" → PostgreSQL with auto-partitioning. The shared data source comes from
// AddNpgsqlDataSource(...) unless the storage pins its own. One key per storage, and each keyed
// storage owns its table (plus its own migration and retention schedules).
builder.Services.AddSaPostgreSqlFileStorage("documents", b => b
.Options(ob => ob.Configure(x =>
{
x.Basket = "documents";
x.TableName = "documents_files";
})));
// Basket "archive" → S3 cloud storage. The call registers the storage's own bucket client.
builder.Services.AddSaS3FileStorage(o => o.Options(ob => ob.Configure(x =>
{
x.Endpoint = "http://minio:9000";
x.AccessKey = "ROOTUSER";
x.SecretKey = "ChangeMe123";
x.Bucket = "company-archive";
x.Basket = "archive";
})));
// Build the hybrid container once: it picks up every IFileStorage registered above.
builder.Services.AddSaHybridFileStorage(cfg => cfg.AddLogging());
Every Add... call registers one storage, and the container collects all of them, so several
baskets (and several storages in one basket) can be wired before a single AddSaHybridFileStorage
call. ConfigureStorage is only needed for a custom IFileStorage implementation.
Once configured, all CRUD operations use basket names — not provider specifics:
// Upload to the "drafts" virtual folder — goes to the filesystem automatically
var result = await storage.UploadAsync("drafts", input, stream, ct);
// File ID: fs://drafts/42/my-notes.txt
// Download from "documents" — routed to PostgreSQL transparently
await storage.DownloadAsync(result.FileId, processStream, ct);
// Copy from "drafts" to "archive" — crosses FS → S3 boundary seamlessly
await storage.CopyToBasketAsync(result.FileId, "archive", ct: ct);
Supported Storage Providers
| Provider | Package | Registration | Use Case |
|---|---|---|---|
| In-Memory | Sa.HybridFileStorage |
AddSaInMemoryFileStorage() |
Testing, ephemeral scenarios |
| File System | Sa.HybridFileStorage.FileSystem |
AddSaFileSystemFileStorage(...) |
Local development, on-premise deployments |
| S3 Compatible | Sa.HybridFileStorage.S3 |
AddSaS3FileStorage(...) |
Cloud storage (AWS S3, MinIO, etc.) |
| PostgreSQL | Sa.HybridFileStorage.Postgres |
AddSaPostgreSqlFileStorage(...) |
Database-embedded files, transactional consistency, partitioning |
Each provider's
Add...method registers anIFileStorage, and the hybrid container picks every registeredIFileStorageup automatically. The provider classes themselves are internal — construct them through theirAdd...method, not withnew.
Key Features
- ✅ Unified API — Single
IHybridFileStorageinterface for all storage providers - ✅ Basket-Tenant isolation — Multi-tenant support with scoped storage containers
- ✅ Failover — Automatic provider switching when one backend fails
- ✅ Streaming support — Memory-efficient file transfers via
Stream - ✅ Native AOT ready — Full compatibility with .NET 10 Native AOT
- ✅ Batch operations — Bulk file processing with configurable parallelism
- ✅ Interceptors — Upload/download/delete lifecycle hooks for cross-cutting concerns
- ✅ Read-only mode — Protect storage from accidental modifications
File ID Format
All files are identified using a unified URI-like format:
{storageType}://{basket}/{tenantId}/{path}
Each storage provider adds its own path depth:
| Provider | File ID Example | Path Structure |
|---|---|---|
| In-Memory | mem://share/42/document.pdf |
{basket}/{tenantId}/{fileName} |
| File System | fs://documents/100/report.xlsx |
{basket}/{tenantId}/{fileName} |
| S3 | s3://uploads/7/invoice.csv |
{basket}/{tenantId}/{fileName} |
| PostgreSQL | pg://files/12/1751347200/photo.jpg |
{basket}/{tenantId}/{unixTimestamp}/{fileName} |
Note: PostgreSQL includes a Unix timestamp in the path because it partitions by date. Other providers omit the timestamp.
Parsing File IDs
Use the static FileIdParser utility:
if (FileIdParser.TryParse("pg://files/42/1751347200/report.pdf", out var basket, out var tenantId, out var timestamp, out var fileName))
{
Console.WriteLine($"Basket={basket}, Tenant={tenantId}, TS={timestamp}, Name={fileName}");
// Basket=files, Tenant=42, TS=1751347200, Name=report.pdf
}
// The parameterless layout is guessed: a numeric second segment is read as the Postgres
// timestamp. State the provider's layout when it is known — otherwise a filesystem name with a
// numeric directory (`fs://documents/100/2024/report.pdf`) loses that directory from `fileName`.
if (FileIdParser.TryParse(
"fs://documents/100/2024/report.pdf",
FileIdFormat.WithoutTimestamp,
out _, out var tenantId, out _, out var fullName))
{
Console.WriteLine($"Tenant={tenantId}, Name={fullName}"); // Tenant=100, Name=2024/report.pdf
}
Quick Start
Registration (Generic Host)
There is no public non-DI constructor: the hybrid storage is created by
AddSaHybridFileStorage, and every IFileStorage registered before it joins the failover chain
in registration order.
using Microsoft.Extensions.Hosting;
using Sa.HybridFileStorage;
using Sa.HybridFileStorage.Domain;
using Sa.HybridFileStorage.FileSystem;
using Sa.HybridFileStorage.S3;
using System.Text;
var builder = Host.CreateApplicationBuilder(args);
// 1. Providers register themselves. Each call adds an IFileStorage — and, for the filesystem,
// its own keyed temp folder. Several providers (and several baskets) can coexist.
builder.Services
.AddSaInMemoryFileStorage()
.AddSaFileSystemFileStorage("files", b => b
.Options(ob => ob.Configure(x => x.Basket = "documents"))
.TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
folder.RootPath = @"C:\data\files"))))
.AddSaS3FileStorage(o => o.Options(ob => ob.Configure(x =>
{
x.Endpoint = "http://localhost:9000";
x.AccessKey = "ROOTUSER";
x.SecretKey = "ChangeMe123";
x.Bucket = "mybucket";
x.Basket = "uploads";
})));
// 2. Build the hybrid container (one call; a second one throws). Logging interceptors are optional.
builder.Services.AddSaHybridFileStorage(cfg => cfg.AddLogging());
var host = builder.Build();
var storage = host.Services.GetRequiredService<IHybridFileStorage>();
// 3. Upload — routed to the first writable provider of the basket ("share" → in-memory).
using var stream = new MemoryStream("Hello, HybridFileStorage!"u8.ToArray());
var result = await storage.UploadAsync(
basket: "share",
input: new UploadFileInput { FileName = "hello.txt", TenantId = 42 },
fileStream: stream,
cancellationToken: ct);
Console.WriteLine(result.FileId); // mem://share/42/hello.txt
// 4. Download — the stream is handed to your callback, never buffered by the hybrid layer.
bool wasFound = await storage.DownloadAsync(result.FileId, async (fileStream, token) =>
{
using var reader = new StreamReader(fileStream, Encoding.UTF8);
Console.WriteLine(await reader.ReadToEndAsync(token)); // Hello, HybridFileStorage!
}, ct);
// 5. Delete — a write, so read-only providers are skipped.
bool deleted = await storage.DeleteAsync(result.FileId, ct);
IHybridFileStorage is a singleton, so inject it into your own services instead of resolving it
inside handlers. Registration stays in one place: the hybrid layer is not additive, and a second
AddSaHybridFileStorage call throws.
Minimal DI registration
For quick setups, each provider has its own extension method:
// In-Memory only
builder.Services.AddSaInMemoryFileStorage();
// File System only
builder.Services.AddSaFileSystemFileStorage("files", b => b
.Options(ob => ob.Configure(x => x.Basket = "documents"))
.TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
folder.RootPath = @"C:\data\files"))));
// S3 only
builder.Services.AddSaS3FileStorage(o => o.Options(ob => ob.Configure(x =>
{
x.Endpoint = "http://localhost:9000";
x.AccessKey = "ROOTUSER";
x.SecretKey = "ChangeMe123";
x.Bucket = "mybucket";
x.Basket = "uploads";
})));
// Then register the hybrid layer
builder.Services.AddSaHybridFileStorage(cfg => cfg.AddLogging());
CRUD Examples
Upload
Upload accepts a basket name (scope/container), metadata, and a Stream. The hybrid layer finds a writable provider matching the basket and uploads the file.
// Upload from a Stream
using var stream = File.OpenRead(@"C:\temp\document.pdf");
var result = await storage.UploadAsync(
basket: "documents",
input: new UploadFileInput { FileName = "document.pdf", TenantId = 42 },
fileStream: stream,
cancellationToken: ct);
Console.WriteLine($"Uploaded: {result.FileId}");
// Output: fs://documents/42/document.pdf
Copy a local file directly:
var result = await storage.CopyFromFileAsync(
filePath: @"C:\temp\image.png",
basket: "images",
input: new UploadFileInput { FileName = "avatar.png", TenantId = 7 },
ct: ct);
Download
Download delegates the file stream to a callback Func<Stream, CancellationToken, Task>. This avoids loading the entire file into memory.
// Process stream inline
bool found = await storage.DownloadAsync(result.FileId, async (stream, token) =>
{
using var reader = new StreamReader(stream, Encoding.UTF8);
string content = await reader.ReadToEndAsync(token);
Console.WriteLine(content);
}, ct);
// Copy to another stream
using var destination = new FileStream(@"C:\output\copy.pdf", FileMode.Create);
await storage.DownloadAsync(result.FileId, async (source, token) =>
await source.CopyToAsync(destination, 81920, token),
ct);
Delete
bool deleted = await storage.DeleteAsync(result.FileId, ct);
if (deleted)
Console.WriteLine("File removed.");
else
Console.WriteLine("File not found.");
Get Metadata
var metadata = await storage.GetMetadataAsync(result.FileId, ct);
if (metadata != null)
{
Console.WriteLine($"Basket: {metadata.Basket}");
Console.WriteLine($"Tenant: {metadata.TenantId}");
Console.WriteLine($"Name: {metadata.FileName}");
Console.WriteLine($"Type: {metadata.StorageType}");
}
Cross-Basket Copying
Move or duplicate files between baskets (even across different providers):
// Copy within same basket
var copied = await storage.CopyToBasketAsync(
fileId: "fs://documents/42/report.pdf",
basket: "archive",
ct: ct);
// Customise upload metadata during copy
var renamed = await storage.CopyToBasketAsync(
fileId: "fs://documents/42/report.pdf",
basket: "backup",
configure: meta => new UploadFileInput
{
TenantId = meta.TenantId,
FileName = $"renamed-{meta.FileName}" // change the name
},
ct: ct);
Batch Operations
High-throughput parallel file operations with progress reporting and error handling:
// Batch copy with parallelism
var batchResult = await storage.CopyToScopeBatchAsync(
fileIds:
[
"fs://documents/1/a.pdf",
"fs://documents/2/b.pdf",
"s3://uploads/3/c.pdf",
],
basket: "archive",
options: new BatchOptions
{
MaxDegreeOfParallelism = 8,
ContinueOnError = true,
OperationTimeout = TimeSpan.FromSeconds(30),
Progress = new Progress<BatchOperationProgress>(p =>
{
Console.WriteLine($"{p.Completed}/{p.Total} — OK:{p.Succeeded} Fail:{p.Failed}");
})
},
ct: ct);
foreach (var ok in batchResult.Succeeded)
Console.WriteLine($"Copied: {ok.FileId}");
foreach (var err in batchResult.Failed)
Console.WriteLine($"Failed #{err.Index}: {err.FileId} — {err.Exception.Message}");
// Or throw on any failure
batchResult.ThrowIfHasErrors(); // throws BatchOperationException<StorageResult>
BatchResult<T>
| Member | Type | Description |
|---|---|---|
Succeeded |
IReadOnlyList<T> |
Successful results |
Failed |
IReadOnlyList<BatchError> |
Errors with file ID and exception |
Total |
int |
Total items processed |
HasErrors |
bool |
Whether any operation failed |
ThrowIfHasErrors() |
void |
Throws BatchOperationException<T> if failures occurred |
BatchOptions
| Property | Description | Default |
|---|---|---|
MaxDegreeOfParallelism |
Concurrent operations | 4 |
ContinueOnError |
Keep going after individual failures | true |
OperationTimeout |
Per-operation timeout (0 = infinite) |
0 |
Progress |
IProgress<BatchOperationProgress> reporter |
null |
Interceptors
Lifecycle hooks for upload/download/delete operations. Implement one of the three interfaces:
public interface IUploadInterceptor
{
// Return false to reject the upload
ValueTask<bool> CanUploadAsync(IFileStorage storage, UploadFileInput input, Stream fileStream, CancellationToken ct);
ValueTask AfterUploadAsync(IFileStorage storage, StorageResult result, CancellationToken ct);
ValueTask OnUploadErrorAsync(IFileStorage storage, Exception exception, CancellationToken ct);
}
public interface IDownloadInterceptor { /* CanDownloadAsync / AfterDownloadAsync / OnDownloadErrorAsync */ }
public interface IDeleteInterceptor { /* CanDeleteAsync / AfterDeleteAsync / OnDeleteErrorAsync */ }
Example — reject uploads of specific filenames:
public class DeniedExtensionInterceptor : IUploadInterceptor
{
private static readonly HashSet<string> DeniedExtensions = ["exe", "bat", "cmd"];
public ValueTask<bool> CanUploadAsync(IFileStorage storage, UploadFileInput input, Stream fileStream, CancellationToken ct)
{
var ext = Path.GetExtension(input.FileName)?.TrimStart('.').ToLowerInvariant();
return ValueTask.FromResult(!DeniedExtensions.Contains(ext));
}
public ValueTask AfterUploadAsync(IFileStorage storage, StorageResult result, CancellationToken ct)
=> ValueTask.CompletedTask;
public ValueTask OnUploadErrorAsync(IFileStorage storage, Exception exception, CancellationToken ct)
=> ValueTask.CompletedTask;
}
Register interceptors through the fluent builder:
builder.Services.AddSaHybridFileStorage(cfg => cfg
.ConfigureInterceptors((sp, container) =>
{
container.AddUploadInterceptor(new DeniedExtensionInterceptor());
container.AddDownloadInterceptor(new LoggingDownloadInterceptor());
}));
Built-in logging interceptors are available via .AddLogging().
Read-Only Mode
Set IsReadOnly = true on any provider to prevent writes. Attempted writes throw HybridFileStorageWritableException:
builder.Services.AddSaFileSystemFileStorage("readonly", b => b
.Options(ob => ob.Configure(x =>
{
x.IsReadOnly = true; // uploads/deletes will fail
}))
.TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
folder.RootPath = @"C:\readonly\data"))));
Settings Reference
FileSystemStorageOptions
| Property | Description | Default |
|---|---|---|
Basket |
Scope/container name | "share" |
StorageType |
Scheme prefix in File ID | "fs" |
IsReadOnly |
Prevent writes | false |
The storage root and the file-I/O policy are configured on the keyed temp folder through the
mandatory TempFolder(...) channel (RootPath, MaxAge, TouchDebounce, ...). The registration
is named-only: AddSaFileSystemFileStorage(name, b => b…TempFolder(…)). See
Sa.HybridFileStorage.FileSystem/Readme.md
for pre/post-initialisation, configuration binding, and migration from BasePath.
S3FileStorageOptions
| Property | Description | Default |
|---|---|---|
Endpoint |
S3 endpoint URL | (required) |
AccessKey |
S3 access key | (required) |
SecretKey |
S3 secret key | (required) |
Bucket |
Bucket name | (required) |
Basket |
Scope/container name | "share" |
Region |
Region for SigV4 signing | "eu-central-1" |
StorageType |
Scheme prefix in File ID | "s3" |
IsReadOnly |
Prevent writes | false |
TotalRequestTimeout |
Per-request timeout | 180 sec |
ConnectionPoolLifetime |
Connection pool lifetime | 15 min |
HandlerLifetime |
HttpClient handler lifetime | ∞ (infinite) |
Configured through the standard options pipeline — see
Sa.HybridFileStorage.S3/Readme.md
for pre/post-initialisation, configuration binding and validation.
PostgresFileStorageOptions
Flat options (no nested PartOptions/CleanupOptions/StorageOptions):
| Property | Description | Default |
|---|---|---|
SchemaName |
PostgreSQL schema (auto-detected from search_path if not set) | "public" |
TableName |
Table for file data | "files" |
StorageType |
Scheme prefix in File ID | "pg" |
Basket |
Scope/container name | "share" |
PgPartBy |
Partitioning granularity | PgPartBy.Day |
MigrationScheduleForwardDays |
Days ahead to pre-create partitions | 2 |
ExpireDays |
Auto-cleanup threshold (days) | 365 * 3 |
IsReadOnly |
Prevent writes | false |
InMemoryFileStorageOptions
| Property | Description | Default |
|---|---|---|
Basket |
Scope/container name | "share" |
MaxSizeBytes |
Total byte limit (0 = unlimited) |
1 GB |
IsReadOnly |
Prevent writes | false |
Domain Types
StorageResult
Returned by upload operations. Contains the canonical File ID and a publicly accessible URL.
public sealed record StorageResult(
string FileId, // e.g. "fs://documents/42/report.pdf"
string AbsoluteUrl, // e.g. "C:\data\files\documents\42\report.pdf"
string StorageType, // e.g. "fs", "s3", "pg", "mem"
DateTimeOffset UploadedAt);
UploadFileInput
Input metadata for uploads.
public sealed record UploadFileInput
{
public int TenantId { get; init; } // defaults to 0
public string FileName { get; init; } = ""; // required at validation
public static UploadFileInput Empty { get; } // pre-created empty instance
}
FileMetadata
Read-only metadata retrieved via GetMetadataAsync.
public sealed class FileMetadata
{
public required string Basket { get; init; }
public required string FileName { get; init; }
public int TenantId { get; init; }
public required string StorageType { get; init; }
}
Exceptions
| Exception | When thrown |
|---|---|
HybridFileStorageNoAvailableException |
No storage is available for the operation: no provider is registered for the basket, no storage recognizes the file ID, or every candidate was vetoed by an interceptor |
HybridFileStorageWritableException |
A write (upload or delete) was attempted while every candidate storage is read-only |
HybridFileStorageAggregateException |
Provider errors were collected during failover and no storage produced an answer |
BatchOperationException<T> |
Thrown by BatchResult.ThrowIfHasErrors() when at least one batch item failed |
License
MIT
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.14)
- Microsoft.Extensions.Logging.Abstractions (>= 10.0.14)
- Microsoft.Extensions.Options (>= 10.0.14)
- Microsoft.Extensions.Options.ConfigurationExtensions (>= 10.0.14)
NuGet packages (3)
Showing the top 3 NuGet packages that depend on Sa.HybridFileStorage:
| Package | Downloads |
|---|---|
|
Sa.HybridFileStorage.Postgres
File storage management in Pg |
|
|
Sa.HybridFileStorage.FileSystem
File storage management |
|
|
Sa.HybridFileStorage.S3
File storage management in S3 |
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 0.20.0 | 102 | 11/13/2026 |
| 0.18.0 | 124 | 11/3/2026 |
| 0.17.0 | 127 | 10/29/2026 |
| 0.16.0 | 130 | 10/14/2026 |
| 0.15.0 | 132 | 10/2/2026 |
| 0.12.0 | 171 | 7/28/2026 |
| 0.11.0 | 155 | 7/14/2026 |
| 0.10.0 | 170 | 7/2/2026 |
| 0.9.1 | 165 | 5/8/2026 |
| 0.9.0 | 153 | 4/29/2026 |
| 0.8.3 | 174 | 3/11/2026 |
| 0.8.1 | 167 | 3/6/2026 |
| 0.8.0 | 158 | 2/10/2026 |
| 0.7.0 | 161 | 1/27/2026 |
| 0.6.0 | 154 | 1/15/2026 |
| 0.5.0 | 169 | 12/29/2025 |
| 0.4.0 | 344 | 12/17/2025 |
| 0.3.0 | 234 | 10/17/2025 |
| 0.1.0 | 293 | 9/1/2025 |
| 0.0.3 | 248 | 6/16/2025 |