Accipitech.DedupeDoc 0.1.0

Accipitech.DedupeDoc

Tenant-aware document de-duplication for .NET: content-addressed blob storage (FluentStorage: local disk, AWS S3 or any S3-compatible endpoint) with SHA-256 dedup, version chains, and a metadata store that provisions its own tables at startup on SQL Server, MySQL, or PostgreSQL.

Extracted from and battle-tested against two production document pipelines.

How it works

• Every ingest hashes the content (streaming SHA-256) and stores the blob under a sharded, content-addressed key: {collection}/{aa}/{bb}/{cc}/{hash}. Identical content is stored once; every upload still gets its own metadata row (id, filename, tenant, timestamps).
• Tenancy lives in the metadata database only (EF Core global query filters). Blobs are shared across tenants; a blob is deleted only when the last row in any tenant referencing it is deleted.
• Versioning is a linear chain: IngestNewVersionAsync resolves any chain member to the root, retroactively marks the root as v1, and appends. Re-uploading a version's identical content is a no-op that returns the existing version.

Quick start

// Program.cs — pick ONE provider package:
//   Accipitech.DedupeDoc.SqlServer | .MySql | .PostgreSql
builder.Services.AddDedupeDoc(builder.Configuration, d =>
{
    d.UseSqlServer(builder.Configuration.GetConnectionString("DedupeDoc")!);
    d.UseLocalDiskStorage("/var/app-storage/documents");   // or d.UseS3Storage(...)
    d.UseTenantProvider<MyClaimsTenantProvider>();          // omit for single-tenant apps
});

On startup the library creates/upgrades its own tables (dedupe.documents, dedupe.document_versions — dedupe_-prefixed on MySQL) with its own migrations history (__dedupe_migrations), never touching the host's EF migrations. Set DedupeDocOptions.AutoMigrate = false to run IDedupeSchemaManager.MigrateAsync() from a deployment job instead.

public sealed class MyClaimsTenantProvider(IHttpContextAccessor ctx) : IDedupeTenantProvider
{
    public string? TenantId => ctx.HttpContext?.User.FindFirstValue("tenant");
}

Use it:

var doc = await dedupe.IngestAsync(stream, "cv.pdf");             // doc.BlobWasDeduplicated
var v2  = await dedupe.IngestNewVersionAsync(doc.Id, newStream);
await using var content = await dedupe.OpenReadAsync(v2.Id);
await dedupe.DeleteAsync(doc.Id);                                  // blob freed with last ref
var stats = await dedupe.GetDeduplicationStatsAsync();             // bytes saved, per tenant

Configuration (appsettings.json, section DedupeDoc)

{
  "DedupeDoc": {
    "DefaultCollection": "documents",
    "AutoMigrate": true,
    "IncludeExtensionInBlobKey": false,
    "Storage": {
      "Provider": "S3",
      "BasePath": "/var/app-storage/documents",
      "S3": { "ServiceUrl": "https://minio.internal:9000", "Bucket": "docs",
              "AccessKey": "...", "SecretKey": "...", "Region": null }
    }
  }
}

Host-owned migrations mode

If you'd rather ship the tables inside your app's own migrations pipeline:

// in your DbContext:
protected override void OnModelCreating(ModelBuilder modelBuilder)
    => modelBuilder.AddDedupeDocEntities(this, ctx => ctx.CurrentTenantId, schema: "dedupe");

// registration — no provider package needed:
builder.Services.AddDedupeDoc(d =>
{
    d.UseHostDbContext<MyAppDbContext>();
    d.UseLocalDiskStorage("/var/app-storage/documents");
});

Then dotnet ef migrations add in your app picks the tables up. Schema/table prefix are free parameters here (they are fixed in auto-migrate mode).

Notes and limits

• MySQL: net8.0/net9.0 hosts use Pomelo (8.0.3 / 9.0.0); net10.0 hosts use the drop-in Microting fork of Pomelo (upstream Pomelo is unmaintained and its EF9 build fails at runtime against EF Core 10 — verified). Same API and namespaces; no code or migration changes between TFMs.
• EF Core 8 (net8.0) has no migration locking; on multi-node net8.0 deployments run migrations from a single job, or rely on the built-in retry.
• DocumentResult.BlobWasDeduplicated can reveal that identical content exists in another tenant — treat it as internal telemetry, don't expose it to end users.
• Blob-delete failures are logged and swallowed (an orphaned blob beats a dangling reference); reclaim orphans with IDedupeMaintenanceService.SweepOrphanedBlobsAsync (dry-run by default).
• Non-seekable upload streams are spooled to a delete-on-close temp file automatically.