Dtde.Core 1.0.1

<div align="center">

DTDE - Distributed Temporal Data Engine

Transparent horizontal sharding and temporal versioning for Entity Framework Core

📚 Documentation · 🚀 Quick Start · 💡 Samples · 📊 Benchmarks

</div>

Overview

DTDE is a NuGet package that adds transparent horizontal sharding and optional temporal versioning to Entity Framework Core applications. Write standard LINQ queries — DTDE handles data distribution, query routing, and result merging automatically.

// Standard EF Core LINQ - DTDE routes queries transparently
var euCustomers = await db.Customers
    .Where(c => c.Region == "EU")
    .ToListAsync();  // Automatically queries only EU shard

// Point-in-time queries for temporal entities
var ordersLastMonth = await db.ValidAt<Order>(DateTime.Today.AddMonths(-1))
    .Where(o => o.Status == "Completed")
    .ToListAsync();

✨ Key Features

📦 Installation

# All-in-one package (recommended)
dotnet add package Dtde.EntityFramework

# Or install individual packages
dotnet add package Dtde.Abstractions  # Core interfaces
dotnet add package Dtde.Core          # Core implementations
dotnet add package Dtde.EntityFramework  # EF Core integration

Requirements: .NET 8.0+ / 9.0+ / 10.0+, Entity Framework Core 8.0+ / 9.0+ / 10.0+

🚀 Quick Start

1. Define your entity

public class Customer
{
    public int Id { get; set; }
    public string Name { get; set; } = string.Empty;
    public string Region { get; set; } = string.Empty;  // shard key
    public DateTime CreatedAt { get; set; }
}

2. Inherit DtdeDbContext and declare the shard key in OnModelCreating

public class AppDbContext : DtdeDbContext
{
    public DbSet<Customer> Customers => Set<Customer>();

    public AppDbContext(DbContextOptions<AppDbContext> options) : base(options) { }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        modelBuilder.Entity<Customer>()
            .ShardBy(c => c.Region);
    }
}

3. One call in Program.cs

builder.Services.AddDtdeDbContext<AppDbContext>(
    db => db.UseSqlite("Data Source=app.db"),
    dtde => dtde.AddShards("EU", "US", "APAC"));

That's it — three logical shards over a single SQLite connection. Need separate databases per shard? Swap AddShards(...) for one AddShard("EU", euConnStr) per region.

4. Use standard LINQ — DTDE routes for you

// Insert: routed by Region
db.Customers.Add(new Customer { Name = "Acme Corp", Region = "US" });
await db.SaveChangesAsync();

// Query without filter: fans out across all shards, merges results
var all = await db.Customers.ToListAsync();

// Query with a Where on the shard key: pruned to one shard
var euOnly = await db.Customers
    .Where(c => c.Region == "EU")
    .ToListAsync();

That's the entire mental model. See the Getting started guide for date / hash sharding, temporal entities, and per-shard databases.

🔀 Sharding Strategies

DTDE supports multiple sharding strategies to match your data access patterns:

Property-Based Sharding

Distribute data by a property value (region, tenant, category):

modelBuilder.Entity<Customer>(entity =>
{
    entity.ShardBy(c => c.Region);
});

Use cases: Multi-region deployments, GDPR compliance, data residency

Date-Based Sharding

Partition data by date for time-series workloads:

modelBuilder.Entity<Transaction>(entity =>
{
    entity.ShardByDate(t => t.TransactionDate, DateShardInterval.Month);
});

Use cases: Financial transactions, audit logs, metrics, event sourcing

Hash-Based Sharding

Even distribution across shards using consistent hashing:

modelBuilder.Entity<UserProfile>(entity =>
{
    entity.ShardByHash(u => u.UserId, shardCount: 8);
});

Use cases: High-volume data, preventing hotspots, horizontal scaling

Manual Sharding

Map an entity to pre-existing tables managed by your DBA or a SQL project:

modelBuilder.Entity<Order>(entity =>
{
    entity.UseManualSharding(config =>
    {
        config.AddTable("dbo.Orders_2023", o => o.OrderDate.Year == 2023);
        config.AddTable("dbo.Orders_2024", o => o.OrderDate.Year == 2024);
        config.MigrationsEnabled = false;
    });
});

Use cases: Brownfield databases, regulatory schemas you don't control, archive tables.

⏱️ Temporal Versioning

Track entity history and query data at any point in time:

Enable Temporal Tracking

modelBuilder.Entity<Contract>(entity =>
{
    entity.HasTemporalValidity(
        validFrom: nameof(Contract.ValidFrom),
        validTo: nameof(Contract.ValidTo));
});

Temporal Queries

// Current data
var current = await _context.ValidAt<Contract>(DateTime.UtcNow).ToListAsync();

// Historical data (as of a specific date)
var historical = await _context.ValidAt<Contract>(new DateTime(2024, 1, 1)).ToListAsync();

// All versions (bypass temporal filtering)
var allVersions = await _context.AllVersions<Contract>()
    .Where(c => c.ContractNumber == "C-001")
    .OrderBy(c => c.ValidFrom)
    .ToListAsync();

// Data within a date range
var rangeData = await _context.ValidBetween<Contract>(startDate, endDate).ToListAsync();

Temporal Operations

// Add with effective date
_context.AddTemporal(contract, effectiveFrom: DateTime.UtcNow);

// Create new version (closes old, opens new)
var newVersion = _context.CreateNewVersion(existing, changes, effectiveDate);

// Terminate (close validity)
_context.Terminate(contract, terminationDate: DateTime.UtcNow);

await _context.SaveChangesAsync();

🔄 Mixed Usage: Sharded + Regular Entities

DTDE works seamlessly alongside regular EF Core entities:

public class AppDbContext : DtdeDbContext
{
    public DbSet<Customer> Customers => Set<Customer>();   // Sharded
    public DbSet<Contract> Contracts => Set<Contract>();   // Temporal
    public DbSet<AuditLog> AuditLogs => Set<AuditLog>();   // Regular EF Core
}

🗄️ Shard Tiers

Organize shards by access frequency for cost optimization:

dtde.AddShard(s => s
    .WithId("2024-current")
    .WithTier(ShardTier.Hot)         // Frequently accessed, recent data
    .WithConnectionString(fastStorage));

dtde.AddShard(s => s
    .WithId("2023-archive")
    .WithTier(ShardTier.Cold)        // Archived, rarely accessed
    .WithConnectionString(cheapStorage)
    .AsReadOnly());                   // Prevent accidental writes

⚙️ Configuration Options

Fluent API

options.UseDtde(dtde =>
{
    // Entity configuration
    dtde.ConfigureEntity<Order>(e => e.ShardByDate(o => o.OrderDate));

    // Shard definitions
    dtde.AddShard(s => s.WithId("primary").WithConnectionString("..."));

    // Load from configuration file
    dtde.AddShardsFromConfig("shards.json");

    // Default temporal context
    dtde.SetDefaultTemporalContext(() => DateTime.UtcNow);

    // Performance tuning
    dtde.SetMaxParallelShards(10);

    // Diagnostics
    dtde.EnableDiagnostics();
});

JSON Configuration (shards.json)

{
  "shards": [
    {
      "shardId": "2024-q4",
      "name": "Q4 2024 Data",
      "connectionString": "Server=...;Database=Data2024Q4",
      "tier": "Hot",
      "dateRangeStart": "2024-10-01",
      "dateRangeEnd": "2024-12-31",
      "isReadOnly": false,
      "priority": 100
    },
    {
      "shardId": "2024-archive",
      "name": "2024 Archive",
      "connectionString": "Server=...;Database=Archive2024",
      "tier": "Cold",
      "isReadOnly": true,
      "priority": 10
    }
  ]
}

📊 Performance Benchmarks

Comprehensive benchmarks comparing single table vs sharded approaches:

Test Environment

Key Results

Concurrent Access Performance

Write Operations

Key insight: Sharded queries benefit significantly from partition pruning — queries that target a specific shard key value only scan relevant partitions. Concurrent workloads show even greater improvements due to reduced contention.

When to Use Sharding

✅ Good candidates:

• Large datasets (millions+ rows)
• Time-series / temporal data
• Multi-tenant applications
• Geographic distribution requirements
• High write throughput needs
• Hot/cold data patterns
• Concurrent read-heavy workloads

⚠️ Consider carefully:

• Small datasets (<100K rows)
• Random access patterns
• Complex cross-entity joins
• Simple CRUD applications

# Run benchmarks
cd benchmarks/Dtde.Benchmarks
dotnet run -c Release

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Your Application                          │
├─────────────────────────────────────────────────────────────┤
│                   Dtde.EntityFramework                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │DtdeDbContext│  │Query Engine │  │   Update Engine     │  │
│  │  - ValidAt  │  │ - Rewriter  │  │  - VersionManager   │  │
│  │  - AllVersions│ │ - Executor  │  │  - ShardRouter     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│                      Dtde.Core                               │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  Metadata   │  │  Sharding   │  │     Temporal        │  │
│  │  Registry   │  │  Strategies │  │     Context         │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│                   Dtde.Abstractions                          │
│         Interfaces • Contracts • Exceptions                  │
└─────────────────────────────────────────────────────────────┘

📁 Project Structure

dtde/
├── src/
│   ├── Dtde.Abstractions/        # Core interfaces and contracts
│   ├── Dtde.Core/                # Core implementations
│   └── Dtde.EntityFramework/     # EF Core integration
├── tests/
│   ├── Dtde.Core.Tests/          # Unit tests
│   ├── Dtde.EntityFramework.Tests/
│   └── Dtde.Integration.Tests/
├── samples/                       # Sample applications
│   ├── Dtde.Sample.WebApi/       # Basic getting started
│   ├── Dtde.Samples.RegionSharding/   # Property-based sharding
│   ├── Dtde.Samples.DateSharding/     # Date-based sharding
│   ├── Dtde.Samples.HashSharding/     # Hash-based sharding
│   ├── Dtde.Samples.MultiTenant/      # Multi-tenancy
│   └── Dtde.Samples.Combined/         # Mixed strategies
├── benchmarks/
│   └── Dtde.Benchmarks/          # Performance benchmarks
└── docs/                         # Documentation (MkDocs)

💡 Sample Projects

Explore working examples for each sharding strategy:

🧪 Testing

# Run the full suite (~403 tests across Core, EntityFramework, Integration)
dotnet test -c Release

# Run with coverage
dotnet test -c Release --collect:"XPlat Code Coverage"

# Run a single project
dotnet test tests/Dtde.Core.Tests/Dtde.Core.Tests.csproj -c Release

📚 Documentation

• Full Documentation — Complete guides, API reference, and tutorials
• Quick Start Guide — Get running in 5 minutes
• Sharding Guide — Deep dive into sharding strategies
• Temporal Guide — Temporal versioning explained
• API Reference — Complete API documentation
• Architecture — Design decisions and architecture

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide before submitting PRs.

# Clone and build
git clone https://github.com/yohasacura/dtde.git
cd dtde
dotnet build
dotnet test

See also:

• Code of Conduct
• Security Policy
• Changelog

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

🙏 Acknowledgments

• Built with Entity Framework Core
• Inspired by temporal database concepts and bi-temporal data modeling
• Benchmarked with BenchmarkDotNet

<div align="center">

Made with ❤️ for the .NET community

⭐ Star on GitHub · 🐛 Report Bug · 💬 Discussions

</div>