Winnow 1.2.0

Winnow

Separate the good saves from the bad.

Batch operations for Entity Framework Core with per-entity failure isolation.

var saver = new Winnower<Product, int>(context);
var result = saver.Insert(products);

if (!result.IsCompleteSuccess)
{
    foreach (var failure in result.Failures)
        _logger.LogWarning("Insert failed: {Message}", failure.ErrorMessage);
}

Motivation

Entity Framework Core's SaveChanges() operates atomically: a single invalid entity causes the entire batch to fail. Winnow — named for the process of separating grain from chaff — winnows out the failures, allowing valid entities to persist while capturing detailed failure information for invalid ones.

Installation

dotnet add package Winnow

Requirements: .NET 8.0+, Entity Framework Core 8.0+

Table of Contents

• Quick Start
• Choosing Your Operation
• Basic Operations
• Strategies
• Handling Results
• When to Use What
• When NOT to Use This
• Parallel Batch Processing
• Advanced Topics
• Common Mistakes

Quick Start

// Create a saver for your entity type and key type
var saver = new Winnower<Product, int>(context);

// Insert new entities
var insertResult = saver.Insert(newProducts);
Console.WriteLine($"Inserted: {insertResult.SuccessCount}, Failed: {insertResult.FailureCount}");

// Update existing entities
var updateResult = saver.Update(existingProducts);

// Delete entities
var deleteResult = saver.Delete(productsToRemove);

// Always check for failures
foreach (var failure in updateResult.Failures)
{
    Console.WriteLine($"Entity {failure.EntityId} failed: {failure.ErrorMessage}");
}

// All operations have async versions
var asyncResult = await saver.InsertAsync(newProducts, cancellationToken);

Supported Key Types

Winnower supports any key type that implements IEquatable<TKey>:

var saver = new Winnower<Product, int>(context);      // Integer keys
var saver = new Winnower<Order, long>(context);       // Long keys
var saver = new Winnower<Document, Guid>(context);    // GUID keys
var saver = new Winnower<Setting, string>(context);   // String keys

For composite keys, see Composite Keys.

Dependency Injection

Register Winnow with your DI container:

services.AddWinnow<AppDbContext>();

Then inject IWinnower<TEntity, TKey> or IWinnower<TEntity>:

public class ProductService(IWinnower<Product, int> saver)
{
    public WinnowResult<int> ImportProducts(List<Product> products)
        => saver.Update(products);
}

Choosing Your Operation

What do you need to do?
├── Insert new entities only ──────────────────→ Insert / InsertGraph
├── Update existing entities only ─────────────→ Update / UpdateGraph
├── Delete entities ───────────────────────────→ Delete / DeleteGraph
└── Insert OR update (based on key) ───────────→ Upsert / UpsertGraph

Do your entities have children (navigation properties)?
├── No  → Use the non-graph method (Insert, Update, etc.)
└── Yes → Use the graph method (InsertGraph, UpdateGraph, etc.)

Basic Operations

Insert

var result = saver.Insert(products, new InsertOptions
{
    Strategy = BatchStrategy.DivideAndConquer
});

// Access generated IDs
foreach (var inserted in result.InsertedEntities)
{
    Console.WriteLine($"Index {inserted.OriginalIndex} got ID {inserted.Id}");
}

Update

var result = saver.Update(products);

Console.WriteLine($"Updated: {result.SuccessCount}");

Delete

var result = saver.Delete(productsToRemove);

Upsert

Performs INSERT or UPDATE based on key value (default key → INSERT, non-default → UPDATE).

Warning: Not a database MERGE. See Upsert Operations for race condition details.

var products = new[]
{
    new Product { Id = 0, Name = "New" },   // INSERT (Id=0)
    new Product { Id = 42, Name = "Updated" } // UPDATE
};

var result = saver.Upsert(products, new UpsertOptions
{
    DuplicateKeyStrategy = DuplicateKeyStrategy.RetryAsUpdate  // Handle race conditions
});
Console.WriteLine($"Inserted: {result.InsertedCount}, Updated: {result.UpdatedCount}");

To route by a business key (e.g. an ExternalId or Sku) rather than the primary-key default-value check, use MatchBy:

saver.Upsert(products, new UpsertOptions()
    .WithMatchBy<Product>(p => p.Sku));

See Custom Match Expressions for composite keys, race-condition behavior, and limitations.

Graph Operations

Handle parent entities with their children:

var orders = new List<CustomerOrder>
{
    new CustomerOrder
    {
        CustomerName = "Alice",
        OrderItems = new List<OrderItem>
        {
            new OrderItem { ProductId = 1, Quantity = 2 }
        }
    }
};

var result = saver.InsertGraph(orders);

// For updates, specify orphan behavior
var updateResult = saver.UpdateGraph(orders, new GraphOptions
{
    OrphanedChildBehavior = OrphanBehavior.Delete
});

// Filter which navigations are traversed
var filter = NavigationFilter.Include()
    .Navigation<CustomerOrder>(o => o.OrderItems);

var filteredResult = saver.InsertGraph(orders, new InsertGraphOptions
{
    NavigationFilter = filter  // Only traverses OrderItems, skips deeper levels
});

See Graph Operations for full documentation.

Strategies

Benchmarked Performance

DivideAndConquer adds minimal overhead vs raw SaveChanges() (within measurement noise at most batch sizes) while providing error isolation.

Flat insert, DivideAndConquer (milliseconds):

OneByOne is 5-530x slower depending on provider and batch size. The gap widens at scale because DivideAndConquer scales sub-linearly while OneByOne scales linearly.

Failure Rates Change Everything

DivideAndConquer's advantage erodes sharply when entities fail validation:

At 25% failures, the strategies perform nearly the same. Pre-validate your entities if you expect failures above ~5% — this preserves DivideAndConquer's speed advantage.

Graph and Memory

Graph operations (parent + children) use 2-3x more memory per entity (~20-31 KB vs ~9-11 KB for flat). UpsertGraph is the most expensive due to loading existing entities and tracking changes. DivideAndConquer speedups are compressed but still significant (2-76x depending on provider).

For full results, see SQLite, PostgreSQL, and SQL Server benchmarks.

Handling Results

Every batch operation winnows out the failures, giving you detailed results for each entity:

var result = saver.Update(products);

// Check overall status
if (result.IsCompleteSuccess) { /* all succeeded */ }
if (result.IsPartialSuccess) { /* some succeeded, some failed */ }
if (result.IsCompleteFailure) { /* all failed */ }

// Access successes
foreach (var id in result.SuccessfulIds)
{
    Console.WriteLine($"Saved: {id}");
}

// Access failures with details
foreach (var failure in result.Failures)
{
    Console.WriteLine($"Failed {failure.EntityId}: {failure.ErrorMessage}");
}

// Performance metrics
Console.WriteLine($"Round trips: {result.DatabaseRoundTrips}");
Console.WriteLine($"Duration: {result.Duration}");

For full result type documentation, see Results Reference. To trade reporting detail for lower allocation (especially on graph operations, where per-entity tracking dominates memory), set ResultDetail on the options object — see ResultDetail.

When to Use What

When NOT to Use This

Winnow is not the right choice for every scenario:

• Single entity operations: Standard EF Core Add/Update/Remove + SaveChanges() is simpler
• All-or-nothing transactions: If a single failure should roll back everything, use standard SaveChanges()
• True database MERGE: For high-concurrency upserts, use raw SQL with MERGE (SQL Server) or ON CONFLICT (PostgreSQL)
• Bulk operations without failure tracking: Libraries like EFCore.BulkExtensions offer higher throughput when you do not need per-entity failure isolation
• Read-heavy workloads: This library focuses on write operations

Parallel Batch Processing

ParallelWinnower distributes work across multiple DbContext instances:

// Requires a factory that creates a new DbContext on each call
var saver = new ParallelWinnower<Product, int>(
    () => new AppDbContext(options),
    maxDegreeOfParallelism: 4);

var result = await saver.InsertAsync(products, cancellationToken);

Or use IDbContextFactory<TContext>:

var saver = factory.CreateParallelWinnower<Product, int, AppDbContext>(maxDegreeOfParallelism: 4);

Benchmark reality check: In our benchmarks, ParallelWinnower showed no consistent benefit across any provider. SQLite uses file-level locking so parallel writes contend. PostgreSQL and SQL Server showed improvements (26-36%) at DOP 4 for small batches (1K entities), but the gains disappeared or reversed at larger sizes due to connection pool contention. For most workloads, standard Winnower with DivideAndConquer is faster and simpler. See the benchmark docs for details.

ParallelWinnower may still help with high-latency remote databases (cloud SQL with cross-region latency) where the round-trip cost dominates — a scenario our local Docker benchmarks don't capture. If you use it, note that each partition commits independently: if one fails, others that already committed will NOT be rolled back.

Advanced Topics

Detailed documentation for complex scenarios:

• Composite Keys - Multi-column primary keys
• Graph Operations - Parent-child hierarchies
• Many-to-Many Relationships - Skip navigations and join entities
• Reference Navigation - Many-to-one relationships
• Self-Referencing Hierarchies - Recursive entity structures
• Upsert Operations - Insert-or-update with race condition handling
• Results Reference - Full result type API
• Options Reference - All configuration options
• SQLite Benchmarks - Performance data and strategy guidance
• PostgreSQL Benchmarks - Network database performance
• SQL Server Benchmarks - Network database performance

Common Mistakes

1. Ignoring Failures

// Wrong: Assuming success
saver.Insert(products);

// Correct: Always check results
var result = saver.Insert(products);
if (!result.IsCompleteSuccess)
{
    foreach (var f in result.Failures)
        _logger.LogError("Failed to insert: {Error}", f.ErrorMessage);
}

2. Using Graph Methods for Simple Entities

// Unnecessary overhead
saver.InsertGraph(simpleProducts);

// Better: Use non-graph method
saver.Insert(simpleProducts);

3. Forgetting OrphanBehavior on Updates

// Throws exception if children were removed
saver.UpdateGraph(orders);

// Explicit about what happens to removed children
saver.UpdateGraph(orders, new GraphOptions
{
    OrphanedChildBehavior = OrphanBehavior.Delete
});

4. Assuming Upsert Is Atomic

// Race condition possible between key check and save
saver.Upsert(products);

// Better: identify entities by a unique business key
saver.Upsert(products, new UpsertOptions()
    .WithMatchBy<Product>(p => p.Sku));

// Best for high-concurrency: combine MatchBy with RetryAsUpdate.
// See docs/upsert-operations.md for race condition details.

5. Using DivideAndConquer with High Failure Rates

// Slower than OneByOne when many failures expected
saver.Insert(untrustedData, new InsertOptions
{
    Strategy = BatchStrategy.DivideAndConquer
});

// Better: Use OneByOne for untrusted/validation-heavy data
saver.Insert(untrustedData, new InsertOptions
{
    Strategy = BatchStrategy.OneByOne
});