TaskProcessor 1.1.2

TaskProcessor

A lightweight .NET library for controlled concurrent task processing with built-in retry and error handling.

🚀 Overview

TaskProcessor helps you process large collections of tasks efficiently while preventing system overload.

It provides:

• Controlled parallel execution
• Retry mechanisms for transient failures
• Error handling strategies
• Support for both execution and transformation workloads

🎯 Problem It Solves

In real-world backend systems, processing large datasets (thousands of records, API calls, or jobs) can lead to:

• CPU spikes
• Database connection exhaustion
• API throttling
• Unstable performance

Naively running all tasks in parallel (Task.WhenAll) can overwhelm the system.

TaskProcessor solves this by limiting concurrency and adding reliability controls.

⚙️ Features

• ✅ Configurable concurrency (MaxDegreeOfParallelism)
• ✅ Retry support with delay
• ✅ Continue or fail on error
• ✅ Error callbacks for logging/monitoring
• ✅ Supports async operations
• ✅ Generic design (works with any task)

📦 Installation

Install via NuGet:

dotnet add package TaskProcessor

🧠 Usage

1. Process tasks (no return)

var processor = new TaskProcessor(new TaskProcessorOptions
{
    MaxDegreeOfParallelism = 3,
    RetryCount = 2
});

await processor.ProcessAsync(data, async item =>
{
    await ProcessItem(item);
});

2. Transform data (with return value)

var results = await processor.ProcessAsync<Source, Destination>(
    sourceData,
    async item =>
    {
        return mapper.Map<Destination>(item);
    });

🔁 How It Works

TaskProcessor internally:

1. Limits concurrent execution using a semaphore
2. Executes tasks in parallel within defined limits
3. Retries failed tasks based on configuration
4. Collects results (for transformation scenarios)

📊 Use Cases

• Processing large datasets (data migration, ETL)
• Calling external APIs with rate limits
• Background job processing
• Bulk data transformations
• File or message processing

⚠️ Notes

• Result order is not guaranteed in parallel execution
• Best suited for independent tasks
• Avoid very high parallelism for CPU-bound workloads

🧩 Design Philosophy

TaskProcessor follows a simple principle:

You define the work — TaskProcessor manages how it runs.

It keeps business logic separate from execution control, making it reusable across different domains.

🔮 Future Improvements

• Batch processing support
• Exponential backoff retry strategy
• Metrics and performance tracking
• Rate limiting

📄 License

MIT License