ToolWheel.Extensions.JobManager.Resilience 1.0.0-preview6

ToolWheel.Extensions.JobManager.Resilience

Adds resilience patterns to the ToolWheel Job Manager: circuit breaker, configurable retry strategies and per-operation/total timeouts.

Package Info

Auto Configuration

This package ships JobManagerResilienceConfigurator implementing IAutoFeatureConfigurator. When AddJobManager() scans loaded assemblies it automatically:

1. Registers IRetryService, ICircuitBreakerService and ITimeoutService as singletons.
2. Adds CircuitBreakerMiddleware, TotalTimeoutMiddleware, RetryMiddleware and OperationTimeoutMiddleware to the pipeline (in that order).
3. Adds CircuitBreakerCondition as an IExecutionCondition.

No manual registration is required – referencing the package is sufficient.

Per-job configuration flows through each feature's Apply method called by IJobService.Add:

Fluent Configuration

All resilience features are reached via a single .Resilience() entry point:

jobs.Add("import-job", worker.RunImport)
    .Name("Data Import")
    .Enabled()
    .Resilience()
        .WithCircuitBreaker(failureThreshold: 3, openDuration: TimeSpan.FromMinutes(2))
        .WithExponentialRetry(maxAttempts: 3, initialDelay: TimeSpan.FromSeconds(2))
        .WithTimeout(t => t
            .WithOperationTimeout(TimeSpan.FromSeconds(30))
            .WithTotalTimeout(TimeSpan.FromMinutes(2)))
    .JobBuilder;

Circuit Breaker

Prevents a job from starting when the circuit is open (too many recent failures). The circuit transitions through three states:

Configuration

// Shorthand
.WithCircuitBreaker(failureThreshold: 5, openDuration: TimeSpan.FromMinutes(1))

// Full configuration
.WithCircuitBreaker(cb => cb
    .WithFailureThreshold(3)
    .WithOpenDuration(TimeSpan.FromSeconds(30))
    .WithSuccessThreshold(2)                        // successes in half-open to close
    .WithSamplingDuration(TimeSpan.FromMinutes(5))  // rolling failure window
    .OnlyFor(typeof(HttpRequestException), typeof(TimeoutException)))

How It Works

1. CircuitBreakerCondition blocks execution when the circuit is open.
2. CircuitBreakerMiddleware records the outcome of each execution (RecordFailure / RecordSuccess).
3. After FailureThreshold failures the circuit opens.
4. After OpenDuration elapses, the circuit transitions to half-open and allows one probe.
5. After SuccessThreshold consecutive successes in half-open state, the circuit closes.

Retry

Automatically retries a failed job execution with configurable delay strategies.

Configuration

// Exponential backoff
.WithExponentialRetry(maxAttempts: 3, initialDelay: TimeSpan.FromSeconds(1))

// Linear delay
.WithLinearRetry(maxAttempts: 5, delay: TimeSpan.FromSeconds(10))

// Full configuration
.WithRetry(r => r
    .WithMaxAttempts(4)
    .WithStrategy(RetryStrategy.Exponential)
    .WithInitialDelay(TimeSpan.FromSeconds(2))
    .WithMaxDelay(TimeSpan.FromMinutes(5))
    .WithBackoffMultiplier(2.0)
    .OnlyFor(typeof(HttpRequestException)))

Retry Strategies

Retries stop when the maximum number of attempts is reached, a non-retryable exception is thrown, or the total timeout (if configured) is exceeded.

Timeout

Two independent timeouts can be configured:

Configuration

.WithTimeout(t => t
    .WithOperationTimeout(TimeSpan.FromSeconds(30))
    .WithTotalTimeout(TimeSpan.FromMinutes(2))
    .WithTimeoutException(true))  // throw JobTimeoutException instead of OperationCanceledException

Middleware Execution Order

The pipeline order is fixed by JobManagerResilienceConfigurator:

Request
  └─ CircuitBreakerMiddleware       (1 – blocks if circuit is open)
       └─ TotalTimeoutMiddleware    (2 – deadline for all attempts)
            └─ RetryMiddleware      (3 – retry loop)
                 └─ OperationTimeoutMiddleware  (4 – timeout per attempt)
                      └─ Job TargetMethod

API Reference

Extension Method

IJobDescriptionResilienceBuilder Resilience(this IJobDescriptionBuilder builder)

Services

Features

Conditions

Middleware