LowCodeHub.Webhooks 0.0.6

LowCodeHub.Webhooks

A reliable webhook delivery system for ASP.NET Core with database-driven subscriptions, outbox-based at-least-once delivery, two-level resilience (Polly + durable retries), and multi-tenant support. Webhooks that survive crashes, restarts, and extended outages — without losing a single event.

Why This Library?

Feature	LowCodeHub.Webhooks	Fire-and-Forget HTTP	Queue-Based
Delivery guarantee	At-least-once (outbox)	None	At-least-once
Survives crashes	Yes — persisted before sending	No	Depends on broker
Retry strategy	Two-level — immediate + durable backoff	Manual	Single-level
Subscription storage	Database-driven — dynamic CRUD	Hardcoded	Hardcoded
Multi-tenant	Built-in	Manual	Manual
Infrastructure	Your existing DB	None	Requires message broker
Observability	OpenTelemetry metrics + traces	Manual	Varies

Installation

dotnet add package LowCodeHub.Webhooks

Quick Start

// Program.cs
builder.Services
    .AddWebhooks(builder.Configuration, typeof(Program).Assembly)  // scans assembly for handlers
    .AddWebhooksSqlServer(builder.Configuration);     // or AddWebhooksPostgreSql

// Dispatch a webhook from anywhere via DI
public class OrderService(IWebhookDispatcher webhookDispatcher)
{
    public async Task CreateOrderAsync(Order order, CancellationToken ct)
    {
        // ... create order logic ...

        await webhookDispatcher.DispatchAsync(
            eventType: "order.created",
            payload: new { order.Id, order.Total, order.CreatedAt },
            tenantId: order.TenantId,   // optional
            cancellationToken: ct);
    }
}

That's it. The library finds all active subscriptions matching "order.created", snapshots their config, persists delivery attempts to the outbox, and the background worker picks them up — with automatic retries, exponential backoff, and graceful shutdown.

---

Table of Contents

• How It Works
• Configuration
  • appsettings.json
  • Code-Based Configuration
  • PostgreSQL Configuration
• Dispatching Webhooks
  • With Multi-Tenancy
  • Without Multi-Tenancy
• Two-Level Resilience
• Per-Subscription Proxy
• Subscription Management
• Delivery Lifecycle
• Health Checks
• Graceful Shutdown
• Automatic Cleanup
• Observability
• Database Migrations
  • With LowCodeHub.Migration.SqlServer
  • With LowCodeHub.Migration.PostgreSql
  • With EF Core Migrations
• Delivery Handlers
• Custom Repository Implementations
• Requirements
• License

---

How It Works

┌──────────────────────────────────────────────────────────────────┐
│  Your Code                                                       │
│  webhookDispatcher.DispatchAsync("order.created", payload)       │
└───────────────────────────┬──────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│  DefaultWebhookDispatcher                                        │
│  1. Query active subscriptions matching event type + tenant      │
│  2. Snapshot each subscription's config (URL, headers, method)   │
│  3. Persist WebhookDeliveryAttempt records (State = Pending)     │
└───────────────────────────┬──────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│  WebhookDeliveryWorker (BackgroundService, polls every 2s)       │
│  1. Claim batch of pending deliveries (pessimistic lock)         │
│  2. Send HTTP requests via resilience-enhanced HttpClient        │
│     ├── Success (2xx) → MarkDelivered                            │
│     └── Failure → MarkFailed + exponential backoff retry         │
│         └── Max retries exhausted → permanently Failed           │
└──────────────────────────────────────────────────────────────────┘

Key design principle: snapshot-at-dispatch. When a webhook is dispatched, the subscription's URL, headers, HTTP method, and timeout are copied into the delivery attempt. Later subscription changes don't affect in-flight deliveries — ensuring consistency.

---

Configuration

appsettings.json

{
  "Webhooks": {
    "Enabled": true,
    "BatchSize": 50,
    "MaxParallelism": 4,
    "PollInterval": "00:00:02",
    "InitialRetryDelay": "00:00:05",
    "MaxRetryDelay": "00:05:00",
    "DefaultMaxRetries": 5,
    "DefaultTimeoutSeconds": 30,
    "DrainOnShutdown": true,
    "DrainTimeout": "00:00:30",
    "HealthStaleAfter": "00:02:00",
    "HttpRetryCount": 3,
    "HttpRetryBaseDelay": "00:00:01",
    "HttpTotalTimeout": "00:01:00",
    "CleanupRetentionPeriod": "30.00:00:00",
    "CleanupInterval": "01:00:00",
    "CleanupBatchSize": 1000,
    "SqlServer": {
      "ConnectionString": "Server=...;Database=...;",
      "SubscriptionSchema": "dbo",
      "SubscriptionTable": "WebhookSubscriptions",
      "DeliverySchema": "dbo",
      "DeliveryTable": "WebhookDeliveryAttempts",
      "LeaseDuration": "00:02:00"
    }
  }
}

All Options

Option	Default	Description
Enabled	true	Enable/disable the delivery worker
BatchSize	50	Delivery attempts claimed per poll cycle
MaxParallelism	4	Concurrent HTTP calls per batch
PollInterval	2s	Time between polling cycles
InitialRetryDelay	5s	Delay before first durable retry
MaxRetryDelay	5min	Maximum delay between durable retries
DefaultMaxRetries	5	Default max retries for new deliveries
DefaultTimeoutSeconds	30	Default HTTP timeout for new subscriptions
DrainOnShutdown	true	Wait for in-flight deliveries on shutdown
DrainTimeout	30s	Maximum drain wait time
HealthStaleAfter	2min	Worker heartbeat staleness threshold
HttpRetryCount	3	Immediate Polly retries per HTTP call
HttpRetryBaseDelay	1s	Base delay for immediate retries
HttpTotalTimeout	60s	Total timeout including all immediate retries
CleanupRetentionPeriod	30 days	How long to keep completed records (null to disable)
CleanupInterval	1 hour	Time between cleanup cycles
CleanupBatchSize	1000	Max records deleted per cleanup cycle

Code-Based Configuration

builder.Services.AddWebhooks(options =>
{
    options.BatchSize = 100;
    options.MaxParallelism = 8;
    options.PollInterval = TimeSpan.FromSeconds(1);
    options.HttpRetryCount = 5;
}, typeof(Program).Assembly);  // optional: scan for delivery handlers

builder.Services.AddWebhooksSqlServer(options =>
{
    options.ConnectionString = "Server=...;Database=...;";
    options.SubscriptionTable = "MyWebhookSubscriptions";
});

PostgreSQL Configuration

{
  "Webhooks": {
    "PostgreSql": {
      "ConnectionString": "Host=...;Database=...;",
      "SubscriptionSchema": "public",
      "SubscriptionTable": "webhook_subscriptions",
      "DeliverySchema": "public",
      "DeliveryTable": "webhook_delivery_attempts",
      "LeaseDuration": "00:02:00"
    }
  }
}

builder.Services
    .AddWebhooks(builder.Configuration)
    .AddWebhooksPostgreSql(builder.Configuration);

---

Dispatching Webhooks

Inject IWebhookDispatcher and call DispatchAsync:

await webhookDispatcher.DispatchAsync(
    eventType: "order.created",
    payload: new { orderId, total, createdAt },
    tenantId: "tenant-123",         // optional
    cancellationToken: ct);

With Multi-Tenancy

Each subscription has an optional TenantId. When you pass a tenantId to DispatchAsync, only subscriptions matching that tenant are triggered:

// Only subscriptions for tenant "acme" listening to "order.created" will receive this
await webhookDispatcher.DispatchAsync("order.created", payload, tenantId: "acme");

Without Multi-Tenancy

Omit tenantId — matches subscriptions with no tenant assigned:

await webhookDispatcher.DispatchAsync("user.registered", new { userId, email });

---

Two-Level Resilience

Level	Mechanism	Scope	Typical Duration
Immediate	Polly v8 (via Microsoft.Extensions.Http.Resilience)	Per HTTP call	Seconds — retry transient 5xx, network blips
Durable	Outbox worker + exponential backoff	Across process restarts	Minutes to hours — extended outages

Immediate retries handle transient failures (503, network timeouts) within a single HTTP call attempt. Configured via HttpRetryCount, HttpRetryBaseDelay, and HttpTotalTimeout.

Durable retries handle persistent failures. When an HTTP call fails after all immediate retries, the delivery attempt is marked as failed and scheduled for retry with exponential backoff:

Retry 1: +5s     (InitialRetryDelay)
Retry 2: +10s    (×2)
Retry 3: +20s    (×2)
Retry 4: +40s    (×2)
Retry 5: +80s    (×2, capped at MaxRetryDelay)

After MaxRetries is exhausted, the delivery is permanently marked as Failed.

---

Per-Subscription Proxy

Proxy settings are per-subscription, not global. Each subscription can route through a different proxy — useful for multi-tenant deployments where tenants have their own network egress requirements.

Set ProxyUrl on the subscription record (supports embedded credentials):

-- SQL Server
UPDATE [dbo].[WebhookSubscriptions]
SET [ProxyUrl] = 'http://proxy.tenant-a.example.com:8080'
WHERE [TenantId] = 'tenant-a';

-- With embedded credentials
UPDATE [dbo].[WebhookSubscriptions]
SET [ProxyUrl] = 'http://user:password@proxy.example.com:8080'
WHERE [Id] = '...';

When ProxyUrl is NULL, the library uses a direct connection through the factory-managed HttpClient (with Polly resilience). When set, a standalone HttpClient with the specified WebProxy is created per delivery attempt.

Note: Proxy-routed deliveries bypass the factory-managed resilience pipeline. The per-delivery timeout from TimeoutSeconds still applies.

---

Subscription Management

Subscriptions are stored in the database. The library reads them — you manage them (insert/update/delete) through your own API or admin UI. Here's the subscription model:

Field	Type	Default	Description
Id	Guid	NewGuid()	Unique identifier
TenantId	string?	null	Optional tenant for multi-tenant filtering
EventType	string	required	Event type to listen for (e.g. "order.created")
Name	string	required	Human-readable name
WebhookUrl	string	required	Target URL for delivery
HttpMethod	string	"POST"	HTTP method (POST, PUT, PATCH, etc.)
ContentType	string	"application/json"	Request body content type
Headers	string?	null	Custom headers as JSON dictionary
TimeoutSeconds	int	30	Per-request timeout
MaxRetries	int	5	Maximum durable retry attempts
ProxyUrl	string?	null	Proxy URL for outbound HTTP calls (supports embedded credentials)
IsActive	bool	true	Whether this subscription receives events

Custom Headers

Headers are stored as a JSON dictionary and added to every delivery request:

{
  "Authorization": "Bearer eyJ...",
  "X-Webhook-Secret": "my-secret-key",
  "X-Custom-Header": "custom-value"
}

---

Delivery Lifecycle

Each delivery attempt progresses through these states:

Pending → Processing → Delivered
                    └→ Pending (retry scheduled)
                    └→ Failed  (max retries exhausted)

State	Description
Pending	Waiting to be picked up by the delivery worker
Processing	Claimed by a worker, currently being sent (locked via lease)
Delivered	Successfully delivered (2xx response received)
Failed	Permanently failed after exhausting all retries

Pessimistic Locking

Workers claim deliveries atomically using database-level locking:

• SQL Server: UPDLOCK, READPAST, ROWLOCK — claimed rows are invisible to other workers
• PostgreSQL: FOR UPDATE SKIP LOCKED — same behavior

A LockedUntilUtc lease prevents stuck deliveries. If a worker crashes mid-delivery, the lease expires (default: 2 minutes) and another worker reclaims the attempt.

---

Health Checks

Three health checks are registered automatically:

Check	Tag	What It Monitors
webhook-delivery-worker	webhooks, liveness	Worker running + heartbeat freshness
webhooks-sqlserver	webhooks, sqlserver, readiness	SQL Server connectivity
webhooks-postgresql	webhooks, postgresql, readiness	PostgreSQL connectivity

// Map health check endpoints filtered by tag
app.MapHealthChecks("/health/live", new() { Predicate = hc => hc.Tags.Contains("liveness") });
app.MapHealthChecks("/health/ready", new() { Predicate = hc => hc.Tags.Contains("readiness") });

The worker health check reports unhealthy if:

• The worker is not running
• The worker has no heartbeat yet
• The heartbeat is older than HealthStaleAfter (default: 2 minutes)

---

Graceful Shutdown

When the application is stopping:

1. The worker stops accepting new batches
2. If DrainOnShutdown is true (default), it waits for the current in-flight batch to complete
3. If the drain exceeds DrainTimeout (default: 30s), in-flight deliveries are force-cancelled
4. Unfinished deliveries remain in the database as Processing — their lease expires, and they're reclaimed on the next startup

---

Automatic Cleanup

The WebhookCleanupWorker runs alongside the delivery worker and periodically deletes old completed records:

Option	Default	Description
CleanupRetentionPeriod	30 days	Records older than this are deleted. Set to null to disable.
CleanupInterval	1 hour	How often the cleanup runs
CleanupBatchSize	1000	Max records deleted per cycle

Only Delivered and Failed records are deleted — Pending and Processing records are never touched.

---

Observability

Built-in OpenTelemetry instrumentation under the LowCodeHub.Webhooks source:

Metrics

Metric	Type	Description
webhooks.deliveries.dispatched	Counter	Delivery attempts queued
webhooks.deliveries.succeeded	Counter	Successful deliveries (2xx)
webhooks.deliveries.failed	Counter	Failed deliveries (will retry or permanent)
webhooks.deliveries.permanently_failed	Counter	Deliveries that exhausted all retries
webhooks.deliveries.retried	Counter	Deliveries scheduled for retry
webhooks.deliveries.cleaned	Counter	Old records removed by cleanup
webhooks.worker.batches_claimed	Counter	Batches claimed by the worker
webhooks.dispatch.no_subscriptions	Counter	Dispatches with no matching subscriptions
webhooks.delivery.duration	Histogram (ms)	HTTP call duration
webhooks.worker.batch_size	Histogram	Deliveries per batch

Traces

Activity	Kind	Description
webhooks.dispatch	Producer	Dispatching an event to subscriptions
webhooks.deliver	Client	Sending a single HTTP delivery
webhooks.cleanup	Internal	Cleanup cycle

// Wire up OpenTelemetry
builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddSource("LowCodeHub.Webhooks"))
    .WithMetrics(m => m.AddMeter("LowCodeHub.Webhooks"));

---

Database Migrations

LowCodeHub.Webhooks does not create or migrate its own database schema during service registration. The schema scripts ship as embedded resources in the package, but the consuming application owns when and how they are applied.

Embedded resource prefixes:

Provider	Embedded resource prefix
SQL Server	LowCodeHub.Webhooks.Repositories.SqlServer.Scripts.
PostgreSQL	LowCodeHub.Webhooks.Repositories.PostgreSql.Scripts.

The scripts create webhook subscriptions, delivery attempts, and the required dispatch indexes.

With LowCodeHub.Migration.SqlServer

using LowCodeHub.Migration.SqlServer.Extensions;
using LowCodeHub.Webhooks.Options;

builder.Services.AddMigration(o =>
{
    o.ConnectionString = builder.Configuration.GetConnectionString("Webhooks")!;
    o.Directories = ["LowCodeHub.Webhooks.Repositories.SqlServer.Scripts."];
});

await app.RunDatabaseMigrationAsync<SqlServerWebhookOptions>();

With LowCodeHub.Migration.PostgreSql

using LowCodeHub.Migration.PostgreSql.Extensions;
using LowCodeHub.Webhooks.Options;

builder.Services.AddMigration(o =>
{
    o.ConnectionString = builder.Configuration.GetConnectionString("Webhooks")!;
    o.Directories = ["LowCodeHub.Webhooks.Repositories.PostgreSql.Scripts."];
});

await app.RunDatabaseMigrationAsync<PostgreSqlWebhookOptions>();

Do not scan the whole LowCodeHub.Webhooks assembly without a directory filter, because the package contains scripts for both providers.

With EF Core Migrations

EF Core will not discover these embedded scripts automatically. Create an application-owned EF migration and execute the embedded scripts from Up.

using LowCodeHub.Webhooks.Migrations;
using Microsoft.EntityFrameworkCore.Migrations;

public partial class AddLowCodeHubWebhooksSchema : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        foreach (var resource in SqlWebhooks.SqlServerResources)
        {
            migrationBuilder.Sql(SqlWebhooks.ReadFromResource(resource));
        }
    }

    protected override void Down(MigrationBuilder migrationBuilder)
    {
        // Drop Webhooks objects here if your application's migration policy requires reversible migrations.
    }
}

For PostgreSQL, loop over SqlWebhooks.PostgreSqlResources instead.

---

Delivery Handlers

React to webhook delivery results with strongly-typed handlers. Each handler targets a specific event type and receives the remote API response deserialized into your model.

1. Define a handler

Extend WebhookDeliveryHandler<TMessage> where TMessage is the type you expect from the remote API's success response body:

public class OrderCreatedHandler(
    IOrderService orderService,
    ILogger<OrderCreatedHandler> logger) : WebhookDeliveryHandler<OrderApiResponse>
{
    public override string EventType => "order.created";

    public override async Task OnSuccessAsync(
        WebhookDeliveryAttempt delivery,
        OrderApiResponse? message,
        CancellationToken cancellationToken = default)
    {
        if (message is not null)
        {
            await orderService.MarkNotifiedAsync(message.OrderId, cancellationToken);
        }
    }

    // Optional — override only if you need to handle failures
    public override Task OnFailureAsync(
        WebhookFailureContext failure,
        CancellationToken cancellationToken = default)
    {
        if (failure.PermanentlyFailed)
        {
            logger.LogCritical(
                "Webhook permanently failed for {Url}, last error: {Error}",
                failure.Delivery.WebhookUrl, failure.Error);
        }

        return Task.CompletedTask;
    }
}

2. Register via assembly scanning

Pass the assembly containing your handlers to AddWebhooks — all WebhookDeliveryHandler<T> implementations are discovered and registered automatically:

builder.Services
    .AddWebhooks(builder.Configuration, typeof(Program).Assembly)
    .AddWebhooksSqlServer(builder.Configuration);

You can pass multiple assemblies:

builder.Services.AddWebhooks(builder.Configuration,
    typeof(Program).Assembly,
    typeof(SharedHandlers).Assembly);

If no assemblies are provided, no handlers are registered and the system works exactly as before.

API

OnSuccessAsync(delivery, message) — called on 2xx responses:

Parameter	Type	Description
delivery	WebhookDeliveryAttempt	Full delivery entity (event type, URL, payload, tenant, retry count, etc.)
message	TMessage?	API response body deserialized into your type. Null if deserialization failed or no body.

OnFailureAsync(failure) — called on failures (optional override, default is no-op):

Property	Type	Description
failure.Delivery	WebhookDeliveryAttempt	Full delivery entity.
failure.StatusCode	int?	HTTP status code. Null on timeout/network error.
failure.Error	string	Error description.
failure.ResponseBody	string?	Raw response body string.
failure.PermanentlyFailed	bool	True when all retries are exhausted.

How it works

• Handlers are scoped DI services — inject any service you need (DbContext, HttpClient, etc.)
• Only handlers whose EventType matches the delivery's event type are invoked (case-insensitive)
• Handlers run after the delivery result is persisted to the database
• A failing handler is caught and logged — it never breaks the delivery pipeline or other handlers
• Multiple handlers can listen to the same event type

---

Custom Repository Implementations

The library uses two repository interfaces. Implement them to use a different database:

public interface IWebhookSubscriptionRepository
{
    Task<IReadOnlyList<WebhookSubscription>> GetActiveByEventTypeAsync(
        string eventType, string? tenantId = null, CancellationToken ct = default);
}

public interface IWebhookDeliveryRepository
{
    Task AddAsync(WebhookDeliveryAttempt delivery, CancellationToken ct = default);

    Task<IReadOnlyList<WebhookDeliveryAttempt>> ClaimPendingAsync(
        int batchSize, DateTimeOffset utcNow, CancellationToken ct = default);

    Task MarkDeliveredAsync(Guid id, int statusCode, DateTimeOffset deliveredAtUtc, CancellationToken ct = default);

    Task MarkFailedAsync(Guid id, string error, int? statusCode, int retryCount,
        DateTimeOffset? nextAttemptAtUtc, CancellationToken ct = default);

    Task<int> DeleteCompletedBeforeAsync(DateTimeOffset cutoffUtc, int batchSize, CancellationToken ct = default);
}

Register your implementations before calling AddWebhooks:

builder.Services.AddScoped<IWebhookSubscriptionRepository, MyMongoWebhookSubscriptionRepo>();
builder.Services.AddScoped<IWebhookDeliveryRepository, MyMongoWebhookDeliveryRepo>();
builder.Services.AddWebhooks(builder.Configuration);
// No need to call AddWebhooksSqlServer/AddWebhooksPostgreSql

The library uses TryAddScoped, so your registrations take precedence.

---

Requirements

• .NET 10 or later
• SQL Server (via Microsoft.Data.SqlClient) or PostgreSQL (via Npgsql) — or your own repository implementation
• Microsoft.Extensions.Http.Resilience 10.1+ (included as a dependency)

---

License

MIT © Ahmed Abuelnour