LowCodeHub.Webhooks 0.0.2

LowCodeHub.Webhooks

A reliable webhook delivery system for ASP.NET Core with database-driven subscriptions, outbox-based at-least-once delivery, two-level resilience (Polly + durable retries), and multi-tenant support. Webhooks that survive crashes, restarts, and extended outages — without losing a single event.

Why This Library?

Installation

dotnet add package LowCodeHub.Webhooks

Quick Start

// Program.cs
builder.Services
    .AddWebhooks(builder.Configuration)
    .AddWebhooksSqlServer(builder.Configuration);     // or AddWebhooksPostgreSql

// Dispatch a webhook from anywhere via DI
public class OrderService(IWebhookDispatcher webhookDispatcher)
{
    public async Task CreateOrderAsync(Order order, CancellationToken ct)
    {
        // ... create order logic ...

        await webhookDispatcher.DispatchAsync(
            eventType: "order.created",
            payload: new { order.Id, order.Total, order.CreatedAt },
            tenantId: order.TenantId,   // optional
            cancellationToken: ct);
    }
}

That's it. The library finds all active subscriptions matching "order.created", snapshots their config, persists delivery attempts to the outbox, and the background worker picks them up — with automatic retries, exponential backoff, and graceful shutdown.

Table of Contents

• How It Works
• Configuration
  • appsettings.json
  • Code-Based Configuration
  • PostgreSQL Configuration
• Dispatching Webhooks
  • With Multi-Tenancy
  • Without Multi-Tenancy
• Two-Level Resilience
• Per-Subscription Proxy
• Subscription Management
• Delivery Lifecycle
• Health Checks
• Graceful Shutdown
• Automatic Cleanup
• Observability
• Database Migrations
  • SQL Server
  • PostgreSQL
• Custom Repository Implementations
• Requirements
• License

How It Works

┌──────────────────────────────────────────────────────────────────┐
│  Your Code                                                       │
│  webhookDispatcher.DispatchAsync("order.created", payload)       │
└───────────────────────────┬──────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│  DefaultWebhookDispatcher                                        │
│  1. Query active subscriptions matching event type + tenant      │
│  2. Snapshot each subscription's config (URL, headers, method)   │
│  3. Persist WebhookDeliveryAttempt records (State = Pending)     │
└───────────────────────────┬──────────────────────────────────────┘
                            │
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│  WebhookDeliveryWorker (BackgroundService, polls every 2s)       │
│  1. Claim batch of pending deliveries (pessimistic lock)         │
│  2. Send HTTP requests via resilience-enhanced HttpClient        │
│     ├── Success (2xx) → MarkDelivered                            │
│     └── Failure → MarkFailed + exponential backoff retry         │
│         └── Max retries exhausted → permanently Failed           │
└──────────────────────────────────────────────────────────────────┘

Key design principle: snapshot-at-dispatch. When a webhook is dispatched, the subscription's URL, headers, HTTP method, and timeout are copied into the delivery attempt. Later subscription changes don't affect in-flight deliveries — ensuring consistency.

Configuration

appsettings.json

{
  "Webhooks": {
    "Enabled": true,
    "BatchSize": 50,
    "MaxParallelism": 4,
    "PollInterval": "00:00:02",
    "InitialRetryDelay": "00:00:05",
    "MaxRetryDelay": "00:05:00",
    "DefaultMaxRetries": 5,
    "DefaultTimeoutSeconds": 30,
    "DrainOnShutdown": true,
    "DrainTimeout": "00:00:30",
    "HealthStaleAfter": "00:02:00",
    "HttpRetryCount": 3,
    "HttpRetryBaseDelay": "00:00:01",
    "HttpTotalTimeout": "00:01:00",
    "CleanupRetentionPeriod": "30.00:00:00",
    "CleanupInterval": "01:00:00",
    "CleanupBatchSize": 1000,
    "SqlServer": {
      "ConnectionString": "Server=...;Database=...;",
      "SubscriptionSchema": "dbo",
      "SubscriptionTable": "WebhookSubscriptions",
      "DeliverySchema": "dbo",
      "DeliveryTable": "WebhookDeliveryAttempts",
      "LeaseDuration": "00:02:00"
    }
  }
}

All Options

Code-Based Configuration

builder.Services.AddWebhooks(options =>
{
    options.BatchSize = 100;
    options.MaxParallelism = 8;
    options.PollInterval = TimeSpan.FromSeconds(1);
    options.HttpRetryCount = 5;
});

builder.Services.AddWebhooksSqlServer(options =>
{
    options.ConnectionString = "Server=...;Database=...;";
    options.SubscriptionTable = "MyWebhookSubscriptions";
});

PostgreSQL Configuration

{
  "Webhooks": {
    "PostgreSql": {
      "ConnectionString": "Host=...;Database=...;",
      "SubscriptionSchema": "public",
      "SubscriptionTable": "webhook_subscriptions",
      "DeliverySchema": "public",
      "DeliveryTable": "webhook_delivery_attempts",
      "LeaseDuration": "00:02:00"
    }
  }
}

builder.Services
    .AddWebhooks(builder.Configuration)
    .AddWebhooksPostgreSql(builder.Configuration);

Dispatching Webhooks

Inject IWebhookDispatcher and call DispatchAsync:

await webhookDispatcher.DispatchAsync(
    eventType: "order.created",
    payload: new { orderId, total, createdAt },
    tenantId: "tenant-123",         // optional
    cancellationToken: ct);

With Multi-Tenancy

Each subscription has an optional TenantId. When you pass a tenantId to DispatchAsync, only subscriptions matching that tenant are triggered:

// Only subscriptions for tenant "acme" listening to "order.created" will receive this
await webhookDispatcher.DispatchAsync("order.created", payload, tenantId: "acme");

Without Multi-Tenancy

Omit tenantId — matches subscriptions with no tenant assigned:

await webhookDispatcher.DispatchAsync("user.registered", new { userId, email });

Two-Level Resilience

Immediate retries handle transient failures (503, network timeouts) within a single HTTP call attempt. Configured via HttpRetryCount, HttpRetryBaseDelay, and HttpTotalTimeout.

Durable retries handle persistent failures. When an HTTP call fails after all immediate retries, the delivery attempt is marked as failed and scheduled for retry with exponential backoff:

Retry 1: +5s     (InitialRetryDelay)
Retry 2: +10s    (×2)
Retry 3: +20s    (×2)
Retry 4: +40s    (×2)
Retry 5: +80s    (×2, capped at MaxRetryDelay)

After MaxRetries is exhausted, the delivery is permanently marked as Failed.

Per-Subscription Proxy

Proxy settings are per-subscription, not global. Each subscription can route through a different proxy — useful for multi-tenant deployments where tenants have their own network egress requirements.

Set ProxyUrl on the subscription record (supports embedded credentials):

-- SQL Server
UPDATE [dbo].[WebhookSubscriptions]
SET [ProxyUrl] = 'http://proxy.tenant-a.example.com:8080'
WHERE [TenantId] = 'tenant-a';

-- With embedded credentials
UPDATE [dbo].[WebhookSubscriptions]
SET [ProxyUrl] = 'http://user:password@proxy.example.com:8080'
WHERE [Id] = '...';

When ProxyUrl is NULL, the library uses a direct connection through the factory-managed HttpClient (with Polly resilience). When set, a standalone HttpClient with the specified WebProxy is created per delivery attempt.

Note: Proxy-routed deliveries bypass the factory-managed resilience pipeline. The per-delivery timeout from TimeoutSeconds still applies.

Subscription Management

Subscriptions are stored in the database. The library reads them — you manage them (insert/update/delete) through your own API or admin UI. Here's the subscription model:

Custom Headers

Headers are stored as a JSON dictionary and added to every delivery request:

{
  "Authorization": "Bearer eyJ...",
  "X-Webhook-Secret": "my-secret-key",
  "X-Custom-Header": "custom-value"
}

Delivery Lifecycle

Each delivery attempt progresses through these states:

Pending → Processing → Delivered
                    └→ Pending (retry scheduled)
                    └→ Failed  (max retries exhausted)

Pessimistic Locking

Workers claim deliveries atomically using database-level locking:

• SQL Server: UPDLOCK, READPAST, ROWLOCK — claimed rows are invisible to other workers
• PostgreSQL: FOR UPDATE SKIP LOCKED — same behavior

A LockedUntilUtc lease prevents stuck deliveries. If a worker crashes mid-delivery, the lease expires (default: 2 minutes) and another worker reclaims the attempt.

Health Checks

Three health checks are registered automatically:

// Map health check endpoints filtered by tag
app.MapHealthChecks("/health/live", new() { Predicate = hc => hc.Tags.Contains("liveness") });
app.MapHealthChecks("/health/ready", new() { Predicate = hc => hc.Tags.Contains("readiness") });

The worker health check reports unhealthy if:

• The worker is not running
• The worker has no heartbeat yet
• The heartbeat is older than HealthStaleAfter (default: 2 minutes)

Graceful Shutdown

When the application is stopping:

1. The worker stops accepting new batches
2. If DrainOnShutdown is true (default), it waits for the current in-flight batch to complete
3. If the drain exceeds DrainTimeout (default: 30s), in-flight deliveries are force-cancelled
4. Unfinished deliveries remain in the database as Processing — their lease expires, and they're reclaimed on the next startup

Automatic Cleanup

The WebhookCleanupWorker runs alongside the delivery worker and periodically deletes old completed records:

Only Delivered and Failed records are deleted — Pending and Processing records are never touched.

Observability

Built-in OpenTelemetry instrumentation under the LowCodeHub.Webhooks source:

Metrics

Traces

// Wire up OpenTelemetry
builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddSource("LowCodeHub.Webhooks"))
    .WithMetrics(m => m.AddMeter("LowCodeHub.Webhooks"));

Database Migrations

SQL Server Migration

CREATE TABLE [dbo].[WebhookSubscriptions]
(
    [Id]              UNIQUEIDENTIFIER NOT NULL PRIMARY KEY DEFAULT NEWID(),
    [TenantId]        NVARCHAR(200)    NULL,
    [EventType]       NVARCHAR(200)    NOT NULL,
    [Name]            NVARCHAR(200)    NOT NULL,
    [WebhookUrl]      NVARCHAR(2000)   NOT NULL,
    [HttpMethod]      NVARCHAR(10)     NOT NULL DEFAULT 'POST',
    [ContentType]     NVARCHAR(100)    NOT NULL DEFAULT 'application/json',
    [Headers]         NVARCHAR(MAX)    NULL,
    [TimeoutSeconds]  INT              NOT NULL DEFAULT 30,
    [MaxRetries]      INT              NOT NULL DEFAULT 5,
    [ProxyUrl]        NVARCHAR(2000)   NULL,
    [IsActive]        BIT              NOT NULL DEFAULT 1,
    [CreatedAtUtc]    DATETIMEOFFSET(7) NOT NULL DEFAULT SYSDATETIMEOFFSET(),
    [UpdatedAtUtc]    DATETIMEOFFSET(7) NULL
);

CREATE INDEX [IX_WebhookSubscriptions_EventType_IsActive]
    ON [dbo].[WebhookSubscriptions] ([EventType], [IsActive])
    INCLUDE ([TenantId]);

CREATE TABLE [dbo].[WebhookDeliveryAttempts]
(
    [Id]                 UNIQUEIDENTIFIER NOT NULL PRIMARY KEY DEFAULT NEWID(),
    [SubscriptionId]     UNIQUEIDENTIFIER NOT NULL,
    [TenantId]           NVARCHAR(200)    NULL,
    [EventType]          NVARCHAR(200)    NOT NULL,
    [WebhookUrl]         NVARCHAR(2000)   NOT NULL,
    [HttpMethod]         NVARCHAR(10)     NOT NULL,
    [ContentType]        NVARCHAR(100)    NOT NULL,
    [Headers]            NVARCHAR(MAX)    NULL,
    [TimeoutSeconds]     INT              NOT NULL DEFAULT 30,
    [ProxyUrl]           NVARCHAR(2000)   NULL,
    [Payload]            NVARCHAR(MAX)    NOT NULL,
    [State]              INT              NOT NULL DEFAULT 0,
    [RetryCount]         INT              NOT NULL DEFAULT 0,
    [MaxRetries]         INT              NOT NULL DEFAULT 5,
    [NextAttemptAtUtc]   DATETIMEOFFSET(7) NULL,
    [LastError]          NVARCHAR(MAX)    NULL,
    [ResponseStatusCode] INT             NULL,
    [CreatedAtUtc]       DATETIMEOFFSET(7) NOT NULL DEFAULT SYSDATETIMEOFFSET(),
    [DeliveredAtUtc]     DATETIMEOFFSET(7) NULL,
    [LockedUntilUtc]     DATETIMEOFFSET(7) NULL
);

CREATE INDEX [IX_WebhookDeliveryAttempts_State_NextAttempt]
    ON [dbo].[WebhookDeliveryAttempts] ([State], [NextAttemptAtUtc])
    INCLUDE ([LockedUntilUtc], [CreatedAtUtc]);

PostgreSQL Migration

CREATE TABLE "public"."webhook_subscriptions"
(
    id               UUID         NOT NULL PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id        VARCHAR(200) NULL,
    event_type       VARCHAR(200) NOT NULL,
    name             VARCHAR(200) NOT NULL,
    webhook_url      VARCHAR(2000) NOT NULL,
    http_method      VARCHAR(10)  NOT NULL DEFAULT 'POST',
    content_type     VARCHAR(100) NOT NULL DEFAULT 'application/json',
    headers          JSONB        NULL,
    timeout_seconds  INT          NOT NULL DEFAULT 30,
    max_retries      INT          NOT NULL DEFAULT 5,
    proxy_url        VARCHAR(2000) NULL,
    is_active        BOOLEAN      NOT NULL DEFAULT TRUE,
    created_at_utc   TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    updated_at_utc   TIMESTAMPTZ  NULL
);

CREATE INDEX ix_webhook_subscriptions_event_type_active
    ON "public"."webhook_subscriptions" (event_type, is_active)
    INCLUDE (tenant_id);

CREATE TABLE "public"."webhook_delivery_attempts"
(
    id                   UUID         NOT NULL PRIMARY KEY DEFAULT gen_random_uuid(),
    subscription_id      UUID         NOT NULL,
    tenant_id            VARCHAR(200) NULL,
    event_type           VARCHAR(200) NOT NULL,
    webhook_url          VARCHAR(2000) NOT NULL,
    http_method          VARCHAR(10)  NOT NULL,
    content_type         VARCHAR(100) NOT NULL,
    headers              JSONB        NULL,
    timeout_seconds      INT          NOT NULL DEFAULT 30,
    proxy_url            VARCHAR(2000) NULL,
    payload              TEXT         NOT NULL,
    state                INT          NOT NULL DEFAULT 0,
    retry_count          INT          NOT NULL DEFAULT 0,
    max_retries          INT          NOT NULL DEFAULT 5,
    next_attempt_at_utc  TIMESTAMPTZ  NULL,
    last_error           TEXT         NULL,
    response_status_code INT          NULL,
    created_at_utc       TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    delivered_at_utc     TIMESTAMPTZ  NULL,
    locked_until_utc     TIMESTAMPTZ  NULL
);

CREATE INDEX ix_webhook_delivery_attempts_state_next_attempt
    ON "public"."webhook_delivery_attempts" (state, next_attempt_at_utc)
    INCLUDE (locked_until_utc, created_at_utc);

Custom Repository Implementations

The library uses two repository interfaces. Implement them to use a different database:

public interface IWebhookSubscriptionRepository
{
    Task<IReadOnlyList<WebhookSubscription>> GetActiveByEventTypeAsync(
        string eventType, string? tenantId = null, CancellationToken ct = default);
}

public interface IWebhookDeliveryRepository
{
    Task AddAsync(WebhookDeliveryAttempt delivery, CancellationToken ct = default);

    Task<IReadOnlyList<WebhookDeliveryAttempt>> ClaimPendingAsync(
        int batchSize, DateTimeOffset utcNow, CancellationToken ct = default);

    Task MarkDeliveredAsync(Guid id, int statusCode, DateTimeOffset deliveredAtUtc, CancellationToken ct = default);

    Task MarkFailedAsync(Guid id, string error, int? statusCode, int retryCount,
        DateTimeOffset? nextAttemptAtUtc, CancellationToken ct = default);

    Task<int> DeleteCompletedBeforeAsync(DateTimeOffset cutoffUtc, int batchSize, CancellationToken ct = default);
}

Register your implementations before calling AddWebhooks:

builder.Services.AddScoped<IWebhookSubscriptionRepository, MyMongoWebhookSubscriptionRepo>();
builder.Services.AddScoped<IWebhookDeliveryRepository, MyMongoWebhookDeliveryRepo>();
builder.Services.AddWebhooks(builder.Configuration);
// No need to call AddWebhooksSqlServer/AddWebhooksPostgreSql

The library uses TryAddScoped, so your registrations take precedence.

Requirements

• .NET 10 or later
• SQL Server (via Microsoft.Data.SqlClient) or PostgreSQL (via Npgsql) — or your own repository implementation
• Microsoft.Extensions.Http.Resilience 10.1+ (included as a dependency)

License

MIT © Ahmed Abuelnour