Gizza.Retention.Elasticsearch 1.2.1

Gizza.Retention.Elasticsearch

Policy-driven Elasticsearch retention for .NET hosted services.

Türkçe dokümantasyon

The package supports two independent retention methods:

• DeleteByQuery (the backward-compatible default) runs bounded, scheduled _count and _delete_by_query operations against indices, aliases, or data streams.
• DataStreamIlm manages an Elasticsearch ILM policy and attaches it to exact data streams so Elasticsearch can roll over and delete whole backing indices.

Both methods support dry-run validation, protected targets, retry controls, execution locks, and process-local observable state. A target cannot be owned by both methods.

Supported runtimes and Elasticsearch

• .NET 8, .NET 9, and .NET 10
• Elasticsearch 9.3 is covered by the opt-in live contract test
• The package uses modern Elasticsearch _count, _delete_by_query, ILM, data stream, and case-insensitive term query APIs. Validate the exact Elasticsearch version used by your application before enabling deletion.

Installation

dotnet add package Gizza.Retention.Elasticsearch --version 1.2.0

Registration

using Gizza.Retention.Elasticsearch;

builder.Services.AddElasticsearchRetention(builder.Configuration);

Registration validates the ElasticsearchRetention configuration during host startup and starts one independent scheduler loop for every enabled job.

Configuration example

{
  "ElasticsearchRetention": {
    "LevelNormalization": {
      "CaseInsensitive": true,
      "Mappings": [
        { "To": "Trace", "From": [ "Trace", "Verbose" ] },
        { "To": "Debug", "From": [ "Debug" ] },
        { "To": "Information", "From": [ "Information", "Informational", "Info" ] },
        { "To": "Warning", "From": [ "Warning", "Warn" ] },
        { "To": "Error", "From": [ "Error" ] },
        { "To": "Critical", "From": [ "Critical", "Fatal" ] }
      ]
    },
    "Connections": {
      "Default": {
        "Node": "https://elasticsearch.example.com:9200",
        "UsernameSecretRef": "ELASTIC_RETENTION_USERNAME",
        "PasswordSecretRef": "ELASTIC_RETENTION_PASSWORD"
      }
    },
    "Policies": {
      "ServiceLogs": {
        "TimestampField": "@timestamp",
        "LevelFields": [ "log.level" ],
        "DefaultLevelBehavior": "Keep",
        "CaseInsensitiveLevels": true,
        "Levels": {
          "Trace": "10d",
          "Debug": "10d",
          "Information": "30d",
          "Warning": "30d",
          "Error": "keep",
          "Critical": "keep"
        }
      }
    },
    "Jobs": [
      {
        "Name": "service-logs",
        "Enabled": true,
        "Connection": "Default",
        "Policy": "ServiceLogs",
        "DeletionMethod": "DeleteByQuery",
        "Cron": "15 3 * * *",
        "TimeZone": "UTC",
        "RunOnStartup": false,
        "DryRun": true,
        "TargetKind": "DataStream",
        "Targets": [ "logs-default" ],
        "ProtectedTargets": [ ".*", "kibana*", "fleet*" ],
        "IgnoreMissingTargets": true,
        "ContinueOnError": true,
        "DeleteBatchSize": 1000,
        "MaxDocumentsPerBatch": 5000,
        "MaxDocumentsPerRun": 50000,
        "MaxRunDurationSeconds": 900,
        "RequestsPerSecond": 100,
        "RequestTimeoutSeconds": 300,
        "Retry": {
          "MaxAttempts": 4,
          "InitialDelayMilliseconds": 500,
          "MaxDelayMilliseconds": 5000
        },
        "ExecutionLockEnabled": true
      }
    ]
  }
}

Data stream ILM example

Use ILM mode when a data stream contains one retention class and volume makes document-by-document deletion impractical:

{
  "ElasticsearchRetention": {
    "Connections": {
      "Default": {
        "Node": "https://elasticsearch.example.com:9200",
        "UsernameSecretRef": "ELASTIC_RETENTION_USERNAME",
        "PasswordSecretRef": "ELASTIC_RETENTION_PASSWORD"
      }
    },
    "Jobs": [
      {
        "Name": "trace-debug-streams-10d",
        "Connection": "Default",
        "DeletionMethod": "DataStreamIlm",
        "DataStreamIlm": {
          "PolicyName": "logs-trace-debug-10d",
          "RolloverMaxAge": "1d",
          "RolloverMaxPrimaryShardSize": "25gb",
          "Retention": "10d"
        },
        "Cron": "*/5 * * * *",
        "TimeZone": "UTC",
        "RunOnStartup": true,
        "DryRun": true,
        "TargetKind": "DataStream",
        "Targets": [ "logs-trace", "logs-debug" ],
        "ProtectedTargets": [ ".*", "kibana*", "fleet*" ],
        "IgnoreMissingTargets": true,
        "ContinueOnError": true,
        "MaxRunDurationSeconds": 900,
        "RequestTimeoutSeconds": 300,
        "Retry": {
          "MaxAttempts": 4,
          "InitialDelayMilliseconds": 500,
          "MaxDelayMilliseconds": 5000
        },
        "ExecutionLockEnabled": true
      }
    ]
  }
}

Set ILM Retention to keep to keep rolled-over backing indices indefinitely while still applying rollover. A non-dry-run job creates or updates its ILM policy before probing targets, so the policy exists even when the writer has not created a data stream yet. The job does not create a missing data stream because its mapping and index template belong to the writer. It attaches the policy on the first scheduled reconciliation after the stream appears; use a short, idempotent reconciliation schedule such as every five minutes. DryRun=true probes exact targets but does not create or update ILM policies or data stream settings.

ILM delete age is measured from rollover. Consequently, the oldest document can outlive the configured retention by up to the rollover window. This is the cost of efficient whole-index deletion and must be included in compliance calculations.

Durations accept s, m, h, d, and w suffixes, for example 900s, 12h, 10d, or 2w. Use keep to retain a level indefinitely. Zero and negative durations are rejected.

DefaultLevelBehavior=Keep preserves documents whose level is unknown or missing. DeleteUsingShortestRetention applies the shortest finite policy duration to those documents.

Configuration validation fails when credentials referenced by UsernameSecretRef or PasswordSecretRef are missing. Inline Username and Password values are also supported, but a dedicated secret source is recommended.

Safe rollout

1. Start with DryRun=true.
2. Verify targets, cutoff timestamps, matched document counts, and level mappings.
3. Use a dedicated Elasticsearch principal limited to the intended targets.
4. Confirm MaxDocumentsPerRun, duration, timeout, and throttle limits against cluster capacity.
5. Enable real deletion for one environment first and monitor multiple scheduled executions.

The engine treats Elasticsearch failures, version conflicts, timeouts, malformed responses, and non-success HTTP responses as unsuccessful operations. Probe and count requests use bounded retry with exponential backoff. Delete requests are not automatically retried because their server-side completion is unknown after a transport failure.

Execution locking

The default lock prevents overlapping executions only inside one process. Applications running multiple retention replicas must register an IElasticsearchRetentionExecutionLockProvider implementation backed by a distributed lock before calling AddElasticsearchRetention.

using Gizza.Retention.Elasticsearch.Execution;

builder.Services.AddSingleton<IElasticsearchRetentionExecutionLockProvider, MyDistributedLockProvider>();
builder.Services.AddElasticsearchRetention(builder.Configuration);

The registration uses TryAdd, so the application-owned provider remains authoritative.

Observability

IElasticsearchRetentionState exposes the latest process-local snapshot for each job:

using Gizza.Retention.Elasticsearch.Execution;

app.MapGet("/retention/jobs", (IElasticsearchRetentionState state) => state.GetJobs());

Snapshots include start, completion and last-success timestamps, status, matched/deleted counts, unmanaged data stream warnings, and the latest error. This is runtime state, not a durable audit log.

Elasticsearch permissions

Use a separate least-privilege principal. DeleteByQuery needs target-specific permissions to probe targets, execute _count, and execute _delete_by_query. DataStreamIlm needs data stream probe access, the manage_ilm cluster privilege, and manage on its exact target streams so it can update their settings. Do not reuse an administrative or log-writer credential.

Method ownership boundary

Do not point DeleteByQuery and DataStreamIlm jobs at the same data. Startup validation rejects explicit or wildcard target overlap between enabled jobs using different methods. This check cannot discover that differently named aliases resolve to the same backing data, so alias ownership remains the application's responsibility. Two enabled DataStreamIlm jobs also cannot own the same target, and a shared ILM policy name cannot carry conflicting settings. An indefinite keep policy can still produce unbounded storage growth.

Live contract test

The live Elasticsearch test is skipped unless GIZZA_RETENTION_ELASTICSEARCH_URL is set. Optional Basic authentication uses GIZZA_RETENTION_ELASTICSEARCH_USERNAME and GIZZA_RETENTION_ELASTICSEARCH_PASSWORD. The test creates uniquely named index, alias, and data stream resources; verifies actual mixed-case expired-level deletion through every target kind; and removes all resources in finally blocks.

dotnet test tests/Gizza.Retention.Elasticsearch.UnitTests/Gizza.Retention.Elasticsearch.UnitTests.csproj --filter "Category=ExternalIntegration"

License

MIT