LlmKeyPool 0.1.0

LlmKeyPool

A provider-agnostic API key pool for .NET. Rotation, quota-aware cooldown, per-key concurrency limits, health tracking, and state that survives a restart — for applications holding several LLM API keys that need requests to keep flowing when one key is rate limited.

                       ┌──────────────────────────────┐
   request ───────────▶│          IKeyPool            │
                       │                              │
                       │  select ──▶ lease ──▶ report │
                       └───────┬──────────────────────┘
                               │
        ┌──────────┬───────────┼───────────┬──────────┐
        ▼          ▼           ▼           ▼          ▼
     key #1     key #2      key #3      key #4     key #5
    cooling     cooling    available    disabled   available
   RPD · 6h12m  TPM · 18s  in-flight:2    401      in-flight:0

Why not just Polly?

Polly answers "this request failed — how should I retry it?". LlmKeyPool answers "which credential should this request use, and which ones are burnt right now?" They compose; this is not a replacement.

A 429 from an LLM provider is not a generic transient fault. It names a dimension — requests or tokens, per minute or per day — and implies a recovery window that ranges from 20 seconds to the provider's next daily reset. Backing off on the same key is the wrong move: the key is out of budget, not momentarily unlucky.

// What retry-only resilience does with a spent daily quota:
key A → 429 → wait 1s → 429 → wait 2s → 429 → wait 4s → 429 → give up

// What a key pool does:
key A → 429 "requests per day" → cool A until 00:00 Pacific → key B → 200

Install

dotnet add package LlmKeyPool

Targets net8.0 and net9.0. Depends only on Microsoft.Extensions.* abstractions.

Quick start

builder.Services.AddLlmKeyPool("gemini", pool =>
{
    pool.Provider = "gemini";
    pool.AddKeys(builder.Configuration["Gemini:ApiKeys"]!);   // CSV, newlines or semicolons
    pool.FailurePolicy = KeyFailurePolicies.Gemini;
    pool.Selector = KeySelectors.LeastRecentlyUsed;
    pool.MaxConcurrencyPerKey = 4;
});

Then run work through the pool. It picks the key, you make the call, you report what happened:

public sealed class Translator(IKeyPoolProvider pools, HttpClient http)
{
    private readonly IKeyPool _pool = pools.Get("gemini");

    public Task<string> TranslateAsync(string text, CancellationToken ct) =>
        _pool.ExecuteAsync(async (lease, token) =>
        {
            using var request = new HttpRequestMessage(HttpMethod.Post, Endpoint)
            {
                Content = JsonContent.Create(new { text }),
            };
            request.Headers.Add("x-goog-api-key", lease.Key.Value);

            using var response = await http.SendAsync(request, token);

            if (!response.IsSuccessStatusCode)
            {
                var body = await response.Content.ReadAsStringAsync(token);
                lease.ReportFailure((int)response.StatusCode, response.ToHeaderDictionary(), body);
                response.EnsureSuccessStatusCode();   // throw to rotate onto the next key
            }

            var result = await response.Content.ReadFromJsonAsync<Result>(token);
            lease.ReportSuccess(result!.Usage.TotalTokens);
            return result.Text;
        }, cancellationToken: ct);
}

ExecuteAsync rents a key, runs your delegate, classifies any throw through the pool's failure policy, cools the key for the right window, and retries on the next key — up to MaxKeyAttempts.

What the failure policy knows

This is the part a generic resilience library deliberately leaves open.

Built-in profiles: Generic, Gemini, OpenAI. Add your own with QuotaPolicyProfile, or mix providers in one pool with CompositeFailurePolicy.

Selection strategies

The round-robin cursor is reserved with Interlocked.Increment when the order is built, not when a response arrives. Advancing on completion lets every concurrent caller read the same cursor and stampede one key while the rest of the pool sits idle.

Inspecting the pool

app.MapGet("/keypool", (IKeyPoolProvider pools) => pools.Get("gemini").Inspect());

key           status     quota   resumes in   ok/err   tokens
────────────  ─────────  ──────  ───────────  ───────  ──────
AIza~4f2c1d9a cooling    RPD     6h 12m       412/3    1.2M
AIza~9a30bb17 cooling    TPM     18s          388/1    980K
AIza~7c1e0d55 available  —       —            401/0    1.1M
AQ.A~d1b93f60 disabled   Auth    —            0/1      0

KeyState carries no secret material, so it is safe to serialise into a status endpoint. Keys are identified by a fingerprint — a four-character provider prefix plus a hash — and ApiKey.ToString() is masked, so a stray log line cannot leak a credential.

Persistence

A day-class cooldown is worthless if it evaporates on redeploy: the next request goes straight back into an empty budget.

pool.PersistTo(new InMemoryKeyStateStore());   // FileKeyStateStore and Redis land in v0.2/v0.3

Snapshots are keyed by fingerprint and contain no key material, so they are safe to write to a mounted volume or a shared cache. On import, cooldowns merge forward — a restored snapshot can extend a lock but never shorten one.

Testing

Every cooldown computation goes through TimeProvider, so the suite drives the clock:

var time = new FakeTimeProvider(start);
// ... cool a key for 8 hours ...
time.Advance(TimeSpan.FromHours(8));
Assert.Equal(KeyStatus.Available, pool.Inspect()[0].Status);

93 tests covering day-long quota windows and DST transitions run in about 100 ms.

Scope

Out of scope on purpose:

• Sending LLM requests or parsing responses — that belongs to your provider SDK.
• Retry and backoff within one key — Polly owns that and composes cleanly.
• Cost accounting and model routing — the pool reports token counts and stops there.
• Running as a gateway process — this is an in-process library.

A note on quotas

This library is for applications that legitimately hold several credentials: multi-tenant products where each tenant brings a key, separate dev/staging/prod projects, multi-region Azure OpenAI deployments, paid-to-backup failover chains. Whether a particular set of keys may be pooled is governed by each provider's terms. Nothing here makes a quota larger.

License

MIT — see LICENSE.