Batchwatch 0.2.2

Batchwatch - .NET client

Client for batchwatch.dev: crowdsourced measurement of queue time on LLM batch APIs.

Batch endpoints cost 50% of the synchronous ones, but "completes within 24 hours" is impossible to plan around. batchwatch measures what the queue actually does and answers one question: should I use batch for this job?

net8.0, no package dependencies - HttpClient and System.Text.Json only.

Status: compiled and tested. dotnet build and dotnet test run in CI on every push (the dotnet job), alongside the other first-class clients. The suite is xunit; it captures every outbound request and asserts on it.

Install

dotnet add package Batchwatch

On NuGet. Or add the project directly from a clone:

dotnet add reference ../clients/dotnet/src/Batchwatch/Batchwatch.csproj

Two lines

using Batchwatch;

var bw = new BatchwatchClient(new BatchwatchOptions { Token = "bw_..." });

// 1. before you submit - does this belong in the queue?
if (await bw.ShouldBatchAsync("gpt-5.6-sol", new AdviceOptions { MaxWait = "15m" }))
{
    var job = await openai.Batches.CreateAsync(...);
}

// 2. measure it, so the next person gets a better answer
using var t = bw.Track("gpt-5.6-sol", inputTokens: 9720);
var result = await WaitFor(job);
t.Done(outputTokens: result.Usage.CompletionTokens);

using closes the measurement at the end of the block, so the only way to lose one is to leave the process. MeasureAsync wraps a delegate and records failed if it throws.

Get a key with no email and no card:

curl -X POST https://batchwatch.dev/v1/keys -d '{"label":"my pipeline"}'

The deadline guard — batch when you can, sync when you must

Batch is half the price, but a queue that misses your deadline can take down a product. bw.Batch(...) gets you both: it runs your batch, watches the clock, and if the batch has not finished by your deadline it cancels it and runs your synchronous fallback instead — so your job always gets an answer, on time.

var job = bw.Batch("gpt-5.6-sol", new BatchJobOptions
{
    Deadline = "15m",
    OnDeadline = () => openai.Chat.Completions.Create(...),
});
job.Submit(() => openai.Batches.Create(...));
var result = await job.ResultAsync();

You hand over two callables — the batch-create and the sync fallback — and the client runs them. It never sees or builds your provider payload; there is no field for it, exactly as with Track(). If the batch finishes in time you get its result; if the deadline fires you get the fallback's result.

Every fallback is a measured prediction outcome: "batching would have missed — running sync was right." It goes down the same accuracy path ShouldBatchAsync() already feeds, so the server can score how often the guard was needed. Nothing new leaves the machine.

Deadline speaks the same duration language as AdviceOptions.MaxWait — "15m", "6h", "30s", a bare number of seconds, or null for no guard. The defaults duck-type the OpenAI / Anthropic batch shape; a different provider passes Poll / Cancel / ResultOf callables on BatchJobOptions.

The poll loop is ours, not yours

ResultAsync() owns the wait so you do not write the same sleep/backoff loop everyone else does. It polls with exponential backoff and jitter, never faster than a rate-limit floor (a naive one-second loop against a 24-hour job is 86,400 requests and an angry provider) and never slower than a ceiling. The first interval is informed by the model's own measured p50 — no reason to poll every five seconds against a model whose median queue time is forty minutes — and reading that p50 fails open: if batchwatch is unreachable, polling simply continues on the fixed fallback schedule.

The 24-hour expiry is a distinct terminal state, never silently a timeout: job.Expired is true when the batch hit the cutoff, so you can tell "the queue was slow" from "the batch failed".

Driving an async or worker loop yourself instead of blocking a thread? Step the same state machine one poll at a time:

var state = job.PollOnce();      // PollState.Running / Done / Expired / Failed
if (state == PollState.Running)
{
    await Task.Delay(TimeSpan.FromSeconds(job.NextInterval));  // backoff + jitter, applied
}

Partial completion — landed, failed, expired

A batch of 20,000 requests is not binary: some land, some fail per-request, and some never return before the 24-hour expiry. job.Split(...) separates the three, mapped back to your own objects by custom_id (never by index — provider ordering is not guaranteed):

var result = job.Split(downloadedLines, everySubmittedId);
result.Landed;        // count that came back clean
result.Failed;        // count that failed per-request
result.Expired;       // count still outstanding at the 24h cutoff
result.Complete;      // true only if EVERYTHING landed — never a silent success
result.ResultFor("my-request-42");     // your object for one id, mapped correctly

// retry only what failed — idempotent, so a second call submits nothing new
var child = job.RetryFailed(failedIds => openai.Batches.Create(...));

An expired job is reported to batchwatch as status="expired", recorded but kept out of the percentiles (only completed rows count) — so a slow queue neither pollutes p90 nor loses the "the queue was slow" signal. On misuse — a result before a submit, a fired deadline with no OnDeadline, a double submit — the job throws BatchJobError rather than stranding you on a result that never comes.

Alerts — subscribe to outcome notifications

Get told when the queue for a provider or model degrades. Three per-key routes, all keyed to your own token:

// webhook: omit the secret and the server mints one, returned ONCE — read it here.
using var sub = await bw.SubscribeAsync("webhook", "https://example.com/hook",
                                        providers: "openai", minSeverity: "severe");

// your active subscriptions (never the secret), and revoke one by id
var subs = await bw.SubscriptionsAsync();
await bw.UnsubscribeAsync(5);

Like MyCallsAsync, these require a key and do not fail open — subscribing is an explicit authed action, so with no token they throw BatchwatchAuthError before touching the network, rather than pretending you subscribed when you did not.

It fails open, always

If batchwatch is down, slow, or broken, your job must not notice. That is the first requirement, ahead of collecting any data at all.

• Track() and Done() return immediately. Submissions run on background tasks; no network I/O happens on your call path.
• Two-second timeout by default (BATCHWATCH_TIMEOUT in seconds).
• Every batchwatch exception is swallowed and handed to BatchwatchOptions.OnError. Nothing is logged unless you wire it up.
• ShouldBatchAsync is the one call you await, because you want the answer. If it cannot answer you get AdviceOptions.Fallback back - never a guess. The default is false, "run it synchronously": being wrong that way costs money, being wrong the other way blows a deadline.
• An exception from your own delegate in MeasureAsync is recorded as failed and re-thrown untouched. We swallow our errors, never yours.

FailOpenTests covers this against a port nothing listens on and against a socket that accepts but never answers.

It never sends your content

No prompts, no completions, no system prompts, no tool calls, no file names. The request body is built from a fixed allowlist - provider, model, mode, endpoint, request count, token counts, timestamps, status - and everything else is dropped by Scrub() on the way out. There is no field to put text in.

NoContentTests asserts it on the serialized request bodies, with a positive control so the test cannot pass by the client simply sending nothing. Note that unlike the Python and TypeScript suites, which run against a real loopback HTTP server, these tests capture the request through an HttpMessageHandler - one layer above the socket. HttpListener needs a URL ACL on Windows, so a real server would not run unelevated.

outputTokens defaults to null, never 0

You know your input tokens. You cannot know your output tokens before the model has answered. So the parameter is long? and the default is absence, not zero.

Zero is not a harmless placeholder: output costs five to six times as much as input, so a saving computed on zero output is systematically too low - measured at 3.4x too low on a real model - and nothing in the response would tell you. If you know a ceiling, pass MaxTokens and the answer comes back labelled as a ceiling.

Done(outputTokens: 0) really does send 0: zero is a measurement, absence is not.

Spooling

When a measurement cannot be delivered, the completed record is appended to a JSONL file and replayed later through POST /v1/calls/complete. Losing measurements exactly when the network is bad means losing them exactly when they are most interesting.

• Default path: $BATCHWATCH_SPOOL, or batchwatch-spool.jsonl in the temp directory. SpoolEnabled = false turns it off.
• Replayed automatically, at most once a minute, right after a successful call - that is the moment we know the network is up. Call FlushSpoolAsync() yourself on shutdown if you want it drained on exit.
• Spooling requires a token. /v1/calls/complete takes your own timestamps, so it is closed to anonymous callers; without a key a spool file could never be sent, and writing one would just leak disk.
• The file is capped at 5 MB. Beyond that, measurements are dropped rather than filling your disk.
• A replayed measurement can arrive twice if the original PATCH reached the server but the response did not. Deliberate: a duplicate is visible in the dataset, a lost measurement is not.
• The file format is identical across the Python, TypeScript and .NET clients, so a spool written by one can be flushed by another.

Read back your own contributions

Two per-key read routes let you see what your key sent and where it stands. They are the readback for Track() — there is no per-id route, and none to anyone else's rows.

var bw = new BatchwatchClient(new BatchwatchOptions { Token = "bw_..." });

// Everything this key has contributed. Returns { label, count, next, calls,
// note } verbatim; `next` is a ready-made relative URL for the following page
// (null on the last). after/limit paginate and are omitted unless you pass them.
using var mine = await bw.MyCallsAsync(after: 1787666964, limit: 100);

// This key's tier, contribution status and quota, verbatim.
using var status = await bw.KeyStatusAsync();

Both require a key and, unlike the measurement path, do not fail open: with no key they throw BatchwatchAuthError before touching the network, because reading your own calls is an explicit authed action, not a no-op.

Configuration

Tests

dotnet test tests/Batchwatch.Tests

xunit. Run in CI on every push - see the status note at the top.

Licence

MIT