ManagedCode.LlmTestServer 1.0.0

LLM Test Server

A local, deterministic LLM provider for .NET tests, with a dashboard that shows where every token goes.

What it is · Token usage and cache hits · Quick start · Capabilities · Providers · TypeSafe / Jev · Docs

What it is

LLM Test Server is a small ASP.NET Core server that speaks the HTTP APIs of 13 LLM providers, each under its own namespace: /openai/v1, /anthropic/v1, /gemini/v1beta, /azure-openai, and so on. Your application keeps its official SDK or Microsoft.Extensions.AI client and only changes the base URL. Every answer, tool call, stream chunk and failure comes from a fixture your test configures, so the same test produces the same result on every run — with no external model calls and no production credentials. Run it from an Aspire AppHost, inside your own ASP.NET Core host, or as a standalone container.

Native decision APIs are included too: TypeSafe Jev, Kev through System One, and Cloudflare Clef. Use the typed client from ManagedCode.LlmTestServer.Client with the contracts in ManagedCode.LlmTestServer.SystemOne — see the TypeSafe/Jev example.

Why use it

• Deterministic tests for LLM features. Script answers, tool calls, multi-turn conversations, streams, rate limits and content filters instead of mocking SDK internals.
• Your real integration code runs. Requests travel through the same provider SDKs and wire formats your application uses in production.
• Proof of what your application sent. Every request is recorded with its messages, payloads, usage and outcome; tests can query, export and verify that journal.
• Token and cache visibility. See how much of each prompt is system text, tool schemas or tool results, and how much of it the prompt cache served.
• Safe in parallel. Isolated sessions give each test its own configuration, queues, cache and journal on one shared server.

Token usage and prompt-cache hits

Every chat request gets deterministic token usage from the server's tokenizer, and the provider response carries it in that provider's own usage fields. Three things are tracked for each request:

• What the input is made of. Input tokens are split by role — system prompt, tool schemas, user, assistant text, tool-call arguments and tool results. The parts add up exactly to the reported input and are recorded as InputComposition on the request's usage.
• How the prompt cache served it. A repeated prompt prefix reports cache reads; the first request that makes a prefix cacheable reports cache writes. Thresholds follow each provider (for example 1,024 tokens in 128-token steps on OpenAI-compatible routes, 2,048 on Gemini, and explicit cache markers on Anthropic and Bedrock).
• What the model produced. Output tokens, with reasoning tokens for reasoning-capable fixture models reported separately.

Read it from a test through the request journal (GET /admin-api/requests):

var journal = await control.GetRequestsAsync();
foreach (var request in journal.Requests)
{
    if (request.Usage is not { InputComposition: { } parts } usage) continue;
    Console.WriteLine($"{request.ModelId}: {usage.InputTokens} input = {parts.SystemTokens} system + " +
        $"{parts.ToolDefinitionTokens} tool schemas + {parts.UserTokens} user + {parts.AssistantTokens} assistant + " +
        $"{parts.ToolCallTokens} tool calls + {parts.ToolResultTokens} tool results + {parts.OtherTokens} other; " +
        $"{usage.CachedInputTokens} read from cache, {usage.CacheCreationInputTokens} written");
}

Or open the dashboard: the Tokens tab draws the flow above for one request or a whole labeled conversation, with context-mix and prompt-cache donuts and per-request context growth. Traffic insights (/?view=insights) charts all recorded traffic by provider, model, outcome, latency, cache and context mix; clicking a slice filters the inbox.

Details: dashboard analytics · token usage and prompt-cache accounting.

Quick start

Status: the v1.0.0 NuGet packages and container image are not published yet. Until they are, build and pack the repository (version 0.1.18) and use the local feed — see installation.

Install the libraries in your existing AppHost or integration-test project from the locally packed feed:

dotnet add package ManagedCode.LlmTestServer.Aspire --version 0.1.18 --no-restore
dotnet add package ManagedCode.LlmTestServer.Client --version 0.1.18 --no-restore
dotnet restore --source /absolute/path/to/LlmTestServer/artifacts/packages \
  --source https://api.nuget.org/v3/index.json

In an Aspire AppHost

AddLlmTestServer() starts the packaged .NET service; Docker is not required.

using ManagedCode.LlmTestServer.Aspire;

var builder = DistributedApplication.CreateBuilder(args);
var llm = builder.AddLlmTestServer().WithApiKey("test-key");

builder.AddProject<Projects.Api>("api")
    .WithReference(llm)
    .WithEnvironment("OPENAI_ENDPOINT", llm.GetOpenAiEndpoint())
    .WithEnvironment("OPENAI_API_KEY", "test-key")
    .WaitFor(llm);

builder.Build().Run();

In a test

Add ManagedCode.LlmTestServer.Aspire, ManagedCode.LlmTestServer.Client and Aspire.Hosting.Testing, then configure a response and call it:

using Aspire.Hosting.Testing;
using ManagedCode.LlmTestServer.Aspire;
using ManagedCode.LlmTestServer.Client;
using Microsoft.Extensions.AI;
using ChatMessage = Microsoft.Extensions.AI.ChatMessage;

var builder = DistributedApplicationTestingBuilder.Create([]);
builder.AddLlmTestServer("llm").WithApiKey("test-key");
await using var app = await builder.BuildAsync();
await app.StartAsync();
await app.ResourceNotifications.WaitForResourceHealthyAsync("llm");

var control = new LlmTestServerClient(app.CreateHttpClient("llm", "http"), "test-key");
await control.ConfigureAsync(config => config.RequireBearerToken("test-key").UseGpt41Mini()
    .UseChatScenario("greeting", scenario => scenario.WhenUserContains("hello").Responds("Hello from your test!")));
using var chat = control.CreateChatClient();
var reply = await chat.GetResponseAsync([new ChatMessage(ChatRole.User, "hello")]); // "Hello from your test!"

Your application can use its official SDK against the same server. In the Aspire test above, use its allocated endpoint and the same fixture key — for example with OpenAI:

using System.ClientModel;
using OpenAI;
using OpenAI.Chat;

await control.ResetAsync(); // Reuse the configured fixture for this separate SDK example.
using var providerHttp = app.CreateHttpClient("llm", "http");
var openAiChat = new ChatClient("gpt-4.1-mini", new ApiKeyCredential("test-key"),
    new OpenAIClientOptions { Endpoint = new Uri(providerHttp.BaseAddress!, "openai/v1") });
ChatCompletion completion = await openAiChat.CompleteChatAsync([new UserChatMessage("hello")]);
Console.WriteLine($"{completion.Usage.InputTokenCount} input tokens, " +
    $"{completion.Usage.InputTokenDetails.CachedTokenCount} from cache");

As a standalone container

Build the image with the .NET SDK (use linux-x64 in both RID arguments on amd64), run it, then configure it over HTTP — any language can drive it:

dotnet publish samples/ManagedCode.LlmTestServer.Service/ManagedCode.LlmTestServer.Service.csproj \
  --configuration Release --runtime linux-arm64 --self-contained false \
  /t:PublishContainer -p:ContainerRuntimeIdentifiers=linux-arm64 -p:ContainerImageTag=local -p:UseAppHost=false
docker run --rm -p 8080:8080 -e LlmTestServer__RequiredBearerToken=local-test-key managedcode/llm-test-server:local

curl --fail-with-body http://localhost:8080/admin-api/configure \
  -H 'Authorization: Bearer local-test-key' -H 'Content-Type: application/json' \
  --data '{"requiredBearerToken":"local-test-key","models":[{"id":"gpt-4.1-mini","kind":"chat"}],"chatScenarios":[{"id":"hello","modelId":"gpt-4.1-mini","responses":[{"content":"Hello from your fixture"}]}]}'
curl --fail-with-body http://localhost:8080/openai/v1/chat/completions \
  -H 'Authorization: Bearer local-test-key' -H 'Content-Type: application/json' \
  --data '{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"Hello"}]}'

Open http://localhost:8080/ for the dashboard. When a key is configured it protects provider routes and /admin-api/*. More: Docker distribution, embedding in ASP.NET Core.

What you can do

Everything is a fixture you choose: nothing is hidden behind retries or randomness, and the server never calls a real model.

Dashboard tour

The dashboard at / is a request inbox: one row per provider call, with the full request, response, conversation, payloads, stream chunks, tokens and runtime result one click away.

Request overview	Conversation	Runtime diagnostics
What was sent, what came back, and what it cost in tokens.	Each turn with its role and token share.	Fixtures, provider routes, assertions and the event timeline.

Providers

Each provider family lives under its own namespace and keeps its native request and response shapes.

Operation coverage differs by provider and grows over time. A provider being listed does not mean every API of that provider is implemented — check the operation coverage and routes reference for the exact contracts.

TypeSafe and Jev

TypeSafe is the provider; Jev is its model family. System One is the native API. Use ManagedCode.LlmTestServer.SystemOne for its request and response contracts, and ManagedCode.LlmTestServer.Client for control.CreateSystemOneClient(). The native server route is POST /systemone/v1/systemone.

Configure deterministic answers for predicates, choices, scores and Jev 1.14 bounding boxes. JSON image assets and multipart uploads are supported with explicit image opt-in. Kev uses the same protocol package with its own compatibility profile.

Configure and call Jev from a test · Jev 1.14 images and limits · Native decision fixtures

Packages

The full list with NuGet links is in providers, packages and routes.

Learn more

• Installation and hosting — local feed, container, ASP.NET Core, Aspire, building from source
• Configuration and scenarios — models, scenarios, modalities, faults, assertions
• Clients and SDKs — control client, official SDKs, Microsoft.Extensions.AI, TypeSafe client
• Token usage and prompt cache and dashboard analytics
• Providers, packages and routes and operation coverage
• Architecture, test inventory and delivery plan

Build from source

dotnet restore ManagedCode.LlmTestServer.slnx
dotnet build ManagedCode.LlmTestServer.slnx --configuration Release --no-restore
dotnet test --project tests/ManagedCode.LlmTestServer.Tests/ManagedCode.LlmTestServer.Tests.csproj --configuration Release --no-build

Packing, coverage and package verification are described in installation.

License

MIT