RagObservability 0.1.1

RagObservability

A drop-in, self-hosted RAG-pipeline observability dashboard for .NET / Blazor — Hangfire Dashboard, but for your RAG pipelines.

Capture every RAG run and its stages (embedding → retrieval → [rerank] → generation), then browse them live, in-process, at /rag-dashboard: the retrieved chunks + similarity scores, the entire assembled prompt, the model response, token usage, cost, and per-stage latency. No external platform — nothing leaves your process.

Provider-agnostic. Works with any .NET LLM/agent framework or SDK — Azure OpenAI, AWS Bedrock, LangChain.NET, AutoGen, Semantic Kernel / Microsoft Agent Framework, raw provider SDKs, custom HTTP. Microsoft.Extensions.AI clients get convenience overloads; everything else uses the generic API below.

<img width="3789" height="1657" alt="image" src="https://github.com/user-attachments/assets/592a685f-c457-487e-9ce2-2453f2359937" /> <img width="3380" height="1795" alt="image" src="https://github.com/user-attachments/assets/6bf5ec70-105e-4600-8aad-59b7a2fd8ec3" /> <img width="3375" height="1778" alt="image" src="https://github.com/user-attachments/assets/4f42ee76-c135-40b5-a82b-378eb7c11a8b" />

Install

dotnet add package RagObservability

Targets .NET 10. Works in any ASP.NET Core host (Web API, MVC, Blazor) — you do not need an existing Blazor app, Node, or Tailwind.

Quickstart

1. Register + mount (Program.cs):

builder.Services.AddRagObservability(options =>
{
    options.Capacity       = 1_000;            // in-memory ring buffer
    options.CaptureContent = true;             // store prompts / responses / chunk texts
    options.Pricing.Set("gpt-4o", inputPerMTok: 2.50m, outputPerMTok: 10.00m);
});

var app = builder.Build();
app.UseAntiforgery();                  // required by the interactive dashboard
app.MapRagDashboard("/rag-dashboard"); // browse to /rag-dashboard
app.Run();

2. Instrument your RAG flow — wrap it in a using block:

public sealed class RagService(
    IRagTracer rag,
    IEmbeddingGenerator<string, Embedding<float>> embeddings,
    IVectorStore store,
    IChatClient chat)
{
    public async Task<string> AnswerAsync(string query)
    {
        using var run = rag.BeginRun(query);                       // start a run

        var vector = await run.Embed(query, embeddings);           // embedding stage
        var chunks = await run.Retrieve(                           // retrieval stage (chunks + scores)
            () => store.SearchAsync(vector, k: 5));

        // optional rerank stage — pass the reranker used so it shows on the timeline
        chunks = await run.Rerank(() => RerankAsync(query, chunks), strategy: "cohere-rerank-3");

        var prompt = BuildPrompt(query, chunks);
        var resp   = await run.Generate(prompt, chat);             // generation stage

        return resp.Text;                                          // run recorded on dispose
    }
}

Retrieve returns RetrievedChunks — construct them from your store's result, e.g. new RetrievedChunk(text, score, id, source), or use the generic Retrieve(search, map) overload to project your own type. Rerank is optional; when omitted, the dashboard shows the rerank step as skipped.

Works with any LLM/agent framework

The example above uses the Microsoft.Extensions.AI convenience overloads (run.Embed(generator), run.Generate(chat)) — these cover any provider exposed as IChatClient/IEmbeddingGenerator (Azure OpenAI, AWS Bedrock, Ollama, Microsoft Agent Framework, …).

For everything else, wrap your own call with the generic Generate<T> / Embed<T> and map the native response to a normalized capture. No dependency on the framework's packages — you call it, we time it, capture failures, and record the data:

// Any client: LangChain.NET, AutoGen, raw Azure.AI.OpenAI, AWS Bedrock Converse, custom HTTP…
var result = await run.Generate(
    prompt,
    () => myClient.CompleteAsync(prompt),                     // your call, your response type
    r  => new GenerationCapture(                              // map it to normalized data
        Response: r.Text,
        InputTokens: r.PromptTokens,
        OutputTokens: r.CompletionTokens,
        Model: "claude-3-5-sonnet",
        Provider: "bedrock"));

Embed<T> works the same way (map to EmbeddingCapture). When you can't wrap the call, use the manual escape hatches:

run.RecordEmbedding(dimensions: 1536, inputTokens: 42, model: "text-embedding-3-large");
run.RecordRetrieval(chunks);                                 // IEnumerable<RetrievedChunk>
run.RecordRerank(reranked, strategy: "cohere-rerank-3");
run.RecordGeneration(prompt, response, inputTokens: 1200, outputTokens: 180, model: "gpt-4o");

Cost is estimated for any provider as long as the model you pass is in the pricing table (options.Pricing.Set(...)).

See samples/Sample.MinimalApi.Rag/INTEGRATIONS.md for copy-paste snippets per provider: OpenAI, Azure OpenAI, AWS Bedrock, Anthropic Claude, OpenRouter, Google Gemini, LangChain.NET, Semantic Kernel / Agent Framework, and custom HTTP.

Multi-agent flows

A run can be a tree of steps, not just a flat pipeline. Open agent scopes with BeginAgent; nested captures (embed/retrieve/generate/tool/sub-agent) attach under that agent, and Handoff records the message/contract passed between agents:

using var run = rag.BeginRun(query);
using var planner = run.BeginAgent("Planner", input: query);

IReadOnlyList<RetrievedChunk> chunks;
using (var retriever = planner.BeginAgent("Retriever"))      // sub-agent (the RAG part)
{
    var vector = await retriever.Embed(query, embeddings);
    chunks = await retriever.Retrieve(() => store.SearchAsync(vector, k: 4));
    retriever.Complete($"{chunks.Count} chunks");
}

planner.Handoff("Writer", "Answer grounded only in the retrieved context.");

using (var writer = planner.BeginAgent("Writer"))
{
    await writer.Tool("format_citations", () => FormatAsync(chunks), r => r); // function/tool call
    var resp = await writer.Generate(BuildPrompt(query, chunks), chat);
    writer.Complete(resp.Text);
}

The run detail renders this as an agent timeline — nested steps with each agent's input/output, the handoff message, tool args/result, and tokens/cost/latency per agent (aggregated) and per child. The single-flow API above (run.Embed/Retrieve/Generate) is unchanged; a run with no agents still renders as the flat pipeline.

What you get at /rag-dashboard

• Runs list — every run with Succeeded/Failed status, a Run stages progress (e.g. 3/4), search, status filters, and a table grouped by day. Headline cards (total runs, succeeded, failed, tokens, cost, avg latency) plus 24-hour charts for runs (with the failed split), cost and latency. New runs appear live, without a refresh.
• Run detail — the stage timeline (Embedding → Retrieval → Rerank → Generation). Each stage shows its model/provider, tokens, cost and latency. Expand to see the retrieved/reranked chunks with similarity scores, the full assembled prompt and the model response. Stages that did not run are shown disabled (the optional rerank as skipped, later stages as not reached when a run failed early).

Try the sample (no API key)

dotnet run --project samples/Sample.MinimalApi.Rag --urls http://localhost:5099
# open http://localhost:5099/rag-dashboard   — seeded with demo runs (some with rerank, one failed)
# create more: /ask?q=your+question        (add &rerank=true to include a rerank stage)

The sample is a plain Web API with an in-process fake RAG (deterministic embeddings + in-memory vector store + canned chat client), proving the drop-in story end-to-end.

Pricing & cost (multiple models)

Cost is estimated per stage, using the model that stage actually ran — so different agents / calls using different models are each priced correctly. Register a price for every model you use (Set is chainable):

builder.Services.AddRagObservability(options =>
{
    options.Pricing
        .Set("claude-3-haiku",         inputPerMTok: 0.25m, outputPerMTok: 1.25m)  // simple/fast tasks
        .Set("claude-3-opus",          inputPerMTok: 15.00m, outputPerMTok: 75.00m) // complex tasks
        .Set("gpt-4o",                 inputPerMTok: 2.50m, outputPerMTok: 10.00m)
        .Set("text-embedding-3-small", inputPerMTok: 0.02m, outputPerMTok: 0.00m);  // embeddings
});

A run's total cost is the sum of its stages; a multi-agent run that uses Haiku for cheap steps and Opus for hard ones reflects exactly that. The model id comes from the provider response (or client metadata), or from what you pass to RecordGeneration(..., model: "…") / GenerationCapture.

If a stage's model has no registered price, its cost is left empty (the run still records tokens). Register the model to include it in cost totals.

Securing the dashboard

The dashboard is open by default. Two opt-in ways to lock it (use either or both):

1. Authorization (recommended) — uses your app's existing login. MapRagDashboard returns the endpoint builder, so chain RequireAuthorization:

app.MapRagDashboard("/rag-dashboard").RequireAuthorization();            // any signed-in user
app.MapRagDashboard("/rag-dashboard").RequireAuthorization("RagAdmin");  // a specific policy

This leverages whatever the host already uses (cookies, OpenID Connect / Entra ID, JWT, Identity).

2. Hangfire-style predicate — for hosts with no auth stack (shared secret, IP allow-list, …):

builder.Services.AddRagObservability(options =>
{
    options.Authorize = ctx =>
        System.Net.IPAddress.IsLoopback(ctx.Connection.RemoteIpAddress!) ||
        ctx.Request.Headers["X-Rag-Key"] == builder.Configuration["RagDashboard:Key"];
});

The predicate runs on every dashboard request; returning false denies access. (For strong auth, prefer option 1 — the predicate gates page loads, not the raw SignalR endpoint.)

How it works

• A single Razor Class Library packs to one NuGet package. MapRagDashboard self-hosts a Blazor Server (InteractiveServer) circuit at /rag-dashboard — even in a host with no Blazor of its own.
• Default storage is a bounded in-memory ring buffer (ITraceStore); nothing is persisted.
• Cost is estimated per stage from token usage × a configurable per-model price table.
• The dashboard CSS (Tailwind v4) is compiled at build time and shipped as a static web asset, so consumers need no Node/Tailwind toolchain.

License

MIT — see LICENSE.