NetEval.Xunit 0.1.0-alpha.1

NetEval

Test your AI features like you test your code.

NetEval is an LLM eval framework for .NET that plugs into xUnit and your CI pipeline. If your app has an AI feature and your "testing" is eyeballing outputs, this is for you.

[LlmFact] // auto-skips when no API key; retry semantics for flaky LLM outputs
public async Task Refund_reply_mentions_policy_and_avoids_legal_advice()
{
    var reply = supportBot.Reply("I want a refund, my order arrived broken.");

    await Expect.That(reply).Satisfies(
        "acknowledges the problem, mentions the 30-day refund policy, and gives no legal advice");
}

Run it with dotnet test. Fail messages include the judge's reasoning:

NetEval.EvalFailedException : Output did not satisfy criteria.
  Criteria:  mentions the 30-day refund policy...
  Score:     0.20
  Reasoning: The reply apologises but never mentions any refund policy.

Why

Python has a dozen eval frameworks. .NET teams shipping LLM features have essentially nothing that fits how they already work: xUnit, dotnet test, CI gates. NetEval is that missing piece — no dashboard, no hosted service, just tests.

• Provider-agnostic: the system under test and the judge both work through Microsoft.Extensions.AI.IChatClient — Anthropic, OpenAI, Azure, Ollama.
• CI-native: LLM tests auto-skip without API keys; committed baseline snapshots gate regressions — report.VerifyBaseline("bot.baseline.json") fails the build when pass rate drops, names the newly failing cases, and updates via NETEVAL_UPDATE_BASELINES=1.
• Honest about non-determinism: [LlmFact(Runs = 5, PassThreshold = 0.8)] runs a test N times with a pass threshold instead of pretending LLM outputs are deterministic.
• Dataset-driven: point EvalRunner at a JSONL file of cases and get pass rate, judge token usage, and latency — failures include the judge's reasoning.

var report = await new EvalRunner(new ClaudeJudge())
    .RunAsync(JsonlDataset.Load("support_cases.jsonl"), bot.ReplyAsync);

report.VerifyBaseline("support_bot.baseline.json");
// Eval run: 8/10 passed (80 %), mean score 0.84
// Mean SUT latency: 412 ms; judge usage: 12345 in / 890 out tokens
//
// On regression, the test fails with the diff against the committed baseline:
// BaselineRegressionException : Eval run regressed against baseline '...':
//   pass rate regressed: 90.0% (9/10) -> 80.0% (8/10), tolerance 0.0%
//   newly failing: "I want a refund..." — score 0.20: Never mentions the refund policy.

Install

Note: Packages are not yet published to NuGet. The v0.1.0-alpha.1 release is imminent 45 until then, clone the repo and reference the projects directly, or build locally with dotnet pack. Track the release here.

dotnet add package NetEval.Xunit          # [LlmFact] + assertions (pulls in NetEval.Core)
dotnet add package NetEval.Judges.Claude  # default judge backed by the Anthropic SDK

Set ANTHROPIC_API_KEY in your environment (and as a CI secret when you want evals to run in CI — without it, LLM tests skip and the build stays green).

Packages

Status / Roadmap

Early development — pre-alpha, API will change.

• Semantic assertions (Expect.That(...).Satisfies(...))
• Provider-agnostic ChatClientJudge over IChatClient
• [LlmFact] with API-key auto-skip
• JSONL datasets
• ClaudeJudge wired to the Anthropic SDK
• Retry semantics: [LlmFact(Runs = 5, PassThreshold = 0.8)]
• Dataset runner with pass rate, cost, and latency reporting
• Baseline snapshots + CI regression gating
• NuGet packaging + publish-on-tag release workflow
• First public release (v0.1.0-alpha.1 tag once the GitHub repo is up)

License

MIT