CatExperiments 0.0.1

CatExperiments

.NET SDK and runner for cat-experiments — a CLI-driven evaluation engine for LLM applications.

A cat-experiments project is an Exe that references this package. Mark task and evaluator methods with attributes; the cat-experiments CLI discovers, runs, and scores them.

Install

dotnet add package CatExperiments

Minimal project

MyExperiment/
├── MyExperiment.csproj
├── Program.cs
└── Experiment.cs

MyExperiment.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <RollForward>Major</RollForward>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="CatExperiments" Version="*" />
  </ItemGroup>
</Project>

Program.cs:

await Cat.Experiments.Runner.RunAsync(args);

Experiment.cs:

using Cat.Experiments;

public static class MyExperiment
{
    public static string Name = "My Experiment";

    [Task]
    public static TaskOutput Run(TaskInput input)
    {
        var question = input.Input.TryGetValue("question", out var q) ? q.GetString() ?? "" : "";
        return new TaskOutput(output: new Dictionary<string, object?>
        {
            ["answer"] = question.ToUpperInvariant(),
        });
    }

    [Evaluator]
    public static EvalOutput ExactMatch(EvalInput input)
    {
        var expected = input.ExpectedOutput is { } ex && ex.TryGetProperty("answer", out var e)
            ? e.GetString() ?? "" : "";
        var actual = input.ActualOutput.ValueKind == System.Text.Json.JsonValueKind.Object &&
                     input.ActualOutput.TryGetProperty("answer", out var a)
            ? a.GetString() ?? "" : "";
        return new EvalOutput(
            score: expected == actual ? 1.0 : 0.0,
            label: expected == actual ? "match" : "mismatch");
    }
}

Run

Install the CLI (Homebrew or go install), then:

cat-experiments run MyExperiment/MyExperiment.csproj --dataset data.jsonl --output-dir results/

The CLI builds the project (incrementally), spawns the compiled assembly, and drives it over a JSON-lines protocol on stdin/stdout.

License

MIT