AgentTestTools 2.0.7

AgentTestTools

AgentTestTools is a lightweight .NET library for unit and integration testing of AI agents. It provides LLM response mocking, workflow simulation, handoff validation, regression testing, and xUnit helpers, lowering friction for teams adopting the Microsoft Agent Framework in CI/CD pipelines.

🔄 Architecture flow

Microsoft Agent Framework ---> AgentTestTools ---> CI/CD Pipeline (agents & workflows) (tests & asserts) (automation & validation)

🚀 Quick Start

1. Install the package

dotnet add package AgentTestTools

2. Create a Console App

dotnet new console -n AgentTestDemo
cd AgentTestDemo

3. Add a simple test

Program.cs:

using AgentTestTools;

class Program
{
    static async Task Main()
    {
        // Define your own prompt and mocked response
        var mock = AgentTestManager.CreateMock("Hello", "Mocked response");
        string response = await mock.GetResponseAsync();

        // Validate the response
        AgentAssert.ResponseContains(response, "Mocked");

        Console.WriteLine("✅ Quick Start test passed!");
    }
}

4. Run the app

dotnet run

Output:

[AgentTestTools] 2026-06-15T17:25:00Z: Mock created for prompt 'Hello'
✅ Quick Start test passed!

5. (Optional) Add xUnit tests

Note: Test classes should be placed in a dedicated test project (e.g., AgentTestTools.Test). This ensures that xUnit can discover and execute them automatically.

dotnet new xunit -n AgentTestTools.Tests
cd AgentTestTools.Tests
dotnet add reference ../AgentTestDemo/AgentTestDemo.csproj

AgentWorkflowTests.cs:

using AgentTestTools;
using Xunit;

public class AgentWorkflowTests
{
    [AgentTest]
    public async Task Workflow_Should_Run_In_Order()
    {
        var result = await AgentTestManager.SimulateWorkflowAsync("Step1", "Step2");
        AgentAssert.WorkflowOrder(result, new[] { "Step1", "Step2" });
    }
}

Run tests:

dotnet test

Test Runner Output (xUnit):

[AgentTestTools] 2026-06-16T16:38:00Z: Simulating workflow with 2 steps
✓ AgentWorkflowTests.Workflow_Should_Run_In_Order [<1ms]

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.

📌 Why AgentTestTools

Modern agent frameworks introduce complexity: multi-agent orchestration, LLM dependencies, and unpredictable workflows.
Testing these systems is expensive and difficult without proper tooling.

AgentTestTools solves this by offering:

• Mocking: replace costly LLM calls with predictable responses.
• Simulation: validate workflows before deploying to production.
• Validation: ensure agent handoffs are correct and reliable.
• Regression testing: record and replay agent scenarios.
• CI/CD integration: run tests automatically in pipelines.

How It Differs from Other Libraries

• Compared to AgentEval:

  • AgentEval focuses on evaluation and benchmarking (latency, RAG metrics).
  • AgentTestTools focuses on lightweight unit/integration testing with mocks, workflow simulation, and regression replay.

• Compared to Microsoft.Agents.Builder.Testing:

  • Builder.Testing is designed for conversational flow testing.
  • AgentTestTools adds workflow simulation, handoff validation, regression testing, and extensibility for multi-agent scenarios.

Together, these tools complement each other, but AgentTestTools fills the gap for lightweight CI/CD testing.

Features

Core Assertions

• ResponseContains
• WorkflowOrder
• HandoffValid

Others:

• LLM Response Mocking
• Workflow Simulation (async)
• Handoff Validation with exceptions
• Scenario Loader (JSON/YAML)
• Logging with timestamps
• Extensibility via interfaces (ILlmMockProvider)
• Regression Testing (record/replay scenarios)
• xUnit Helper Attribute ([AgentTest])
• Error Classes (AssertionFailedException, ScenarioReplayException)

How AgentTestTools works (Flow Diagram)

1. User provides values (text, steps, agents)
2. Library runs validations & simulations
3. Result: Pass / Fail assertion

📌 Explanation

• User provides values --> prompts, mocked responses, workflow steps, agent names, handoff messages, or regression events.
• Library runs validations --> checks order, content, handoff correctness, and replay consistency.
• Result: Pass/Fail --> assertions confirm whether the behavior matches expectations.

🧪 What does AgentTestTools actually test?

AgentTestTools focuses on validating agent behaviors and interaction flows. Even though most parameters we provide are text, this is highly useful because agent systems are driven by prompts, responses, and event logs.

📌 Types of tests

• Prompt tests --> validate that an agent responds correctly to a given input.
  Example: prompt "What is your name?" should produce a response containing "name".

• Workflow tests --> check that steps in a process are executed in the correct order.
  Example: "Login" --> "Validation" --> "Dashboard".

• Handoff tests --> ensure that a message is correctly transferred from one agent to another.
  Example: "SalesBot" passes "customer transfer" to "SupportBot".

• Regression tests --> record events and replay them to confirm the system behaves consistently after changes.
  Example: "User logs in" and "System validates credentials" replay in the same order.

• Negative tests --> verify that assertions fail when behavior is missing (important for CI/CD pipelines).

Where do the test values come from?

When using AgentTestTools, you provide your own values as parameters. These values represent the scenarios you want to test. The library does not generate them — it validates and simulates based on what you pass in.

• Prompts --> come from the questions or inputs you would normally send to an LLM.
  Example: "Hello", "What is your name?".

• Mocked responses --> are the outputs you want to simulate instead of calling a real LLM.
  Example: "I am a test mock".

• Workflow steps --> represent the sequence of actions in your agent system.
  Example: "Login", "Validation", "Dashboard".

• Agent names --> are the identifiers of the agents in your architecture.
  Example: "SalesBot", "SupportBot".

• Handoff messages --> are the payloads or context passed when one agent hands off to another.
  Example: "customer transfer".

• Events for regression --> are the logs or checkpoints you want to record and replay later.
  Example: "User logs in", "System validates credentials".

Data types

Strings --> most parameters are plain text (string). String arrays --> workflows are passed as string[]. JSON/YAML --> scenarios can be loaded from structured files using ScenarioLoader. Custom objects --> if you implement your own ILlmMockProvider, you can generate richer responses and then serialize them to string. Other types (numbers, dates, booleans) --> can be embedded inside JSON scenarios or events, and the library will handle them as serialized text.

🎯 Why is text-based testing useful?

• Prompts, responses, workflow steps, and handoff messages in agent systems are textual by nature.

• Validating these strings ensures that:

  • Agents respond with the keywords.
  • Workflows follow the correct sequence.
  • Messages are handed off without losing context.

• In real projects, these strings represent user inputs, chat messages, system events, or JSON payloads.

• The library provides the testing infrastructure, while you define the specific content to validate.

⚠️ When can tests fail?

AgentTestTools tests agent interactions, not algorithms. Even though parameters are mostly text, tests can fail when the actual behavior does not match the scenario.

📌 Typical failure cases

• Content mismatch
  If you assert that a response must contain "mock" but the actual string is "Text1", the test fails because the keyword is missing.

• Wrong workflow order
  If you simulate steps ["Login", "Validation", "Dashboard"] but the result comes back as "Validation --> Login --> Dashboard", the assertion fails.

• Invalid handoff
  Handoff means the transfer of a message or context from one agent to another inside a multi-agent system. If you expect "SalesBot" to hand off "customer transfer" to "SupportBot", but the message is "Text2" or the agents don’t match, the test fails.

• Regression mismatch
  If you replay a recorded scenario and the events don’t appear in the same order or with the same text, the regression test fails.

• Negative tests
  These are designed to fail intentionally when behavior doesn’t meet expectations — useful to confirm that your assertions are working correctly.

✅ Why this matters

Even if inputs are “just text,” they represent real-world agent interactions:

• Prompts --> user inputs.
• Responses --> agent outputs.
• Steps --> workflow sequences.
• Messages --> payloads between agents.
• Events --> logs for regression.

If any of these don’t match expectations, the test fails — helping you catch errors in agent orchestration, prompt handling, or workflow consistency.

Usage Examples

Example 1: Console Application (Program.cs)

This example demonstrates the core features of AgentTestTools in a simple console app. It shows how to mock LLM responses, simulate multi‑step workflows, validate agent handoffs, and perform regression testing with recorded scenarios. The output illustrates both the logging and replay capabilities, confirming that all validations pass successfully.

using AgentTestTools;

class Program
{
    static async Task Main()
    {
        // Mocking
        var mock = AgentTestManager.CreateMock("Hello", "Mocked response");
        string response = await mock.GetResponseAsync();
        AgentAssert.ResponseContains(response, "Mocked");

        // Workflow simulation
        string workflow = await AgentTestManager.SimulateWorkflowAsync("Step1", "Step2", "Step3");
        AgentAssert.WorkflowOrder(workflow, new[] { "Step1", "Step2", "Step3" });

        // Handoff validation
        AgentAssert.HandoffValid(() => AgentTestManager.ValidateHandoff("AgentA", "AgentB", "handoff message"));

        // Regression testing
        AgentTestManager.RecordEvent("Mock created for prompt 'Hello'");
        AgentTestManager.RecordEvent("Workflow simulated with Step1, Step2, Step3");
        AgentTestManager.SaveScenario("scenario.json");

        AgentTestManager.ReplayScenario("scenario.json", e => Console.WriteLine($"Replaying: {e}"));

        Console.WriteLine("All tests passed successfully!");
    }
}

Output:

[AgentTestTools] 2026-06-15T16:54:00Z: Mock created for prompt 'Hello'
[AgentTestTools] 2026-06-15T16:54:00Z: Simulating workflow with 3 steps
[AgentTestTools] 2026-06-15T16:54:00Z: Validating handoff from AgentA to AgentB
Replaying: Mock created for prompt 'Hello'
Replaying: Workflow simulated with Step1, Step2, Step3
All tests passed successfully!

Example 2: Unit Test Project (AgentTestTools.Tests)

This example shows how AgentTestTools integrates directly with xUnit. By using the [AgentTest] attribute, agent-specific validations such as workflow order and handoff correctness are executed as standard unit tests. The output demonstrates how logs are captured during test execution and how results are reported consistently in the test runner.

Note: Test classes should be placed in a dedicated test project (e.g., AgentTestTools.Test). This ensures that xUnit can discover and execute them automatically.

AgentWorkflowTests.cs:

using AgentTestTools;
using Xunit;

public class AgentWorkflowTests
{
    // The [AgentTest] attribute integrates with xUnit
    // and ensures agent-specific logs and validations
    // are captured consistently during test execution.
    [AgentTest]
    public async Task Workflow_Should_Run_In_Order()
    {
        var result = await AgentTestManager.SimulateWorkflowAsync("Step1", "Step2");
        AgentAssert.WorkflowOrder(result, new[] { "Step1", "Step2" });
    }

    [AgentTest]
    public void Handoff_Should_Be_Valid()
    {
        AgentAssert.HandoffValid(() => AgentTestManager.ValidateHandoff("AgentA", "AgentB", "handoff message"));
    }
}

Test Runner Output (xUnit):

[AgentTestTools] 2026-06-16T16:39:00Z: Simulating workflow with 2 steps
[AgentTestTools] 2026-06-16T16:39:00Z: Validating handoff from AgentA to AgentB

Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
Test Run Successful.

Example 3: Loading Mocks from JSON (Program.cs)

This example demonstrates how to load mock data from a JSON file to simulate different agent scenarios. It shows how agents can handle various data types—strings, numbers, booleans, and structured objects—without relying on a real LLM. By using ScenarioLoader, you can inject predefined responses and validate agent behavior consistently across multiple cases.

mocks.json:

json
{
  "Hello": "Mocked response from JSON",
  "Bye": "Goodbye response",
  "AgeCheck": 42,
  "IsActive": true,
  "UserProfile": {
    "Name": "Mark",
    "Role": "Developer",
    "Joined": "2026-06-15"
  }
}

Program.cs:

using AgentTestTools;
using System.Text.Json;

class Program
{
    static async Task Main()
    {
        var loader = new ScenarioLoader();
        var mocks = loader.LoadMocks("mocks.json");// File name

        // String mock
        var mockHello = AgentTestManager.CreateMock("Hello", mocks["Hello"].ToString());
        string responseHello = await mockHello.GetResponseAsync();
        AgentAssert.ResponseContains(responseHello, "Mocked");
        Console.WriteLine(responseHello);

        // Numeric mock
        var mockAge = AgentTestManager.CreateMock("AgeCheck", mocks["AgeCheck"].ToString());
        string responseAge = await mockAge.GetResponseAsync();
        AgentAssert.ResponseContains(responseAge, "42");
        Console.WriteLine(responseAge);

        // Boolean mock
        var mockActive = AgentTestManager.CreateMock("IsActive", mocks["IsActive"].ToString());
        string responseActive = await mockActive.GetResponseAsync();
        AgentAssert.ResponseContains(responseActive, "True");
        Console.WriteLine(responseActive);

        // Object mock (serialize to string)
        var userProfile = JsonSerializer.Serialize(mocks["UserProfile"]);
        var mockProfile = AgentTestManager.CreateMock("UserProfile", userProfile);
        string responseProfile = await mockProfile.GetResponseAsync();
        AgentAssert.ResponseContains(responseProfile, "Mark");
        Console.WriteLine(responseProfile);
    }
}

---

Output:

[AgentTestTools] 2026-06-15T16:56:00Z: Mock created for prompt 'Hello'
Mocked response from JSON

[AgentTestTools] 2026-06-15T16:56:01Z: Mock created for prompt 'AgeCheck'
42

[AgentTestTools] 2026-06-15T16:56:02Z: Mock created for prompt 'IsActive'
True

[AgentTestTools] 2026-06-15T16:56:03Z: Mock created for prompt 'UserProfile'
{"Name":"Mark","Role":"Developer","Joined":"2026-06-15"}

Example 4: Custom Mock Provider

This example illustrates how to create a custom mock provider by implementing the ILlmMockProvider interface. It shows how developers can define their own logic for generating mocked responses, ensuring flexibility and extensibility. By following a common contract, different providers can be swapped seamlessly without changing the rest of the library logic.

CustomMockProvider.cs:

class CustomMockProvider : ILlmMockProvider
{
    public Task<string> GetMockResponseAsync(string prompt)
    {
        return Task.FromResult($"Custom mock for: {prompt}");
    }
}

Program.cs:

using AgentTestTools;

class Program
{
    static async Task Main()
    {
        ILlmMockProvider provider = new CustomMockProvider();
        string response = await provider.GetMockResponseAsync("Hello");

        AgentAssert.ResponseContains(response, "Custom mock");
        Console.WriteLine(response);
    }
}

Output:

Custom mock for: Hello

Example 5: Negative Test Case (Assertion Failure in xUnit)

This example shows how to validate negative cases by asserting a missing keyword. The failure is expected and reported as a passed test in xUnit.

Note: Test classes should be placed in a dedicated test project (e.g., AgentTestTools.Test). This ensures that xUnit can discover and execute them automatically.

NegativeTests.cs:

using AgentTestTools;
using Xunit;

public class NegativeTests
{

    // The [AgentTest] attribute integrates with xUnit
    // and ensures agent-specific logs and validations
    // are captured consistently during test execution.
    [AgentTest]
    public void ResponseContains_ShouldFail_WhenKeywordMissing()
    {
        Assert.Throws<AssertionFailedException>(() =>
            AgentAssert.ResponseContains("Test response", "MissingKeyword"));
    }
}

Test Runner Output (xUnit):

[AgentTestTools] 2026-06-16T16:47:00Z: Validating response contains 'MissingKeyword'
X NegativeTests.ResponseContains_ShouldFail_WhenKeywordMissing [<1ms]
Error: Expected response to contain 'MissingKeyword' but got 'Test response'

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.

🤔 Common Questions

🤖 Who creates the agents?

The agents that can fail are created by the user inside the Microsoft Agent Framework (or any other multi-agent environment).

📌 Who defines the agents

• User-defined agents: the developer creates agents in Microsoft Agent Framework, each encapsulating an LLM, business rules, or API integrations.
• Agent workflows: the user designs the interaction flows between agents (e.g., SalesBot --> SupportBot).
• Custom prompts: the user decides which textual inputs are sent to each agent.
• Handoff logic: the user programs how one agent transfers context or messages to another.

🎯 How AgentTestTools fits in

The library does not create agents. Instead, it:

• Provides infrastructure for mocks and asserts.
• Allows the user to simulate how their agents would respond to certain prompts or workflows.
• Validates that the agents created in Microsoft Agent Framework behave as expected.

📖 Quick Reference

Support the Project

If you find this library useful, consider supporting its development:

Buy Me a Coffee

⚖️ License

This project is freely available under the MIT license.
You may use it without restrictions, as long as you retain the reference to the original license.