Graphene.AIOrchestrator 1.26.8.18

AIOrchestrator

Deterministic AI-powered automation engine for .NET 10

AIOrchestrator is a C# library that combines Large Language Models with reflection-based tool calling to automate complex tasks. It powers the AIOffice application (and the AgentBridge server) and can be used as a standalone automation framework.

Philosophy: Deterministic AI

Unlike traditional "agentic AI" frameworks that rely on the LLM to generate free-form code or unstructured outputs, AIOrchestrator uses a deterministic reflection-based tool calling approach:

1. Static method discovery — UISupportGeneric.Analyzer.GeToolDefinitions() scans C# classes via reflection and generates a Markdown + JSON Schema catalog of available methods (API interaction mode; CLI mode instead exposes a single "execute shell command" tool — see Agent interaction modes)
2. Structured tool calling — the LLM receives the catalog and responds with {"method": "FillFields", "fieldsJson": "..."} — JSON that maps 1:1 to real C# method signatures
3. Type-safe execution — UISupportGeneric.API.CallMethod() invokes the method via reflection with full parameter validation, returning structured JSON
4. Snake_case → PascalCase — LLMs naturally produce fill_fields; the system auto-converts to FillFields

The LLM never writes code, never invents parameters, and never hallucinates method names. It selects from a fixed, real catalog.

Why This Approach Wins

Architecture

┌─────────────────────────────────────────────────────┐
│                  AIOffice                           │
│  (Blazor Server App — UI & Business Logic)          │
└──────────┬──────────────────────────┬───────────────┘
           │                          │
     Panels/WebAgent.cs         API.cs, Voice.cs
     (Agent orchestration)      (Public endpoints,
                                 streaming chat)
           │
┌──────────▼──────────────────────────────────────────┐
│                 AIOrchestrator                       │
│                                                      │
│  LLMUtility.cs        Setup.cs       ProviderConfig  │
│  (LLM communication,  (API keys,     (Ollama,        │
│   history, tool       provider       DeepSeek,       │
│   calling loop)       selection)     Z.ai, Gemini)   │
│                                                      │
│  AgentHarness.cs  API/WebTool  FileTool     │
│  (agent loop,          (Playwright   (document       │
│   subagents,           browser       search)         │
│   attachments,         automation)                   │
│   streaming)                                         │
│                                                      │
│  ZaiOcrConverter.cs    AllToMarkdown                 │
│  (Z.ai GLM-OCR for     (docs → Markdown)             │
│   images/PDF)                                        │
└──────────┬──────────────────────────────────────────┘
           │
┌──────────▼──────────┐  ┌────────────────────────────┐
│  UISupportGeneric   │  │     ReverseMarkdown         │
│  (Reflection API,   │  │  (HTML→MD + Interactive     │
│   code generation,  │  │   JSON converter)           │
│   tool definitions, │  │                             │
│   FileAttachment)   │  │                             │
└─────────────────────┘  └────────────────────────────┘
           │
┌──────────▼──────────────────────────────────────────┐
│  AgentBridge (OpenAI-compatible HTTP server)     │
│  hosts AIOrchestrator for standalone clients        │
│  (e.g. Giraffe AI): /v1/chat/completions, /v1/files │
└─────────────────────────────────────────────────────┘

Agent Architecture

API.IAgentTool — Agent Interface

All agent classes implement the IAgentTool interface (internal, same assembly):

internal interface IAgentTool
{
    bool SupportsAsyncTasks => false;          // long-running background operations
}

When a new agent class is added (e.g. SpreadsheetTool), it automatically:

• Gets its methods discovered as tools via reflection
• Is described to the agent by its class-level summary and its method XML docs (the tool definitions block)
• Becomes available as a subagent for delegation — introduced to the main agent as - Delegation to the sub-agent of: <first line of the class-level XML summary> (via UISupportGeneric.Terminal.GetClassDescriptionFirstLine)

AgentHarness — Agent Loop

The orchestrator runs a deterministic loop with dynamic prompt composition:

public AgentResult ExecuteAction(
    string prompt,
    Type[] agentTypes,
    Type[]? subagents = null,    // optional subagents for delegation
    int maxIterations = 50,
    IEnumerable<FileAttachment>? attachments = null);  // optional user files

Prompt composition: the Steps block contains the trivial-question fast-path plus the optional subagent delegation lines; the tool definitions block (class summaries + method definitions, in agentTypes order) is the agent's manual for the registered tools.

File Attachments & Streaming

The orchestrator supports multi-file attachments in both chat modes:

• Non-streaming — ExecuteAction(..., attachments) (tool-calling agent loop).
• Streaming — ExecuteChatStream(prompt, attachments, cancellationToken) yields text chunks as the LLM generates them (used by the AIOffice Voice panel for TTS playback).

The golden rule: files are never converted client-side. The user's files arrive in their ORIGINAL binary format inside a FileAttachment (FileName + Content). The orchestrator converts each one to Markdown server-side:

The converted Markdown is cached in FileAttachment.MarkdownContent, wrapped into a context block by BuildAttachmentsContext, truncated to the configured budget (MaxAttachmentContextChars, default 30.000 chars, split equally per file), and injected into the prompt sent to the LLM. Files that fail to convert are simply skipped — they never break the chat. See ARCHITECTURE.md → "File Attachment Pipeline" for the full diagram.

AgentBridge — OpenAI-compatible Server

AgentBridge (part of the AIOffice solution) is a small web app that hosts this library and exposes ExecuteAction as standard OpenAI endpoints (/v1/chat/completions with SSE streaming, /v1/files for attachments, /v1/models, /health), so standalone clients such as Giraffe AI can drive the agents with any OpenAI SDK. See AgentBridge/README.md.

Subagent Dialogue

The orchestrator supports multi-turn delegation between agents without context bleed:

Main agent (no tools) → launch_subagent("Analyze budget.xlsx")
  → Subagent runs in isolated context (own LLM, tools, history)
  → Subagent: {"done":true, "question":"Which sheet?", "sessionId":"sub_1"}
  → Main agent: provide_subagent_input("sub_1", "Summary sheet")
  → Subagent continues from where it paused
  → Subagent: {"result": "Budget total: $10,000"}

Key properties:

• Isolated context — each subagent has its own LLMUtility, agents, and message history
• No context bleed — subagent conversations don't pollute the main agent's window
• Auto-cleanup — subagent resources (LLM, workbook handles, browser) freed on completion
• Auto-detect session — if sessionId is omitted and only one session exists, it's auto-selected
• Extensible — any class implementing IAgentTool is automatically available as a subagent

API.SpreadsheetTool — Spreadsheet Manipulation

Full XLSX manipulation via Aspose.Cells.FOSS, sandboxed within Setup.DocumentsPath.

Security sandbox: the agent's world is a Unix-style filesystem — every path starts with "/" and uses "/" separators, relative to the workspace root (Setup.DocumentsPath maps to "/"). Absolute paths escaping this directory are rejected. The host filesystem (drives, backslashes) is never exposed to the agent.

Workflow:

1. Open(filePath) or Create(filePath) to load/create a workbook
2. GetSheetNames() → DescribeWorksheet() for a compact JSON overview
3. Use cell/range/style/chart/table methods to manipulate data
4. Save() to persist (automatically creates numbered .NNN.bak backup)
5. Restore() to roll back to the most recent backup
6. Dispose() when done

Example:

var agent = new SpreadsheetTool("report.xlsx");
var sheets = agent.GetSheetNames();
var json = agent.DescribeWorksheet("Sales");
agent.SetCellValue("Sales", "A1", "Hello");
agent.FormatHeaderRow("Sales");
agent.AddChart("Sales", "Column", "Sales!$A$1:$B$10", 0, 5, 15, 8);
agent.Save();  // backup created: report.001.bak

Backup system: Save() and SaveAs() automatically create numbered backups (filename.001.bak, .002.bak, ...) before overwriting. Restore() finds the most recent backup and restores it, preserving the backup file.

Setup.DocumentsPath — Sandbox Workspace (Unix path standard)

All agent file operations are sandboxed within Setup.DocumentsPath. This is the only directory agents can access for file operations. The agent sees it as the Unix root "/": paths it provides start with "/" and use "/" separators (converted by AIOrchestrator.UnixPath):

// Sandbox path — the agent's root "/" maps to this directory
Setup.DocumentsPath = @"C:\path\to\workspace";

// Agent path "/report.xlsx" → resolves to C:\path\to\workspace\report.xlsx
var agent = new SpreadsheetTool("/report.xlsx");

// Absolute host path outside the sandbox → rejected with error
var agent2 = new SpreadsheetTool("C:/secret/data.xlsx");  // throws

⚠️ Startup indexing cost (coding agents, read this): Setup.RagDocumentProcessor is created lazily on first use, and its initialization indexes DocumentsPath immediately (full build if no index exists, incremental refresh in Release). On a large folder this takes minutes with no log output — it looks hung but is working. When the code under test does not need document searches, set Setup.SkipIndexingOnStartup = true before any Setup member is touched (e.g. AgentBridge's --SkipIndexingOnStartup true does this before Setup.Load()): the index is neither built nor refreshed and the file watcher is not started, so file searches return empty. Never construct a second RagDocumentProcessor — always use Setup.RagDocumentProcessor.

Mermaid Diagram Rendering

PDF documents can include Mermaid diagrams, rendered by the MermaidRendering project (Node.js + Mermaid.js v11).

The LLM prompt in CreateDocument.cs uses MarkdownToPdfConverter.SupportedMermaidTypes for the list of supported diagram types.

LLMUtility — LLM Communication

The central class for talking to LLM providers:

var llm = new LLMUtility(LLMProvider.DeepSeek);
var (response, hResult) = llm.SendQuery(
    prompt,
    role: SystemRole.ApiAgent,
    toolDefinitions: toolDefinitions);

• Multi-provider: Ollama (local), DeepSeek, Z.ai, Gemini
• Conversation history: maintained automatically, supports tool results injection
• System roles: DocumentPreparer (document generation), ApiAgent (tool calling)
• JSON enforcement: forceJsonResponse mode for structured outputs
• Anonymization (anonymize: true): NameOrKey elements (company/personal names, document keys) found in the prompt and in the support document contents are replaced with [HEXID_PLACEHOLDER] tokens before the request is sent — HEXID is the element's index in a session-long registry (uppercase hexadecimal; the registry only grows, so indices stay stable across turns and history placeholders remain resolvable). The replacement is case-insensitive and never lands adjacent to letters/digits, so words that merely contain the term are left intact. The role system message instructs the model to output the tokens exactly as-is; tool results (AddToolResult) and subagent messages (AddUserMessage) are anonymized the same way; the assistant response is translated back to the real names before being returned or stored in history. SendQueryStream is not supported with anonymization (throws NotSupportedException).

API.WebTool — Browser Automation Tool

Playwright-based web automation with 9 tools exposed to the LLM:

Key features:

• Persistent browser profile — uses real Edge profile (cookies, logins, history)
• CDP fallback — connects to already-running Edge via Chrome DevTools Protocol
• Smart content extraction — CSS-hidden elements filtered, base64 images skipped, Angular custom elements handled
• Interactive element JSON — all inputs, buttons, selects, checkboxes, tabs serialized with metadata (XPath, state, attributes)
• PDF detection — auto-converts PDFs to Markdown via browser or HTTP

UISupportGeneric.API — Deterministic Tool Calling

// 1. Generate tool catalog from a C# class
var catalog = Analyzer.GeToolDefinitions(typeof(API.WebTool));

// 2. LLM responds with: {"method": "navigate", "url": "https://..."}
// 3. Execute deterministically
var result = API.CallMethod(typeof(API.WebTool), "Navigate", response, instance);

Features:

• Parameter validation against real method signatures
• Snake_case → PascalCase auto-conversion
• Default parameter support (optional params work seamlessly)
• JSON repair for common LLM formatting errors

Key Design Decisions

1. Persistent Browser, Not Headless

The agent uses the user's real Edge profile (--profile-directory=Default). This means:

• Already logged into Google, GitHub, etc.
• No CAPTCHA or bot detection issues
• Seamless "do it for me" UX

2. Markdown + Interactive JSON, Not Raw HTML

Page content is converted to structured Markdown with interactive elements as inline JSON:

(cookie banners auto-dismissed)

Dashboard
<<<INTERACTIVE::button::{"id":"submit-btn","type":"button","text":"Save","state":{"enabled":true},"path":"/html/body/..."}>>>

The LLM sees readable text AND machine-parseable interactive metadata.

3. One-Pass Content Filtering

Before conversion, a JS pass marks CSS-hidden elements with data-hidden attributes. The converter then skips them and drops unknown Angular tags via Bypass mode. Result: 2.8MB HTML → ~20KB usable Markdown.

4. Snake_Case Method Names

LLMs naturally produce fill_fields and web_search. The system converts these to C# PascalCase (FillFields, WebSearch) automatically. No prompt engineering needed.

Quick Start

using AIOrchestrator;

// Configure API key
Setup.DeepSeekApiKey = "sk-...";
Setup.ProviderConfig = ProviderConfigs.DeepSeek;

// Create LLM utility
var llm = new LLMUtility(LLMUtility.LLMProvider.DeepSeek);

// Send a query
var (response, _) = llm.SendQueryAsync("Explain quantum computing in 3 sentences.");
Console.WriteLine(response);

Web Agent Example

using AIOrchestrator;

// Non-streaming agent loop with attachments:
var orchestrator = new AgentHarness(LLMUtility.LLMProvider.DeepSeek);
var result = orchestrator.ExecuteAction(
    "Summarize the attached report",
    new[] { typeof(AIOrchestrator.API.WebTool), typeof(AIOrchestrator.API.FileTool) },
    attachments: new[] { new FileAttachment("report.pdf", File.ReadAllBytes("report.pdf")) });
Console.WriteLine(result.Message);

// Streaming chat (chunk by chunk, e.g. for TTS):
await foreach (var chunk in orchestrator.ExecuteChatStream("Tell me a story"))
    Console.Write(chunk);

Configuration

API keys can be set via Setup class or environment variables:

Setup.DeepSeekApiKey = "...";   // or env: DeepSeekApiKey
Setup.GeminiApiKey = "...";     // or env: GeminiApiKey
Setup.ZaiApiKey = "...";        // or env: ZaiApiKey
Setup.ProviderConfig = ProviderConfigs.DeepSeek;

Provider configuration — providers.json

LLM providers are configured in providers.json, copied next to the executable at build time (an embedded factory-default copy is used as fallback when the file is missing or corrupt — keep the two in sync when editing the repo copy). Adding a provider is a JSON edit, no code change required. The full reference — every field, its default, the cache semantics, the interaction mode and worked examples — is in docs/providers-config.md (shipped into the build output of every consumer, including AgentBridge). Each entry supports the following fields:

Agent interaction modes — API vs CLI

The tool classes (EMailTool, WebTool, ...) can be exposed to the LLM in two ways:

• API (default for large models) — every public method of every registered class is rendered as a JSON tool definition (GeToolDefinitions) and the model calls {"method": "get_emails", ...} directly.
• CLI (default for small models) — the model gets a single "execute shell command" tool plus the list of allowed commands (one per class). It drives the terminal as a command line: {"method": "cli", "command": "EMailTool get_emails --count 5"} — like git commit ... or docker container ls .... The terminal (UISupportGeneric.Terminal) supports -h/--help/-help/? help with per-subcommand detail, positional or named arguments (--count 5, --flag for booleans), and JSON/comma arrays.

When AgentInteractionMode is omitted the default follows the model size: CLI for models with a context window below 128 000 tokens, API for larger ones — so local small models get the CLI automatically and big cloud models keep the full API tool set. See API/API_interactionMode.md for the complete design.

CacheType drives how SendQuery handles the ephemeral nonCashableData (the dynamic context) and which blocks may be marked as cacheable:

• noCacheSupported — no client-manageable cache. Activates the dynamic-context mechanism: nonCashableData — per-call data that may change completely on every request (e.g. agent instructions, auto-context ToC) and is never stored in history — is injected at the start of the prompt. It provides a temporary context view that keeps the stored history compact; being outside the history it also sits outside any cached prefix (no cache benefit by design).
• PrefixCache — prefix-based cache: the provider serves repeated prompt prefixes from cache (Ollama/ExLlamaV2 KV cache in VRAM, DeepSeek/Z.ai context caching, Gemini implicit caching). The prompt prefix must stay stable, so nonCashableData is appended to the last user message instead of the system prompt, keeping the cached prefix intact.
• AnthropicCache — Anthropic prompt caching (Claude): stable blocks get a single cache_control breakpoint each (last system block, last tool), so repeated prefixes are served from cache at 10% of the normal token price. Anthropic allows at most 4 breakpoints per request. nonCashableData is never cached — it would invalidate the prefix.

The default when the field is omitted is PrefixCache. The factory presets assign PrefixCache to the local inference engines (Ollama, ExLlamaV2) and to the cloud APIs (DeepSeek, Z.ai, Gemini) — all have prefix-based caches; only DeepSeekBridge (a proxy to DeepSeek's web backend) stays noCacheSupported. To use an Anthropic provider you must also resolve its API key in Setup.ApiKey (only DeepSeek/Zai/Gemini have a key slot today).

Debug-only convenience (Z.ai key): in Debug builds, when Setup.ZaiApiKey is empty the library loads the key from a local, git-ignored file (see ZaiOcrConverter.EnsureApiKeyLoaded): zai_api_key.local at the repository root, or %USERPROFILE%\.aioffice\zai_api_key. Release builds never contain this fallback. An explicitly configured key always wins.

SMTP settings for email notifications:

Setup.SmtpServer = "smtp.example.com";
Setup.SmtpPort = 587;
Setup.SmtpUser = "user@example.com";
Setup.SmtpPassword = "...";

Persisting settings in headless hosts — Setup.Load / Setup.Save

Host apps that consume the library without a settings UI (console apps, AgentBridge) can persist Setup directly. AIOffice does not use these — its settings panel persists values itself (via UISupportGeneric).

Setup.Load();                      // at startup: reads the per-app settings file
Setup.SmtpServer = "...";          // configure as needed...
Setup.Save();                      // persist after configuration changes

• File: %LocalAppData%\{entry-assembly-name}\setup.json — one folder per host executable, so different apps never share credentials. DocumentsPath is not part of this file (it persists itself via rag_settings.json inside PersistentData/).
• Secrets (API keys, SMTP/IMAP passwords) are DPAPI-encrypted on Windows (bound to the current user); on other platforms they are stored as-is.
• The persisted ProviderName restores Setup.ProviderConfig (used for API-key resolution); omit it from the file if the host app selects the provider itself.

Debug/test setup — Setup.LoadDebugPreset (DEBUG builds only)

Test harnesses and scratch apps that must talk to real services (e.g. EMailTool reading a real mailbox) can preload developer credentials without touching the host app:

Setup.LoadDebugPreset();   // compiles only in DEBUG

reads debug_setup.json next to the executable (e.g. bin/Debug/net10.0/debug_setup.json), plain text, PascalCase property names:

{
  "ImapServer": "imap.gmail.com",
  "ImapPort": 993,
  "ImapUser": "me@gmail.com",
  "ImapPassword": "app-password",
  "SmtpServer": "smtp.gmail.com",
  "SmtpPort": 587,
  "SmtpUser": "me@gmail.com",
  "SmtpPassword": "app-password",
  "Email": "me@gmail.com"
}

Add **/debug_setup.json to .gitignore — the file is developer-local and must never reach source control. The method is compiled out of Release builds, so it cannot run in production.

Universal Tool System (UTS)

The agent's tools are plugins: compiled .NET assemblies (IAgentTool implementations) that hosts (AgentBridge, AIOffice, this library's consumers) load dynamically from their Tools/ folder — at startup and hot-added via a recursive filesystem watcher (30 s debounce). Plugins live in their own subdirectory (Tools/<Plugin>/, primary) or directly in Tools/ (both are supported); the host skips assemblies it already provides (e.g. AIOrchestrator.dll), so plugins never carry redundant host binaries. Tools run natively (no interpreters, no Docker/chroot — the sandbox is structural), are built from the deterministic UISupportGeneric reflection engine + the generative model, and are cross-platform (AnyCPU, RID-neutral).

• End-user guide (install/remove/hot-add in plain language): docs/universal-tool-system.md
• Developer contract (how to write a tool): API/AGENT_TOOLS_GUIDE.md
• Official tools: Graphene.DocumentTool, Graphene.SpreadsheetTool (NuGet, Graphene-Lab).

Requirements

• .NET 10
• Microsoft Edge (uses system-installed browser)
• Playwright (Microsoft.Playwright NuGet)
• One of: Ollama (local), DeepSeek API key, Z.ai API key, or Gemini API key

License

Proprietary — internal project.