UiGuideAgent.Desktop 0.7.0

UiGuide Agent

An AI help agent you add to any application with a user interface. It guides your users through the UI: it answers "how do I…?" with concrete steps (which button to press, what happens next), in the language of the question, starting from the screen the user is on.

• Server (UiGuideAgent.Server, UiGuideAgent.Server.Host, Docker image) — the agent, knowledge bundles, sessions and the connection to your OpenAI-compatible LLM (LiteLLM, vLLM, Ollama, OpenAI…). Your LLM, your key: UiGuide Agent does not include or pay for a model.
• Client SDKs — UiGuideAgent.Desktop (WPF / WinForms; generates the knowledge bundle at build), UiGuideAgent.Web (Blazor, Razor, .NET MAUI Blazor Hybrid) and ui-guide-agent.js / @ui-guide-agent/web (any web page).
• Companion — UiGuideAgent.Companion, command uiguide-companion: the assistant next to any other Windows application (Python / Qt / Tk, Java, Delphi, any exe), without changing it.
• Developer tool — UiGuideAgent.Cli, command uiguide: init, scan, record, bundle, test, doctor, key, update, remove.
• Logic graph — which control opens which window / page, with nothing written by you: read from your code at build (event handlers, XAML Click, MVVM Command="{Binding …}" / [RelayCommand], Python Qt / Tk connect / command=, web links and router calls — Blazor, Vue, React, Angular, Next, Nuxt, SvelteKit) and completed while your application is used (the SDKs report which window / page a control opened; the server uses an edge after 3 sightings). The server gives the LLM the shortest route from the user's screen, so the assistant copies real routes.
• Answer cache — a first question already answered for the same application version, language and screen is answered by the server at once, without the LLM; kept 180 days, never the question text (see "Answer cache").

Privacy: only the UI structure and labels leave your application — control types, labels, states and the open screens. Never field values, passwords, list / table / tree contents, window titles or screenshots. The filter runs inside your application. The knowledge bundle contains window / control names, UI texts and your descriptions, read from your source at build — never user data.

Quick start — desktop (WPF / WinForms)

1. Add the package — in your project folder: dotnet add package UiGuideAgent.Desktop, then build and run: a round button appears in the bottom-right corner of your main window — no code needed. The first build writes uiguide.json (the application id from the project's name; copied next to the executable) and uiguide.map.json; until the assistant has a server it answers "not configured yet" and the build warns UIG003. Your application works as before either way.
   • .NET Framework 4.8 with the default C# version: add <LangVersion>latest</LangVersion>, or call UiGuide.Attach(this); in your main window's constructor.
   • A login window shown before the main window gets the button too; it moves to the main window once that appears.
2. Connect a server, then rebuild:
   • On your computer: dotnet tool install -g UiGuideAgent.Server.Host, then uiguide-server. Its first start creates your settings, %LOCALAPPDATA%\ui-guide-agent\server.json (Linux / macOS: ~/.local/share/ui-guide-agent/server.json; kept when the server is updated): set your LLM there (Llm → BaseUrl, Model, optional EmbeddingModel for the answer cache; the key only as the environment variable Llm__ApiKey), start it again and check http://localhost:5180/api/v1/health → "llm": "ok". The next build finds it, writes its address in uiguide.json and registers the application (%LOCALAPPDATA%\ui-guide-agent\data\apps\<appId>.json with { "profile": { "name": "…", "description": "…", "supportContact": "…", "rules": [ … ] } }).
   • Your company's: dotnet tool install -g UiGuideAgent.Cli, then in the project folder uiguide init --server https://… (see "Server for a company network").
   • The Release build never carries an address of this computer: until your users have one, every build warns UIG003 (<WarningsAsErrors>UIG003</WarningsAsErrors> in the project makes it an error).
   • uiguide.json: server, appId; optional "language": "de-DE" (default: the UI culture), "autoStart": false (you start it from code — see API), "position": "bottom-left", "theme": { "accent": "#0b7a75", "background", "foreground", "userBubble", "botBubble", "font" } (colors #rgb / #rrggbb / #rrggbbaa), "texts": { "greeting": "…" } or per language { "de": { "greeting": "…" } } (keys: open — the round button, title, greeting, placeholder, send, stop, clear, close, thinking, queued, stopped, error…).
3. Ask: click the button and ask "How do I …?". Something wrong? uiguide doctor checks the whole setup and says what to do; the debug output has lines starting with UiGuide: — or set the environment variable UIGUIDE_LOG to a file path before starting the application, and the SDK writes there what it did (configuration, window, button shown or why not).
4. Teach it your application:
   • Name every control after what it does. The assistant sees only names and ids, so it cannot point users to a control without a name. Every build warns about each one — warning UIG001 with file and line — in XAML, WinForms designer files and controls created in C# code. Name it yourself, in the words a user would use: AutomationProperties.Name="Export orders" (in code AutomationProperties.SetName(button, "Export orders")), WinForms AccessibleName, or by its text / label. Names are never invented: a generated name that doesn't match what the button really does misleads the assistant. uiguide scan lists the same controls in the console (exit code 1 while any is left — for CI); --fix only copies the tooltip / hint / inner text you already wrote (check it says what the control does). <WarningsAsErrors>UIG001</WarningsAsErrors> makes the warnings errors; <NoWarn>UIG001</NoWarn> hides them.
   • Every build generates uiguide.bundle.json from your windows, controls and UI texts (XAML, WinForms designer, resource dictionaries, .resx). In uiguide.map.json you describe what the code cannot say — per control (by automationId): action, enabledWhen, opens, label; and flows (goal + steps with {controlId} placeholders). uiguide bundle --check for CI.
   • UiGuide.SetContext("orderOpen", true) for state the UI doesn't show (state, never user data).
   • Flows without writing them: uiguide record --goal "export the orders" starts your built application; do the task once and close it — the controls you used become a flow in uiguide.map.json ({"goal": …, "steps": ["{fileMenuItem}", "{exportMenuItem}"]}). Only control ids, names and types are recorded, never what you type or pick in a list. Other Windows applications: through the Companion (--run "python app.py").
5. Check the answers: write the questions your users ask in uiguide.tests.json with the names the answer must contain — { "tests": [ { "question": "How do I export the orders?", "screen": "main", "expect": ["File", "Export to CSV"] } ] } ("A|B" = either, "avoid" = must not contain, "language" per question) — and run uiguide test against the server and its LLM (--repeat 3 for stability; exit code 1 when an answer misses; never answered from the server's answer cache).

The uiguide tool: the build runs it from the package; for the commands above dotnet tool install -g UiGuideAgent.Cli (npm projects: npx uiguide).

API: UiGuide.Attach(window | form), Detach(), SetLanguage(…), SetToken(() => token) (signed-in user, server "auth": "External"), SetContext / RemoveContext / ClearContext, MapScreen(windowClass, screenId), IsAttached. Start it yourself (your code picks the window): "autoStart": false in uiguide.json, then UiGuide.Attach(window).

uiguide exit codes: 0 ok, 1 problems found (scan, bundle --check, doctor), 2 wrong usage; uiguide help <command> lists the options.

Quick start — web (Blazor, Razor, .NET MAUI Blazor Hybrid, any web page)

1. Server: step 2 above.
2. .NET: dotnet add package UiGuideAgent.Web, then build. Every build writes uiguide.json (when missing) and wwwroot/uiguide.config.json — what the page reads — and, when a UiGuide server answers on this computer, registers the application with the web pages allowed to use the server ("origins": https://0.0.0.1 for MAUI Blazor Hybrid, the launchSettings.json URLs for Blazor / ASP.NET). The Release build (dotnet publish) leaves an address of this computer out. Your company's server: uiguide init --server https://…. Web apps without .NET (React, Vue, Svelte, Angular, Next.js…): npm install -D @ui-guide-agent/web and one line in the bundler's configuration — Vite: plugins: [uiGuide()] (import uiGuide from '@ui-guide-agent/web/vite'), webpack: new UiGuidePlugin() (@ui-guide-agent/web/webpack), Next.js: export default withUiGuide(nextConfig) (@ui-guide-agent/web/next) plus import '@ui-guide-agent/web/auto' in instrumentation-client.ts; every build (and dev start) then does the same as above and Vite / webpack put the assistant in the page; no .NET needed (npm installs the uiguide tool of your computer). Without a bundler: ui-guide-agent.js from the release, next to the uiguide.config.json that uiguide init writes (dotnet tool install -g UiGuideAgent.Cli).
3. Every build also warns (UIG001) about controls without a name in .razor / .cshtml / .html and keeps wwwroot/uiguide.bundle.json up to date.
4. One line before </body> (wwwroot/index.html, App.razor, _Layout.cshtml):

   <script src="_content/UiGuideAgent.Web/ui-guide-agent.js"></script>
   

   Other web apps: <script src="ui-guide-agent.js"></script>. Everything else is in uiguide.json, the same keys as on the desktop: language (default: <html lang>, followed when it changes), position, theme, texts, key, "autoStart": false. Without a server address the page works and the assistant says it is not configured yet. From code: uiGuide.start({ server, appId, bundle, language, position, theme, texts, token }), uiGuide.setContext(key, value), uiGuide.setLanguage(…), uiGuide.setToken(() => token), uiGuide.stop() (Blazor: IJSRuntime.InvokeVoidAsync("uiGuide.setContext", "orderOpen", true)).
5. The page is read like a screen reader reads it (roles, labels — never values). Name controls the browser way: aria-label, <label for>, the button's text; a <div @onclick> / <span onClick> is reported because it is not a control. The UI map's screens get their route from @page "/…", from the router configuration (Vue Router, React Router, Angular routes) or from the folders (Next pages/ and app/…/page.tsx, Nuxt pages/, SvelteKit src/routes/…/+page.svelte).

Quick start — any other Windows application (Companion)

Python (PyQt / PySide / Tkinter), Java, Delphi, Electron or any exe, without changing it:

1. Server: step 2 above. The tools: dotnet tool install -g UiGuideAgent.Cli and dotnet tool install -g UiGuideAgent.Companion.
2. In the application's folder: uiguide init (writes uiguide.json), uiguide scan (Python Qt / Tk: setAccessibleName, label.setBuddy(control)), uiguide bundle.
3. uiguide-companion --config uiguide.json --run "python app.py" — starts the application and puts the assistant on its window; or --process python / --window "My App*" to attach to a running one. The Companion reads the window through UI Automation (the same privacy filter) and ends with the application it started.

Updating

When a newer version is out you are told, with the lines to run, in three places: as a build warning UIG002 in Visual Studio / dotnet build (projects with uiguide.json), after any uiguide command, and in the server log at start. The check runs at most once a day. To update:

1. the project — the line the notice gives: dotnet add package UiGuideAgent.Desktop / UiGuideAgent.Web, or npm install -D @ui-guide-agent/web@latest (uiguide update in the project folder does the same, and replaces a copied ui-guide-agent.js);
2. the tools: dotnet tool update -g UiGuideAgent.Cli (and UiGuideAgent.Companion); the server on your computer: dotnet tool update -g UiGuideAgent.Server.Host (your server.json and the data are kept); Docker: docker pull, then the same docker run.

What changed is in the release notes ("What's new"). Turn the check off on a machine with UIGUIDE_NO_UPDATE_CHECK=1 (it is also off when CI=true).

Removing it

• From one project — in its folder: uiguide remove takes out the package reference, uiguide.json, the generated bundle, the <script> line (web) and the registration on the local server; it keeps uiguide.map.json / uiguide.tests.json (yours — --all deletes them too). If your code still calls the SDK (UiGuide.SetContext, uiGuide.start, …) it changes nothing and lists those lines. --dry-run shows what it would do. With your company's server it also prints the one line for its administrator (app remove <appId>).
• From this computer: dotnet tool uninstall -g UiGuideAgent.Server.Host (and UiGuideAgent.Cli, UiGuideAgent.Companion); the server's settings and data are in %LOCALAPPDATA%\ui-guide-agent (Linux / macOS: ~/.local/share/ui-guide-agent) — delete the folder to remove them too. Run uiguide remove in your projects first.

Server for a company network (authentication, limits, metrics, Docker)

Where things run: the user's computer has only the library in your application (button, chat window, reading and highlighting controls — no prompt, no LLM key); the agent runs on the server (system prompt built into the server, completed per application with its knowledge bundle — sent by the application itself — and its registration), which calls your LLM. A registration (apps/<appId>.json) is a few lines, not code.

Who does what: the administrator installs the server once — docker pull ghcr.io/domitorai/ui-guide-agent-server, then the docker run below (container name uiguide); without Docker, dotnet tool install -g UiGuideAgent.Server.Host with "Urls" in its server.json set to the network address; the developer hands over nothing but the lines below. The developer runs uiguide init --server https://… in the project: it writes uiguide.json with an application key and prints one line for the administrator — docker exec uiguide dotnet uiguide-server.dll app set <appId> <base64> (without Docker: uiguide-server app set …) — which registers the application with its name, description, web pages, the key's hash and default limits; no file to edit, no restart; run again it keeps what the administrator changed. Likewise uiguide key → app add-key <appId> sha256:… and uiguide remove → app remove <appId> (registration, bundles, cached answers, learned navigation). On the server: app list; app clear-cache <appId> forgets an application's cached answers.

Learned navigation

The code reading cannot see every route (a window opened through a service or a plugin, a type chosen at run time). The SDKs fill the gap while your application is used: after the user activates a control and, within 3 seconds, another screen of the UI map opens, they note "this control of that screen opens this screen" and send these notes every 2 minutes (desktop; also when the assistant closes) or when the page is hidden (web) — POST /api/v1/apps/{appId}/navigation. What is sent: screen ids; the control's id and its label only when they are in your UI map (never e.g. a customer's name on a button or an id like order-5531-edit), never values or titles. The server keeps only notes whose screens and control are in the bundle (data/navigation/<appId>/<bundle>.json: ids, counts, dates and salted fingerprints of the reporters — no user name, no address) and uses an edge after 3 different users / computers reported it (UiGuide:Navigation:MinObservations; AppKey: the client's address, External: the signed-in user; on the server's own computer every report counts); at most 60 reports per hour per user / address and 5,000 per application (ReportsPerClientPerHour, ReportsPerAppPerHour, above → 429); an edge from the code always wins. Each edge is deleted 180 days after it was last seen (UiGuide:Navigation:RetentionDays) and with its bundle. Off, on the server: "learnNavigation": false in the application's registration, or UiGuide__Navigation__Learn=false for all — the clients then stop observing. Metrics: uiguide_navigation_edges_total. Desktop: WPF / WinForms windows of the application's own process, and with the Companion the other application's windows (controls from UI Automation events, or classic Win32 controls at the start of a mouse click; Tk learns nothing), also from context menus, double-click in lists and the default button; web: page changes through the router or a full page load (the click and an unsent batch are carried to the next page).

Answer cache

The first question of a conversation already answered for the same application version (bundle), language and screen state (active screen, open screens, the application's context) is answered by the server at once, without the LLM. Different applications and versions never share answers. Kept: only first questions answered without a screen capture; never the question text (its hash, and with Llm:EmbeddingModel its meaning as a vector — then questions in other words match too, cosine ≥ UiGuide:Cache:Similarity 0.92), nothing about the user. Where: data/cache/<appId>/<bundle>.jsonl on the server. Each answer is deleted 180 days after it was stored (UiGuide:Cache:RetentionDays), at most 2,000 per version (MaxEntries), and with its bundle (deleted 180 days after its last use, UiGuide:Bundles:RetentionDays). Off: UiGuide__Cache__Enabled=false, or "cache": false in an application's registration. uiguide test never uses it (Cache-Control: no-cache). Metrics: outcome cached.

On its own computer the server needs no authentication. From the network every application needs one of:

• Application key — uiguide init with a company server sets it up (on your own computer: uiguide key, which writes the local apps/<appId>.json): the server keeps only the key's SHA-256 hash, the project's uiguide.json gets "key" (a web page gets it at the next build, in uiguide.config.json). The key is inside your application, so it identifies the application, not the user — always set limits. Run it again to rotate (old keys stay valid until you remove them from "keys").
• External check — for signed-in users: "auth": "External", "validation": { "url": "https://your-app/api/me" }. The client sends the user's token (UiGuide.SetToken(() => session.AccessToken) on desktop, uiGuide.setToken(() => token) on the web); the server sends GET to your URL with it as Bearer: 2xx = valid (optional body { "user": "id" } names the user for the per-user limit), 401/403 = refused. Valid tokens are remembered cacheSeconds (default 300). The URL must be https (http only on the same computer); conversations are kept per user.

Limits per application: "limits": { "questionsPerHour": 1000, "questionsPerUserPerHour": 30, "maxConcurrent": 10 } (0 = none; the user is the one named by the external check, otherwise the client's IP address). Over a limit the client shows "busy, try again in …" (429 rate_limited). Only real questions count.

Metrics: GET /metrics (Prometheus) — questions per application and outcome, refusals per reason, answer durations, prompt tokens and LLM prefix-cache hits, answers running, the queue, conversations; also as the .NET meter UiGuideAgent.Server. Readable from the same computer; from the network only with Authorization: Bearer <UiGuide__MetricsKey>. The text of questions is never logged or counted.

Docker: docker pull ghcr.io/domitorai/ui-guide-agent-server:<version>; docker run -d --name uiguide -p 8080:8080 -v uiguide-data:/data -e Llm__BaseUrl=… -e Llm__Model=… -e Llm__ApiKey=… ghcr.io/domitorai/ui-guide-agent-server:<version> — or packaging/docker-compose.yml. Applications are in /data/apps/<appId>.json on the volume (written by the app … lines). Update: docker pull the new version, docker rm -f uiguide, the same docker run — the volume keeps everything. Behind a reverse proxy set UiGuide__TrustedProxies to the proxy's IP address(es) so the limits see the real client address; a request relayed by a proxy never counts as "same computer".

Limits

• Desktop SDK: Windows 10/11, WPF and WinForms, .NET 8+ or .NET Framework 4.8. The chat panel needs the Microsoft Edge WebView2 Runtime (if missing, the user is offered the download).
• Automatic start needs C# 9 (.NET 5+, or .NET Framework with LangVersion 9+); one window at a time (the main window — before it appears, the window shown first, e.g. a login — or the one you attach); while a modal dialog is open the button stays on the window below it; the button in a bottom corner (right or left), its icon and the header text white.
• The tool and the bundle generation at build need the .NET 8+ runtime on the build machine (also for .NET Framework projects). They read the project and the projects it references (ProjectReference, e.g. a shared library of windows): WPF .xaml, WinForms *.Designer.cs / .resx and controls created in C# code (new Button { … }, button.Content = …, AutomationProperties.SetName(…); read as text, in the same file); templates, third-party controls not named like the standard ones and controls built by helper methods in other files are not seen (the assistant still sees them on screen).
• Without authentication ("auth": "None", the default) the server answers only on its own computer; from the network: AppKey or External (above). Web pages: only the origins listed in the application's registration ("origins"; never a wildcard). Limits are counted in memory: a restart starts them again.
• Web SDK: modern browsers / WebView2; the page snapshot skips tables, lists (outside navigation, menus, toolbars — data-uiguide="ui" includes one), hidden elements and data-uiguide="ignore"; open shadow DOM is read. Web markup is scanned as text (.razor, .cshtml, .html, .vue, .svelte, .jsx, .tsx).
• Companion: Windows 10/11, .NET 8+ Desktop Runtime, WebView2 Runtime; one application window at a time; the application must expose UI Automation (Qt, Java Access Bridge, Win32, WPF, WinForms, Chromium do).
• Questions up to 2,000 characters; the panel keeps the last 100 messages; a conversation is kept 30 minutes after the last question, in the server's memory only; the agent remembers its last 10 questions and answers (above that, the 5 latest — dropped in one block so the LLM server's prefix cache keeps working); the screen state and captures are sent with the current question only, never kept.
• Panel texts built in for English only. The application's other languages (its resource files, "language"): the build has the server's LLM translate the panel's texts once and adds them to uiguide.json "texts" — correct them there, they are never overwritten; until then that language's panel is in English (warning UIG004). An application in one language without resource files: set "language". Answers follow the language of the question.
• Screen capture: at most 400 controls and 3 seconds; controls with neither an accessible name nor an id are left out. Knowledge bundle at most 2 MB, deleted after 180 days unused.
• Logic graph from the code (text / syntax only, nothing is compiled or run): C# event handlers (XAML Click, x.Click += …, lambdas, methods of the same class they call, up to 4 calls deep) and MVVM commands (Command="{Binding X}" → X = new RelayCommand(M) / DelegateCommand / [RelayCommand] M() / ReactiveUI ReactiveCommand.Create(M) → what M opens: a window, or a view model shown by its view — DataTemplate DataType or the name, SettingsViewModel → SettingsWindow / SettingsView; Caliburn.Micro x:Name / Message.Attach → the view model's method) that create one of your screens; Python Qt x.clicked.connect(…) / Tk command= (also over several lines; the method of the class that wires it) → a window class; web links and router calls with a literal route (href, to, routerLink, NavigateTo / navigate / router.push / goto, also in the method a click handler calls), router configuration with child routes, folder routing when package.json has next / nuxt / @sveltejs/kit. Also: Angular loadChildren, ReactiveUI BindCommand, WPF KeyBinding shortcuts (shown in the route), Python windows built in a method (tk.Toplevel / QDialog, named after their title; the Companion recognises Tk / Qt windows by the title written in the code, compared locally and never sent). Routes built at run time and windows chosen by reflection come from the learned navigation instead; "opens" in uiguide.map.json still wins. At most 40 routes per question.
• SetContext: at most 48 keys, text up to 200 characters, lists up to 50 items; openScreens / activeScreen are reserved.
• Answers are plain text.
• uiguide record: WPF / WinForms through the SDK (exact ids, menus with their parents); through the Companion only what the application reports to UI Automation (WinForms tool strips report nothing there). Drop-down list items and typed text are never recorded. Web: write flows in uiguide.map.json.
• uiguide test checks words in the answer, not its meaning; the LLM may phrase answers differently between runs — use --repeat.

Full step-by-step guide, troubleshooting and license: https://github.com/DomitorAI/ui-guide-agent

Free to use without limits, including commercially; ship the client libraries unmodified inside your own application (also one you sell). Selling UiGuide Agent itself, or offering it as a paid hosted service, needs the author's permission. Provided as is, without warranty — whether and how to use it in your project is your decision. Permissions, questions and versions tailored to a specific project: https://github.com/DomitorAI/ui-guide-agent