Standard.Agents.Decision.Brains.LlamaSharp 0.1.0

Standard.Agents.Decision.Brains.LlamaSharp

A local GGUF brain for The Standard AI Agent Framework.

Part of The Standard for Agents. In Agent = Orchestration(Data, Decision, Direction), this package supplies the Decision nature's brain — from a local GGUF model, with no HTTP and no API key. It is an optional adapter: it plugs into the core Standard.Agents package through .UseGenerator(...), and everything else — skills, tools, guardians, memory, knowledge, streaming — comes from the core, unchanged.

The core Standard.Agents package is deliberately dependency-free. The native weight (LLamaSharp / llama.cpp) lives only here, so you take the local-inference dependency only if you want it. Prefer a hosted endpoint? Use the core's .Brain(url, key, model) instead and skip this package entirely.

Install

dotnet add package Standard.Agents                               # the core framework
dotnet add package Standard.Agents.Decision.Brains.LlamaSharp    # this local brain
dotnet add package LLamaSharp.Backend.Cpu                        # or .Cuda12 / .Vulkan for GPU

Use

using Standard.Agents;
using Standard.Agents.Decision.Brains.LlamaSharp;

var agent = new StandardAgent()
    .UseGenerator(new LlamaSharpGeneratorBroker("path/to/model.gguf"));

string answer = await agent.ProcessPromptAsync("What is 47 * 89?");

That's the whole difference from a hosted agent — one line. LlamaSharpGeneratorBroker implements the core's IGeneratorBroker (both GenerateAsync and streaming GenerateStreamAsync), so the rest of the API is identical to Standard.Agents — add skills, tools, guardians, memory, knowledge, and streaming exactly as the core how-to describes:

var agent = new StandardAgent()
    .UseGenerator(new LlamaSharpGeneratorBroker("model.gguf"))   // local brain (this package)
    .Skills("Skills")                                           // everything below is core
    .Tool(new CalculatorTool())
    .Memory("agent-memory.txt")
    .Knowledge("Knowledge")
    .LogTo("log.txt");

Constructor options: contextSize, gpuLayerCount (0 = CPU), maxTokens, and promptTemplate (below). You supply the .gguf model and a backend package; llama.cpp does the rest, entirely on your machine.

Choosing the prompt template (important)

An instruct model only generates when it's given its own chat format. If generation comes back empty, the template is almost always the reason. The default is ChatML (Qwen and many modern GGUFs); switch it to match your model:

using Standard.Agents.Decision.Brains.LlamaSharp;

// ChatML (default) — Qwen, many modern instruct models
new LlamaSharpGeneratorBroker("model.gguf");

// Llama 3 instruct
new LlamaSharpGeneratorBroker("model.gguf", promptTemplate: PromptTemplates.Llama3);

// NVIDIA Nemotron
new LlamaSharpGeneratorBroker("model.gguf", promptTemplate: PromptTemplates.Nemotron);

// Base / completion model
new LlamaSharpGeneratorBroker("model.gguf", promptTemplate: PromptTemplates.Plain);

// Your own — any (systemPrompt, userPrompt) => prompt
new LlamaSharpGeneratorBroker("model.gguf",
    promptTemplate: (system, user) => $"<s>[INST] {system}\n{user} [/INST]");   // Mistral

Backend setup & troubleshooting

RuntimeError: The native library cannot be correctly loaded is a backend problem, not a model or adapter one:

• Exactly one backend — never install LLamaSharp.Backend.Cpu and .Cuda12 together; they fight over the native llama library. CPU-only box → keep only .Cpu.
• Put the backend in the app that runs, not just a class library — a Backend.* reference on a library doesn't reliably copy its native DLLs into the executable's output. Verify llama.dll / ggml*.dll sit next to your .exe.
• Match versions — LLamaSharp and the Backend must be the same version.
• Force CPU and see the load attempts, at the top of Main:

  LLama.Native.NativeLibraryConfig.All
      .WithCuda(false)
      .WithLogCallback((level, msg) => Console.Write(msg));
  

Where this fits

License

Licensed under the The Standard Software License (TSSL) v1.1 — the same license as Standard.Agents.

LLamaSharp and llama.cpp are MIT-licensed and consumed as NuGet dependencies. The GGUF model weights you supply carry their own licenses (Llama, Gemma, Nemotron, …) — you are responsible for using them within those terms.