SharpMind.Core 1.0.4

<p align="center"> <img src="SharpMind.Core/sharpmind_logo.svg" alt="SharpMind logo" width="512" height="512"/> </p>

<p align="center"><b>SharpMind. A pure C# / .NET LLM engine — inference and agent tooling in one solution.</b></p>

<p> <img alt="status" src="https://img.shields.io/github/v/release/Integral2u/SharpMind?label=status&color=brightgreen"> <img alt="lang" src="https://img.shields.io/badge/language-C%23%20(.NET%2010)-239120"> <img alt="deps" src="https://img.shields.io/badge/dependencies-near--zero-blue"> </p>

What is SharpMind?

SharpMind is an end-to-end LLM stack written entirely in C#, with no dependency on llama.cpp, PyTorch, or any native runtime for its core path. It loads GGUF models, runs quantized CPU/GPU inference with modern decoding acceleration (speculative + Medusa-style drafting), and — unusually for a C# inference engine — also includes its own autograd engine so you can fine-tune (LoRA), distill, and prune models in the same process that serves them.

It ships as a set of composable libraries plus a terminal chat application (SharpMind.CUI) built on top of them.

Chat / conversation view	Model & session welcome view
Runtime options (hardware tier, load mode, sampling)

Why SharpMind

• No native runtime required. Tensor math, quantization kernels, and the training loop are all managed C#. GPU acceleration (via ILGPU) is opt-in and lives in its own assembly — the CPU path never needs it.
• Runs models bigger than your RAM. A disk-streaming load mode pages transformer layers in and out during the forward pass instead of holding the whole model resident (details below).
• Modern decoding, not just greedy/top-p. Both classic speculative decoding and Medusa-style multi-head speculative decoding are implemented from scratch, with careful KV-cache rollback on rejection.
• A genuinely pluggable kernel system. Hardware-specific kernel variants (scalar/SSE/AVX2/FMA/GPU) are wired up at runtime through a small internal dispatch layer JigSawDotNet rather than hand-written if/switch ladders — adding a new backend means adding an assembly, not editing the core.
• Agent tooling included. Tool-calling, permission gating (Never / Ask / Always), and sub-agent orchestration ship in SharpMind.Inference.Agent.
• Inference and training in one codebase. Most C# "LLM" libraries are thin bindings around llama.cpp and only run models. SharpMind can load a GGUF checkpoint, chat with it, or train a model from scratch. LoRA fine-tuning and distillation are also present in the training stack but are still experimental — see Training.

Install the chat app (Windows)

The quickest way to try the terminal chat client is the installer, which sets up SharpMind.CUI, Start Menu + desktop shortcuts, and the app folder:

Download SharpMind Console Setup (MSI)

Requires the .NET 10 Runtime. Running the MSI installs the app and shortcuts; uninstall or repair is available via Apps & Features. Then grab a model as described in Quick Start — the running app also has a built-in model browser.

Quick Start: Load and Run a Model

This walks through the smallest possible program that loads a GGUF model and starts an interactive chat session.

1. Get a model

Download a small instruct model to get started — Qwen3-0.6B-Q8_0 is a good first choice: it's under 1GB, loads quickly, and is known to produce coherent output out of the box.

Place the .gguf file in a folder, e.g. C:\Models\Qwen3-0.6B-Q8_0.gguf.

2. Minimal Program.cs

using SharpMind.Core.Quantization;
using SharpMind.Inference;
using SharpMind.Inference.Chat;
using SharpMind.Model;
using SharpMind.Model.Config;
using SharpMind.Model.Format;
using SharpMind.Tokenization;

var modelPath = @"C:\Models\Qwen3-0.6B-Q8_0.gguf";

// 1. Load model metadata, config, and tokenizer from the GGUF file.
var metaHelper = ModelFormatHelpers.GetModelMetaHelperFor(ModelFormat.Gguf);
metaHelper.Load(modelPath, null, out ModelMetaData meta, out ModelConfig modelConfig, out Tokenizer? tokenizer);

if (tokenizer == null)
{
    Console.WriteLine("No tokenizer data found in this GGUF file.");
    return;
}

// 2. Resolve hardware/quant mapping and load the weights.
var sharpConfig = modelConfig.ForModel();
var qOps = QuantizationFactory.Create(sharpConfig.ResolvedHardware);

using var weights = ModelFactory.CreateWeights(modelConfig, sharpConfig, qOps, modelPath, LoadMode.Full);
weights.InitializeWeights();

// 3. Build the transformer.
using var model = ModelFactory.CreateTransformer(weights, sharpConfig);

// 4. Start a chat session.
await using var session = new ChatSession<StandardGeneratorBuilder<KVCacherBuilder>, KVCacherBuilder>(model, tokenizer, meta)
{
    MaxTokens = 512,
    Temperature = 0.7f,
    TopK = 40,
    TopP = 0.9f,
};
session.InitializeChat();

Console.WriteLine("Chat ready! Type a message (or 'exit' to quit).\n");
var cts = new CancellationTokenSource();

await session.StartChatAsync(Prompt, Response, cts.Token);

async Task<ChatMessage> Prompt()
{
    Console.Write("\nYou: ");
    var input = Console.ReadLine() ?? "exit";
    if (input == "exit") cts.Cancel();
    return new ChatMessage { Content = input, Role = ChatRole.User };
}

void Response(ChatStreamEntry entry) => Console.Write(entry.Token);

What each step does

Notes

• Temperature = 0.7f, TopK = 40, TopP = 0.9f give more natural, varied output than greedy decoding (Temperature = 0). Set Temperature = 0 for deterministic, reproducible output when debugging.
• MaxTokens caps the response length per turn — lower it (e.g. 128) on slower hardware if you don't need long responses.
• On memory-constrained machines, prefer LoadMode.Streaming over LoadMode.Full, and stick to Q4–Q8 quantized models under ~1B–3B params for reasonable throughput.

Compatibility matrix

Validated across two independent runs against 15 model/architecture/quantization combinations. "Runs clean" means the model loaded, built a transformer, and completed all five benchmark prompts with no exceptions or crashes — it is not a quality claim; output coherence varies a lot by model size and quant level, which is expected and not specific to SharpMind.

Tested on: AMD Ryzen 3 2200U (2C/4T, 2.5GHz base) w/ Radeon Vega Mobile Graphics, 12GB RAM — a modest mobile/laptop-class chip, not a workstation. Load times and throughput scale heavily with hardware and quant level — as a rough sense of range on this machine, SmolLM-135M.Q4_K_M loaded in ~2s, while qwen2.5-1.5b-instruct-q8_0 (the largest model tested) took ~2 minutes to load and initialize. That everything above ran clean on a 2-core/4-thread laptop CPU is itself a reasonable data point for SharpMind's baseline hardware requirements — run your own copy of the benchmark against your target hardware before relying on these numbers for capacity planning.

Extensibility: plugins

Beyond JigSaw's compile-time kernel dispatch, SharpMind.CUI has a separate, simpler runtime plugin loader for extending the app itself without touching the core libraries. Drop a .dll into the app's Plugins/ folder and PluginLoader.LoadFrom will scan it and wire up anything it recognizes:

• Tools — any class with a method tagged [ToolDesc("...")] (parameters can carry their own [ToolDesc] too, for per-argument descriptions) is picked up as an agent tool automatically, the same mechanism the built-in EchoTool, FileSystemTool, and WeatherTool use:

  public class WeatherTool
  {
      [ToolDesc("Gets the current weather for a specified city.")]
      public async Task<string> GetCurrentWeather([ToolDesc("The name of the city.")] string city) { ... }
  }
  

• Context compactors — classes implementing IContextCompactor register under their own Name and become selectable alongside the built-in summarizing/truncating compactors.

• Prompt pre/post-processors — IPromptPreProcessor / IPromptPostProcessor implementations get to rewrite a prompt before it's sent or a completion after it comes back.

• Generators — any type implementing IGeneratorBuilder<TCache> is discovered by matching the open generic interface via reflection and added to the generator-strategy list next to Standard/Speculative/Medusa — a third-party decoding strategy can appear in the same options menu as the built-in ones.

The loader is defensive by design: each .dll loads independently (a failure is recorded as a warning, not a crash), duplicate names are rejected rather than silently overwritten, and only concrete classes with a public parameterless constructor are considered. Today this loader is wired into SharpMind.CUI specifically; nothing about the interfaces is CUI-specific, so the same plugin assemblies work if you host the inference/chat libraries directly.

Streaming model loading (LoadMode.Streaming)

Normally (LoadMode.Full) every transformer layer's weights are allocated and loaded up front. LoadMode.Streaming instead:

1. Memory-maps the GGUF file and reads only tensor metadata (offsets, shapes, dtypes) at load time — no weight bytes are touched yet.
2. Before each layer runs in the forward pass, EnsureLayerLoadedSync blocks (if needed) on that layer's data being read from disk.
3. While the current layer computes, the next layer is prefetched on a background task (PreloadLayerAsync), overlapping I/O with compute.
4. Once a layer has been consumed, its weights are released (FreeLayer) — only the current and next layer's weights are ever resident at once.
5. CompleteForward sweeps any remaining resident layers at the end of a pass so memory doesn't creep up across tokens.

The net effect: a model whose full weights don't fit in RAM can still run, trading some throughput for a small, roughly constant memory footprint (current + next layer, rather than all layers). The quantized LM head is also read directly from its raw on-disk bytes in streaming mode rather than materialized as a float tensor, cutting one of the largest single allocations for typical vocab sizes.

Inference deep dive

Decoding strategies

SharpMind ships three interchangeable generators behind a common IGenerator<T> / IGeneratorBuilder interface, so switching strategy is a builder call, not a rewrite:

Medusa in more detail, since it's the more novel of the two: each decoding round, the LM head's own greedy pick becomes token₀, and K trained head projections from the same hidden state produce token₁ … token_K. That draft of length K+1 is run through the model as one batch. Verification then walks the draft left to right — token₀ is always accepted (it's the model's own choice), and each subsequent token is accepted only if the model's forward pass agrees with the head's guess; the walk stops at the first disagreement. If every token in the draft is accepted, a bonus token is generated for free before the next round starts. On partial acceptance, the KV cache is trimmed back to the last accepted position so generation is bit-for-bit identical to plain greedy decoding — Medusa can only change throughput, never correctness. In the ideal case, with K=3 well-calibrated heads, this gives up to a ~2.5× reduction in forward passes per token; today the heads are randomly initialized and need MedusaHeads.Calibrate to be run before that speedup materializes.

Speculative decoding follows the more familiar draft-and-verify pattern with an independent draft model, defaulting to 4 draft tokens per round, and shares the same accept/rollback discipline over the KV cache.

Quantization

Full GGUF-style quant coverage — Q2_K through Q8_0/Q8_1/Q8_K, plus the classic block types (Q4_0/Q4_1/Q5_0/Q5_1) and several 1-bit/ternary formats (IQ1_S, IQ1_M, TQ1_0, TQ2_0) — each with scalar, SSE, AVX2, and FMA kernel variants, and GPU kernels for the most common types.

The "JigSaw" dispatch mechanism

Most inference engines pick a kernel implementation with a big switch over CPU features, duplicated at every call site. SharpMind instead defines each swappable operation (a vec-dot, a quantized matmul, an activation, a norm, an optimizer step, …) once as an abstract method on a small "ops" class, decorated with a [PuzzleCornerPiece] attribute that lists the concrete method for each hardware variant:

[PuzzleCornerPiece(QuantizationKeys.KeyVecDotQ4K, true, null,
    "q4k_fma",    $"{NS}.{nameof(QuantizationKernels.VecDotQ4K_FMA)}",
    "q4k_avx2",   $"{NS}.{nameof(QuantizationKernels.VecDotQ4K_AVX2)}",
    "q4k_sse",    $"{NS}.{nameof(QuantizationKernels.VecDotQ4K_Scalar)}",
    "q4k_scalar", $"{NS}.{nameof(QuantizationKernels.VecDotQ4K_Scalar)}")]
public abstract unsafe float VecDotQ4K(float* input, byte* rawWeights, int col, int inFeatures);

At startup, a MappingBuilder inspects the detected HardwareTier (or an explicit override) and the active SharpMindConfig (activation, attention, gating, quantization scheme, etc.) and produces a Dictionary<string,string> mapping each operation key to the variant name it should use — "q4k_fma", "gpu", and so on. Assembler.CreateInstance<QuantizationOps>(mapping) then builds a concrete implementation of the abstract class at runtime, resolving every abstract method straight to its chosen static kernel. The result is cached by a hash of the mapping, so a given hardware/config combination only pays the assembly cost once.

The part that makes this genuinely extensible rather than just "reflection instead of a switch": other assemblies can contribute additional variants for an existing key without the core project referencing them. SharpMind.GPU declares its own [PuzzlePeice] entries against the same keys (KeyVecDotQ4_0, KeyQuantizedMatMulQ4K, …) pointing at ILGPU-backed kernels. JigSaw discovers these via assembly scanning at startup, so calling WithGpu() — which just causes SharpMind.GPU to be loaded into the process — is enough for GPU variants to become selectable, with no compile-time dependency from SharpMind.Core on the GPU project at all. Adding a future Metal, Vulkan, or SIMD-width-specific backend is the same pattern: a new assembly, new [PuzzlePeice] entries, zero changes to existing call sites.

Training

This half of SharpMind is functional today but earlier in its lifecycle than inference — expect the fastest churn here.

• Autograd (SharpMind.Training/Autograd) — a from-scratch gradient engine (ForwardContext, BlockContext, Gradients) underpinning the training loop.
• Optimizers & schedulers — including AdamW, gradient norm, and LR scheduling, each also dispatched through the JigSaw mapping system so training kernels get the same hardware-tier treatment as inference kernels.
• LoRA (SharpMind.Training/LoRA) — low-rank adapters over attention and FFN layers for parameter-efficient fine-tuning.
• ModelSizer — given a data source, samples it, trains a throwaway tokenizer, and grid-searches architecture hyperparameters under a SizingBudget/SizingConstraints to recommend a model configuration that fits a target parameter budget — a small AutoML step for "how big a model should I even train on this data."
• Synthetic data (SharpMind.Data/Sources/PseudoLanguage) — a generated toy-language pipeline (morphemes, vocabulary, configurable complexity) for exercising the tokenizer/training pipeline without needing a real corpus.
• Data sources — CSV, JSONL, plain text, HuggingFace datasets-server streaming (dependency-free, via HttpClient + System.Text.Json), and a composable cleaning Pipeline with branch/merge nodes.

Expect the training API surface (config records, trainer entry points) to change as this matures.

Training sample with actual data

SharpMind.Samples/Training/Acutal/ ships a complete, reproducible end-to-end training run on real text:

Run SmmRealTextExample to reproduce the full pipeline: BPE tokenizer training → data loading → finite-difference training → .SMM export → reload → greedy/sampled/chat inference. The model is deliberately tiny (finite-difference gradients are O(parameters) forwards per step), but it produces recognisably language-like output and exercises every stage of the training-to-inference path.

To load the pre-trained checkpoint directly in the CUI, point it at shakespeare.smm or use SmmTrainingPipeline.LoadForInference in code. The chat formatter is auto-resolved from the ChatML-style Jinja template embedded in the .SMM file.

IChatClient integration (Microsoft.Extensions.AI)

SharpMind.Extensions.AI bridges SharpMind into the standard .NET AI ecosystem by wrapping an IChatSession behind an IChatClient. This means any library that accepts IChatClient — semantic-kernel planners, ASP.NET minimal APIs, MAUI apps, or your own code — can use a SharpMind-backed model without knowing anything about GGUF, KV caches, or prompt formatters.

What it does

Quick start

using Microsoft.Extensions.AI;
using SharpMind.Core.Quantization;
using SharpMind.Extensions.AI;
using SharpMind.Inference;
using SharpMind.Inference.Agent;
using SharpMind.Inference.Chat;
using SharpMind.Model;
using SharpMind.Model.Config;
using SharpMind.Model.Format;
using SharpMind.Tokenization;

var modelPath = @"C:\Models\Qwen3-0.6B-Q8_0.gguf";

// 1. Load the model (same as the basic Quick Start).
var metaHelper = ModelFormatHelpers.GetModelMetaHelperFor(ModelFormat.Gguf);
metaHelper.Load(modelPath, null, out ModelMetaData meta, out ModelConfig modelConfig, out Tokenizer? tokenizer);
var sharpConfig = modelConfig.ForModel();
var qOps = QuantizationFactory.Create(sharpConfig.ResolvedHardware);
using var weights = ModelFactory.CreateWeights(modelConfig, sharpConfig, qOps, modelPath, LoadMode.Full);
weights.InitializeWeights();
using var model = ModelFactory.CreateTransformer(weights, sharpConfig);

// 2. Create a SharpMind chat session (with optional agent builder for tools).
var agentBuilder = new AgentBuilder("MyAgent");
using IChatSession session = ChatSessionFactory.CreateChatSession(
    typeof(StandardGeneratorBuilder<KVCacherBuilder>),
    typeof(KVCacherBuilder),
    model, tokenizer!, meta,
    agentBuilder: agentBuilder);

// 3. Wrap it in the IChatClient adapter.
await using var client = new SharpMindChatClient(session, agentBuilder);

// 4. Use the standard IChatClient API.
var messages = new List<ChatMessage>
{
    new(ChatRole.System, "You are a helpful assistant."),
    new(ChatRole.User, "What is 2 + 2?"),
};

// Single-shot response.
ChatResponse response = await client.GetResponseAsync(messages);
Console.WriteLine(response.Text);

// Streaming response.
await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync(messages))
    Console.Write(update.Text);

Using MEAI tools with SharpMind

Tools defined as AIFunction instances are automatically routed into SharpMind's agent loop when passed via ChatOptions.Tools:

using Microsoft.Extensions.AI;

// Define a tool using the MEAI factory.
var getWeather = AIFunctionFactory.Create(
    (string city) => $"72°F and sunny in {city}",
    name: "GetWeather",
    description: "Get the current weather for a city.");

// The adapter registers it with the agent builder on first use.
var response = await client.GetResponseAsync(
    new List<ChatMessage>
    {
        new(ChatRole.User, "What's the weather in London?")
    },
    new ChatOptions { Tools = [getWeather] });

Console.WriteLine(response.Text);

The tool executes inside SharpMind's existing tool-loop infrastructure — no separate execution path, no extra dependencies beyond Microsoft.Extensions.AI.Abstractions.

Installation

SharpMind.Extensions.AI is a separate project/package. Add it alongside your SharpMind references:

<ProjectReference Include="..\SharpMind.Extensions.AI\SharpMind.Extensions.AI.csproj" />

Or as a NuGet package (once published):

dotnet add package SharpMind.Extensions.AI

The only additional dependency is Microsoft.Extensions.AI.Abstractions — no full Microsoft.Extensions.AI or Microsoft.Extensions.DependencyInjection required.

Also included

• Agent framework (SharpMind.Inference.Agent) — tool-calling with a three-state permission model (Never / Ask / Always), tool categories, and auto-named sub-agents (temperature → a "Greek tier" naming scheme, e.g. Athena-Alpha at low temperature, Prometheus-Epsilon at high).
• Chat layer (SharpMind.Inference.Chat) — pluggable prompt formatters (ChatML, a small Jinja-template evaluator, a simple formatter), pinned-message-aware context compaction (summarizing or truncating), and a ChatArtifact concept for attaching text/image/code/JSON blocks to a response.
• SharpMind.CUI — a full terminal chat client: model browser, session manager, settings, file picker, plugin loading, and a permission gate UI, shown above.
• SharpMind.GPU — ILGPU-backed kernels for activations, norms, and quantized ops, isolated from the core so the CPU path has zero GPU dependency.
• SharpMind.Extensions.AI — IChatClient adapter for the Microsoft.Extensions.AI ecosystem. Wraps any SharpMind IChatSession into a standard IChatClient, routes MEAI tools through SharpMind's agent loop, and maps chat types bidirectionally — see IChatClient integration.
• SharpMind.Extensions.Tools — optional common tools (grep, git, datetime) packaged as a plugin DLL. Auto-discovered from the CUI's plugins/ folder at runtime — no compile-time dependency required. The CUI build copies it there automatically.
• SharpMind.Benchmarks — evaluation kernels for measuring model/generator performance.

Project layout

SharpMind.Core          Zero-dependency tensor primitives, quantization, activations, memory pooling, MemoryHelpers/BigArray
SharpMind.Model         Architectures, layers, GGUF loading, model config
SharpMind.Inference     Generators (standard/speculative/Medusa), chat, agents, sampling
SharpMind.Training      Autograd, optimizers, LoRA
SharpMind.Tokenization  BPE tokenizer, vocab, serialization
SharpMind.Data          Data sources, cleaning pipeline, batching
SharpMind.Data.Parquet  Parquet data source
SharpMind.GPU           ILGPU-backed GPU kernels (optional)
SharpMind.CUI           Terminal chat application
SharpMind.Extensions.AI Microsoft.Extensions.AI IChatClient adapter
SharpMind.Extensions.Tools Optional common tools (grep, git, datetime) — plugin DLL
SharpMind.Samples       Example programs + training sample data (Shakespeare corpus, checkpoint, job config)
SharpMind.Benchmarks    Evaluation harness
SharpMind.Tests         Test suite

Status & roadmap

See CHANGELOG.md for release history.

Wishlist (not ordered)

• AVX512 Kernels
• Additional Model Support
• Optimiations
• Microsoft IChatClient and or other services. — shipped as SharpMind.Extensions.AI
• Common tools, GREP, GIT etc — shipped as SharpMind.Extensions.Tools
• Limit breaker(Project Goku), int.MaxValue element-count limit workaround. Solutions not excuses. — shipped as MemoryHelpers, BigArray<T>, IWorkspace, BigWorkspace

Issues, questions, and early feedback are welcome.