Fuuga.Server.cpu 1.0.41

Fuuga.Server

OpenAI and Anthropic compatible HTTP server for Fuuga -- the F# LLM framework.

Built with Oxpecker on ASP.NET Core / Kestrel.

Installation

Clone the repo and run locally with dotnet run. The NuGet package exceeds the 250 MB size limit for .NET tools due to native runtime dependencies (libtorch, CUDA), so dotnet tool install is not supported.

git clone https://bitbucket.org/Thorium/fuuga.git
cd fuuga/Fuuga.Server
dotnet run -- --checkpoint ../checkpoints/best --tokenizer ../data/tokenizer --port 8080

Prerequisites: .NET 10 SDK (v10.0.103+).

Usage

fuuga-serve --checkpoint <dir> --tokenizer <dir> [options]
fuuga-serve --package <path.fuuga> [options]
fuuga-serve --decision-model <dir> [--decide-* options]

Examples

# Serve from a checkpoint directory:
fuuga-serve --checkpoint checkpoints/step-5000 --tokenizer data/tokenizer --port 8080

# Serve from a .fuuga package:
fuuga-serve --package my-model.fuuga --port 8080

# With API key authentication and request tracing:
fuuga-serve --package my-model.fuuga --api-key secret123 --observe

# CPU offloading to reduce VRAM:
fuuga-serve --package my-model.fuuga --offload cpu --gpu-layers 12

# Typed decisions only (POST /v1/decide), e.g. an imported Laya; no generative model loaded:
fuuga-serve --decision-model models/laya-fuuga --decide-calibration models/laya-fuuga/calibration.json

Options

A question whose options do not fit the decision model's input is the client's error: 400 on /v1/decide, 422 naming questions.<id>.criteria on /v1/systemone. A /v1/systemone answer carries Jev's confidence (0 for an even split, 1 for certainty); /v1/decide reports the top probability instead, so a threshold carried over from Jev means less there.

API Endpoints

The /admin/* routes exist only when --provider-config is set.

Features

• API compatibility -- serve the OpenAI, Anthropic and Jev (typed decisions) formats side by side, or choose with --api-compat.
• SSE streaming -- token-by-token streaming for chat and text completions.
• MCP tool routing -- connect external tools via the Model Context Protocol for function calling.
• A2A protocol -- Agent-to-Agent task delegation and status tracking.
• Guard rails -- prompt injection detection, PII masking, and content filtering.
• Continuous batching (opt-in) -- with --continuous-batching, concurrent requests are served through the ContinuousBatching scheduler over a shared paged KV cache (BatchedEngine), interleaving token-by-token instead of running one-at-a-time behind a lock. See Serving modes.
• Persistent session memory (opt-in) -- with --memory <dir>, completed turns and pinned notes are remembered per session and relevant ones recalled into later prompts. See Persistent session memory.
• Microsoft.Extensions.AI -- implements IChatClient and IEmbeddingGenerator for ecosystem integration.

Serving modes

By default the server processes one inference at a time behind a lock; concurrent requests queue. The startup banner reports Serving mode: sequential.

Pass --continuous-batching to serve concurrent requests through the continuous-batching engine: requests share one paged KV cache and interleave through the scheduler, so a slow long generation no longer blocks short ones. The banner reports Serving mode: continuous batching.

fuuga-serve --checkpoint checkpoints/best --tokenizer data/tokenizer --continuous-batching --port 8080

Notes and current limits (the mode is experimental):

• Sampling and guardrails apply as usual (per-request temperature / top-k / top-p / repetition penalty; input/output guardrails when --guardrails is set).
• --verify self-verification, consensus voting, and tool-calling are bypassed in this mode — they don't fit token-level batching.
• Compute is per-sequence inside the forward pass (the scheduling interleaves; a single batched-matmul paged kernel over the whole batch is future work), so the win is latency/fairness under concurrency rather than raw single-stream throughput.
• KV-cache pool is memory-budgeted. Because the paged cache pre-allocates its blocks up front, the pool is sized to a memory budget — auto-detected by default (a fraction of free VRAM on CUDA, measured after the model loads, or of system RAM on CPU) — and the concurrent batch size is derived to fit. Override with --cb-cache-gb <G> (or FUUGA_CB_CACHE_GB). The chosen sizing is printed at startup; if only one sequence fits at your context length, a warning suggests raising the budget.

Persistent session memory

OpenAI and Anthropic clients resend the whole conversation on every request, so within one conversation the server needs no memory. What it lacks by default is memory across conversations and restarts. --memory <dir> adds it; without the flag nothing is recorded and nothing changes.

fuuga-serve --checkpoint checkpoints/best --tokenizer data/tokenizer --memory ./memory

• What is remembered. Each completed /v1/chat/completions or /v1/messages turn: the conversation's last user message and the final answer (each cut to 1000 characters). Turns that end in tool calls are not recorded (that is an agent mid-task, not an answer), and an answer the output guardrails block is never stored. You can also pin notes with POST /v1/memory {"text": "..."} — facts that should always be in context. Notes are capped at 2000 characters and, under --guardrails, pass the input checks like user text (a blocked note is refused with 400 guardrails_violation), since they reach every later prompt of the session as a system message.

• What is recalled. Every pinned note, plus the --memory-top-k past exchanges most relevant to the new user message (BM25 keyword relevance; exchanges sharing no word with it are left out). They go into the prompt as a system message after the client's own system prompt, within a 4000-character budget. Memory yields to the conversation: if the final prompt (including any --rag context) would leave too little room for the answer, the request goes out without memory instead of failing with context_length_exceeded. Recall is lexical, so it runs no model forward pass.

• Sessions. The X-Fuuga-Session: <name> header selects the memory session (default when absent), so one server can keep separate memories per user or project. Under --provider-config sessions are additionally namespaced by the caller's API key, so one customer can never read another's memory. Each session is one JSON Lines file in <dir>, named by a hash (session names and keys never appear in paths).

• Per request. X-Fuuga-Memory: read recalls without recording (useful for CI jobs or one-off questions that should not teach the server); X-Fuuga-Memory: off does neither. Any other value is a 400, so a typo cannot silently record a request.

• Managing it. GET /v1/memory lists the session, DELETE /v1/memory/{id} forgets one entry, DELETE /v1/memory forgets the session. These act on the caller's own session only and sit behind the same --api-key / billing auth as the rest of /v1.

• Size. Sessions keep at most --memory-max-entries entries (default 1000); the oldest exchanges are evicted before any note.

• One server per directory. Each server caches sessions in memory and rewrites the files from that cache, so the directory is locked (.fuuga-memory.lock); a second fuuga-serve pointed at it refuses to start.

Access control. Session names are labels, not secrets: anyone who can call /v1 can read, add to or clear any session by naming it. Without --api-key that is anyone who can reach the port (the server warns at startup). Treat the memory directory like the conversations themselves, since it holds user text in plain JSON. For several users, give each an API key under --provider-config (sessions are then isolated per key), or run one server per user.

Semantic response cache

A support queue or an FAQ bot answers the same questions in different words all day. --semantic-cache <dir> lets a chat request that asks what an earlier one asked get the earlier answer without a generation (X-Fuuga-Cache-Result: hit on the response). Without the flag nothing is cached.

fuuga-serve --checkpoint checkpoints/best --tokenizer data/tokenizer \
  --semantic-cache ./cache --decision-model models/laya-fuuga

How a hit is decided, in two stages:

1. Candidates by embedding. The final user message is embedded (the vector RAG retrieval uses too, computed once); cached questions at least --cache-threshold cosine-similar are candidates. Everything else about the request must match exactly: the model, the system prompt, every earlier turn, the chat template, stop sequences. An answer longer than the request's max_tokens is not reused.
2. A judge confirms. The up to three most similar candidates are put to the decision engine as a typed noul, the cached question as the state: "this question asks the same thing as: <new question>". The first with P(true) at least --cache-judge-threshold is served. The judge is --decision-model when given, else the served model scoring the option tags (nothing is generated). A failing judge is a miss.

Measured on 1,000 Quora question pairs (GLUE QQP validation, a hard set: every pair is on one topic, a third are duplicates). Embeddings by the served model (Qwen3-0.6B) or by an embedder (gte-modernbert-base, the one a --rag index records or --embedder):

The embedding alone cannot tell "how do I close my account?" from "how do I open one?": neighbours in an embedding space share a topic, not always a meaning (a generative model's more so than an embedder's). Lower --cache-judge-threshold trades precision for hits. A small served model is a weak judge too; the server warns at startup when either is in use. A dedicated decision model (fuuga import-laya, or one trained with fuuga decide train on your own question pairs) is the judge to use, and fuuga decide calibrate makes its probabilities mean what the threshold assumes.

• What is cached. Complete plain answers: no tools, response_format, verify, consensus, thinking or assistant prefill, finish reason stop, through the output guardrails (a blocked answer is not stored; a hit passes them again). /v1/chat/completions (streamed as one content chunk) and /v1/messages.
• Whose answers. Under --provider-config a tenant reaches only answers made for its own key. Under --memory a request that recalls memory reaches only its session's answers (they were made with that memory, and changing the session's notes forgets them); one sent with X-Fuuga-Memory: off reaches the tenant's memory-free answers. A hit is billed like the answer it replays.
• Per request. X-Fuuga-Cache: refresh generates and replaces; off neither reads nor writes; another value is a 400. DELETE /v1/cache forgets the caller's answers, e.g. after the documents behind them changed.
• What invalidates it. The cache file records a fingerprint of the checkpoint, chat template, RAG index and graph files, retrieval and rerank settings and guardrails; a server started with anything else starts with an empty cache. --cache-ttl expires answers by age.
• Storage. One JSON Lines file in the directory, the oldest answers dropped past --cache-max-entries (about 4 bytes per embedding dimension each, plus the text). The directory is locked to one server process (.fuuga-cache.lock).

Code review in CI/CD

The server can review pull requests in a pipeline: scripts/ci-review.fsx sends each changed file's diff with a response_format JSON schema, so the findings come back machine-readable, and gates the job on their severity. Setup, GitHub Actions and Bitbucket Pipelines examples, and how to train a reviewer model are in docs/ci-code-review.md.

Using an agentic coding client (Crush)

Crush and other clients built on fantasy's openaicompat provider talk to fuuga-serve directly. Start the server, then point a custom provider at it:

fuuga-serve --checkpoint checkpoints/best --tokenizer data/tokenizer --port 8080 --api-key local

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "fuuga": {
      "type": "openai-compat",
      "base_url": "http://localhost:8080/v1",
      "api_key": "local",
      "models": [
        {
          "id": "<the id from GET /v1/models>",
          "name": "Fuuga local",
          "context_window": 8192,
          "default_max_tokens": 2048
        }
      ]
    }
  }
}

Notes:

• Set context_window to the checkpoint's real ContextLength, not an aspirational number — it's what makes the client compact the conversation before it overruns the model. A prompt that exceeds the window is rejected with 400 context_length_exceeded rather than failing inside the kernel.
• Configure the client's small/fast model explicitly. Crush picks the provider's first model for titles and summaries, so it lands on the same endpoint and queues behind your coding requests in sequential mode.
• Don't pair this with --continuous-batching — tool calling is bypassed in that mode (see Serving modes), which is exactly what an agentic client depends on.
• Usage reporting is emitted in a trailing SSE chunk carrying no choices when the client sends stream_options: {"include_usage": true}, which is what clients use to track context pressure. It's withheld otherwise, per the OpenAI spec.
• Prompts that exceed the model's context are rejected with 400 context_length_exceeded on /v1/chat/completions, /v1/completions and /v1/messages, rather than failing inside the generation core.

Hosting as an OpenRouter provider

Want to make money renting out a model you trained? fuuga-serve can be hosted as an OpenRouter provider: it advertises per-token pricing on /v1/models and meters each caller through a prepaid credit ledger — every request is debited by token cost, and a drained key gets HTTP 402. OpenRouter is your customer and settles with you out-of-band, so there's no payment code to write.

fuuga-serve --checkpoint checkpoints/best --tokenizer data/tokenizer \
            --host 0.0.0.0 --port 8080 --provider-config provider.json

provider.json holds per-model pricing, initial per-key balances, and an admin token (guarding /admin/credits/*). Copy provider.example.json to start.

• Full guide: docs/openrouter-provider.md — config schema, endpoints, the credit lifecycle, and security notes.
• ▶ Run it in F#: examples/openrouter-provider.fsx drives the billing ledger end-to-end (no model or GPU needed).

Building from Source

dotnet build                              # CPU (default)
dotnet build -p:TorchBackend=cuda         # GPU/CUDA on Windows x64 (several GB of libtorch)
dotnet build -p:TorchBackend=cuda-linux   # GPU/CUDA on Linux x64 (GPU rentals, servers)
dotnet run -- --checkpoint ../checkpoints/best --tokenizer ../data/tokenizer --port 8080

License

Unlicense (public domain). See the repository root for full license text.