Starlings.Models.Qwen35 0.3.0

Starlings.Models.Qwen35

Chat generation with Qwen/Qwen3.5-0.8B at revision 2fc06364715b967f1860aea9cf38778875588b17, in-process on ONNX Runtime GenAI 0.17.1, offline after an explicit acquisition. The API and the supported platforms may still change. The three registry profiles are Stable since 2026-10-16: ortgenai-fp32x-cpu on Linux x64 CPU, and ortgenai-fp32x-cuda and ortgenai-fp16x-cuda on Linux x64 with an RTX 3090 (see Stable contract and Stable contract: CUDA).

• The pinned chat template is rendered in C# for text and tool-calling conversations (system, user, assistant and tool messages, tool definitions, assistant tool calls, enable_thinking); images are rejected. Prompt ids come from the owned tokenizer Starlings.Tokenizers (profile qwen35-bpe-chat): no Microsoft.ML.Tokenizers, no GenAI tokenizer, no Python.
• Starling's decode loop (Starlings.Generation.OnnxGenAI): greedy or seeded sampling, stop at <|im_end|> / <|endoftext|> and at stop strings, streaming, cancellation between decode steps, usage.
• The package ships the asset locks (sizes and SHA-256), not the model files (fp32x 3.1 GB, fp16x 1.6 GB). Acquisition is an explicit call; generation never downloads.

Maturity

Maturity is per profile, generated from the registry manifests by model-pipeline catalog (do not edit this table by hand). Gates, evidence and capabilities per profile: the model catalog.

Install

dotnet add package Starlings.Models.Qwen35
dotnet add package Microsoft.ML.OnnxRuntimeGenAI --version 0.17.1        # CPU natives
# or, for CUDA: Microsoft.ML.OnnxRuntimeGenAI.Cuda 0.17.1 (reference exactly one)

Use

using Starlings.Models.Qwen35;

Qwen35ModelFiles files = Qwen35ModelAssets.OpenLocal(Qwen35Variant.Fp32x);   // offline, verified, never downloads
using var model = Qwen35ChatModel.Load(files);

var messages = new[] { Qwen35Message.User("What is the capital of France?") };
foreach (var update in model.GenerateStreaming(messages, new Qwen35ChatOptions { MaxNewTokens = 128 }))
{
    Console.Write(update.Text);
    if (update.Result is { } done)
    {
        Console.WriteLine($"\n[{done.FinishReason}, {done.Usage.PromptTokens} + {done.Usage.CompletionTokens} tokens]");
    }
}

On CUDA, pass new Qwen35ModelOptions { Provider = GenAIExecutionProvider.Cuda } (namespace Starlings.Generation.OnnxGenAI) to Load; CudaDeviceId selects the device. CUDA runs Qwen35Variant.Fp32x (TF32 off) or Qwen35Variant.Fp16x; CPU runs only Fp32x, and loading Fp16x on CPU throws NotSupportedException. A provider that cannot be used fails the load; there is no fallback.

The built graphs are not on the Hub: Starling builds them reproducibly from the pinned checkpoint (model-pipeline/reference/generation/build_ortgenai_bundle.py, digests in Qwen35ModelAssets) and hosts every file of the locks (with chat_template.jinja) in the Hugging Face repository Starlinghq/qwen3.5-0.8b-ortgenai, pinned by commit. That repository is public: Qwen35ModelAssets.DownloadAsync(variant) needs no token. Alternatively build the files into the snapshot directory or call Qwen35ModelAssets.OpenDirectory(dir, variant).

For Microsoft.Extensions.AI's IChatClient (middleware, DI, asynchronous streaming, function invocation), add Starlings.Models.Qwen35.Extensions.AI and wrap the model in Qwen35ChatClient.

Tool calling (experimental)

using System.Text.Json;

var weather = Qwen35Tool.Function("get_weather", "Get the current weather for a city.", JsonDocument.Parse(
    """{"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}""").RootElement);
var options = new Qwen35ChatOptions { Tools = [weather] };
var conversation = new List<Qwen35Message> { Qwen35Message.User("What is the weather in Paris today?") };

var result = model.Generate(conversation, options);
var parsed = Qwen35ToolCallParser.Parse(result.Text, options);       // text and Qwen35ToolCall parts
foreach (var call in parsed.ToolCalls)
{
    Console.WriteLine($"{call.Name}({call.Arguments})");                // get_weather({"city":"Paris"})
}

conversation.Add(parsed.ToAssistantMessage());
conversation.Add(Qwen35Message.Tool("18 C and sunny"));               // one tool message per result, in call order
result = model.Generate(conversation, options);                        // the answer

• Qwen35ChatOptions.Tools are rendered into the template's tools block as Python json.dumps does (key order kept). Qwen35Message.ToolCalls (assistant) and Qwen35Role.Tool messages are rendered as the template does: <tool_call> blocks after the content, and consecutive tool results in one user turn of <tool_response> blocks.
• The model answers in the template's format (<tool_call>\n<function=NAME>\n<parameter=ARG>\nVALUE\n</parameter>\n</function>\n</tool_call>). Qwen35ToolCallParser splits a completion, or streamed deltas, into text and calls. Values are cast with the tool's JSON schema as transformers' response parser does (integer, number, boolean, null, object, array; otherwise a string). A malformed or unfinished block is returned as text, verbatim; the parser never throws on model output. With thinking enabled, nothing before </think> is parsed.
• Streaming (Append, Complete) returns text as soon as it cannot start a tool call, and each call when its block closes; the parts equal those of parsing the whole completion.
• Nothing constrains decoding: the model may answer in text, call an undeclared tool, or omit required arguments. Check calls before running them.

Profiles

Semantics

• MaxNewTokens bounds the completion tokens; a stop token id is never part of the text, the ids or the usage count. A stop string ends generation and is cut from the text; its tokens stay in the ids and the count.
• Streaming updates concatenate to the final text and ids; the last update carries the finish reason (Stop, StopSequence, Length) and usage. Cancellation throws OperationCanceledException at the next prefill chunk or decode step; a running chunk or step is not interrupted. Prompts are prefilled in 512-token chunks on CUDA and in one call on CPU (PrefillChunkTokens).
• A conversation the template cannot render (for example no messages, a system message after the first, no user query, or invalid tool data) throws Qwen35TemplateException before any native call.
• Each request renders and prefills its whole conversation; requests share no state. On CUDA concurrent requests are serialized step by step, because concurrent ONNX Runtime GenAI 0.17.1 generators on the CUDA provider produced wrong tokens; on CPU they run in parallel. Conversation state reuse, batching, forced tool calls (requiring a tool) and reasoning extraction are unsupported in this version.

Stable contract

This is the contract of profile qwen3.5-0.8b/ortgenai-fp32x-cpu (fp32x on Linux x64 CPU). Its Stable label (since 2026-10-16) commits to it under the support policy. The statuses are the registry manifest's and Qwen35Capabilities.Profiles, which a test keeps equal key for key. Qwen35Capabilities.All also lists the Stable CUDA profiles' keys (.cuda, fp16x).

Stable contract: CUDA

This is the contract of profiles qwen3.5-0.8b/ortgenai-fp32x-cuda and qwen3.5-0.8b/ortgenai-fp16x-cuda, Stable since 2026-10-16. It is the CPU contract above, with these differences. The statuses are the registry manifests' and Qwen35Capabilities.Profiles, which a test keeps equal key for key.

Evidence: binding CUDA G4 and G5 round, per-release GPU regression, run 38857860186, release-set gates for fp32x CUDA and fp16x CUDA.

Path to Stable

Status 2026-10-16: ortgenai-fp32x-cpu is Stable (contract), and so are ortgenai-fp32x-cuda and ortgenai-fp16x-cuda on an RTX 3090 (contract): every gate is recorded passed, and every check of both passed in the first per-release GPU run (38857860186). Every gate G0 to G7 is recorded passed. G5 passed on the planned quiet-host rerun (evidence) after a first run failed under host load. The weekly qualification workflow reruns its release-set G1 to G3 and a one-hour soak, green in run 38848321624. G2 there reads the full step-0 reference vectors from Starlinghq/starling-release-references, which the protected hosted-reference.json pins. GPU regression runs for every release (an ai-v* tag) and on demand, on an RTX 3090 only, in the qualification workflow's cuda (runpod) job; CPU regression runs weekly. The checklist below predates the release-set gates and is kept as history.

The order of promotion (docs/plan/production-readiness.md) takes ortgenai-fp32x-cpu to Preview first. The fp16x profile's near-tie divergences (18 of 22 development cases, 28 of 32 release prompts exact) are within its protected near-tie rule, which is its contract. Gate evidence is in the catalog once draft PR #4 lands. Agents produce evidence; only the maintainer records a gate or changes maturity.

• G0 provenance: today a checklist (P-016 evidence). The built files are publicly hosted since 2026-10-14 (M3 done); recording the gate result stays with the maintainer.
• G1 and G2: today checklists. On development fixtures: renderings and prompt ids 21/21, greedy 22/22, and step-0 1.38e-3 against a 5e-3 bound, rerun at tokenizer 0.3.0-preview.1 (draft PR #6). The generation evaluator task that machine-checks these against the protected tolerances is in draft PR #5. Still needed: the protected P-030 generation set (M1), and the GPU runs for the CUDA profiles (M6).
• P-029 template corpus beyond the 21 prompts, so the owned renderer can leave experimental (R8). The proposed corpus (121 cases, 41 of them tool calling) awaits the maintainer.
• Tool calling: a tool-use evaluation set and budget, so the model's tool choice is measured rather than shown on one prompt.
• G3 task quality: needs a fixed generation evaluation, a dataset and a budget (M1, M2).
• G4 resource correctness: today a checklist for the LLM semantics and the CUDA concurrency defect (draft PR #6). Needs a one-hour soak and the plan 10.2 tests (R1), long-context tests above 1,666 tokens (done: 1,667 to 8,192 experimental, longer rejected), and the filed upstream report with CUDA serialization confirmed as the documented behavior (M9).
• G5 performance: the harness is in draft PR #7 (an uncontrolled run, not G5 evidence). Needs a controlled run against a budget (M2, M6).
• G6 distribution: the release drill on seeded files is a checklist (draft PR #6). Needs acquisition from public hosting (R9 after M3), and Windows x64 (R6, R9).
• G7 integration: the IChatClient adapter is tested. Needs a representative downstream use (M7).
• aot, long context and fp16x extensions_ai: aot and fp16x extensions_ai experimental; long context split into long_context_1667_to_8192_tokens (experimental) and long_context_over_8192_tokens (unsupported, rejected) (capability closure). The validator refuses Stable with any untested capability.
• Qualification CI (P-018, R6), the compatibility matrix rows, the support policy, and a second release owner (M7).

Attribution and license

Qwen3.5-0.8B by the Qwen team, Alibaba Cloud (Copyright 2026 Alibaba Cloud), Apache-2.0 (LICENSE in the Hub repository at the pinned commit). The built ONNX files are derived from those weights under the same license. Qwen35ChatTemplate is a C# port of the checkpoint's chat_template.jinja (Apache-2.0). This package's code is MIT OR Apache-2.0.

Documentation and evidence: ai-dotnet/README.md.