TensorStack.OnnxRuntime.LLM 0.15.1

Prefix Reserved
dotnet add package TensorStack.OnnxRuntime.LLM --version 0.15.1
                    
NuGet\Install-Package TensorStack.OnnxRuntime.LLM -Version 0.15.1
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="TensorStack.OnnxRuntime.LLM" Version="0.15.1" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="TensorStack.OnnxRuntime.LLM" Version="0.15.1" />
                    
Directory.Packages.props
<PackageReference Include="TensorStack.OnnxRuntime.LLM" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add TensorStack.OnnxRuntime.LLM --version 0.15.1
                    
#r "nuget: TensorStack.OnnxRuntime.LLM, 0.15.1"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package TensorStack.OnnxRuntime.LLM@0.15.1
                    
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=TensorStack.OnnxRuntime.LLM&version=0.15.1
                    
Install as a Cake Addin
#tool nuget:?package=TensorStack.OnnxRuntime.LLM&version=0.15.1
                    
Install as a Cake Tool

TensorStack.OnnxRuntime.LLM

ONNX pipelines for small language, vision and speech models: text generation, speech recognition, text to speech, image captioning/OCR/detection and summarization, with greedy and beam search decoding.

Nuget Package

OnnxRuntime LLM Badge

For GGUF language models with tool calls and image/audio input see TensorStack.LlamaCpp.

Loading and saving images and audio uses TensorStack.Media.


Pipelines

Pipeline Models Use Model files
LlamaPipeline Llama 3.2 Text generation model.onnx, tokenizer.json
Phi3Pipeline Phi-3 mini, medium Text generation model.onnx, tokenizer.json (or tokenizer.model)
WhisperPipeline Whisper tiny to large-v3, turbo Speech recognition and translation encoder_model.onnx, decoder_model_merged.onnx, tokenizer.json (or vocab.json + merges.txt + tokenizer_config.json)
SupertonicPipeline Supertonic, Supertonic 3 Text to speech duration_predictor.onnx, text_encoder.onnx, vector_estimator.onnx, vocoder.onnx, unicode_indexer.json, voice_styles/*.json
FlorencePipeline Florence-2 base, large Captioning, OCR, object detection, grounding and segmentation vision_encoder.onnx, embed_tokens.onnx, encoder_model.onnx, decoder_model_merged.onnx, tokenizer.json (or vocab.json + merges.txt + tokenizer_config.json)
SummaryPipeline T5 Text summarization encoder_model.onnx, decoder_model_merged.onnx, tokenizer.json (or spiece.model)

Model sizes (layers, heads, cache dimensions, mel bins) are read from the ONNX models, the Create model type is only a fallback.

Tokenizers are read from the Hugging Face tokenizer.json when the model folder has one, falling back to the older vocabulary/merges or SentencePiece files.

Execution Providers

Each Create has an overload taking a provider per model. Run vision and audio encoders on the GPU and language model decoders on the CPU, DirectML returns incorrect tokens for some decoders.

var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
var pipeline = WhisperPipeline.Create(gpu, cpu, "M:\\Models\\Whisper-Base", WhisperType.Base);

Decoding

Each text pipeline has two RunAsync overloads:

  • GenerateOptions (WhisperOptions, FlorenceOptions) returns a single GenerateResult using greedy decoding
  • SearchOptions (WhisperSearchOptions, FlorenceSearchOptions) returns a GenerateResult[], best first, using beam search
Option Description
MinLength / MaxLength The generated length in tokens
Beams Number of beams for beam search
TopK / TopP / Temperature Beam search candidate sampling
LengthPenalty Beam score length penalty (1 = none, higher favours longer results)
NoRepeatNgramSize Blocks a repeat of any n-gram of this size, 0 to disable
EarlyStopping None, BestBeam or BeamCount
Seed Random seed for beam search sampling, negative for random

GenerateResult.Score is the sum of the token log probabilities. Progress callbacks report the text as it is generated, IsReset is set when the best beam changes and the full text is reported again.


Llama

var provider = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = LlamaPipeline.Create(provider, "M:\\Models\\Llama-3.2-1B-Instruct"))
{
    var result = await pipeline.RunAsync(new GenerateOptions
    {
        Prompt = "<|start_header_id|>user<|end_header_id|>\n\nWhat is an apple?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
        MaxLength = 200,
        MinLength = 0
    });
    Console.WriteLine(result.Result);
}

BOS is added to the prompt, generation stops at <|eot_id|>, <|eom_id|> or <|end_of_text|>.

GetLastHiddenStateAsync returns the prompt's last hidden state for use as a text encoder.

Phi-3

var provider = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = Phi3Pipeline.Create(provider, "M:\\Models\\Phi-3-medium-128k-instruct", PhiType.Medium))
{
    var results = await pipeline.RunAsync(new SearchOptions
    {
        Prompt = "<|user|>\nWhat is an apple?<|end|>\n<|assistant|>\n",
        Beams = 3,
        TopK = 3,
        MaxLength = 200,
        MinLength = 0
    });
    foreach (var result in results)
        Console.WriteLine(result.Result);
}

Whisper

var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = WhisperPipeline.Create(gpu, cpu, "M:\\Models\\Whisper-Base", WhisperType.Base))
{
    var result = await pipeline.RunAsync(new WhisperOptions
    {
        Task = TaskType.Transcribe,
        Language = LanguageType.EN,
        AudioInput = await AudioInput.CreateAsync("Speech.mp3"), // 16 kHz mono
        MinLength = 0,
        MaxLength = 448,
        NoRepeatNgramSize = 0
    });
    Console.WriteLine(result.Result);
}
  • Audio must be 16 kHz, AudioInput.CreateAsync loads at 16 kHz by default, other audio can be converted with audioTensor.Resample(16000)
  • Long audio is split into chunks of up to ChunkSize seconds (max 30) at the quietest point near each boundary
  • TaskType.Translate translates to English
  • The mel filters are computed, large-v3 and turbo use 128 mel bins (read from preprocessor_config.json or the model)

Supertonic

var provider = Provider.GetProvider(DeviceType.GPU);
using (var pipeline = SupertonicPipeline.Create("M:\\Models\\Supertonic", provider))
{
    var audio = await pipeline.RunAsync(new SupertonicOptions
    {
        TextInput = "On a quiet morning in the old town, a clockmaker named Ellis unlocked his tiny shop.",
        VoiceStyle = "Female1",
        Language = "en", // Supertonic 3, null for Supertonic 1
        Steps = 5,
        Speed = 1f
    });
    await audio.SaveAsync("Output.wav");
}

Text is split into sentences of up to 300 characters (120 for Korean) with SilenceDuration seconds between them. pipeline.VoiceStyles lists the available voices.

Florence-2

var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = FlorencePipeline.Create(gpu, cpu, gpu, gpu, "M:\\Models\\Florence-2-large", FlorenceType.Large))
{
    var result = await pipeline.RunAsync(new FlorenceOptions
    {
        TaskType = TaskType.OD,
        Image = new ImageInput("Image.png"),
        MinLength = 0,
        MaxLength = 1024
    });

    foreach (var coordinate in result.CoordinateResults)
        Console.WriteLine($"{coordinate.Label}: {coordinate.CoordinateBox}");
}
  • Text tasks (CAPTION, DETAILED_CAPTION, MORE_DETAILED_CAPTION, OCR) return Result
  • Region tasks (OD, DENSE_REGION_CAPTION, REGION_PROPOSAL, CAPTION_TO_PHRASE_GROUNDING, OCR_WITH_REGION etc) return CoordinateResults in source image pixels
  • Region input tasks (REGION_TO_CATEGORY, REGION_TO_DESCRIPTION, REGION_TO_OCR, REGION_TO_SEGMENTATION) take Region
  • CAPTION_TO_PHRASE_GROUNDING, OPEN_VOCABULARY_DETECTION and REFERRING_EXPRESSION_SEGMENTATION take Prompt

Summary

var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = SummaryPipeline.Create(gpu, cpu, "M:\\Models\\text_summarization"))
{
    var result = await pipeline.RunAsync(new GenerateOptions
    {
        Prompt = "summarize: " + text,
        MinLength = 10,
        MaxLength = 200,
        NoRepeatNgramSize = 3
    });
    Console.WriteLine(result.Result);
}

T5 models expect the task prefix (summarize: ) on the prompt.

Product Compatible and additional computed target framework versions.
.NET net10.0-windows10.0.17763 is compatible. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
0.15.1 0 11/21/2026
0.15.0 50 11/18/2026
0.14.14 54 11/17/2026
0.14.13 67 11/16/2026
0.14.12 61 11/15/2026
0.14.11 55 11/15/2026
0.14.10 65 11/14/2026
0.14.9 65 11/13/2026
0.14.8 61 11/10/2026
0.14.7 63 11/8/2026
0.14.6 69 11/5/2026
0.14.5 90 11/1/2026
0.14.4 134 10/29/2026
0.14.3 79 10/28/2026
0.14.2 111 10/20/2026
0.14.1 87 10/16/2026
0.13.13 79 10/11/2026
0.13.12 74 10/10/2026
0.13.10 91 10/6/2026
0.13.9 72 10/5/2026
Loading failed