OnnxTextEmbeddings.NET 0.1.1

OnnxTextEmbeddings.NET

Local, CPU-friendly text embeddings and lightweight semantic search for .NET 10.

No Python. No embedding server. No Hugging Face CLI. No vector database required. Install a NuGet package, register one service, and the default Jasper ONNX model is downloaded and cached automatically on first use.

OnnxTextEmbeddings.NET is aimed at wikis, documentation, note apps, games, local tools, and other small-to-medium semantic-search workloads where running a separate AI stack would be ridiculous overhead.

Install

dotnet add package OnnxTextEmbeddings.NET

Optional database-native search integrations are separate packages, so core never drags in database dependencies you do not use:

dotnet add package OnnxTextEmbeddings.NET.PgVector
dotnet add package OnnxTextEmbeddings.NET.SqliteVec
dotnet add package OnnxTextEmbeddings.NET.SqlServer

Five-minute start

builder.Services.AddOnnxTextEmbeddings();

Inject ITextEmbeddingService and embed text:

var embeddings = await embeddingService.EmbedAsync("Hello world");

Short text normally returns one TextEmbedding. Long text is chunked automatically using Markdown structure, paragraphs, sentences, words, and finally token windows when necessary.

Search application records without a vector database:

var results = await semanticSearch.SearchAsync(
    "How do I restore PostgreSQL?",
    pages,
    page => page.Embeddings,
    new SemanticSearchRequest { Top = 10 });

Database-native search

For larger persisted working sets, PostgreSQL/pgvector, SQLite/sqlite-vec, and SQL Server 2025/Azure SQL can keep broad vector candidate search inside the database.

relational filters + native vector KNN
                ↓
        bounded chunk candidates
                ↓
      shared core DefaultV1 reranking
                ↓
          final item results

The database integrations deliberately do not reimplement semantic scoring in SQL. They retrieve plausible direct chunks, then the same DefaultV1 implementation used by in-memory search applies chunk-length confidence, bounded supporting evidence, and semantic-field weights.

Candidate retrieval defaults to over-fetching max(100, Top × 10) chunks so final item scoring has enough evidence. Every provider filters by the query's embedding-space fingerprint during native retrieval.

Plain SQLite remains valid generic persistence; sqlite-vec is the official SQLite-native search integration. See Database-native semantic search.

Per-call token and vector overrides

The application-wide values are defaults, not permanent restrictions.

var compactChunks = await embeddingService.EmbedDocumentAsync(
    text,
    new EmbeddingRequestOptions
    {
        MaxTokens = 512,
        VectorFormat = EmbeddingVectorFormat.Int8
    });

MaxTokens changes the actual document chunk/model-input ceiling for that call. It still cannot exceed the loaded model's hard maximum.

Queries remain exactly one vector and are never chunked. A caller may raise or lower that request's acceptance ceiling independently of the global QueryMaxTokens:

var query = await embeddingService.EmbedQueryAsync(
    queryText,
    new QueryEmbeddingRequestOptions
    {
        MaxTokens = 2048,
        VectorFormat = EmbeddingVectorFormat.Float32
    });

The matching non-throwing validation API accepts the same request options:

var count = await embeddingService.CountQueryTokensAsync(
    queryText,
    new QueryEmbeddingRequestOptions { MaxTokens = 2048 });

if (count.Fits)
    query = await embeddingService.EmbedQueryAsync(
        queryText,
        new QueryEmbeddingRequestOptions { MaxTokens = 2048 });

CPU concurrency and multiple model instances

One ONNX InferenceSession can execute several Run() calls concurrently, so normal concurrency does not require duplicate model weights in memory.

Default model topology:

ModelInstanceCount = 1
ThreadsPerModel = 16
ConcurrentRequestsPerModel = Auto → 8

Automatic concurrency is ThreadsPerModel / 2, minimum one, capped at 8. Explicit positive ConcurrentRequestsPerModel values always win.

If multiple model instances are deliberately loaded, work is routed to the healthy instance with the fewest active requests, with rotating tie-breaking. Two idle model instances receiving two requests therefore receive one request each rather than filling one model first.

Multiple model instances usually do not increase aggregate throughput on a normal CPU host. Once one model instance is busy enough, the bottleneck is commonly somewhere in the shared CPU/memory subsystem—often memory bandwidth/cache/memory-controller or board-level throughput—not a shortage of model copies. ModelInstanceCount > 1 exists for experimentation, unusual hardware/topologies, and future tuning work rather than as a general performance recommendation.

One potentially useful future/experimental direction is CPU-topology-aware isolation: pinning model sessions/threads to distinct CPU groups, NUMA nodes, or architecture-specific regions (for example AMD CCD/NUMA layouts) and keeping their memory locality separate so copies steal less bandwidth/cache from one another. The package does not assume this will help every machine; it is something to benchmark per CPU/platform.

Self-healing model instances

Each model copy has independent health and capacity tracking. A recoverable ONNX session/runtime failure removes that instance from routing, lets its active calls drain, disposes the old session, creates a fresh session generation, and returns the instance to service only after the replacement loads successfully.

If another instance is healthy, traffic continues there at reduced capacity. If every instance is unavailable, the bounded global queue waits for recovery instead of inventing capacity or permanently losing a request slot.

Memory-pressure failures rebuild the affected instance but are not immediately retried on another model copy, avoiding a cascading OOM. This recovery exists inside a live process; if the operating system or container runtime kills the entire .NET process for OOM, process/service-level restart is still required.

See Concurrency, load balancing, and recovery.

Default model

The default is Jasper INT8:

• magiccodingman/Jasper-Token-Compression-600M-ONNX-INT8
• magiccodingman/Jasper-Token-Compression-600M-ONNX-INT4
• magiccodingman/Jasper-Token-Compression-600M-ONNX-FP32

Model precision and returned-vector precision are independent.

Why Jasper?

Jasper Token Compression 600M was chosen because its dynamic token-compression architecture is an unusually good fit for CPU document embedding: measured latency stays remarkably flat as context grows, while the final Dynamic INT8 export remains small and very close to FP32 quality.

The project intentionally defaults to 1024 tokens: local quality testing found Jasper extremely strong through roughly 756 tokens and only slightly degraded around 1024, but with a much sharper long-tail falloff beyond 1024—consistent with the upstream model having been distilled only through 1024 tokens. See Why Jasper is the default model for the complete CPU, memory, concurrency, fidelity, and length-quality results.

Vector formats: FP32 by default, INT8 recommended for compact storage

Both document and query embeddings return FP32 by default for maximum compatibility.

Make INT8 the application-wide document default when compact storage matters:

builder.Services.AddOnnxTextEmbeddings(options =>
{
    options.Vectors.DocumentFormat = EmbeddingVectorFormat.Int8;
});

Or choose dynamically:

var tiny = await embeddingService.EmbedDocumentAsync(text, EmbeddingVectorFormat.Int4);
var compact = await embeddingService.EmbedDocumentAsync(text, EmbeddingVectorFormat.Int8);
var half = await embeddingService.EmbedDocumentAsync(text, EmbeddingVectorFormat.Float16);
var full = await embeddingService.EmbedDocumentAsync(text, EmbeddingVectorFormat.Float32);

Convert vectors you already have

EmbeddingVector fp32 = EmbeddingVector.FromFloat32(values);
EmbeddingVector fp16 = EmbeddingVector.FromFloat32(values, EmbeddingVectorFormat.Float16);
EmbeddingVector int8 = EmbeddingVector.FromFloat32(values, EmbeddingVectorFormat.Int8);
EmbeddingVector int4 = EmbeddingVector.FromFloat32(values, EmbeddingVectorFormat.Int4);

EmbeddingVector smaller = fp32.ConvertTo(EmbeddingVectorFormat.Int8);

Expanding a lower-precision vector back to FP32 changes its representation; it cannot restore fidelity already discarded by quantization.

Combine many chunk embeddings into one

Keeping the original chunk array is the preferred representation for semantic retrieval. When a consumer explicitly requires exactly one vector, the returned IReadOnlyList<TextEmbedding> can be mathematically compressed:

var chunks = await embeddingService.EmbedDocumentAsync(longText);
var single = chunks.CombineToSingle();

For multiple chunks, the default SemanticCoverage-v1 profile performs full-dimensional FP32 semantic aggregation while giving highly repetitive semantic regions diminishing additional influence. It uses persisted source token ranges to avoid double-counting overlap and returns diagnostics such as AggregationCoherence and MinimumSourceSimilarity.

Aggregation, dimension reduction, and numeric conversion are separate stages. A caller can request all three in one operation:

var single = chunks.CombineToSingle(new SingleEmbeddingOptions
{
    OutputDimensions = 512,
    OutputFormat = EmbeddingVectorFormat.Int8
});

That means:

chunk vectors
   ↓
FP32 SemanticCoverage-v1 at full dimensions
   ↓
deterministic SRHT-v1 → 512
   ↓
normalize
   ↓
INT8

Dimension expansion is never allowed. A 2048-dimensional supplied representation cannot be turned into a meaningful 4096-dimensional embedding.

Reduced vectors occupy a new deterministic embedding space, so queries must receive the same transform:

var query = await embeddingService.EmbedQueryAsync("database restore");
var reducedQuery = query.ReduceDimensions(512);

var cosine = EmbeddingVectorMath.CosineSimilarity(
    reducedQuery.Vector,
    single.Vector);

Direct chunk vectors can also be reduced without aggregation:

var reducedChunk = chunks[0].ReduceDimensions(1024);

The reduced direct chunk preserves its source/chunk metadata and enters the same deterministic child space as a query reduced with the same profile.

See Single-embedding aggregation and Dimension reduction.

Native AOT and non-.NET interoperability

The core package declares Native AOT compatibility and is continuously AOT-published/tested on Linux, Windows, and macOS.

The repository also contains a separate OnnxTextEmbeddings.Native facade that publishes the canonical C# implementation as a Native AOT shared library with a versioned C ABI:

OnnxTextEmbeddings.Native
        ↓
.so / .dll / .dylib
        ↓
stable C ABI
        ↓
Rust / C / C++ / Go / Zig / Python FFI / other bindings

This native facade is intentionally not a NuGet package. The project maintains the C ABI, public header, and cross-platform C interoperability tests; third-party language bindings are welcome without implying that every language wrapper becomes a first-party SDK.

See Native AOT compatibility and Native interoperability.

Configuration defaults

Jasper model                  INT8
DocumentChunkMaxTokens        1024
QueryMaxTokens                1024
ModelInstanceCount            1
ThreadsPerModel               16
ConcurrentRequestsPerModel    Auto (8 at 16 threads/model)
QueueCapacity                 256
ChunkOverlapTokens            0
RepeatHeadingContext          true
Document vector format        FP32
Query vector format           FP32
Scoring profile               DefaultV1

Runtime diagnostics

ITextEmbeddingService.ModelInfo reports current model-instance health, active request counts, generation numbers, recovery counts, and the aggregate number of healthy/recovering instances. This is useful for health endpoints and production diagnostics without requiring a monitoring framework.

Persistence

The core package owns no database. Store TextEmbedding or SingleEmbedding records wherever the application already stores data: memory, SQLite BLOBs, SQL Server VARBINARY, PostgreSQL BYTEA, JSON/files, or a native vector column through one of the optional database adapters.

Documentation

• Getting started
• Architecture
• Configuration
• Why Jasper is the default model
• Concurrency, load balancing, and recovery
• Vector formats and conversion
• Single-embedding aggregation
• Dimension reduction
• Model sources
• HTTP model manifest
• Model cache and updates
• Chunking
• Semantic search
• DefaultV1 scoring
• Database-native semantic search
• Persistence
• SQLite persistence
• SQLite/sqlite-vec
• PostgreSQL/pgvector
• SQL Server 2025 / Azure SQL
• Native AOT compatibility
• Native interoperability / C ABI
• Performance
• Deployment
• Troubleshooting

Scope

This library is intentionally not a RAG framework, vector database, ingestion platform, PDF parser, crawler, GPU framework, or distributed embedding service. It is a focused way to add high-quality local text embeddings and good small-scale semantic search to an ordinary .NET application—or, through the Native AOT facade, to another runtime that wants to bind to the same engine.

License

Apache-2.0.