Starlings.Models.E5 0.3.0

Starlings.Models.E5

Sentence embeddings from intfloat/multilingual-e5-small at revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, on ONNX Runtime 1.30.0, in-process and offline after an explicit download. The package is versioned 0.x, so the API may still change between minor versions; maturity is per profile (below).

• Text goes through the owned tokenizer Starlings.Tokenizers (exact against huggingface/tokenizers on the published fixtures). No Microsoft.ML.Tokenizers and no native tokenizer library.
• Batching with right padding, mean pooling over the attention mask, L2 normalization: 384-dimensional unit vectors.
• The package ships the model's asset lock (repository, commit, file sizes and SHA-256), not the 487 MB of model files. Acquisition is an explicit call; inference never downloads.

Maturity

Maturity is per profile, generated from the registry manifests by model-pipeline catalog (do not edit this table by hand). Gates, evidence and capabilities per profile: the model catalog.

Install

dotnet add package Starlings.Models.E5
dotnet add package Microsoft.ML.OnnxRuntime --version 1.30.0          # CPU natives
# or, for CUDA on Linux x64: Microsoft.ML.OnnxRuntime.Gpu.Linux 1.30.0 (reference exactly one)

The library references only Microsoft.ML.OnnxRuntime.Managed 1.30.0; the application chooses the native runtime.

Use

using Starlings.Models.E5;
using Starlings.Runtime.Onnx;

await E5ModelAssets.DownloadAsync();                 // once, explicit, needs the network; verifies SHA-256
E5ModelFiles files = E5ModelAssets.OpenLocal();      // offline, verified, never downloads

using var encoder = E5Encoder.Create(files.ModelPath, new E5EncoderOptions
{
    Session = new OnnxSessionOptions { ExpectedModelSha256 = E5ModelAssets.ModelSha256 },
});
var embedder = new E5TextEmbedder(E5Tokenizer.FromFile(files.TokenizerPath, E5ModelAssets.TokenizerSha256), encoder);

float[] query = embedder.Embed(E5TextEmbedder.QueryPrefix + "how much protein should a female eat");
float[][] passages = embedder.EmbedBatch([E5TextEmbedder.PassagePrefix + "As a general guideline, ..."]);

e5 expects "query: " or "passage: " at the start of every text; nothing is added for you. Vectors are normalized, so cosine similarity is the dot product.

Batching options

EmbedBatch splits a batch into ONNX Runtime calls, each right-padded to its longest row. E5EncoderOptions controls the split:

The defaults are the qualified behavior and give the same outputs as before these options existed, bit for bit. With SortByLength, which rows share a call depends on the whole batch, so a row's embedding can differ from the default by up to the profile's padded-batch-vs-single bound (padded_batch_vs_single_embedding_max_abs; on ONNX Runtime CPU the tests saw no difference). If a call is cancelled, the Span<float> overload may already hold the rows of the chunks that completed: a prefix of the rows by default, the shortest rows with SortByLength.

Status

Profile multilingual-e5-small/onnx-fp32-cpu (Linux x64 CPU) is Stable since 2026-10-15: it passed every qualification gate, G0 to G7, and the weekly qualification workflow reruns G1 to G3 on the release sets and a one-hour soak, green. The hosted soak that failed on 2026-10-15 (about 50 MB/h) did so because of glibc retention after unloading and transparent huge pages set to always on the runner image, not a leak (evidence). On 2026-10-16 one hosted soak in six failed again (58 MiB of unloaded growth against 32): requests from rotating thread-pool threads left freed memory resident in more and more glibc heaps. With dedicated run threads on CPU (below), those heaps stay at 14 to 62 MiB after unloading (evidence). Profile multilingual-e5-small/onnx-fp32-cuda (Linux x64, NVIDIA GeForce RTX 3090) is Stable since 2026-10-16 (Stable contract: CUDA): every gate passed, and the per-release GPU regression run was green for it. GPU regression runs for every release and on demand, on an RTX 3090 only (the qualification workflow's cuda (runpod) job); CPU regression runs weekly. Gate evidence: the catalog. Model license: MIT, stated only in the model card.

Stable contract

This is the contract of profile multilingual-e5-small/onnx-fp32-cpu. Its Stable label (since 2026-10-15) commits to it under the support policy. The statuses are the registry manifest's and E5Capabilities.Profiles, which a test keeps equal key for key.

Stable contract: CUDA

This is the contract of profile multilingual-e5-small/onnx-fp32-cuda (Stable since 2026-10-16). It is the CPU contract above, with these differences. The statuses are the registry manifest's and E5Capabilities.Profiles, which a test keeps equal key for key.

Evidence: binding CUDA G4 and G5 round, per-release GPU regression, run 38857860186, release-set gates, CUDA cancellation.

Memory on Linux

• Heap trim on release (on by default). When a session's native memory is released (after Dispose and the last in-flight run), Starlings.Runtime.Onnx calls glibc's malloc_trim(0), so a process that loads and unloads the model returns the freed memory instead of keeping it resident. It never runs on the request path and changes no allocator setting. Opt out per session with OnnxSessionOptions.TrimNativeHeapOnRelease = false, or for the whole process with the AppContext switch Starlings.Runtime.Onnx.DisableNativeHeapTrim (details in the Starlings.Runtime.Onnx README).
• Dedicated run threads on CPU (default). E5Encoder.Create gives its CPU sessions the runtime's dedicated run threads (OnnxSessionOptions.DedicatedRunThreads, null by default, means on here; set it to false to run on the calling thread; a session passed to the E5Encoder constructor keeps its own setting). Each run allocates and frees memory with malloc on the thread that calls ONNX Runtime, and glibc keeps freed memory resident at the top of that thread's heap. Callers from the .NET thread pool rotate, so that memory spread over 1 to 8 heaps holding 4 to 139 MiB after unloading in the hosted one-hour soak, and one of the six hosted soaks of 2026-10-16 failed the growth rule (unloaded growth 58 MiB against 32, run 38913307558). On a fixed set of threads (at most one per processor, started only when all are busy) the same heaps are reused: 16 to 62 MiB in three soaks out of three. More concurrent calls than processors queue instead of oversubscribing the CPU, and a token cancelled while a call is queued takes effect when it reaches a thread. Embeddings are bit-identical either way (a test checks it), and request latency in the soak was unchanged (p50 17.4 ms before, 16.9 and 17.2 ms after, on the same CPU model). ONNX Runtime's CPU memory arena keeps its default (on). Evidence: e5 soak reliability.
• Transparent huge pages. The qualification soaks ran with /sys/kernel/mm/transparent_hugepage/enabled set to madvise (the Ubuntu default). On hosts set to always, the kernel's khugepaged fills in partly used 2 MiB heap ranges, so resident memory can look larger and rise over time without any leak.
• No MALLOC_MMAP_THRESHOLD_ or other MALLOC_* setting is required for this profile.

Evidence: scheduled regression fix.

Qualification evidence

• G0 provenance: the 0.1.0-preview.1 record, confirmed anonymously in the 2026-10-14 provenance check.
• G1, G2 and G3 on the P-030 release sets: release-set gates.
• G4 one-hour soak: 2026-10-14. G5 matched benchmark: 2026-10-15.
• G6 distribution and G7 integration: G6, G7.
• Regression support: the weekly qualification workflow (jobs release-gates and soak), green in run 38822625245.