TurboQuant 0.3.0

TurboQuant

Near-optimal vector quantization for .NET 8+

Compress embeddings to 2-4 bits with <0.5% quality loss. Zero dependencies. AOT-ready.

Implements Algorithm 1 from TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (Zandieh et al., ICLR 2026). Lloyd-Max quantization on the exact Beta(d/2, 1/2) distribution with randomized rotation.

768-dim float32 embedding  ──>  TurboQuant 4-bit  ──>  512 bytes
       3072 bytes                                       6x smaller
                                                        cosine > 0.995

Install

dotnet add package TurboQuant

30-second Example

using TurboQuant;

var quantizer = TurboQuantBuilder.Create(dim: 768).WithBits(4).BuildMSE();

var packed = quantizer.Quantize(embedding);        // float[768] -> PackedVector
var restored = quantizer.Dequantize(packed);       // PackedVector -> float[768]
var score = quantizer.ApproxSimilarity(a, b);      // LUT-based, zero allocation

// Zero-alloc dequantize into existing buffer
Span<float> buffer = stackalloc float[768];
quantizer.Dequantize(packed, buffer);

// Serialize for storage
var bytes = packed.ToBytes();
var loaded = PackedVector.FromBytes(bytes);

Why TurboQuant?

Performance

Measured on Apple M5 (ARM64), on battery, .NET 10.0.10, Release, BenchmarkDotNet 0.15.8 on 2026-10-17. The comparison used the implementation before these changes, three warmup iterations and five measurement iterations, with statistics disabled and variable input vectors. System power history recorded battery use before the run started; results should be compared under the same power conditions.

Similarity scans used 768d inputs with Hadamard padding to 1024 coordinates. The KV comparison used random rotation for both versions; runtime stayed approximately 16 ms because the same full decompression is performed. Smaller range reads avoid decompressing unrelated entries. Numbers describe this CPU and workload, rather than guarantees for other architectures.

SIMD operations use Vector<T> for automatic ISA selection (NEON on ARM, SSE/AVX2/AVX-512 on x86) with dual accumulators to hide data-dependency latency.

ApproxSimilarity scans packed groups directly at all bit widths, without intermediate index arrays. The product tables are initialized lazily and shared per codebook: 256 KiB for 2/4-bit byte groups, 16 KiB for 3-bit coordinate pairs, plus small squared-norm tables. Zero vectors return zero similarity, and unused packing bits are ignored.

Quantize selects centroids and packs bytes in one pass. QuantizeBatch reuses scratch storage across the batch; QuantizeBatchParallel reuses it per worker and processes small batches sequentially. Each result still allocates its own packed byte array. Batch input length must be a multiple of the dimension.

Per-vector timing and distortion statistics are enabled by default. Use .WithStatistics(false) when diagnostics are unnecessary; this applies to MSE quantizers, asymmetric quantizers and KV caches.

Compression Quality

Validated against the paper's theoretical bounds:

Ratios depend on whether the dimension is a power of 2 (no padding needed) or not.

Use Cases

Embedding Compression (OpenAI, Ollama, Azure AI)

var embedding = await embeddingModel.GenerateAsync(["your text"]);
var vector = embedding[0].Vector.ToArray();

var quantizer = TurboQuantBuilder.Create(dim: 768).WithBits(4).BuildMSE();
var packed = quantizer.Quantize(vector);

// Store packed vectors in your DB, cache, or memory
// Search directly on compressed data (zero allocation):
var score = quantizer.ApproxSimilarity(queryPacked, docPacked);

100K embeddings (768d): 292 MB → 37 MB

RAG Pipeline: Compressed Vector Store

var quantizer = TurboQuantBuilder.Create(dim: 768).WithBits(4).BuildMSE();
var store = new Dictionary<string, PackedVector>();

// Index
foreach (var doc in documents)
{
    var emb = await generator.GenerateAsync([doc.Text]);
    store[doc.Id] = quantizer.Quantize(emb[0].Vector.ToArray());
}

// Search (LUT-based, zero allocation)
var queryPacked = quantizer.Quantize(queryEmbedding);
var topK = store
    .Select(kv => (kv.Key, Score: quantizer.ApproxSimilarity(queryPacked, kv.Value)))
    .OrderByDescending(x => x.Score)
    .Take(10);

LLM KV Cache: Extend Context Length

var (cache, window) = TurboQuantBuilder
    .Create(dim: 128)             // head dimension
    .WithKeyBits(4)               // keys: 4-bit (attention precision)
    .WithValueBits(2)             // values: 2-bit (more aggressive)
    .WithHadamardRotation()      // fast path for the power-of-two head dimension
    .WithStatistics(false)
    .WithResidualWindow(tokens: 128)
    .BuildKVCache();

// Append during generation. Recent tokens stay in float32,
// older tokens are automatically compressed
window.Append(keyVector, valueVector);

// Retrieve for attention computation
// Allocate once, then reuse these buffers while the context size stays the same.
var keys = new float[window.TotalCount * cache.Dimension];
var values = new float[keys.Length];
window.CopyKeysTo(keys);       // contiguous rows: recent exact, older decompressed
window.CopyValuesTo(values);

// Read only the range needed by the caller, including across the residual boundary.
window.CopyKeysTo(startIndex: 0, count: Math.Min(32, window.TotalCount), destination: keys);

The compressed cache also provides CopyKeysTo / CopyValuesTo, single-entry span dequantization, and allocation-free GetKey(index) / GetValue(index) access to packed vectors. The existing GetAllKeys / GetAllValues APIs return independent snapshots. The residual window uses preallocated circular buffers; a zero-sized window compresses immediately. TrimToWindow releases old packed arrays without shifting retained entries. Cache mutation requires external synchronization when shared between threads.

32 layers x 32 heads x 4K tokens x 128d: 32 GB → ~6 GB

Microsoft Agent Framework: Compressed RAG

using Microsoft.Agents.AI;
using Microsoft.Agents.AI.ChatCompletion;
using TurboQuant;

// Compressed document store
var quantizer = TurboQuantBuilder.Create(dim: 768).WithBits(4).BuildMSE();
var docs = new Dictionary<string, (PackedVector Vec, string Title, string Text, string Link)>();

// Index documents (once)
foreach (var doc in documents)
{
    var embedding = await embeddingModel.GenerateAsync(doc.Text);
    docs[doc.Id] = (quantizer.Quantize(embedding), doc.Title, doc.Text, doc.Link);
}

// Search adapter: quantize query, find top matches via ApproxSimilarity
Task<IEnumerable<TextSearchProvider.TextSearchResult>> SearchAdapter(
    string query, CancellationToken ct)
{
    var queryEmbedding = embeddingModel.Generate(query);
    var queryPacked = quantizer.Quantize(queryEmbedding);

    var results = docs.Values
        .Select(d => (d, Score: quantizer.ApproxSimilarity(queryPacked, d.Vec)))
        .OrderByDescending(x => x.Score)
        .Take(5)
        .Select(x => new TextSearchProvider.TextSearchResult
        {
            Text = x.d.Text,
            SourceName = x.d.Title,
            SourceLink = x.d.Link
        });

    return Task.FromResult(results);
}

// Create agent with compressed RAG
var agent = azureOpenAIClient
    .GetChatClient(deploymentName)
    .AsAIAgent(new ChatClientAgentOptions
    {
        ChatOptions = new() { Instructions = "Answer questions using the provided context." },
        AIContextProviders = [new TextSearchProvider(SearchAdapter, new()
        {
            SearchTime = TextSearchProviderOptions.TextSearchBehavior.BeforeAIInvoke
        })]
    });

100K documents (768d): 292 MB → 37 MB of packed payload and norms in the search adapter.

Semantic Kernel: Compressed Memory

public class CompressedMemoryStore : IMemoryStore
{
    private readonly TurboQuantMSE _quantizer;
    private readonly ConcurrentDictionary<string,
        ConcurrentDictionary<string, (PackedVector Vec, MemoryRecord Rec)>> _collections = new();

    public CompressedMemoryStore(int dim) =>
        _quantizer = TurboQuantBuilder.Create(dim).WithBits(4).BuildMSE();

    public Task<string> UpsertAsync(string collection, MemoryRecord record, CancellationToken ct)
    {
        var store = _collections.GetOrAdd(collection, _ => new());
        var packed = _quantizer.Quantize(record.Embedding.Vector.ToArray());
        store[record.Metadata.Id] = (packed, record);
        return Task.FromResult(record.Metadata.Id);
    }

    public Task<MemoryRecord?> GetNearestMatchAsync(
        string collection, ReadOnlyMemory<float> query, double minScore, CancellationToken ct)
    {
        if (!_collections.TryGetValue(collection, out var store))
            return Task.FromResult<MemoryRecord?>(null);

        var qPacked = _quantizer.Quantize(query.Span);
        return Task.FromResult(store.Values
            .Select(d => (d.Rec, Score: (double)_quantizer.ApproxSimilarity(qPacked, d.Vec)))
            .Where(x => x.Score >= minScore)
            .MaxBy(x => x.Score).Rec);
    }
}

ML.NET Pipeline

var quantizer = TurboQuantBuilder.Create(dim: 384).WithBits(4).BuildMSE();

var pipeline = mlContext.Transforms.CustomMapping<Input, Output>(
    (input, output) =>
    {
        var packed = quantizer.Quantize(input.Features);
        output.Compressed = quantizer.Dequantize(packed);
    },
    contractName: "TurboQuant");

ONNX Runtime: Post-Inference Compression

using var session = new InferenceSession("embedding_model.onnx");
var quantizer = TurboQuantBuilder.Create(dim: 768).WithBits(4).BuildMSE();

using var results = session.Run(inputs);
var outputTensor = results.First().AsEnumerable<float>().ToArray();

var compressed = new PackedVector[batchSize];
quantizer.QuantizeBatch(outputTensor, compressed);

Serialization

// Save
var bytes = packed.ToBytes();
File.WriteAllBytes("vector.bin", bytes);

// Load
var loaded = PackedVector.FromBytes(File.ReadAllBytes("vector.bin"));
var restored = quantizer.Dequantize(loaded);

Rotation Strategies

TurboQuant defaults to random orthogonal rotation (QR decomposition of a Gaussian matrix), which matches the paper exactly and works with any dimension without overhead.

For power-of-2 dimensions (128, 256, 512, 1024), the Hadamard rotation is an excellent alternative: identical quality, O(d log d) time, and O(d) memory instead of O(d^2).

For non-power-of-2 dimensions, Hadamard pads internally to the next power of 2. This adds 33% for 384, 768 and 1536 dimensions, and up to nearly 100% for arbitrary dimensions. A Trace.TraceWarning is emitted when padding is needed.

Quantizers created through the builder or the seeded MSE constructor share immutable rotations for the same dimension, seed and rotation strategy while those rotations remain in use. Weak references allow large matrices to be collected after their quantizers are released. Explicitly supplied custom rotations are used as provided.

// Default: random orthogonal (paper-correct, any dimension)
var quantizer = TurboQuantBuilder.Create(dim: 768).WithBits(4).BuildMSE();

// Fast path for power-of-2 dimensions (recommended for KV cache with head dim 64/128)
var fast = TurboQuantBuilder.Create(dim: 1024).WithBits(4).WithHadamardRotation().BuildMSE();

Best practice: if your embedding model supports configurable dimensions (e.g. OpenAI, nomic), choose a power-of-2 dimension (512, 1024) to get the best of both strategies.

Note on seed correlation: the rotation matrix is generated from a random seed. If your test vectors are generated from the same seed (e.g. new Random(42) for both), the rotation and data become correlated, causing degraded quality. This does not happen in practice (embeddings come from a model, not from Random), but use distinct seeds in synthetic benchmarks.

How It Works

                    ┌──────────────┐
   float[768]  ──>  │  Normalize   │ store ||x||
                    └──────┬───────┘
                           v
                    ┌──────────────┐
                    │   Rotate     │ random orthogonal (default) or Hadamard
                    └──────┬───────┘
                           v
                    ┌──────────────┐
                    │  Quantize    │ per-coordinate Lloyd-Max on Beta(d/2, 1/2)
                    └──────┬───────┘
                           v
                    ┌──────────────┐
                    │  Bit-Pack    │ 4-bit: 2 values/byte, 3-bit: 8 values/3 bytes
                    └──────┬───────┘
                           v
                    PackedVector (data + norm)

The rotation ensures each coordinate is approximately Beta-distributed regardless of input, enabling provably near-optimal scalar quantization. D_mse is within factor sqrt(3)pi/2 ~ 2.7x of the information-theoretic lower bound (Theorem 3 in the paper).

ApproxSimilarity computes cosine similarity directly from precomputed products and squared norms of packed groups. It supports 2/3/4-bit data with zero scratch allocation after the shared lookup tables are initialized. Packed metadata must match the quantizer; serialized vectors must be read with the original rotation configuration.

Project Layout

src/TurboQuant/
  Core/
    Rotation/       RandomRotation (default, paper-correct), HadamardRotation (fast, power-of-2)
    Codebook/       BetaCodebook (Lloyd-Max on exact Beta distribution, with LUT)
    Packing/        BitPacker (2/3/4-bit), PackedVector (serializable, immutable)
    Quantizers/     TurboQuantMSE, TurboQuantAsymmetric
    Simd/           DotProductSimd (Vector<T> auto-ISA: NEON/SSE/AVX2/AVX-512)
  Cache/            KVCache, ResidualWindow
  Diagnostics/      CompressionStats (thread-safe), QuantizationBenchmark
  TurboQuantBuilder.cs

Build & Test

dotnet build TurboQuantNet.slnx
dotnet test
dotnet run --project samples/TurboQuant.Demo
dotnet run -c Release --project benchmarks/TurboQuant.Benchmarks

The benchmark runner accepts filters and job options. For example:

dotnet run -c Release --project benchmarks/TurboQuant.Benchmarks -- --list flat
dotnet run -c Release --project benchmarks/TurboQuant.Benchmarks -- --filter '*StatisticsBenchmarks*' --job short
dotnet run -c Release --project benchmarks/TurboQuant.Benchmarks -- --filter '*CacheReadBenchmarks*' --job short

The suite covers all bit widths, both rotations, variable input vectors, buffer-based dequantization, sequential/parallel batches, statistics overhead, rotation reuse and KV cache reads/appends. Run on each target CPU architecture before making throughput claims.

License

MIT. Based on the paper by Zandieh, Daliri, Hadian & Mirrokni (Google Research / NYU / Google DeepMind).