SurrealForge.Vector 1.2.0

SurrealForge

A SurrealDB toolkit for .NET — a client layer over the official SurrealDb.Net SDK, a C#-first schema/migration engine, a vector-search and embedding-pipeline layer with an optional local ONNX embedding engine, and a Roslyn analyzer that keeps your SurrealQL injection-safe. Extracted from a production workflow engine and released under MIT.

From 1.0.0 the transport is the official SDK (SurrealDb.Net 1.0.0): it owns the socket, the RPC framing, the connection pool and the sign-in handshake. SurrealForge keeps what the SDK does not provide — the per-statement SurrealResponse shape, the CBOR marshalling contract, transport retry, transactions, scoped-credential handling on HTTP, and live-query sequencing and replay.

Every package except the analyzer targets net8.0 and net10.0; SurrealForge.Analyzer stays on netstandard2.0 because Roslyn loads analyzers into the compiler host. netstandard2.0 was dropped from the libraries in 1.0.0 — the SDK has a netstandard2.1 floor — so .NET Framework consumers stay on 0.5.0.

Packages

Package	Target	What it does
SurrealForge.Client	net8.0;net10.0	ISurrealConnection over the SurrealDb.Net SDK, a parameterized SurrealQuery builder, a SurrealIdentifier reserved-word denylist, multi-statement composition, explicit BeginTransactionAsync() with identical result-slot indices on both transaction shapes, native-CBOR write marshalling with a System.Text.Json read path, scoped-credential (JWT) authentication on HTTP, live queries with at-least-once delivery, optional change-feed gap-fill and a durable outbox, and jittered retry on idempotent statements.
SurrealForge.Schema	net8.0;net10.0	Mermaid-ER parser, .surql generator, migration runner backed by a schema_migration checksum table, model-driven reconcile (live introspection + diff + DEFINE … OVERWRITE field evolution). The surrealforge command itself ships separately as SurrealForge.Cli.
SurrealForge.Analyzer	netstandard2.0	Roslyn analyzer SRDB0001 (error severity) — bans string-interpolated / concatenated SurrealQL outside the safe query-builder layer, with one-hop variable resolution to close the most common bypass. Stays on netstandard2.0 because Roslyn loads analyzers into the compiler host, which may be any runtime.
SurrealForge.Vector	net8.0;net10.0	Vector search (indexed HNSW KNN + brute-force vector::similarity::*), an IVectorEncoder abstraction with a mandatory batch overload, content-hash embedding cache, token-budgeted TextChunker, schema-declared embedding backfill jobs (incremental / ad-hoc / LIVE-driven dynamic), and a write-time save interceptor. No ONNX dependency — bring your own encoder.
SurrealForge.Vector.Onnx	net8.0;net10.0	The engine package: WordPiece tokenization, genuinely batched ONNX Runtime inference (one InferenceSession.Run per batch), attention-mask mean/CLS/max pooling, L2 normalisation, and exact-token chunking, all behind one IVectorEncoder. Adds Microsoft.ML.OnnxRuntime 1.29.0 + Microsoft.ML.Tokenizers 2.0.0. Bring your own model.
SurrealForge.Vector.Onnx.MiniLM	net8.0;net10.0	The content package: all-MiniLM-L6-v2 (384-dim) model.onnx + vocab.txt, plus UseMiniLm(). The weights are never committed — eng/Get-MiniLmModel.ps1 fetches them SHA-256-pinned at build time and they pack into the nupkg.
SurrealForge.Cli	net10.0	The surrealforge dotnet tool: up, reset, migrate up\|down\|status\|dry-run, generate-from-assembly, flowcharts-from-assembly. Single-target on purpose — PackAsTool does not survive a TFM-conditioned PropertyGroup, which is how 1.1.0 shipped a tool payload NuGet would not install. Install it, do not PackageReference it.

SurrealForge.Client pulls in SurrealDb.Net 1.0.0, which floors the whole Microsoft.Extensions.* graph at 10.0.7 (via Microsoft.Extensions.Http). A net8.0 consumer therefore resolves .NET-10-wave Microsoft.Extensions.* assemblies; anything pinned lower is an NU1605 downgrade error.

Install

dotnet add package SurrealForge.Client
dotnet add package SurrealForge.Schema    # optional: schema + migrations
dotnet add package SurrealForge.Analyzer  # optional: compile-time SurrealQL safety
dotnet add package SurrealForge.Vector    # optional: vector search + embedding pipeline
dotnet add package SurrealForge.Vector.Onnx         # optional: local ONNX encoder, bring your own model
dotnet add package SurrealForge.Vector.Onnx.MiniLM  # optional: ships all-MiniLM-L6-v2 + UseMiniLm()

The CLI tool:

dotnet tool install -g SurrealForge.Cli
surrealforge --help

Moved in 1.2.0. The tool used to be advertised as dotnet tool install -g SurrealForge.Schema. That never worked: through 1.1.0 the Schema package carried the tool payload but no DotnetTool package type, so NuGet declined to install it. SurrealForge.Schema is now a library on both target frameworks — and finally ships a real lib/net10.0 instead of leaving net10.0 consumers on its net8.0 assets — while the command lives in SurrealForge.Cli. If you worked around this by invoking …/surrealforge.schema/<version>/tools/net10.0/any/SurrealForge.Schema.dll from the restored package, that path is gone in 1.2.0; install the tool.

Quick start

using SurrealForge.Client;
using SurrealForge.Client.Query;

services.AddSurrealForgeSdk(o =>
{
    o.Endpoint  = "ws://127.0.0.1:8000/rpc";   // or http://127.0.0.1:8000
    o.Namespace = "app";
    o.Database  = "main";
    o.User      = "root";
    o.Password  = "root";
});

// Elsewhere, injected:
public sealed class PeopleService(ISurrealExecutor executor)
{
    public Task<IReadOnlyList<Person>> AdultsAsync()
    {
        // Parameterized — never string-interpolate user input into SurrealQL.
        var query = SurrealQuery
            .Of("SELECT * FROM person WHERE age > $min")
            .WithParam("min", 18);

        // The executor maps statement results into typed rows.
        return executor.QueryAsync<Person>(query);
    }
}

AddSurrealForgeSdk registers the SDK client, a singleton ISurrealConnection (SurrealDbNetConnection) and a singleton ISurrealExecutor (DefaultSurrealExecutor). Dispose the container asynchronously: the SDK's SurrealDbClient implements IAsyncDisposable and not IDisposable, and Microsoft's container refuses to dispose such a singleton synchronously. Generic-host and ASP.NET Core apps already do this; a hand-built provider needs await using.

AddSurrealForge(...), the original registration over SurrealForge's own HttpSurrealConnection, was removed in 1.0.0 together with that transport. AddSurrealForgeSdk(...) is the replacement; see the changelog's breaking-changes section.

The SurrealForge.Analyzer package flags any raw interpolation ($"SELECT * FROM {table}") at compile time as SRDB0001, so unsafe query construction fails the build rather than shipping. From 1.0.0 it also covers ExecuteRawAsync / ExecuteMarshalledAsync on connection types and the SDK's RawQuery, and its allowlist is keyed on assembly identity plus an exact namespace — declaring a namespace named after a SurrealForge one no longer exempts anything. The one safe interpolation is the SDK's own Query(QueryInterpolatedStringHandler, …) overload as declared in the real SurrealDb.Net assembly, which binds each hole as a parameter; a look-alike handler in your own code (including a test double) is flagged, because such a type is free to concatenate the holes into the query text.

If you have written your own parameterized query builder — the role SurrealQuery plays inside SurrealForge — you can now grant it the same trust from your own .editorconfig:

[*.cs]
dotnet_diagnostic.SRDB0001.safe_layers = MyApp.Data.Queries,MyApp.Infra.Sql

This cannot reopen the hole the built-in allowlist exists to close, and the reason is structural rather than a check: Roslyn resolves .editorconfig per compilation, so a value set here is only ever visible to analysis of your own project's build and can therefore only exempt code that ends up in your own assembly. Naming SurrealForge.Client.Query in your own file grants trust inside your assembly and nowhere else. Matching is exact-or-proper-sub-namespace (MyApp.Data.Queries covers …Queries.Internal but not …QueriesExtra); a malformed entry discards the whole value and the rule keeps firing everywhere; a blank value is never a wildcard; and the global namespace can never be allowlisted, because code with no namespace declaration has nothing to compare against. Full detail: src/SurrealForge.Analyzer/README.md.

Usage guide

1. Connect

SurrealConnectionOptions configures endpoint, namespace/database, credentials, pool size, timeout, and retry. Depend on ISurrealConnection, not on the concrete transport.

// Switch namespace/database at runtime:
await conn.UseAsync("analytics", "events");

Embedded engines. AddSurrealForgeInMemory() and AddSurrealForgeRocksDb(path) select SurrealDB's in-process engines by endpoint scheme (mem://, rocksdb://). The SurrealDb.Embedded.InMemory / SurrealDb.Embedded.RocksDb packages are deliberately not dependencies — they carry large platform-specific native binaries — so reference the one you want and call AddInMemoryProvider() / AddRocksDbProvider() yourself. If you do not, resolving ISurrealConnection fails with a message naming the package and the call. The RocksDB path is resolved to an absolute path at registration time.

Authentication

Root credentials work on every transport. Namespace- and database-scoped system users must say so:

options.AuthenticationScope = SurrealAuthenticationScope.Database;

Over ws:// those credentials go through the SDK's signin RPC and work unchanged. Over http:// they do not: SurrealDb.Net 1.0.0 authenticates an HTTP endpoint by putting HTTP Basic credentials on POST /rpc, and SurrealDB accepts Basic Auth only for root — a scoped user gets a 401 whose plaintext body then fails the SDK's CBOR decoder. There is no newer SDK to upgrade into.

SurrealForge works around it: it signs in on a root bootstrap client, takes the resulting JWT, and builds a second client with that token — then renews the token before it expires. That is what TokenIssuerUser/TokenIssuerPassword are for, and why they must name a root user:

options.AuthenticationScope = SurrealAuthenticationScope.Database;
options.TokenIssuerUser     = "root";      // mints the token; never queried with
options.TokenIssuerPassword = rootPassword;
// options.ScopedAuthentication defaults to Automatic: exchange on http(s),
// pass the credentials straight through everywhere else.

A missing issuer on a path that needs one fails at registration, not at first query. The container never sees the issuer credentials: the registration stores options.WithoutIssuerCredentials(), and the real values go to the token source as constructor arguments. TokenRenewalMargin (5 min) and TokenClockSkew (30 s) tune renewal; a failed request is retried against a fresh token at most once, and only when the statement is idempotent.

Debug logging leaks credentials. SurrealDb.Net logs its serialized RPC frames as hex at Debug, and carries the templates Connection signed as root, user={username}, password={password}. and Connection signed via token, token={token}. SurrealForge wraps the two clients it builds itself — the root bootstrap client and the per-token client — in a logger factory that drops everything below Information. The SDK client the container registers is not wrapped, because SurrealForge does not construct it, so on the root path Debug on the SurrealDb category still writes credentials to your log.

Rationale and the measured failure modes: src/SurrealForge.Client/Auth/AGENTS.md.

2. Raw parameterized queries

ExecuteRawAsync returns a SurrealResponse — an IReadOnlyList<SurrealStatementResult>, one entry per semicolon-separated statement. Read typed rows with GetValues<T>(index).

var response = await conn.ExecuteRawAsync(
    "SELECT * FROM person WHERE city = $city; SELECT count() FROM person",
    new { city = "Cairo" });

response.EnsureAllOk();                                 // throw on any statement error
IReadOnlyList<Person> matches = response.GetValues<Person>(0);
long total = response.GetValues<long>(1).FirstOrDefault();

Bind parameters with an anonymous object or an IDictionary<string, object?>. On the SDK transport they are marshalled to native CBOR (see §3); on the legacy HTTP transport they are serialized with SurrealJsonOptions.Default. Prefer the fluent SurrealQuery builder to keep parameters and SQL together:

var q = SurrealQuery.Of("SELECT * FROM person")
    .Where("age >= $min", new { min = 21 })
    .OrderBy("name")
    .Fetch("company");                                 // resolve a record link

var adults = await executor.QueryAsync<Person>(q);
var one    = await executor.QuerySingleAsync<Person>(SurrealQuery<Person>.Key("person:jade"));

3. The wire format is CBOR — what that means for your POCOs

The SDK speaks CBOR, and SurrealDB does not coerce CBOR text into typed columns the way it coerces JSON. Measured against 3.2.4 on SCHEMAFULL fields:

Couldn't coerce value for field `amount`: Expected `decimal` but found `'123.45'`
Couldn't coerce value for field `owner`:  Expected `record`  but found `'spike_poco:probe'`

So the write path emits native types rather than the text forms the JSON transport got away with:

• decimal, DateTime/DateTimeOffset, TimeSpan and record ids go out as native CBOR values.
• A string property carrying [References] or [Id] whose value looks like table:id is auto-promoted to a native record id, so existing code needs no change. Opt out per column with [Column(Type = "string")] (also option<string>) or [References(EmitAsString = true)] — the shapes the schema emitter already renders as TYPE string, which is what RELATE edge tables use for in/out.
• A bare id with no table: prefix is not promoted. [Id] properties normally hold the bare form and the table comes from type::record($_t, $_id) in the SQL.
• An ambiguous id half — one that parses as a number, is true/false/ null, or starts with {/[ — is refused rather than promoted. num:42 cannot say whether the row's id is the number 42 or the string "42", and guessing produced a dangling FK that rendered byte-identically in every JSON view. Read-side, a string id that would read back as a number comes out as num:⟨42⟩ using SurrealDB's own escape, so it still round-trips.
• Guid, Uri, char, byte[], DateOnly and TimeOnly stay text on purpose: promoting a Guid to a native CBOR uuid would change the column type of every id field.

The read path is unchanged in shape. SurrealStatementResult.Result is still a JsonElement, so [JsonPropertyName], JsonElement-typed POCO properties and GetValues<T>(int, JsonSerializerOptions) all keep working — CborJsonBridge re-emits each CBOR result as JSON rather than putting a second, divergent decoder in the product.

Two numeric widths changed, both to stop losing data silently: a ulong above long.MaxValue now arrives as decimal (it used to persist as -1), and a high-precision embedded JSON number arrives as decimal rather than a drifted double (0.1234567890123456789012345 used to arrive as 0.12345678901234568).

Things that now throw where they used to be accepted silently — SurrealCborException, or ArgumentException for the parameter-bag shape: non-finite doubles on read and write, default(JsonElement), two properties resolving to the same wire name, an unmapped System.* type, a scalar parameter bag (5m used to bind {"scale":0}, an enum bound zero parameters), a JSON number with no lossless CBOR carrier, and a datetime or duration outside DateTime/TimeSpan range on read. If you were relying on the old silence, you now get an exception at the call site.

The full contract — every SurrealJsonOptions.Default behaviour and how it is reproduced — is src/SurrealForge.Client/Cbor/AGENTS.md. Extra Dahomey converters can be registered from your composition root via SurrealConnectionOptions.ConfigureCborOptions.

4. Strongly-typed query builder

SurrealQuery<T> translates C# expressions to SurrealQL:

var q = SurrealQuery<Person>.From()
    .Where(p => p.Age >= 21 && p.City == "Cairo")
    .OrderByDescending(p => p.Age);

var rows = await executor.QueryAsync<Person>(q);

Collection membership translates to INSIDE in both directions — a constant collection containing a column (statuses.Contains(w.Status) → status INSIDE $status) and a value inside a column array — on net8.0 and net10.0 alike. A string receiver is deliberately excluded and maps to string::contains instead, since INSIDE means collection membership. The net10.0 case did not work before; see the changelog's Unreleased §Fixed for why the two TFMs disagreed.

Typed writes cover inserts, record-shaped upserts, conditional partial updates, and conditional deletes. Mutation builders require a predicate; updates accept multiple typed assignments and use an explicit Unset for SurrealDB's NONE sentinel. Fluent calls are immutable, so a common predicate can safely branch into independent mutations:

var insert = SurrealWriter.Create(new Person { Id = "jade", Name = "Jade" });
var upsert = SurrealWriter.Upsert(new Person { Id = "jade", Name = "Jade A." });

var update = SurrealWriter.UpdateOnly<Person>("person:jade")
    .Where(p => p.City == "Cairo" && p.Age >= 21)
    .Set(p => p.Name, "Jade A.")
    .Build();

var response = await executor.ExecuteAsync(update);
if (response[0].AffectedCount() != 1)
    throw new InvalidOperationException("The conditional update lost its race.");

var delete = SurrealWriter.DeleteOnly<Person>("person:jade")
    .Where(p => p.Name == "Archived")
    .Build();

Prefer these typed primitives for single-table reads and mutations. Raw parameterized SurrealQL remains the standing escape hatch only for atomic multi-table or multi-statement transactions the typed builders cannot preserve. An unsupported single statement, DDL operation, or dynamic administrative query requires a documented owner, reason, and expiry for removal. Never interpolate identifiers or values.

5. EF-style context (SurrealContext)

For a DbContext-like experience with LINQ and change tracking. T must implement ISurrealRecord.

var ctx = new SurrealContext(conn);

// Query — SurrealQueryable<T> is IQueryable<T> with async terminals:
List<Person> adults = await ctx.Set<Person>()
    .Where(p => p.Age >= 18)
    .ToListAsync();

// Change tracking:
ctx.Add(new Person { Id = "person:new", Name = "Sam", Age = 30 });
await ctx.SaveChangesAsync();

FirstOrDefaultAsync, SingleOrDefaultAsync, CountAsync, and AnyAsync terminals are also available.

6. Transactions

The default shape is buffered, on every transport: statements are held client-side while the handle is open and flushed as a single BEGIN; …; COMMIT; on commit. Disposing without committing discards the buffer — nothing was sent — so no server-side transaction can leak. RollbackAsync() is the named, auditable form of the same thing.

await using var tx = await conn.BeginTransactionAsync();
// While the transaction handle is open, calls on the connection are buffered:
await conn.ExecuteRawAsync("CREATE account:a SET balance = 100");
await conn.ExecuteRawAsync("UPDATE account:a SET balance -= 10");
await tx.CommitAsync();   // flushes BEGIN; …; COMMIT; — omit, or RollbackAsync(), to discard

The transaction belongs to the execution context that opened it. ISurrealConnection is registered as a singleton, so SurrealConnectionOptions.StrictTransactionEnlistment (default on) refuses a statement issued from an unrelated flow rather than silently enlisting it — which previously either acknowledged a write that never left the process, or committed it inside a stranger's transaction. Ownership flows down through every await, so the ordinary shape is unaffected however deep the stack.

The two shapes now agree on result-slot indices. A buffered call hands back a response over one pending slot per statement — status OK, no payload — and the commit flush strips the BEGIN and COMMIT slots off its own response and deals the real results back into those same slots, in order. The response object you have been holding since the call fills itself in. So GetValues<T>(i) means the same thing on buffered, server-side and no-transaction paths; only the timing differs, and it differs in the one way it cannot not: reading a result before committing a buffered transaction has nothing to read, because nothing has been sent. Attribution is checked rather than assumed — if the predicted slot count does not match the flush, the pending slots are left alone rather than filled with mis-attributed results.

SurrealDbNetConnection.UseServerSideTransactions opts in to a real server-held transaction on a stateful session. It stays off by default, now for the one reason that survives result-slot parity:

Transaction duration. A server-held transaction's statements reach the server as they are issued, so it is open on the server for the whole life of the handle — including whatever your code does between statements. The buffered shape sends nothing until commit, so it holds nothing open. That is a real operational difference and it should be a decision, not a default.

Parameter names must be in the canonical form on every path. All three — buffered, server-side, and no-transaction — take the naming convention's spelling (see Cbor/AGENTS.md), and a non-canonical spelling such as $SomeLabel fails loudly rather than binding to NONE. This is a change: the buffered flush used to rewrite the raw CLR spelling onto its scoped name as well, so a PascalCase parameter bound correctly inside a buffered transaction and bound to NONE everywhere else — the same code meaning different things depending on a setting, failing silently in the direction that is hardest to notice. See the changelog for the migration note.

Turning it on only takes effect where the endpoint can hold a transaction: measured on 3.2.4 with SDK 1.0.0, ws:// can and http:// cannot — the HTTP engine does not implement the SDK's transaction interface at all (SupportsTransactions() answers false, CreateSession() succeeds and session.BeginTransaction() throws NotSupportedException), because /sql is one request with no session continuity. That is a protocol fact, not a gap: enabling the option on HTTP degrades to the buffered shape and records why in ServerSideTransactionUnavailableReason. Set RequireServerSideTransactions to turn that fallback into an error instead.

Commit atomicity is enforced, not inherited. Measured on SurrealDB 3.2.4, a BEGIN; …; COMMIT; script aborts wholesale on the first statement error, but a session-held transaction does not — a duplicate-record ERR between two CREATEs left both of them committed. So a server-side transaction that produced a statement-level ERR now refuses to commit, throwing the same SurrealStatementException the buffered flush would have thrown, with the same slot index; the handle is left open and un-spent for your RollbackAsync/DisposeAsync. ExecuteRawAsync itself still never throws on a per-statement ERR in either shape, so a caller that reads a duplicate-insert ERR as an expected race-loss is unaffected. A failed buffered commit now blames the culprit statement rather than SurrealDB's collateral slot.

Composing a transaction as one BEGIN; …; COMMIT; string through ExecuteRawAsync with no handle open is unaffected by any of this. Issuing a bare BEGIN/COMMIT/CANCEL while a handle is open is refused in both shapes, because SurrealDB rejects the nesting statement-by-statement and the extra ERR slots would shift every indexed read.

7. Live queries

SurrealLiveClient opens LIVE SELECT subscriptions with at-least-once delivery, sequencing, an outbox and explicit acknowledgement on top of the SDK's live-query transport. It needs a stateful transport: the SDK answers every live method on an http(s):// endpoint with NotSupportedException, so point the client at ws(s)://…/rpc.

// Registration — in addition to AddSurrealForgeSdk, which supplies the client.
// Registers a singleton SurrealLiveClient and a singleton SurrealLiveOptions.
// Resolving the client fails fast, naming the endpoint, if it is not ws(s)://.
services.AddSurrealForgeLive(o => o.MaxUnacknowledgedNotifications = 4096);

// …then inject SurrealLiveClient and SurrealLiveOptions (the options are passed
// per subscription, not held by the client, so a subscription can override them).
await using var subscription = live.Subscribe<Person>(
    "LIVE SELECT * FROM person WHERE city = $city",
    new { city = "Cairo" },
    options);

// Implicit acknowledgement: notification N is released when the body comes
// back for N+1, so a body that throws leaves N replayable.
await foreach (LiveNotification<Person> n in subscription.ReadNotificationsAsync())
    Console.WriteLine($"{n.Action}: {n.Record.Name}");   // Create / Update / Delete

For batching, hand-off to another thread, or anything that writes durably before admitting it handled a notification, use the explicit shape — nothing is released until you say so:

await foreach (SurrealLiveDelivery<Person> d in subscription.ReadAsync())
{
    await HandleAsync(d.Record);
    d.Acknowledge();          // cumulative and monotonic
}

The guarantee, and its limits, in the same breath:

• Every notification this client received is delivered at least once, in sequence order, and is not released until acknowledged. It survives socket drops, transport faults, re-subscription, an abandoned reader, and a loop body that throws.
• Anything delivered but not acknowledged before a re-subscribe or reader restart can be duplicated. SurrealLiveDelivery<T>.IsRedelivery, .DeliveryAttempt and LiveNotification<T>.Sequence make that visible; consumers must be idempotent.
• Out of the box it is not end-to-end. Changes committed while the socket was down are never sent by SurrealDB and no client-side outbox can invent them. SurrealLiveDelivery<T>.FollowsServerGap marks the first notification on the far side of that window so you can resynchronise with a query; it flags the gap, it does not fill it. Gap-fill closes that window when — and only when — the table has a change feed (see below).
• The outbox is in memory and bounded by default. SurrealLiveOptions.MaxUnacknowledgedNotifications defaults to 1024, counting delivered-but-unacknowledged and received-but-not-yet-pulled notifications. With the default SurrealLiveMemoryOutboxStore the reader throws SurrealLiveOutboxOverflowException at the bound, the server-side query is KILLed and Status becomes Faulted — everything already accepted is delivered first. Drop-oldest would manufacture the exact silent gap this exists to prevent; backpressure would stall every other subscription multiplexed on the same socket. Nothing survives the process: disposing discards unacknowledged notifications. A durable store changes both halves of that (see below).
• The SDK reconnects the socket; it does not reconnect the live query. SurrealForge re-subscribes with a doubling backoff (200 ms → 30 s, indefinitely); ReconnectCount and LastReconnectError are the diagnostics. The first open is deliberately not retried, so a bad table name or an http:// endpoint fails loudly instead of becoming a subscription that silently never delivers.

Gap-fill: recovering the socket-down window from the change feed. Opt in with SurrealLiveOptions.GapFill. On reconnect the subscription reads SHOW CHANGES FOR TABLE … SINCE <versionstamp> from its watermark to the feed's head and replays those changes through the same sequencing, outbox and acknowledgement path as live frames, flagged FollowsRecoveredGap with Source == ChangeFeed.

// The table must have been defined with a change feed. INCLUDE ORIGINAL is
// optional but strongly preferred — see the fidelity caveats below.
// DEFINE TABLE person CHANGEFEED 1h INCLUDE ORIGINAL;
await using var subscription = live.SubscribeToTable<Person>(
    "person",
    new SurrealLiveOptions { GapFill = new SurrealLiveGapFillOptions() });

It refuses out loud rather than quietly under-delivering. SurrealLiveSubscription<T>.LastGapFillOutcome is the honest answer, and only Recovered and NothingToRecover mean the window is covered — Disabled, ChangeFeedUnavailable, WatermarkExpired, WindowTooLarge and Failed each name a reason, and every one of them falls back to exactly the pre-gap-fill FollowsServerGap behaviour. A failed gap-fill degrades; it does not fault, because turning the feature on must never be riskier than leaving it off.

What it costs and what it cannot do, all measured against SurrealDB 3.2.4:

• It re-reads every change on the watched table once per GapFill.WatermarkInterval (default 30 s). There is no "current versionstamp" function and SHOW CHANGES cannot be used as a subquery, so reading the feed is the only way to learn where its head is.
• A recovered change is a new sequence carrying a change you may have already seen — not a redelivery — so IsRedelivery is false on it. Deduplicate on the record, not the sequence.
• On a plain feed a recovered create is reported as Update, because CREATE t:a SET n = 1 and an update record the same {"update": {…}} shape; and a recovered delete carries only the record id. CHANGEFEED … INCLUDE ORIGINAL fixes both. Prefer it on any table you intend to gap-fill.
• Subscribe<T> (raw LIVE SELECT) cannot infer a table, so GapFill.Table must be set explicitly or the constructor throws. SubscribeToTable<T> fills it in. A filtered raw subscription recovers the whole table's window, not the filtered slice — more, never less.
• Watermarks are versionstamps only: SINCE <datetime> does not resolve on 3.2.4, and SINCE is inclusive.

A durable outbox: surviving the process. ISurrealLiveOutboxStore has two implementations. SurrealLiveMemoryOutboxStore is the default described above. SurrealLiveTableOutboxStore keeps rows in a SurrealDB table (sf_live_outbox by default), keyed by (subscriptionId, sequence) and deleted on acknowledgement, with one checkpoint row per subscription holding the ack mark, the received mark and the gap-fill watermark.

var options = new SurrealLiveOptions
{
    OutboxStore = new SurrealLiveTableOutboxStore(live.QueryExecutor, "worker-1"),
};

A new process opening the same SubscriptionId resumes the same sequence line, redelivers everything unacknowledged (flagged IsRedelivery, because the process that died may have handed it over first) and picks the watermark up, so a restart is just another gap-fill window. Being durable, it also lifts the bound: MaxUnacknowledgedNotifications becomes the resident page size rather than a fault threshold, and nothing overflows. The costs are real and worth stating: one round trip per received notification, a table that grows if the consumer never acknowledges, and no lease or fencing token — nothing stops two live processes opening the same SubscriptionId and interleaving on one sequence line, so treat a subscription id as owned by exactly one writer.

Design notes and the measured SDK behaviour behind each choice: src/SurrealForge.Client/Live/AGENTS.md.

WebSocketSurrealConnection.LiveAsync<T> — the hand-rolled JSON-RPC socket that carried live queries before 1.0.0 — was removed with the rest of the legacy transport. The IQueryable<T> extension ExecuteLiveAsync<T> survives and now takes a SurrealLiveClient: it folds the deferred chain, rewrites the leading SELECT to LIVE SELECT, and streams LiveNotification<T> with implicit acknowledgement, disposing the subscription when enumeration ends.

Idempotency ledger

SurrealForge.Client.Idempotency ships an exactly-once execution ledger for irreversible operations, backed by a SurrealDB table with a UNIQUE index on the key (SurrealIdempotencyLedger — TryClaimAsync / CompleteAsync / FailAsync / GetAsync). Behaviour that used to live in each consuming app is now folded into the package and turned on through options:

using SurrealForge.Client.Idempotency;

var ledger = new SurrealIdempotencyLedger(executor, new IdempotencyLedgerOptions
{
    // Retry the claim on SurrealDB 3.x RocksDB transient write-write conflicts
    // ("Transaction conflict: Resource busy … can be retried"). Off by default.
    RetryOnTransientConflict = true,
    MaxConflictRetries       = 8,

    // Base64url-encode stored keys that contain a ':' (the record-id separator),
    // transparently decoded on read. Off by default; the deterministic record id
    // is always SHA-256 of the ORIGINAL key.
    EncodeColonKeys = true,
});

// Or bind from appsettings.json under SurrealDb:Idempotency:
services.Configure<IdempotencyLedgerOptions>(config.GetSection("SurrealDb:Idempotency"));

Two more application-agnostic helpers round out the surface:

• SurrealTransientConflict — the standalone bounded retry primitive (RetryOnConflictAsync + IsRetryableConflict) for any contended single-winner claim, usable independently of the ledger.
• IdempotencyReplay — the content-hash key (ContentHash), the JSON round-trip (SerializeForReplay / DeserializeForReplay), and the replay state machine (ReplayFromRecord). The state machine is generic over your own result envelope via IReplayResultFactory<T, TResult> (or two lambdas), so the package never depends on any app's result type.

Schema as C# — source of truth

Decorate POCOs with the schema attributes from SurrealForge.Client; the SurrealForge.Schema generator reflects over a compiled assembly and emits deterministic (byte-stable) .surql schema files. Regeneration is idempotent, so a CI drift-check keeps the generated SQL and the C# in lockstep.

surrealforge generate-from-assembly path/to/YourApp.dll
surrealforge migrate up --endpoint http://127.0.0.1:8000

Model-driven migrations — evolving an existing field

The checksum-tracked file applier can create tables/fields, but the idempotent DEFINE … IF NOT EXISTS it emits can never alter an existing one — so a change like option<string> → array<string> on a live table used to silently no-op. surrealforge up now closes that gap with a reconcile pass:

1. introspect the live schema (INFO FOR DB / INFO FOR TABLE) into the same model shape the C# scanner produces,
2. diff it against the desired model (parsed from the generated .surql, or from --assembly <dll>),
3. emit DEFINE FIELD OVERWRITE … TYPE <new> for each drifted field/index and apply it — OVERWRITE replaces the definition, so the type actually evolves.

# Apply schema files + migrations, THEN reconcile drifted field types/defaults:
surrealforge up --connection http://127.0.0.1:8000 \
                --namespace app --database main \
                --schemas-dir ./Schemas --migrations-dir ./Migrations

surrealforge up ... --dry-run            # print the OVERWRITE plan, write nothing
surrealforge up ... --allow-destructive  # also apply drops / narrowing type changes
surrealforge up ... --no-reconcile       # legacy additive-only apply (skip reconcile)

Reconcile runs by default; --force guarantees it. Destructive changes (field/table/index removal, narrowing type changes) are planned but skipped unless --allow-destructive is set. If existing row data can't coerce to a new type, the apply fails with a clear SchemaCoercionException (exit code 4) naming the field — never a silent corruption. See src/SurrealForge.Schema/Migration/AGENTS.md.

Vector search and embeddings

SurrealForge.Vector adds KNN search and an embedding pipeline on top of the client. It has no ONNX or model dependency — you supply the encoder, so the package stays small and you keep control of where embedding CPU is spent. If you want a local encoder rather than an API call, add SurrealForge.Vector.Onnx (engine) and optionally SurrealForge.Vector.Onnx.MiniLM (weights); see §Local ONNX encoder.

Searching

Declare the vector column and its index on the model, reconcile as usual, then search through any ISurrealExecutor:

[SurrealTable("article")]
public sealed class Article
{
    [Column(Order = 1)]
    public string? Title { get; set; }

    [Column(Order = 2, Type = "array<float>")]
    [HnswIndex("hnsw_article_embedding", Dimension = 384, Distance = "COSINE")]
    public float[]? Embedding { get; set; }
}

// Indexed HNSW KNN — the fast path.
var hits = await executor.VectorSearchAsync<Article>("embedding", queryVector, k: 10);

foreach (var hit in hits)
    Console.WriteLine($"{hit.Record.Title} (dist {hit.Distance:F4})");

The embedding is always bound as $q, never interpolated — the SurrealForge.Vector namespace is on the SRDB0001 allowlist for exactly this reason. K and EF are range-checked ints (SurrealDB cannot parameterize them).

Both search paths are available:

// Brute-force, for un-indexed or small tables — no index required.
var hits = await executor.VectorSearchAsync<Article>(
    "embedding", queryVector,
    new VectorSearchOptions
    {
        K = 10,
        Strategy = VectorSearchStrategy.BruteForce,
        Metric = VectorMetric.Cosine,          // or Euclidean / Manhattan
        Filter = SurrealQuery.Of("status = $s").WithParam("s", "published"),
    });

Filter merges an extra parameterized predicate into the same WHERE on either path. On the indexed path the index's own metric applies, so Metric is ignored there; Ef (the HNSW beam width) defaults to max(K, 40).

SurrealDB 3.x note: the single-argument <|K|> operator was removed, so the indexed path always emits <|K,EF|>. Verified live against SurrealDB 3.2.4.

Embedding pipeline

Encoders implement IVectorEncoder; the batch overload is mandatory, because batching is the only real throughput lever:

public interface IVectorEncoder
{
    int Dimension { get; }
    ValueTask<float[]> EncodeAsync(string text, CancellationToken ct = default);
    ValueTask<float[][]> EncodeAsync(IReadOnlyList<string> texts, CancellationToken ct = default);
}

Search embeds one string per query — negligible. Inserts embed N documents — the entire cost center. So where embedding happens is a per-field choice declared on the schema, not baked into the write path.

Both modes now execute. Batched writes land with a null or stale vector and a hosted worker fills them in; WriteTime encodes inline as part of the save.

[SurrealTable("article")]
public sealed class Article
{
    // Source text -> target vector column, with the mode declared inline.
    [Column(Order = 1)]
    [Embedded("embedding", Mode = EmbeddingMode.Batched)]   // or WriteTime
    public string? Body { get; set; }

    [Column(Order = 2, Type = "array<float>")]
    [HnswIndex("hnsw_article_embedding", Dimension = 384, Distance = "COSINE")]
    public float[]? Embedding { get; set; }
}

The vector column can also be schema-only — put [ExtraSurrealField("embedding", "array<float>")] on the class instead, and the column exists in SurrealDB with no CLR property surfacing it.

EmbeddingSchemaScanner turns those declarations into job definitions, and the hosted EmbeddingBackfillService drains them through a bounded channel (backpressure, not unbounded memory):

services.AddSurrealForgeSdk(...);              // executor, from SurrealForge.Client
services.AddSurrealVectorSearch(o =>
{
    o.AddEncoder(new MyEncoder());             // default profile
    o.AddJobsFrom<Article>();                  // Batched fields become jobs
});

Backfill runs in three shapes — incremental (checkpointed pages over a historical window, resumable), ad-hoc (run-once over an explicit range, e.g. after a model upgrade), and dynamic (LIVE-query driven). Re-embedding unchanged text is a no-op: IEmbeddingCache is keyed by content hash, so write-time, incremental, and ad-hoc paths all skip work they've already done. TextChunker splits long documents into overlapping token-budgeted windows using char/token heuristics — no model dependency (SurrealForge.Vector.Onnx ships an exact-token OnnxTextChunker that drops into the same shape).

Dynamic jobs are wired onto SurrealLiveClient, and you no longer supply the adapter. o.AddLiveJobsFrom<Article>(client) — where client is the SDK's ISurrealDbClient — is the one-liner; EmbeddingBackfillJob.CreateDynamic(definition, client) and EmbeddingLiveSource.ForTable(client, definition) are the lower-level forms. Any IAsyncEnumerable<EmbeddingSourceChange> still works, since the adapter is a supplier of that delegate, not a new pipeline. Two properties of it are load-bearing:

• Acknowledgement follows the write, not the read. The adapter uses the explicit ReadAsync() shape and hands the runner an OnCommitted callback, so a notification is released only once its embedding write has committed. Live's acknowledgement is cumulative, so getting this backwards would release earlier unwritten rows too.
• A job watching the table it writes to no longer re-encodes its own echo. EmbeddingLiveSourceOptions.SkipRowsAlreadyEmbedded is on by default: the adapter reads the notification's own {target}_hash column and hands it over as StoredHash, so the consumer's existing content-hash comparison catches the echo. Measured against SurrealDB 3.2.4, one scripted run went from five write statements to three. It can only ever suppress a no-op — the two hashes compared are independently derived from whatever text each write actually saw — and a missing or non-string hash column simply falls through to an unconditional write. Set it false to opt back into that everywhere.

Live queries need ws(s):// and ForTable throws at configuration time on an http(s):// client, because a subscription that can never deliver should fail where it was written. And a dynamic job's guarantee is Live's guarantee: it is at-least-once from the socket inward, so it cannot be the only thing keeping a vector column current. Pair it with a sweep — FollowsServerGap and DynamicBackfillConfig.OnServerGap are the triggers, and which sweep to run is deliberately not defaulted.

Write-time mode runs through ISurrealSaveInterceptor, a seam SurrealContext offers and Vector fills (Client must not learn about vectors). AddSurrealVectorSearch registers the interceptor with TryAddSingleton; attach it with context.UseWriteTimeEmbeddings(interceptor). Three decisions inside it are worth knowing:

• Encoding happens before the transaction opens. One PrepareAsync sees the whole save, so every row goes through the encoder in a single batch call per profile, and no model call is ever made while a server-held transaction is open. The cost is a wasted encode if the commit later fails — the cheaper failure.
• The vector is a second statement, not a rewritten first one. Each embedded column contributes an UPDATE … SET target = $v, target_hash = $h immediately after the entry's own CREATE/UPSERT and inside the same BEGIN..COMMIT, so row and vector commit or roll back together. On the buffered shape that costs no extra round trip.
• A failed encode fails the save (WriteTimeEncodeFailure.FailSave, the default) — SaveChangesAsync leaves the change tracker intact, so a retry is the natural next move. WriteTimeEncodeFailure.LeaveForBackfill degrades instead, and writes neither the vector nor the hash: writing the hash alone would permanently poison the backfill's content-hash skip for that row.

Design rationale and the full job/config contract: src/SurrealForge.Vector/AGENTS.md.

Current limitation: the schema emitter does not yet auto-define the vector and hash columns from [Embedded] — the attribute is inert to AttributeSchemaScanner — so declare target and {target}_hash yourself (or run the table SCHEMALESS).

Local ONNX encoder

SurrealForge.Vector.Onnx is the engine: tokenize → infer → pool → normalise, exposed as one IVectorEncoder. SurrealForge.Vector.Onnx.MiniLM is the content package that ships all-MiniLM-L6-v2 and registers it.

services.AddSurrealVectorSearch(o =>
{
    o.UseMiniLm();                 // default profile, 384-dim, mean-pooled, L2-normalised
    o.AddJobsFrom<Article>();
});

o.UseOnnxEncoder(profile, …) takes your own model.onnx + vocab.txt instead. Both have a container-owned variant (services.AddSurrealOnnxEncoder(...) / AddSurrealMiniLmEncoder(...) plus o.UseRegisteredOnnxEncoder(profile)) for hosts that build and tear down providers repeatedly; the self-owned form caches the session for the life of the process. Either way session construction is deferred to first use, so a container built before the model is on disk still starts.

• The batch overload is the whole point. EncodeAsync(IReadOnlyList<string>) slices into MaxBatchSize groups and issues one InferenceSession.Run per group, padding each group to the longest sequence in that group rather than to MaxSequenceLength. Padded positions are masked out of pooling, so a short input embeds identically alone and beside a long one.
• EncodeAsync returns ValueTask and runs synchronously. session.Run() is CPU-bound and blocking; wrapping it in Task.Run would buy a thread hop and no concurrency. Real asynchrony is EmbeddingMode.Batched plus batching.
• IntraOpNumThreads defaults to 1. ORT defaults to every core, which thrashes on a fractional-vCPU container.
• CPU execution provider only. GPU providers are not wired.
• The MiniLM weights are a build input, never a committed file. eng/Get-MiniLmModel.ps1 fetches them SHA-256-pinned into a git-ignored assets/ directory and they pack into the nupkg from there; -p:SurrealForgeRequireMiniLmAssets=true turns a missing asset into a pack-time error. A local dotnet pack without them produces a package with no model, so pass that flag when it matters. At runtime the probe order is explicit option → SURREALFORGE_MINILM_MODEL/_VOCAB → <app>/minilm/<file> → <app>/<file>, and failing to resolve raises MiniLmAssetsNotFoundException listing every path probed and all three fixes.
• The fp32 export is the default, not the quantized one, and that is a deliberate deviation from the original plan. Dynamic quantization derives its scales from the runtime tensor, so padding rows shift the dynamic range and a stored vector depends on what else was in the batch (measured batched-vs-alone cosine 0.982927, max component delta 0.032379; fp32 is 1.000000 / 0.000000) — which defeats the content-hash idempotency the whole embedding pipeline rests on. -Variant Quantized is one flag away for memory-constrained deployments that accept the trade. See src/SurrealForge.Vector/DESIGN.md §Deviation.
• Mean pooling, L2 normalisation, lower-casing and Dimension = 384 are not configurable on MiniLM. They are properties of the published checkpoint, not preferences, and the model is not Matryoshka — a truncated vector is a broken vector, not a cheaper one.

Building from source

dotnet restore SurrealForge.slnx
dotnet build   SurrealForge.slnx -c Release
dotnet test    SurrealForge.slnx

• Unit tests (SurrealForge.Client.Tests, .Schema.Tests, .Analyzer.Tests, .Vector.Tests) run with no external dependencies.
• Integration tests (SurrealForge.Client.IntegrationTests) need a running SurrealDB.
• ONNX tests (SurrealForge.Vector.Onnx.Tests) run entirely against stubs and a hand-serialized few-KB ONNX graph; only LiveMiniLmEncoderTests wants the real 90 MB weights.

Run the live suites deliberately, and check the duration. Both suites can be run without their dependency, and a suite that never reached its dependency is not evidence of anything:

# SurrealDB on 8020 — NOT 8000, which on the maintainer's machine is a
# different service that answers health checks and is not SurrealDB.
podman run -d --name surrealforge-test -p 8020:8000 \
  surrealdb/surrealdb:v3.2.4 start --user root --pass root memory

export SURREALFORGE_TEST_ENDPOINT=http://127.0.0.1:8020
dotnet test tests/SurrealForge.Client.IntegrationTests

# The ONNX suite's live tests need the weights fetched first.
pwsh eng/Get-MiniLmModel.ps1
dotnet test tests/SurrealForge.Vector.Onnx.Tests

The fixtures fail rather than report green when their dependency is unreachable; skipping is an explicit opt-out that reports Skipped, not passed — set SURREALFORGE_ALLOW_MISSING_LIVE=1 to skip the integration suite or SURREALFORGE_ALLOW_MISSING_MINILM=1 to skip the ONNX suite's live-model tests, and note that the live suite needs SURREALFORGE_TEST_USER=root / SURREALFORGE_TEST_PASS=root — dbuser/dbpass fails root-level /version auth on the dev container and corrupts CBOR parsing. The history here is worth carrying: with SURREALFORGE_TEST_ENDPOINT unset the integration suite used to probe 127.0.0.1:8000, find something that was not SurrealDB, and report 38 passed in 43 ms — a real run takes ~14 s. Duration is the tell.

Versioning

The version lives in one place — Directory.Build.props — and applies to all seven packages, which are released in lockstep. Publishing happens via the publish GitHub Actions workflow on a release tag (v0.1.0, …).

License

MIT — see LICENSE.

Status

1.0.0 moved the transport onto the official SurrealDb.Net 1.0.0 SDK and was the first release with a stable public API commitment; it carried a large set of breaking changes (netstandard2.0 dropped from three packages, a JSON→CBOR wire change with native-type marshalling, marshalling faults that throw instead of corrupting silently, a tightened SRDB0001, a Microsoft.Extensions.* floor of 10.0.7). Read CHANGELOG.md before upgrading from 0.x.

Since then: schema reconcile reaches a fixed point (1.0.1), and unreleased on main — live-query gap-fill and a durable outbox, transaction result-slot parity plus enforced commit atomicity, the Contains → INSIDE fix on net10.0, a consumer-configurable analyzer allowlist, executed EmbeddingMode.WriteTime and first-class LIVE wiring for dynamic embedding jobs, and the two new ONNX packages. That set contains one breaking change — buffered transactions no longer accept raw CLR parameter spellings. See the changelog's Unreleased section.

Known limitations / roadmap:

• Live-query delivery is end-to-end only with gap-fill on and a change feed present. By default the guarantee is at-least-once from the socket inward: changes committed while the socket was down are never sent, and FollowsServerGap flags the window rather than filling it. Turning on SurrealLiveOptions.GapFill against a table defined with CHANGEFEED closes that window from the feed — and LastGapFillOutcome names the reason whenever it cannot (Disabled, ChangeFeedUnavailable, WatermarkExpired, WindowTooLarge, Failed), falling back to exactly the old behaviour. Even when it works, the fidelity is bounded: without CHANGEFEED … INCLUDE ORIGINAL a recovered create is reported as Update and a recovered delete carries only the record id, Subscribe<T> (raw LIVE SELECT) cannot infer a table so GapFill.Table must be set explicitly, and gap-fill costs one extra read of the watched table's feed per watermark interval. See §7.
• The durable outbox is single-writer and unfenced. SurrealLiveTableOutboxStore carries unacknowledged notifications across a process restart and lifts the in-memory bound, but nothing stops two processes opening the same SubscriptionId and interleaving on one sequence line. There is no lease and no fencing token. The default SurrealLiveMemoryOutboxStore still faults at MaxUnacknowledgedNotifications.
• Server-side transactions are opt-in (UseServerSideTransactions) and unavailable over HTTP — but no longer for result-index reasons: the two shapes now hand back identical result slots. What remains is transaction duration: a server-held transaction's statements reach the server as they are issued, so it is open for the whole life of the handle, including whatever your code does between statements. Parameter-name spelling is no longer a reason to prefer one shape over the other, either: all three paths now require the naming convention's canonical form, and a non-canonical spelling such as $SomeLabel fails loudly on every one of them instead of binding correctly on buffered and silently to NONE everywhere else.
• HTTP genuinely cannot hold a server-side transaction. The SDK's HTTP engine does not implement transactions at all, because /sql is one request with no session continuity, so the option degrades to buffered there and records why in ServerSideTransactionUnavailableReason. RequireServerSideTransactions turns that fallback into an error.
• SurrealDB 3.2.4 commits a session-held transaction containing a statement error. SurrealForge enforces the abort client-side rather than inheriting it: a server-side transaction that produced a statement-level ERR refuses to commit. That is a client-side guard over a server behaviour, not a fix to the server — anyone reaching the same server through the SDK directly still gets the partial commit.
• The schema emitter does not surface [Embedded]. The attribute is inert to AttributeSchemaScanner, so both write-time and backfill paths assume the target vector column and its {target}_hash sibling already exist (or that the table is SCHEMALESS). Declaring them yourself is the workaround.
• The vector packages have no live integration suite of their own. The dynamic-job and write-time paths are proven by unit tests over the ISurrealLiveChannel seam plus a scripted live run against SurrealDB 3.2.4; the repo's live suite (SurrealForge.Client.IntegrationTests) has no Vector reference, so a SurrealForge.Vector.IntegrationTests project is outstanding.
• The ONNX encoder is CPU-only. GPU execution providers are not wired, and EncodeAsync is synchronous under a ValueTask — see §Local ONNX encoder for why dressing it up would be a lie.
• The legacy HttpSurrealConnection / WebSocketSurrealConnection / SurrealConnectionPool / SurrealTransaction and the AddSurrealForge(...) registration are gone. 1.0.0 removed them outright rather than shipping two transports; SurrealDbNetConnection and AddSurrealForgeSdk(...) are the whole surface.
• Still on the roadmap: a sample app, and TextChunkerOptions growing a TokenCounter seam so SurrealForge.Vector.Onnx's exact-token chunker can be an adapter over the core one instead of a duplicate.