QueryFarm.Vgi 0.18.1

<p align="center"> <img src="https://raw.githubusercontent.com/Query-farm/vgi-csharp/main/docs/vgi-logo.png?v=2" alt="Vector Gateway Interface logo" width="320"> </p>

<h1 align="center">VGI for .NET</h1>

<p align="center"> Add your own functions and tables to DuckDB with C# and Apache Arrow.<br> Built by <a href="https://query.farm">🚜 Query.Farm</a> </p>

<p align="center"> <a href="https://github.com/Query-farm/vgi-csharp/actions/workflows/ci.yml"><img src="https://github.com/Query-farm/vgi-csharp/actions/workflows/ci.yml/badge.svg" alt="CI"></a> <a href="https://www.nuget.org/packages/QueryFarm.Vgi"><img src="https://img.shields.io/nuget/v/QueryFarm.Vgi" alt="NuGet"></a> <a href="https://www.nuget.org/packages/QueryFarm.Vgi"><img src="https://img.shields.io/nuget/dt/QueryFarm.Vgi" alt="NuGet downloads"></a> </p>

A VGI worker is a small .NET program that DuckDB talks to over Apache Arrow IPC. It can expose scalar / table / table-in-out / table-buffering / aggregate functions and whole catalogs (schemas, tables, views, macros) that behave like native DuckDB objects. DuckDB launches your worker for you when a query needs it — you never run a server by hand.

This repo is the C# worker SDK (QueryFarm.Vgi). It is wire-compatible with the canonical Python SDK and the Go/Rust/Java/TypeScript ports, so a C# worker drops in behind the same ATTACH ... (TYPE vgi). Built on vgi-rpc-csharp; targets .NET 10.

Status: full parity. All 356 sqllogictests in the canonical ~/Development/vgi/test/sql/integration/** suite pass — the same unmodified suite the Python/Go/Rust/Java ports are graded against. See docs/roadmap.md for the milestone history.

Why a worker instead of a C++ extension?

Reach for it when you want to: call REST APIs or external services from SQL, run ML inference (ML.NET, ONNX Runtime, etc.), expose an external database/API/filesystem as a queryable catalog, or ship domain-specific functions to your team as one binary.

Your first worker

1. Add the package:

dotnet add package QueryFarm.Vgi

2. Write a function and serve it:

using Apache.Arrow;
using Apache.Arrow.Types;
using QueryFarm.Vgi;
using QueryFarm.Vgi.Attributes;
using QueryFarm.Vgi.Scalar;

var worker = new Worker()
    .CatalogName("example")
    .DefaultSchema("main")
    .RegisterScalar(new UpperCaseFunction());

await worker.RunFromArgsAsync(args);

public sealed class UpperCaseFunction : ScalarFn
{
    public override string Name => "upper_case";

    private void Compute([Param] StringArray value, StringArray.Builder result)
    {
        for (var i = 0; i < value.Length; i++)
        {
            if (value.IsNull(i)) { result.AppendNull(); continue; }
            result.Append(value.GetString(i).ToUpperInvariant());
        }
    }
}

ScalarFn reflects Compute's parameters once per subclass and dispatches per batch — no manual Arrow-schema bookkeeping needed for the common case.

Scalar functions can override ArgumentMonotonicity with an IReadOnlyList<ArgumentMonotonicity>. The optional list is in ArgumentsSchema declaration order and must have exactly one entry per field. Fixed, defaulted, and constant arguments each occupy one slot; a vararg declaration occupies one slot regardless of call-time expansion. Named SQL invocation order does not reorder the claims; null makes no claims.

3. Build it (dotnet build -c Release), then call it from a DuckDB engine that has the vgi extension. The vgi extension currently ships with Query Farm's Haybarn DuckDB distribution, which starts with no install via uvx haybarn-cli. Stock duckdb works too via INSTALL vgi FROM community.

INSTALL vgi FROM community;
LOAD vgi;

-- LOCATION is the command DuckDB runs to launch the worker; the first ATTACH argument names
-- the catalog it appears under (independent of what the worker itself calls itself).
ATTACH 'example' AS example (TYPE vgi, LOCATION './my-worker');

SELECT example.upper_case('hello'); -- => 'HELLO'

Troubleshooting

• ATTACH can't find the worker — LOCATION is resolved relative to DuckDB's working directory, not your project. Use an absolute path if in doubt.
• Catalog Error: ... does not exist — qualify with the attach alias (example.upper_case) or run USE example;.
• Runtime / type errors — exceptions thrown from Bind/Compute (and bind-time type-bound checks) surface directly in DuckDB's error message.

Function shapes

Each raw interface is a small, direct implementation surface (see any fixture under fixtures/QueryFarm.Vgi.ExampleWorker/ for real examples); ScalarFn is the one convenience base class with attribute-driven parameter binding ([Param], [ConstParam], [Setting], [OutputLength] — see "Your first worker" above). Projection/filter pushdown (including genuine expression/spatial-predicate pushdown, evaluated via an embedded DuckDB engine — see Internal/ExpressionFilterEvaluator.cs), ORDER BY/TABLESAMPLE hints, settings, secrets, splits, and cross-process state storage are all handled by the framework, not something each function reimplements.

Filter-capable table functions advertise FilterSemanticProfiles (the default is vgi.duckdb.standard.v1 when FilterPushdown is true), plus optional versioned extension-function, runtime-filter, and evaluation-context capabilities. Protocol 2.0 filter payloads use only the strict vgi.filters.v2 snapshot/delta encoding: literals remain typed Arrow payload fields, field_ref can nest to any depth, and IN sets can be inline Arrow lists or externally indexed join-key batches. PushdownFilterCodec requires the unprojected bind output schema and rejects index/name mismatches, malformed payloads, or legacy encodings atomically; PushdownFilterEvaluator applies required predicates exactly and may ignore a complete advisory predicate it cannot evaluate.

Beyond functions: full catalogs

A worker can expose more than bare functions — a complete catalog of schemas, function-backed tables, views, and macros that behave like native DuckDB objects:

var worker = new Worker()
    .CatalogName("example")
    .DefaultSchema("main")
    .RegisterScalar(new UpperCaseFunction())      // ScalarFn, as above
    .RegisterTable(new MyGeneratorFunction())     // ITableFunction — see Function shapes above
    .RegisterSchema("data", comment: "Reference tables")
    .RegisterCatalogTable(myTable, identity: "data");

ATTACH 'external_db' (TYPE vgi, LOCATION './my-catalog-worker');

SELECT * FROM external_db.data.users;            -- a catalog table
SELECT * FROM external_db.main.upper_case(name)  -- a function
FROM (VALUES ('alice')) t(name);

identity scopes a registration to a specific catalog identity when a worker serves more than one logical catalog from the same process (see Worker.RegisterCatalog); most workers only need the default.

Attach options and credentials

A catalog can declare typed ATTACH-time options (ATTACH ... (TYPE vgi, LOCATION ..., region 'eu')) by building AttachOptionSpecs with AttachOptionSpecBuilder.Build and passing them, encoded, as the CatalogInfo.AttachOptionSpecs it registers via Worker.RegisterCatalog:

AttachOptionSpecs =
[
    EmbeddedIpc.Encode(AttachOptionSpecBuilder.Build("api_key", "API key", StringType.Default,
        defaultValue: null, required: true, secret: true)),
    EmbeddedIpc.Encode(AttachOptionSpecBuilder.Build("region", "Region", StringType.Default,
        new StringArray.Builder().Append("us-east-1").Build())),
],

Credential options (API keys, tokens, passwords) MUST be declared secret: true. The caller passes the value inline as an ordinary attach option. The DuckDB extension redacts it from duckdb_databases(), keeps only a salted hash of it in its cache key, and never logs it; clients mask it and keep it out of exported configuration. To keep the credential out of the SQL text, pass it as an expression:

ATTACH 'sales' (TYPE vgi, LOCATION 'https://sales.example.com', api_key getenv('SALES_API_KEY'));

secret combines with required. A secret option may declare a default, but normally shouldn't. required is advertised, not enforced by the extension: reject a missing required option in your OnAttach handler.

Transports

await worker.RunStdioAsync();                          // default — DuckDB's plain LOCATION
await worker.RunUnixSocketAsync("/tmp/my-worker.sock"); // AF_UNIX, for the launch: pool
await worker.RunFromArgsAsync(args);                    // parses --unix/--idle-timeout/etc. from argv

LOCATION also accepts http://…/https://… for an HTTP worker, or a launch:<argv> prefix for the pooled AF_UNIX launcher transport (a worker process reused across every DuckDB connection that shares the same (argv, cwd, VGI_RPC_*-env) identity, rather than cold-spawned per ATTACH).

Critical rule: stdout is the wire channel for stdio-transport workers. Every diagnostic/log line must go to Console.Error, never plain Console.WriteLine — a stray stdout write corrupts the Arrow IPC stream.

Hosting additional protocols

A worker can host further application protocols beside vgi.v2 — a reporting protocol, a shared fixture — through one hook, called exactly once when the server is built:

new Worker()
    .RegisterScalar(...)
    .HostedProtocols(() => [HostedProtocol.For<IReports>(new Reports(config))])
    .RunFromArgsAsync(args);

The protocols are hosted, in the order returned, on every transport (stdio, unix, the Iroh raw upstream, HTTP), and listed by vgi_rpc.Reflection.v1 after vgi.v2. They cannot change vgi.v2: every request is routed on its vgi_rpc.protocol key with no fallback, so DuckDB dispatches exactly as before. Each interface needs its own [ProtocolName] (not vgi.v2, not under the reserved vgi_rpc. prefix); a bad entry fails startup with an error naming the hook.

Token introspection (vgi_rpc.Identity.v1)

A worker may opt into hosting vgi_rpc.Identity.v1 over HTTP, for a reverse proxy that must resolve an opaque bearer credential to a principal:

worker
    .Identity(
        resolveToken: token => apiKeys.Lookup(token) is { } row
            ? new TokenIdentity(row.Principal, row.Label)
            : null,                                  // "the store answered: unknown"
        introspectPrincipals: ["proxy@example.com"]) // or VGI_INTROSPECT_PRINCIPALS
    .HttpAuthenticate(myAuthenticate);               // who the caller is

Only the methods whose hooks you supply are hosted (resolveToken → introspect_token, mintGrant → issue_grant); with neither, the protocol is absent. A worker that supplies resolveToken without an introspector allowlist refuses to start: authenticating and introspecting are different capabilities, and "any authenticated caller" would let any user resolve any other user's credential to its owner.

For a transient failure — the store is down, a timeout, a 5xx — throw QueryFarm.VgiRpc.Errors.AuthUnavailableException("store unreachable", retryAfterSeconds: 10), the same error an HTTP authenticate delegate throws to get a 503 with Retry-After. The framework reports it as identity_unavailable with your retry hint, which callers know not to negative-cache. Never throw an ArgumentException (or return null) for an outage: that reads as "this credential is unknown".

Grants as bearer credentials

issue_grant mints a credential for automation to present later as an ordinary bearer, and the HTTP worker accepts it back:

worker.SealedGrants(GrantKeys.Parse([Environment.GetEnvironmentVariable("MY_GRANT_KEY")!]));
// or: --grant-key KEY (repeatable, first mints), or VGI_RPC_GRANT_KEYS=key1,key2

With grant keys configured (SealedGrants(...), --grant-key, or VGI_RPC_GRANT_KEYS with the optional VGI_RPC_GRANT_AUDIENCE / VGI_RPC_GRANT_MAX_TTL_SECONDS, default 7 days), the HTTP worker hosts vgi_rpc.Identity.v1 even without hooks, mints sealed vgig1. grants through issue_grant (unless Identity(mintGrant: ...) supplies a minter), and accepts them as Authorization: Bearer credentials on every call, as the grant's owner. Keys are standard base64 of exactly 32 bytes; a malformed one stops the worker at startup. No keys, no change. A caller must have authenticated recently (auth_time) to mint; a grant carries none, so grants never mint grants. Sealed grants are not individually revocable: keep the lifetime short, and rotate by adding the new key first, then removing the old one after its grants expire.

A worker that supplies resolveToken also has it consulted for bearers the earlier authenticators did not accept. Order: your HttpAuthenticate delegate (throw AuthFailure for a credential that is not yours), then sealed grants, then resolveToken. A bad vgig1. token is a 401 that never reaches the resolver; a resolver outage (AuthUnavailableException) is a 503. Behind the Iroh bridge the bearers are checked inside the peer-identity policy, never beside it.

Attach tickets (vgi.attach_tickets.v1)

An attach ticket lets a runner reattach a user's catalog later, as that user, without ever seeing the user's attach options. While the user is attached and logged in, the client calls seal_attach. The worker seals the catalog name, the options (secret ones included) and the version specs into a vgia1. ticket that only this worker can open. Later a runner presents Authorization: Bearer <grant> plus a single option:

ATTACH 'ticket_probe' (TYPE vgi, LOCATION 'https://…', vgi_attach_ticket 'vgia1.…');

catalog_attach redeems the ticket before any catalog code runs. Any other option beside it is invalid_request. The ticket opens only under the caller's principal (attach_ticket_invalid otherwise) and only within its lifetime (attach_ticket_expired). The sealed request then replaces the incoming one, so your OnAttach handler sees exactly what the user attached with, and the name the runner typed is ignored. A ticket carries no authority of its own: without a grant or login for the same principal, it attaches nothing.

worker
    .SigningKey(Encoding.UTF8.GetBytes(Environment.GetEnvironmentVariable("MY_SIGNING_KEY")!)) // or VGI_SIGNING_KEY
    .SealedGrants(grantKeys);                                                             // or VGI_RPC_GRANT_KEYS

The protocol is hosted on HTTP only, and only when the signing key is configured explicitly (SigningKey(...) or a non-empty VGI_SIGNING_KEY) and the worker can issue grants (grant keys, or Identity(mintGrant: ...)). Otherwise it is absent, so a client learns from reflection that it is unavailable. Tickets expire at the worker's grant maximum (ttl_seconds = 0 asks for the maximum); with no maximum they don't expire. Rotating the signing key invalidates every ticket. vgi_attach_ticket is a reserved attach-option name, so declaring it in any letter case fails at definition or at RegisterCatalog. Neither the ticket nor a restored option is ever logged. Spec: vgi-python docs/protocol/vgi-attach-tickets.md. The format lives in AttachTickets, checked byte for byte against the cross-SDK vectors in test/QueryFarm.Vgi.Tests/Vectors/.

The example worker serves the cross-SDK ticket_probe catalog. It has options region (default 'us-east-1') and api_key (required, secret), and table main.probe returns region and the first 12 hex characters of sha256(api_key). Over HTTP the worker accepts the test bearers vgi-test-alice and vgi-test-bob as fresh logins, so a client can mint grants. It hosts tickets when VGI_SIGNING_KEY and VGI_RPC_GRANT_KEYS are both set.

Protocol overview

VGI uses vgi_rpc, an Apache Arrow IPC-based RPC framework, for all client-worker communication — you don't write to this directly (Worker/ScalarFn/the function-kind interfaces handle it), but here's what happens per query:

DuckDB (client)                      VGI worker
  │──── bind(request) ─────────────▶ │  function name, args, input schema
  │◀─── BindResponse ───────────────  │  output schema (your Bind/ResolveOutputSchema)
  │──── init(request) ─────────────▶ │  start the processing stream
  │◀─── stream header ──────────────  │  execution_id, max_workers
  │──── exchange/tick(batch) ──────▶ │
  │◀─── output batch ───────────────  │  your Compute/Produce
  │──── [stream close] ────────────▶ │

See docs/roadmap.md and inline doc comments in Internal/VgiServiceImpl.cs for the full RPC surface (catalog DDL, transactions, splits, secrets, etc.) beyond this per-query happy path.

Repo layout

src/QueryFarm.Vgi/                    the published package
  Attributes/                         [Param]/[ConstParam]/[Setting]/[OutputLength]
  Scalar/ Table/ TableInOut/          per-function-kind interfaces + ScalarFn
  Buffering/ Aggregate/
  Catalog/                            CatalogTable/CatalogView/CatalogMacro
  Protocol/                           IVgiService + wire DTOs (Generated/, from vgi-python)
  Internal/                           VgiServiceImpl (the IVgiService dispatcher), pushdown
                                       filter codec/evaluator, argument codecs, storage
fixtures/QueryFarm.Vgi.ExampleWorker/ the ~170-function conformance-driving fixture worker
fixtures/QueryFarm.Vgi.SimpleWritableWorker/  writable-catalog write-path fixture
fixtures/QueryFarm.Vgi.BadProtocolWorker/     malformed-protocol negative-test fixture
examples/01-minimal-scalar-worker/    "Your first worker" above, as a buildable project
test/QueryFarm.Vgi.Tests/             xUnit unit tests
scripts/run_tests.sh                  fast local sqllogictest runner (see CLAUDE.md)
ci/                                   GitHub Actions integration-test harness

Read fixtures/QueryFarm.Vgi.ExampleWorker/ for a working example of every function kind and catalog feature — it's the fixture the full sqllogictest suite is graded against.

Testing your own worker

The fastest check is to call your function from a DuckDB session (see "Your first worker" above). For automated tests, drive the worker directly with QueryFarm.VgiRpc's client, or shell out to a DuckDB session from your test harness. test/QueryFarm.Vgi.Tests/ shows the former pattern for this SDK's own unit tests (schema derivation, dispatch, codecs, storage).

Build & test

make build                # dotnet build vgi-csharp.slnx
make test                 # unit tests (test/QueryFarm.Vgi.Tests)
make format_check         # dotnet format --verify-no-changes
make test_integration      # full sqllogictest suite against $(VGI_DIR), default ../vgi (launcher transport)

See CLAUDE.md for the full local-development workflow, including the fast sqllogictest iteration loop and the wire-protocol conventions worth knowing before touching Protocol/.

Architecture notes

• No IDL — RPC method dispatch and versioning ride as vgi_rpc.* custom metadata on Arrow IPC batches, not a schema-defined wire format. The protocol's request/response types are generated from the canonical vgi-python dataclasses (make regen_protocol), and a conformance test checks what this port serializes against the protocol's schemas.
• Two-tier dataclass rule: a method's own top-level parameter/return type embeds as IPC inside a binary field; a property nested inside another dataclass is a native Arrow struct.
• Positional vs. name-based decoding: request types (C++ → worker) decode positionally — property declaration order must exactly match the C++ generated schema's field order. Response types (worker → C++) are validated with a strict arrow::Schema::Equals against the C++ extension's generated schema factories.
• Cross-process storage: table-buffering and per-transaction state must survive landing on a different worker process than the call that wrote it (the worker-pool/launcher owns process lifetime, not the caller) — see IFunctionStorage's doc comment for the durable, execution-id/transaction-id-scoped storage contract this requires.

See inline doc comments throughout src/QueryFarm.Vgi/ and fixtures/QueryFarm.Vgi.ExampleWorker/ for the deeper "why" behind specific design choices — most non-obvious decisions are documented at the point of use, cross-referencing the specific sqllogictest file(s) they exist to satisfy.

See Iroh operations for native clients and bridge-ready raw or HTTP workers.

License

Copyright 2025, 2026 Query Farm LLC.

Licensed under the Query Farm Source-Available License, Version 1.0 — see LICENSE for the full terms. In brief, you may use, modify, and redistribute the software freely for non-production use, and for production use except where it would constitute a Competing Offering or a Commercial Marketplace as defined in the license. Each version converts to the Apache License, Version 2.0 on the tenth anniversary of its public release.

For uses not permitted under this license, contact hello@query.farm for a commercial license.

Browser catalog

HTTP workers serve the shared browser catalog at their HTTP root, with vgi-client.js alongside it. The page discovers schemas, tables, views, and functions through the normal VGI RPC endpoints. ?format=json returns worker identity instead. Both assets use the worker's authentication callback and support conditional requests and HEAD.

Applications hosting their own ASP.NET Core server can call app.MapVgiLandingPage(name, serverId, prefix, authenticate: authenticate) alongside app.MapVgiRpc(...), using the same prefix and authentication.