QueryFarm.VgiRpc.Client.Http 0.8.0

<p align="center"> <img src="https://raw.githubusercontent.com/Query-farm/vgi-rpc-csharp/main/assets/vgi-logo.png" alt="Vector Gateway Interface logo" width="320"> </p>

<h1 align="center">vgi-rpc for .NET</h1>

<p align="center"> Transport-agnostic RPC framework built on <a href="https://arrow.apache.org/">Apache Arrow</a> IPC serialization.<br> Built by <a href="https://query.farm">🚜 Query.Farm</a> </p>

<p align="center"> <a href="https://github.com/Query-farm/vgi-rpc-csharp/actions/workflows/ci.yml"><img src="https://github.com/Query-farm/vgi-rpc-csharp/actions/workflows/ci.yml/badge.svg" alt="CI"></a> <a href="https://www.nuget.org/packages/QueryFarm.VgiRpc"><img src="https://img.shields.io/nuget/v/QueryFarm.VgiRpc" alt="NuGet"></a> <a href="https://www.nuget.org/packages/QueryFarm.VgiRpc"><img src="https://img.shields.io/nuget/dt/QueryFarm.VgiRpc" alt="NuGet downloads"></a> <a href="https://github.com/Query-farm/vgi-rpc-csharp/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Query-farm/vgi-rpc-csharp" alt="License"></a> </p>

Define RPC contracts as ordinary C# interfaces. vgi-rpc derives Apache Arrow schemas from those interfaces and provides reflection-based server dispatch and typed unary client proxies. There are no .proto files or code-generation steps, and structured data remains in Arrow's columnar format instead of being converted to JSON.

This implementation is wire-compatible with the canonical Python implementation and the other vgi-rpc ports, allowing clients and servers written in different supported languages to interoperate.

Key features:

• Interface-based contracts — define services with standard C# interfaces and async methods
• Apache Arrow IPC wire format — efficient serialization for structured and batch-oriented data
• Cross-language interoperability — compatible with the Python, Go, Rust, TypeScript, and Java implementations
• Unary and streaming dispatch — producer and exchange streaming patterns are supported server-side
• Multiple transports — in-process pipes, stdio, Unix domain sockets, TCP, shared memory, and HTTP
• Automatic schema inference — CLR primitives, collections, enums, POCOs, and Arrow record batches map to Arrow types
• HTTP security — bearer authentication, mTLS, JWT/JWKS validation, OAuth 2.0 PKCE, CORS, and proxy proof
• Large-payload offload — transparent externalization to Amazon S3, S3-compatible stores, or Google Cloud Storage
• Observability — access logs, OpenTelemetry-compatible tracing and metrics, and Sentry instrumentation

Installation

Install the core package:

dotnet add package QueryFarm.VgiRpc

Install a client package when this process calls a vgi-rpc worker:

dotnet add package QueryFarm.VgiRpc.Client
# For HTTP:
dotnet add package QueryFarm.VgiRpc.Client.Http

Add integrations as needed:

The packages target .NET 10 and require the .NET 10 SDK to build from source.

0.8 migration: client types moved out of QueryFarm.VgiRpc into QueryFarm.VgiRpc.Client. Add that package and update client-side using directives to QueryFarm.VgiRpc.Client; server-only applications do not need the new dependency.

Quick start

Define a service, implement it, and connect a typed client to the server over an in-process pipe:

using QueryFarm.VgiRpc.Client;
using QueryFarm.VgiRpc.Server;
using QueryFarm.VgiRpc.Transport;

public interface IGreeter
{
    Task<string> GreetAsync(string name);
}

public sealed class Greeter : IGreeter
{
    public Task<string> GreetAsync(string name) =>
        Task.FromResult($"Hello, {name}!");
}

var (clientTransport, serverTransport) = PipeTransport.CreatePair();

var server = new RpcServer(typeof(IGreeter), new Greeter());
var serveTask = server.ServeAsync(serverTransport);

var connection = new RpcConnection<IGreeter>(clientTransport);
IGreeter client = connection.CreateProxy();

Console.WriteLine(await client.GreetAsync("World")); // Hello, World!

clientTransport.Output.Close();
await serveTask;

Methods must return Task or Task<T>. By default, method names are converted to snake_case on the wire and a trailing Async suffix is removed, so GreetAsync becomes greet. Use [RpcName("...")] to override a method, parameter, or property name.

See the complete 01-hello-world example for a runnable project.

Service contracts

Parameters and return values may use CLR primitives, common generic collections, enums, Arrow record batches, or POCOs with a parameterless constructor and public settable properties. Nested POCOs map to nested Arrow structs.

A service method may also declare a trailing optional ICallContext parameter. The server injects it for access to request-scoped logging and HTTP sticky-session state; it is excluded from the wire schema.

Transports

Both client classes expose exact-schema methods (CallUnaryAsync, OpenProducerAsync, and OpenExchangeAsync) and CreateProxy<TContract>(). Use IRpcProducerSession and RpcExchangeSession<TInput> in portable typed client contracts; the same contract can then be driven over persistent byte streams or stateless HTTP.

WorkerPool keeps healthy subprocess connections in a command-keyed LIFO pool, bounds idle workers, evicts them after an idle timeout, and exposes borrow/spawn/reuse/discard metrics.

Streaming

Streaming service methods return RpcStream<TState>, where TState derives from ProducerState or ExchangeState. The server invokes ProduceAsync or ExchangeAsync for each stream iteration:

public sealed class CounterState(long count) : ProducerState
{
    private long _current;

    public override Task ProduceAsync(
        OutputCollector output,
        ICallContext? context,
        CancellationToken cancellationToken)
    {
        if (_current >= count)
        {
            output.Finish();
            return Task.CompletedTask;
        }

        output.Emit(ValueCodec.BuildRow(CounterSchema.Output, [_current++]));
        return Task.CompletedTask;
    }
}

public interface ICounterService
{
    Task<RpcStream<CounterState>> CountToAsync(long count);
}

Producer and exchange streaming are conformance-tested over pipe, Unix socket, TCP, and HTTP transports.

HTTP and authentication

QueryFarm.VgiRpc.Http integrates with ASP.NET Core through MapVgiRpc(...). Its authentication delegate can validate bearer tokens, client certificates, or application-specific credentials before dispatch. QueryFarm.VgiRpc.Http.OAuth adds JWT/JWKS validation, protected-resource metadata, and an OAuth 2.0 PKCE browser flow.

The HTTP package also includes CORS handling, request and response size limits, zstd content encoding, sticky sessions, token introspection, and proxy-proof validation.

The separate QueryFarm.VgiRpc.Client.OAuth package performs OIDC discovery, Authorization Code with PKCE (including constant-time state validation), Device Authorization polling, token refresh, and bearer injection through OAuthBearerHandler.

External storage

Large Arrow batches can be uploaded to object storage and replaced on the wire with an external location descriptor. The receiving peer resolves the descriptor with parallel range requests.

using QueryFarm.VgiRpc.Http;
using QueryFarm.VgiRpc.S3;

var storage = S3Storage.CreateBuilder("my-bucket")
    .WithKeyPrefix("rpc-data/")
    .WithRegion(Amazon.RegionEndpoint.USEast1)
    .Build();

var externalization = new ExternalizationOptions
{
    External = new ServerExternalConfig
    {
        Storage = storage,
        ExternalizeThresholdBytes = 1_048_576,
        Compression = new Compression(),
    },
};

app.MapVgiRpc(server, externalization: externalization);

S3Storage supports Amazon S3 and configurable S3-compatible endpoints. GcsStorage provides the equivalent integration for Google Cloud Storage. Both implementations support server-managed uploads and signed upload/download URL pairs.

Error handling

Remote errors surface as RpcException and include a stable error kind, the remote exception type, message, and traceback. A failed call does not invalidate an otherwise healthy persistent connection. Common protocol conditions have typed exceptions, including MethodNotImplementedException, ProtocolVersionException, SessionLostException, ServerDrainingException, and PayloadTooLargeException.

Examples

Development

The SDK version is pinned in global.json.

dotnet restore
dotnet build -c Release
dotnet test -c Release
dotnet format --verify-no-changes --exclude third_party

Run the cross-language conformance suite with:

./run_tests.sh

The suite uses the canonical Python implementation and covers all supported transports and wire features in the server direction. The CI client-conformance lane also reverses the direction: the C# client drives canonical Python, Rust, Java, Go, and TypeScript workers. Each lane uses only the transports that worker actually implements (including SHM for Python/Rust/Java/Go and stdio/HTTP/Unix/TCP for TypeScript). The xUnit port of Python's native-client worker tests also validates exact all-null/zero-row/nested schemas, producer continuations, sticky sessions, external payloads, verified TLS, and stdio/SHM/Unix/TCP transports. Run that focused native-client suite against a Python checkout with:

VGI_PYTHON_BIN=/path/to/vgi-rpc/.venv/bin/python \
  dotnet test test/QueryFarm.VgiRpc.Http.Tests \
  --filter FullyQualifiedName~PythonClientWorkerTests

See docs/wire-protocol.md for the wire format and third_party/apache-arrow-dotnet/README.md for details about the narrowly patched Arrow dependency.

Related projects

• vgi-rpc — canonical Python implementation and conformance suite
• vgi-rpc-go — Go implementation
• vgi-rpc-rust — Rust implementation
• vgi-rpc-typescript — TypeScript implementation
• vgi-rpc-java — Java implementation

License

Apache License 2.0 — see LICENSE and NOTICE.