InferHub.Client 0.4.0

InferHub.Client

A small, typed .NET client for InferHub — a self-hosted, Ollama-compatible inference mesh.

Point it at a coordinator, pass a Bearer token, and call chat, generate, model listing and status from C# with typed requests, dependency injection, and no heavy dependencies.

v0.4.0 — blocking + streaming inference, embeddings (batch + legacy), and the vector data plane (upsert / query / retrieve / get / delete). RAG retrieval and admin follow in later phases.

Install

dotnet add package InferHub.Client

Targets net9.0 and net10.0.

Quick start

using InferHub.Client;
using InferHub.Client.Configuration;
using InferHub.Client.Models.Ollama;
using Microsoft.Extensions.DependencyInjection;

var services = new ServiceCollection();
services.AddInferHubClient(o =>
{
    o.BaseAddress = new Uri("http://localhost:5080");
    o.ApiKey = "<your-client-api-key>";
});

var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<IInferHubClient>();

var chat = await client.ChatAsync(new ChatRequest
{
    Model = "llama3",
    Messages = new[]
    {
        new ChatMessage { Role = "user", Content = "Say hi in one word." }
    },
    Stream = false
}, CancellationToken.None);

Console.WriteLine(chat.Message?.Content);

What ships in v0.4.0

Streaming

await foreach (var chunk in client.ChatStreamAsync(new ChatRequest
{
    Model = "llama3",
    Messages = new[] { new ChatMessage { Role = "user", Content = "Stream me a haiku." } }
}, cancellationToken))
{
    Console.Write(chunk.Message?.Content);
}

The enumerator stops as soon as a chunk arrives with done:true. A terminal error chunk ({ "error": …, "done": true }) is surfaced as InferHubException — the client never retries mid-stream, so a partial answer plus a clean exception is the contract. Cancelling the token throws promptly out of the await foreach.

Request models carry an extension bag (AdditionalProperties), so any unknown fields from the Ollama contract pass through untouched — you can hand-set options, format, tool definitions, etc. without waiting on the client to type them.

Embeddings

// Single input.
var single = await client.EmbedAsync(
    EmbedRequest.FromText("nomic-embed-text", "hello, world"));

// Batch — one vector per input, same order.
var batch = await client.EmbedAsync(EmbedRequest.FromTexts(
    "nomic-embed-text",
    new[] { "InferHub", "self-hosted", "inference mesh" }));

Console.WriteLine(batch.Embeddings.Length);   // 3
Console.WriteLine(batch.Embeddings[0].Length); // model dimension

EmbedAsync targets the modern batch endpoint (/api/embed); EmbedLegacyAsync wraps /api/embeddings for drop-in Ollama callers. An empty vector list on a 200 response is treated as malformed and surfaced as InferHubException — the client never returns a silent zero-vector result.

Vectors

Text in, ranked matches out. The coordinator embeds text on a node for you, so you never have to hold a model client-side. Needs the coordinator running with VectorStore:Enabled=true and the collection already created (admin plane, later phase).

using InferHub.Client.Models.Vector;

// Upsert — embed text on a node, keep the original as an opaque payload.
await client.UpsertAsync("docs", VectorUpsert
    .FromText("doc-1", "InferHub is a self-hosted inference mesh.", "nomic-embed-text")
    .WithPayload(new { title = "About" })
    .WithMetadata(new Dictionary<string, string> { ["kind"] = "doc" }));

// Query — text in, closest matches out.
var matches = await client.QueryAsync("docs",
    VectorQuery.FromText("what is InferHub?", "nomic-embed-text", k: 3));

foreach (var m in matches)
    Console.WriteLine($"{m.Score:F3}  {m.Id}");

// Read the payload back into your own type.
var record = await client.GetRecordAsync("docs", "doc-1"); // null if absent
var title = record?.Payload.As<Doc>()?.Title;

await client.DeleteRecordAsync("docs", "doc-1"); // false if it wasn't there

Pass a raw vector instead of text with VectorUpsert.FromVector / VectorQuery.FromVector. payload is exposed as a JsonElement?; call .As<T>() to deserialize it. GetRecordAsync returns null and DeleteRecordAsync returns false on a 404; every other non-success status is an InferHubException. RetrieveAsync is the same call as QueryAsync under the RAG-oriented name. See samples/MiniRag for a runnable embed-then-query loop.

Auth

• Non-loopback calls need ApiKey (attached as Authorization: Bearer <key> by a DelegatingHandler).
• Loopback calls to the coordinator skip auth by default (unless the coordinator sets Auth:RequireAuthForLoopback=true).
• /health is always open.
• Admin routes require a separate AdminApiKey and land on a dedicated interface in a later phase; a client key alone never surfaces admin methods.

Errors

Any non-success HTTP response is surfaced as InferHubException, carrying:

• StatusCode — the HTTP status
• Message — the coordinator's { "error": "…" } body if present

The client treats 404 (model missing) and 424 Failed Dependency (retrieval unavailable, added in a later phase) as distinct signals worth checking with StatusCode.

Links

• InferHub server: https://github.com/Dev-Art-Solutions/InferHub
• Product page: https://inferhub.devart.solutions
• Blog: https://blog.devart.solutions

License

MIT — see LICENSE.