OpenTail.Stingray.Server 1.0.6

OpenTail.Stingray.Server

ASP.NET Core endpoints, options, and DI extensions that expose OpenTail.Stingray as a drop-in OpenAI- and Anthropic-compatible HTTP API. Bring your own host (Kestrel, IIS, YARP, …); this package only ships the routes, request/response shapes, and DI wiring.

For the bare inference library, use OpenTail.Stingray. For the standalone CLI, use OpenTail.Stingray.Cli.

Install

dotnet add package OpenTail.Stingray.Server

This transitively pulls in OpenTail.Stingray (the bundled inference engine + CPU/Vulkan/CUDA backends). You must be on the Microsoft.NET.Sdk.Web SDK — the package's Microsoft.AspNetCore.App framework reference is propagated.

Quick start

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTailStingray(opt =>
{
    opt.ModelPath = "models/SmolLM2-1.7B-Instruct-Q4_K_M.gguf";
    opt.GpuLayers = -1; // -1 = all layers on GPU; 0 = pure CPU
});

var app = builder.Build();
app.MapOpenTailStingray();
app.Run();

Deployment safety

OpenTail.Stingray.Server supplies inference routes; it deliberately does not choose an authentication scheme for the host. The bundled demo host therefore listens only on 127.0.0.1:8080 and admits one generation at a time by default.

Do not bind it directly to a LAN or public interface without an authentication boundary. For remote use, place it behind an authenticated reverse proxy, or add your application's authentication/authorization middleware and protect the mapped routes before calling Run(). Set OpenTail.Stingray:MaxQueuedRequests (or STINGRAY_MAX_QUEUE) to bound how many requests may wait behind the active batch. The server returns HTTP 429 once MaxBatchSize + MaxQueuedRequests is reached, rather than retaining unbounded HTTP connections. The default is 16 waiting requests. MaxConcurrentRequests (or STINGRAY_MAX_CONCURRENT) remains available as a legacy fixed in-flight cap and takes precedence when explicitly set.

Bind from configuration instead:

// appsettings.json: { "OpenTail.Stingray": { "ModelPath": "...", "GpuLayers": -1 } }
builder.Services.AddOpenTailStingray(builder.Configuration);

Complete API Surface

Streaming (SSE) is enabled for every chat/completion endpoint, and the JSON pipeline is wired through a source-generated JsonSerializerContext so the package is AOT-friendly even though the project itself is not AOT-published.