TXTextControl.AI.LlamaServer 0.1.0-beta.2

TX Text Control AI llama-server Runtime

TXTextControl.AI.LlamaServer runs GGUF inference in an isolated llama-server process and supplies a Microsoft.Extensions.AI chat client and embedding generator. It handles process ownership, startup health checks, streaming, runtime acquisition and hardware reporting. It does not include model weights, a document engine, web endpoints or an administration UI.

Most applications should start with TXTextControl.AI. Use this lower-level package when you need direct runtime control or a separate embedding process.

Contents

• Getting started
• Model options
• Runtime administration
• Download configuration
• Embedding generation
• Deployment and troubleshooting

Getting started

Targets .NET 8. Install from your approved package feed:

dotnet add package TXTextControl.AI.LlamaServer --prerelease

On Windows x64 or supported Linux x64/glibc, prepare a managed runtime explicitly:

using Microsoft.Extensions.AI;
using TXTextControl.AI.LlamaServer;

var options = new LlamaServerModelOptions
{
    HardwareBackend = LlamaServerHardwareBackend.Cpu,
    ForceCpu = true,
    ContextSize = 8192
};
await LlamaServerRuntime.InstallAsync(options); // One-time setup.
using var model = await LlamaServerModel.LoadAsync(
    Path.GetFullPath("Models/chat.gguf"), options);
using IChatClient chat = model.CreateChatClient();
var response = await chat.GetResponseAsync("Describe a document review workflow.");
Console.WriteLine(response.Text);

For a compatible preinstalled engine, use ExecutablePath and omit installation. macOS is supported through a host-provisioned compatible executable, not through the managed Windows/Linux download catalog. A Metal-capable native engine runs on the Mac, not inside a Linux Docker container.

LlamaServerModel implements IDisposable, not IChatClient: call CreateChatClient(). Keep the model alive while its clients are in use. Client disposal does not transfer process ownership; model disposal stops its owned process. Callers retain/resubmit chat history and cancellation tokens.

Model options

LlamaServerModelOptions uses the following defaults:

Generation options supplied through ChatOptions override defaults. This package does not perform document chunking or maintain chat history. Its tool-call compatibility handling does not authorize tools or guarantee a model will call them correctly. Add authorized function-invocation middleware separately.

Public model calls:

An external Endpoint is not AI.Service's /api URL. The load API still needs the local GGUF for metadata. Configure TLS and credentials for remote inference; the lower-level package is not an end-user authentication or URL-allowlist layer.

Runtime administration

Progress accepts IProgress<float>. Cancellation and failures do not activate partial installations. The default acquisition policy is NeverDownload. Restoring/building a NuGet consumer does not install the engine.

RuntimeInstallationInfo returns Platform, Version, RecommendedBackend, CacheDirectory, Choices, Installed, ExternalRuntime, Limitation, InstallationBlocker and Download. A recommendation is not a guarantee that a GPU driver works. Inspect blockers before offering installation. Each choice has Backend, Version, DownloadBytes, Supported and Limitation; each installed item has Id, Backend and Version.

LlamaServerExecutableLocator.Resolve(explicitPath, runtimeCacheDirectory) performs executable discovery. EnvironmentVariable identifies TXTEXTCONTROL_AI_LLAMA_SERVER. DefaultRuntimeCacheDirectory is LocalApplicationData/TXTextControl.AI/runtimes/llama.cpp under the running account. Configure a persistent private location for services/containers; it is not the NuGet installation folder. Explicit executables remain host-owned.

Download configuration

RuntimeDownloadOptions:

Every RuntimeDownloadAsset supplies Platform (win-x64/linux-x64), Backend (Cpu/Cuda/Vulkan), FileName (.zip/.tar.gz without a path), Bytes (positive, at most 16 GiB), and Sha256 (64 hex digits). At most 32 assets. Include dependency archives needed by a backend, not just the server archive. Validate() checks and snapshots settings without contacting the server.

Downloads use {BaseUrl}/{Version}/{FileName}. Mirror-only changes retain the same size/hash checks:

var mirrored = options with
{
    RuntimeDownload = new RuntimeDownloadOptions
    {
        Version = RuntimeDownloadOptions.DefaultVersion,
        BaseUrl = "https://downloads.example.com/approved/llama.cpp"
    }
};
// Install only after provisioning that mirror with the exact approved archives.

The pinned catalog includes Windows x64 CPU/CUDA/Vulkan and Linux x64 CPU/Vulkan. It is deliberately not "always latest". A custom manifest cannot make an unsupported operating system supported. Updating engines can alter chat templates, tool calling and memory use; test the selected release before deployment.

Embedding generation

LlamaServerEmbeddingGenerator implements IEmbeddingGenerator<string, Embedding<float>> and owns a separate lazy process. It never replaces the chat model.

using System.Security.Cryptography;

var path = Path.GetFullPath("Models/retrieval.gguf");
await using var file = File.OpenRead(path);
var hash = Convert.ToHexString(await SHA256.HashDataAsync(file));
using var embeddings = new LlamaServerEmbeddingGenerator(
    path, hash, dimensions: 384, pooling: "mean",
    options: new LlamaServerModelOptions
    {
        ForceCpu = true,
        ContextSize = 2048, BatchSize = 2048, UBatchSize = 2048,
        ExecutablePath = "/absolute/path/to/llama-server"
    });
var vectors = await embeddings.GenerateAsync(["A document approval policy"]);

Replace the example dimensions/pooling with the retrieval model's actual requirements. Chat GGUFs are not interchangeable with retrieval GGUFs.

External endpoints and arbitrary AdditionalArguments are rejected for this generator. It sets its own embedding/pooling/single-sequence arguments. GenerateAsync(values, options, cancellationToken) accepts at most 32 strings of at most 16000 characters each; model context can impose a lower limit. GetService returns supported services; Dispose releases owned resources. The generator validates finite, nonzero vectors. It does not invent or apply query/document prefixes: configure them in Knowledge or prepare inputs yourself. The ASP.NET Core embedding administration defaults to a CPU process; explicitly set ForceCpu when directly constructing a generator if that is your policy.

Deployment and troubleshooting

• A restored .NET assembly is not a native runtime. Provision engine, weights and all OS prerequisites separately.
• Linux managed archives require compatible glibc and OpenMP (libgomp.so.1, normally Ubuntu's libgomp1). CPU fallback does not remove that dependency.
• CUDA/Vulkan need matching host drivers. macOS native libraries need appropriate architecture and trust/signing; never disable security checks globally.
• Load once and reuse clients. Dispose deterministically, and do not remove a cached engine used by another host.
• A startup timeout is different from an inference timeout. Check process output, architecture, libraries and available memory before increasing either.
• Increasing context consumes memory. Also budget for embeddings and Docker.
• Endpoint authentication failures belong to the inference host; the MCP server uses its own independent endpoint and credentials.
• No model weights, private keys, GPUs, OS packages or application data are installed by this package's NuGet restore.

License and support

Preview APIs may change before a stable release. Distributed by Text Control GmbH under the Text Control AI Source Available License 1.0, included as LICENSE.txt. Solutions must retain properly licensed TX Text Control products as a core component. This is not an unrestricted open-source license. Commercial SDKs, model weights, inference engines and other dependencies retain their own licenses. Strong-name signing is assembly identity, not Apple notarization or a NuGet package-signature guarantee.

Visit Text Control for product, licensing and support.

Public C# reference

Source-derived public types, signatures, parameter defaults and DTO fields for this version. Bodies are omitted: these signatures are not a standalone compilable program. Concise initializers and JSON-name/required attributes are preserved. Workflow validation may require more than C# nullability alone. External types retain their package contracts. Public engine/service implementation types are advanced integration surfaces; prefer the documented registration and facade APIs for ordinary applications.

TXTextControl.AI.LlamaServer.InstalledRuntime

/// <summary>An installation managed by this application account.</summary>
public sealed record InstalledRuntime(string Id, string Backend, string Version);

TXTextControl.AI.LlamaServer.LlamaServerEmbeddingGenerator

/// <summary>A separate, lazily loaded local embedding process. Never borrows or unloads a chat model.</summary>
public sealed class LlamaServerEmbeddingGenerator : IEmbeddingGenerator<string, Embedding<float>>
{
    public LlamaServerEmbeddingGenerator(string modelPath, string sha256, int dimensions, string pooling = "mean", LlamaServerModelOptions? options = null);
    public async Task<GeneratedEmbeddings<Embedding<float>>> GenerateAsync(IEnumerable<string> values, EmbeddingGenerationOptions? options = null, CancellationToken cancellationToken = default);
    public object? GetService(Type serviceType, object? serviceKey = null);
    public void Dispose();
}

TXTextControl.AI.LlamaServer.LlamaServerExecutableLocator

/// <summary>Locates an installed llama-server without coupling applications to a fixed release.</summary>
public static class LlamaServerExecutableLocator
{
    public const string EnvironmentVariable = "TXTEXTCONTROL_AI_LLAMA_SERVER";
    public static string DefaultRuntimeCacheDirectory { get; }

    public static string? Resolve(string? explicitPath = null, string? runtimeCacheDirectory = null);
}

TXTextControl.AI.LlamaServer.LlamaServerHardwareBackend

public enum LlamaServerHardwareBackend
{
    Auto,
    Cpu,
    Cuda,
    Vulkan,
}

TXTextControl.AI.LlamaServer.LlamaServerModel

/// <summary>
/// Owns an optional llama-server child process and exposes its OpenAI-compatible chat API.
/// </summary>
public sealed class LlamaServerModel : IDisposable
{
    public LlamaServerModelInfo Info { get; }
    public Uri Endpoint { get; }

    public static bool IsAvailable(string? executablePath = null);
    public static async Task<LlamaServerModel> LoadAsync(string modelPath, LlamaServerModelOptions options, IProgress<float>? progress = null, CancellationToken cancellationToken = default);
    public IChatClient CreateChatClient();
    public void Dispose();
}

TXTextControl.AI.LlamaServer.LlamaServerModelInfo

public sealed record LlamaServerModelInfo(string ModelPath, string Name, string? Architecture, bool HasChatTemplate, int TrainingContextSize, ulong SizeInBytes, IReadOnlyDictionary<string, string> Metadata, LlamaServerRuntimeInfo Runtime);

TXTextControl.AI.LlamaServer.LlamaServerModelOptions

/// <summary>Configures an isolated llama-server model process.</summary>
public sealed record LlamaServerModelOptions
{
    public string? ExecutablePath { get; init; }
    public Uri? Endpoint { get; init; }
    public LlamaServerRuntimeAcquisitionPolicy RuntimeAcquisition { get; init; } = LlamaServerRuntimeAcquisitionPolicy.NeverDownload;
    /// <summary>Null, empty or whitespace uses the default application-data runtime directory.</summary>
    public string? RuntimeCacheDirectory { get; init; }
    public RuntimeDownloadOptions RuntimeDownload { get; init; } = new();
    public string? ModelId { get; init; }
    public string ApiKey { get; init; } = "local";
    public int ContextSize { get; init; } = 8_192;
    public int GpuLayers { get; init; } = -1;
    public int? Threads { get; init; }
    public int BatchSize { get; init; } = 512;
    public int UBatchSize { get; init; } = 512;
    public int MaxOutputTokens { get; init; } = 1_024;
    public float Temperature { get; init; } = 0.6f;
    public int TopK { get; init; } = 40;
    public float TopP { get; init; } = 0.95f;
    public float? FrequencyPenalty { get; init; }
    public float? PresencePenalty { get; init; }
    public int? Seed { get; init; }
    public bool? FlashAttention { get; init; }
    public bool UseMemoryMapping { get; init; } = true;
    public bool UseMemoryLocking { get; init; }
    public int MainGpu { get; init; }
    public bool ForceCpu { get; init; }
    public LlamaServerHardwareBackend HardwareBackend { get; init; } = LlamaServerHardwareBackend.Auto;
    public bool EnableReasoning { get; init; }
    public TimeSpan StartupTimeout { get; init; } = TimeSpan.FromMinutes(3);
    public TimeSpan RequestTimeout { get; init; } = TimeSpan.FromMinutes(10);
    public IReadOnlyList<string> AdditionalArguments { get; init; } = [];
}

TXTextControl.AI.LlamaServer.LlamaServerRuntime

/// <summary>Provides explicit runtime prefetching for deployment and offline preparation.</summary>
public static class LlamaServerRuntime
{
    /// <summary>Reads the local catalog and managed installations without downloading or checking upstream releases.</summary>
    public static RuntimeInstallationInfo Inspect(LlamaServerModelOptions? options = null);
    /// <summary>Also obtains a locally probed hardware recommendation; never downloads runtime files.</summary>
    public static Task<RuntimeInstallationInfo> InspectAsync(LlamaServerModelOptions? options = null, CancellationToken cancellationToken = default);
    /// <summary>Explicitly downloads and verifies the curated runtime, independently of model loading.</summary>
    public static Task<string> InstallAsync(LlamaServerModelOptions? options = null, IProgress<float>? progress = null, CancellationToken cancellationToken = default);
    /// <summary>Removes an enumerated managed installation. The caller must ensure it is not in use by any host.</summary>
    public static Task RemoveAsync(string installationId, LlamaServerModelOptions? options = null, CancellationToken cancellationToken = default);
    public static Task<string> EnsureAvailableAsync(LlamaServerModelOptions? options = null, IProgress<float>? progress = null, CancellationToken cancellationToken = default);
}

TXTextControl.AI.LlamaServer.LlamaServerRuntimeAcquisitionPolicy

public enum LlamaServerRuntimeAcquisitionPolicy
{
    DownloadIfMissing,
    NeverDownload,
}

TXTextControl.AI.LlamaServer.LlamaServerRuntimeInfo

public sealed record LlamaServerRuntimeInfo(string Engine, string Backend, string Platform, string Architecture, string? Executable);

TXTextControl.AI.LlamaServer.RuntimeDownloadAsset

/// <summary>An archive approved by the host; CUDA may require multiple archives per platform/backend.</summary>
public sealed record RuntimeDownloadAsset
{
    public string Platform { get; init; } = "";
    public string Backend { get; init; } = "";
    public string FileName { get; init; } = "";
    public long Bytes { get; init; }
    public string Sha256 { get; init; } = "";
}

TXTextControl.AI.LlamaServer.RuntimeDownloadChoice

/// <summary>A supported, pinned runtime option. No network request is needed to read this catalog.</summary>
public sealed record RuntimeDownloadChoice(string Backend, string Version, long DownloadBytes, bool Supported, string? Limitation);

TXTextControl.AI.LlamaServer.RuntimeDownloadOptions

/// <summary>Host-approved release and mirror settings. Custom releases require pinned assets.</summary>
public sealed record RuntimeDownloadOptions
{
    public const string DefaultVersion = "b10621";
    public const string DefaultBaseUrl = "https://github.com/ggml-org/llama.cpp/releases/download";
    public string Version { get; init; } = DefaultVersion;
    /// <summary>HTTPS root. Downloads use {BaseUrl}/{Version}/{FileName}.</summary>
    public string BaseUrl { get; init; } = DefaultBaseUrl;
    /// <summary>Empty uses the bundled catalog, only for DefaultVersion. Include all dependency archives.</summary>
    public IReadOnlyList<RuntimeDownloadAsset> Assets { get; init; } = [];

    /// <summary>Validates and snapshots settings without contacting the download server.</summary>
    public RuntimeDownloadOptions Validate();
}

TXTextControl.AI.LlamaServer.RuntimeInstallationInfo

/// <summary>Runtime installation status; model files are never downloaded by this API.</summary>
public sealed record RuntimeInstallationInfo(string Platform, string Version, string RecommendedBackend, string CacheDirectory, IReadOnlyList<RuntimeDownloadChoice> Choices, IReadOnlyList<InstalledRuntime> Installed, string? ExternalRuntime, string? Limitation)
{
    /// <summary>A missing system prerequisite that prevents a managed download from starting.</summary>
    public string? InstallationBlocker { get; init; }
    /// <summary>Configured release and mirror; changing the mirror never disables checksum verification.</summary>
    public RuntimeDownloadOptions Download { get; init; } = new();
}