MBW.GHLinguist 0.4.0

MBW.GHLinguist

MBW.GHLinguist hosts GitHub Linguist behind a synchronous .NET 10 API. It is designed for complete analysis of an in-memory blob, direct content classification, and access to Linguist's language registry without exposing Ruby or native handles to callers.

This is an independent community project. It is not affiliated with, sponsored by, or endorsed by GitHub, Inc. "GitHub" and "Linguist" are used only to identify the upstream GitHub Linguist library that this package embeds.

Links

• Source repository
• GitHub Actions
• NuGet.org package
• GitHub Packages development feed
• GitHub Linguist
• How Linguist works
• Benchmark methodology and limitations

Status

The managed API, native ABI bridge, and relocatable CRuby dependency closures are implemented for the supported runtime identifiers. The managed package and the selected runtime package do not require a system Ruby installation or runtime downloads.

The supported runtime identifiers are win-x64 and linux-x64. The package targets .NET 10 and currently embeds CRuby 4.0.6 and GitHub Linguist 9.6.0 at revision 196b2a14418cab005065c72c9759370934c184bc. Its explicit Ruby gem closure includes zlib 3.2.3 and resolv 0.7.2. CI enforces the patched minimum versions for the Ruby advisories that motivated those pins and performs a live NuGet vulnerability audit for managed packages.

The checked-in benchmark results are preliminary synthetic Ruby-only measurements of the bridge, taken before ClassifyDotNet and LinguistClassifier existed, and are not end-to-end managed performance results. They show generated-check allocation reductions of about 23%, 30%, and 33% for the sampled cases. See the benchmark methodology and limitations.

Stable releases are published to NuGet.org. Development packages are published to GitHub Packages as incrementing authenticated prereleases.

Prerequisites

• The .NET 10 SDK to restore, build, test, or publish from source
• The .NET 10 runtime to run a framework-dependent deployed application
• An x64 Windows or Linux development and deployment environment
• A consuming executable that targets net10.0
• A RuntimeIdentifier of win-x64 or linux-x64

Run dotnet --info if the SDK or host architecture is uncertain. Linux packages are built and exercised on Debian Bookworm and use the host ELF loader and glibc family; they do not support musl-based distributions such as Alpine. A glibc 2.35 observation covered only the bridge and libruby, not every staged ELF dependency, so a minimum glibc version is not declared pending a full closure audit. Test the published output directory on the production distribution before deployment.

macOS, ARM64, .NET 9 and earlier, NativeAOT, trimming, and single-file deployment are not supported.

Install

Install the stable package from NuGet.org:

dotnet package add MBW.GHLinguist

Set the deployment runtime identifier on the executable project (a RID-neutral class library can reference this package):

<PropertyGroup>
  <RuntimeIdentifier>win-x64</RuntimeIdentifier>
</PropertyGroup>

Use linux-x64 when building and deploying on Linux. The managed package has exact dependencies on both RID packages because NuGet cannot condition package dependencies on a runtime identifier. Only the native bridge and closure matching the selected RuntimeIdentifier are included in build and publish output. Do not reference the runtime packages directly.

Restore, build, and publish with the same RID. A missing or unsupported RID fails the build with an actionable error for executable projects (Exe and WinExe) and for test projects (IsTestProject is true). Class libraries build without a RID because the application that hosts them chooses it.

Development packages

Development prereleases are published to GitHub Packages and require a classic GitHub personal access token with read:packages. Add this source to an existing NuGet.config, or create one beside the solution:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="github_lordmike" value="https://nuget.pkg.github.com/LordMike/index.json" />
  </packageSources>
</configuration>

Keep credentials outside the file. Set them for the current PowerShell session, then install the prerelease managed package:

$env:GITHUB_PACKAGES_TOKEN = "YOUR_CLASSIC_PAT"
$env:NuGetPackageSourceCredentials_github_lordmike = "Username=YOUR_GITHUB_USERNAME;Password=$env:GITHUB_PACKAGES_TOKEN;ValidAuthenticationTypes=Basic"

dotnet package add MBW.GHLinguist --prerelease --source github_lordmike

The environment-variable name must end with the exact source key from NuGet.config: github_lordmike. Do not commit a token or a generated configuration containing a clear-text password. In GitHub Actions, prefer the workflow's GITHUB_TOKEN with packages: read after granting that repository access to the package. dotnet package add writes the resolved prerelease version to the project; review and commit the resulting PackageReference.

Common authentication failures:

• 401 or NU1301: verify the feed URL, token, and the exact NuGetPackageSourceCredentials_github_lordmike environment-variable name.
• 403: the token's GitHub account usually lacks access to the package. The username must be the GitHub login, not an email address.

Quick start

Start with LinguistRuntime. Create one runtime, reuse it for calls, and dispose it when the application no longer needs Linguist. Replace the consuming console project's Program.cs with:

using MBW.GHLinguist;

using LinguistRuntime runtime = LinguistRuntime.Create();

byte[] source = "puts 'Hello'\n"u8.ToArray();
BlobAnalysis analysis = runtime.Analyze(
    source,
    new BlobInput
    {
        Path = "src/hello.rb",
        Name = "hello.rb",
    });

Console.WriteLine(analysis.Language?.Name); // Ruby
Console.WriteLine(analysis.Strategy);       // Extension

Run the example from the consuming project directory:

dotnet run --runtime win-x64

Use linux-x64 on Linux. For a long-running application, register one LinguistRuntime as an application-lifetime singleton rather than creating one per request or file.

Expected output:

Ruby
Extension

Returned languages, analyses, classifications, and traces are copied managed objects. They remain usable after the runtime is disposed.

Choose an API

Use Analyze for normal file-language detection. Use Classify when the caller deliberately wants classifier scores without filename, shebang, modeline, XML, or heuristic strategy selection.

Analyze(data) is not equivalent to Classify(data). Without a BlobInput, Analyze still performs blob checks and runs the enabled detection strategies; it simply has no path or filename metadata. Classify performs classifier-only ranking.

Linguist feature coverage

This package exposes a useful subset of Linguist, not the complete repository pipeline used by GitHub.com.

The IsIncludedInLanguageStatistics result is a decision for one supplied blob. It does not aggregate byte counts or reproduce GitHub's repository language bar. Generated, vendored, and documentation results use Linguist's embedded rules for the supplied path and bytes, but repository-specific .gitattributes overrides are not applied. These omissions are the main reason results can differ from GitHub.com even when the same Linguist revision is involved.

The result flags are not mutually exclusive. A generated or vendored blob can still have a detected language, and IsDetectable does not mean that the blob belongs in repository statistics. Check IsIncludedInLanguageStatistics separately.

Classification also has narrower coverage than the registry: a registered language can be absent from classifier results when Linguist has no classifier centroid for it.

Blob metadata

BlobInput groups metadata separately from analysis behavior so path and name cannot be accidentally swapped:

BlobAnalysis result = runtime.Analyze(
    data,
    new BlobInput
    {
        Path = "vendor/generated/client.cs",
        Name = "client.cs",
        IsSymlink = false,
        IsLfsTracked = false,
    },
    new BlobAnalysisOptions
    {
        IncludeLineCounts = true,
        IncludeStrategyTrace = true,
    });

Path should normally be repository-relative and use forward slashes. It affects generated, vendored, documentation, and statistics rules. Name affects filename and extension strategies. If Name is null, the native bridge derives it from Path when possible. Empty strings are passed to Linguist unchanged and are not equivalent to null.

Pass the actual path of one blob, not a glob or .gitattributes pattern. IsSymlink and IsLfsTracked are caller assertions; IsLfsPointer is detected from the supplied content. The API does not inspect Git metadata to infer either input flag or retrieve the target of a symlink or LFS pointer.

Inspect the analysis result directly; a detected language does not imply that the blob should count toward repository statistics:

Console.WriteLine($"Language: {result.Language?.Name ?? "unknown"}");
Console.WriteLine($"Strategy: {result.Strategy}");
Console.WriteLine($"Encoding: {result.Encoding ?? "unknown"}");
Console.WriteLine($"Generated: {result.IsGenerated}");
Console.WriteLine($"Vendored: {result.IsVendored}");
Console.WriteLine($"Include in statistics: {result.IsIncludedInLanguageStatistics}");

Line-count semantics

IncludeLineCounts returns Linguist's rendering-oriented counts, not independent whole-file metrics. LineCount counts physical lines only for viewable text; binary blobs and files larger than 1 MiB return zero even when nonempty. SourceLineCount counts nonblank lines, including comment-only lines. Neither property is a count of executable statements. Use IsEmpty to test emptiness.

Both properties are null when counts are not requested. Disabling this option does not eliminate all line processing: complete analysis still computes flags such as the long-line ratio that depend on Linguist's line representation.

Language lookup

The lookup names and inputs intentionally follow Linguist's Ruby API:

FindByExtension takes a filename, not a bare extension: pass "example.rb", "src/example.rb" or ".rb". A filename with no dot, such as "rb" or "Makefile", has nothing for Linguist to match, so it throws ArgumentException instead of silently returning an empty list. Use FindByFilename for special extensionless names. Linguist considers recognized compound extensions in its own precedence order.

FindById, FindByName, and FindByAlias each have a TryFind... partner that returns false instead of throwing. Use FindBy... when the language is expected to exist, and TryFindBy... when the input may be unknown, such as a language name typed by a user:

if (runtime.TryFindByName(userInput, out LinguistLanguage? language))
{
    Console.WriteLine(language.Id);
}

An empty name or alias finds nothing. Inputs are not trimmed, so whitespace remains significant. Passing null to a required lookup argument throws ArgumentNullException.

LinguistLanguage.Id is the stable language identity used by equality and hashing. Names and aliases are metadata and can change between Linguist revisions. Do not persist IDs across Linguist upgrades without validating them against the newly loaded registry.

Classification filters

Classification is unrestricted by default:

ClassificationResults results = runtime.Classify(source);
ClassificationResult? best = results.Results.FirstOrDefault();
Console.WriteLine(best?.Language.Name);

Restrict classification using language IDs from the same runtime:

LinguistLanguage ruby = runtime.FindByName("Ruby");

ClassificationResults rubyOnly = runtime.Classify(
    source,
    new ClassificationOptions
    {
        CandidateLanguageIds = [ruby.Id],
        AllowedTypes = LanguageTypeMask.Programming,
        MaximumBytes = 16 * 1024,
    });

ClassifyDotNet takes the same options and returns the same results, scores included bit for bit. It tokenizes and scores in .NET against the classifier database Linguist itself loads (lib/linguist/samples.tsv), so it does not take the process-wide Ruby lock and is far cheaper per call. The first call parses the database.

ClassificationResults results = runtime.ClassifyDotNet(source);

When classification is all you need, LinguistClassifier gives the same results without creating a LinguistRuntime, so Ruby never starts. It reads the language registry and classifier database from the same deployed asset directory:

LinguistClassifier classifier = LinguistClassifier.Load();
ClassificationResults results = classifier.Classify(source);

Scores are similarities between 0 and 1. Linguist's floating-point arithmetic can return a score a few units in the last place above 1 (such as 1.0000000000000002) for an input identical to a language's only sample; both methods clamp those to 1.0, and reject anything further out of range with a LinguistException.

Candidate-list semantics are deliberate:

The no-language sentinel (ulong.MaxValue), duplicate IDs, and IDs absent from the active runtime are rejected. Language ID 0 is valid. Classifier scores are similarities, not probabilities or confidence percentages. Applications should establish their own admission policy and inspect ClassificationResults.ConsideredBytes when input truncation matters.

Lifetime and threading

• Runtime creation is synchronous and loads the embedded CRuby and Linguist environment. Create it during application startup when possible.
• A runtime is thread-safe, but calls through one instance are mutually exclusive, and all runtime instances ultimately share one process-wide Ruby worker. Concurrent work queues rather than running Ruby in parallel. ClassifyDotNet is the exception: it runs no Ruby, so concurrent calls run in parallel.
• LinguistClassifier needs no runtime and no disposal. Load() reads the data once per process and returns the same thread-safe instance on every call.
• There is no asynchronous or cancellation API. A caller and Dispose wait for the active native operation to finish.
• Dispose waits for an active call and for the native handle release to finish.
• Calling Dispose repeatedly or concurrently is safe.
• Every state-dependent public member throws ObjectDisposedException after disposal.
• Results contain no native handles and remain readable after disposal.
• Dispose releases that runtime handle, but it does not unload CRuby or stop the process-wide worker.
• The v1 process runtime cannot be unloaded or reinitialized after CRuby starts, and all runtime instances in one process must use the same asset root.

Prefer one long-lived runtime rather than constructing a runtime per file. Creating and disposing runtimes repeatedly does not repeatedly unload the native runtime and provides no parallelism benefit.

After all managed runtime handles are disposed, the native worker thread, queue, CRuby heap, loaded libraries, and Ruby state remain resident until process exit. Disposal is not a native-memory reclamation mechanism.

Native deployment and footguns

The package contains native code and a full private Ruby environment. Treat the RID-specific closure as one immutable deployment unit.

The managed assembly, ghlinguist ABI bridge, and native closure are a matched unit. Do not manually combine files from different package versions.

Select a supported RID

Set the runtime identifier on the executable application, especially for publishing or cross-building:

<PropertyGroup>
  <RuntimeIdentifier>win-x64</RuntimeIdentifier>
</PropertyGroup>

Use linux-x64 on Linux. The managed package supplies both runtime packages as exact transitive dependencies, and build-transitive assets select the matching closure. A missing RID or a RID other than win-x64 or linux-x64 fails the build rather than deploying an incompatible closure.

Restore/build success therefore does not prove that native deployment is valid. Publish for the target RID and run the produced directory on that platform:

dotnet publish --configuration Release --runtime win-x64 --self-contained false

In CI, verify that the publish directory contains MBW.GHLinguist/ghlinguist.dll on Windows or MBW.GHLinguist/ghlinguist.so on Linux, plus the complete Ruby/ Linguist closure in that directory.

Preserve the complete output layout

The selected runtime package exposes its closure as normal transitive content in the MBW.GHLinguist directory beside the managed assembly. Do not copy only ghlinguist.dll or ghlinguist.so, and do not flatten, rename, trim, or selectively clean that directory. The bridge also needs its adjacent CRuby runtime, Ruby standard library, gems, ICU libraries, Linguist sources and data, and tokenizer extension.

Some resources, including classifier data, can be loaded on first use rather than at LinguistRuntime.Create(). Do not replace or delete the native closure while the process is running. Deploy package upgrades as a complete unit and restart the process.

Publish upgrades into a fresh directory or atomically swap whole directories. Do not overlay a new package onto an old output: incremental copying does not guarantee that files removed by a newer closure disappear from the destination.

Unsupported deployment modes

These configurations are not supported:

• Single-file publishing
• NativeAOT
• Trimming
• Collectible AssemblyLoadContext unloading
• Loading different package/asset versions into one process
• Continuing to use an initialized runtime in a forked child process
• win-arm64, linux-arm64, macOS, or any RID other than win-x64 and linux-x64

Directory-based framework-dependent publishing is exercised by CI. A self-contained directory publish is compatible with the intended layout but must be tested by the consumer. Always test the exact publish directory on the same RID used in production.

Do not use plugin isolation to load different package versions or asset roots. Unloading a collectible AssemblyLoadContext cannot unload the process-wide CRuby or native libraries; use one version and one asset root per process.

Bound memory and throughput

Analyze accepts and copies the complete supplied blob into native request storage. It has no streaming interface and no built-in maximum file size. Applications processing untrusted repositories should enforce their own size, queue, timeout, and admission limits before calling the API. Classify is different: it considers at most the leading 50 KiB.

During Analyze, the caller's complete buffer remains live while native request storage holds another copy, before Ruby and result allocations are considered. The operation cannot be interrupted by an in-process timeout or cancellation. For hostile or untrusted workloads, enforce hard per-blob and total in-flight byte limits and consider a separate worker process that the application can terminate.

Because all Ruby work is serialized, adding concurrent callers increases queue depth rather than Linguist throughput. For classification alone, use LinguistClassifier or ClassifyDotNet, which run in parallel. If this becomes a bottleneck, scale with separate processes, not additional LinguistRuntime instances in one process. Thread-safe means serialized, not parallel, bounded, fair, or guaranteed to finish within a particular time. Preliminary synthetic Ruby-only benchmark results are documented in benchmarks/README.md; measure startup, latency, working set, and queueing with representative data.

Version information

LinguistRuntime.Version reports what is loaded: PackageVersion is the managed package version, NativeBridgeRevision is the Git revision the native bridge was built from (or null when the build did not record one), and RubyVersion, LinguistVersion, LinguistRevision, and ClassifierSha256 identify the embedded Ruby and Linguist data.

Testing code that uses the runtime

LinguistRuntime implements ILinguistRuntime. Depend on the interface to substitute the runtime in unit tests without loading CRuby:

services.AddSingleton<ILinguistRuntime>(_ => LinguistRuntime.Create());

Every result type has public init-only properties, so a test double can return hand-built results:

var ruby = new LinguistLanguage { Id = 326, Name = "Ruby", Type = LanguageType.Programming };
var analysis = new BlobAnalysis { Language = ruby, Strategy = DetectionStrategy.Extension, IsText = true };

The interface covers the Ruby-backed operations; ClassifyDotNet is only on LinguistRuntime, and code that depends on the interface uses Classify, which returns the same results. The interface does not include IDisposable; whoever owns the LinguistRuntime instance, such as the dependency-injection container, disposes it.

Exceptions

Discovering the API

Begin with these types in IntelliSense:

• LinguistRuntime for lifecycle and operations
• LinguistClassifier for classification without starting Ruby
• ILinguistRuntime for depending on the runtime and substituting it in tests
• BlobInput for path and filename metadata
• BlobAnalysisOptions for optional trace and line-count work
• ClassificationOptions for classifier bounds and filters
• BlobAnalysis and ClassificationResults for returned values
• LinguistLanguage for registry metadata and stable language identity

The NuGet package includes MBW.GHLinguist.xml, so public members include IntelliSense descriptions, examples, example results, exceptions, and links to the corresponding pinned GitHub Linguist source where available.

Build from source

dotnet restore MBW.GHLinguist.slnx
dotnet build MBW.GHLinguist.slnx --configuration Release --no-restore --nologo
dotnet test --solution MBW.GHLinguist.slnx --configuration Release --no-build --minimum-expected-tests 1

Those commands exercise managed contracts but do not stage or run the embedded CRuby closure. For a Windows native integration run, rebuild and stage a complete closure, copy it to an isolated asset layout, then set the native-integration gate:

./eng/linguist/build-windows.ps1
$env:GHL_RUN_NATIVE_INTEGRATION = "true"
dotnet test --project tests/MBW.GHLinguist.Tests/MBW.GHLinguist.Tests.csproj --configuration Release --runtime win-x64 -p:RunNativeIntegrationTests=true --minimum-expected-tests 1

For Linux, rebuild and stage the Linux closure with ./eng/linguist/build-docker.ps1, use an isolated Linux asset layout, set GHL_RUN_NATIVE_INTEGRATION=true, and run the same command with --runtime linux-x64. A raw checkout also needs generated samples and gem setup; use the complete native-build closure rather than treating this as a standalone command. See eng/linguist/README.md for the native build prerequisites.

Building native assets

RID-specific native builds stage package-ready files beneath .tmp/artifacts/native/<rid>. The corresponding runtime package exposes the ABI bridge under runtimes/<rid>/native and stores its complete relocatable closure under nativeassets/<rid>. Its buildTransitive target contributes the closure as normal content under MBW.GHLinguist with its relative layout intact for build and publish. The managed package requires an explicit supported RuntimeIdentifier.

License

The MBW.GHLinguist source code is licensed under the MIT License. The runtime packages also redistribute CRuby, GitHub Linguist, Ruby gems, and native libraries under their own licenses, including LGPL-licensed libraries. See THIRD-PARTY-NOTICES.md and PACKAGE-LICENSES.md before redistributing an application that contains a runtime closure.