Lodestar.Cluster 0.1.0

Lodestar

A data-science toolkit for C#/.NET, built on an honest premise:

Don't rewrite Python. Use the .NET ecosystem where it's strong, and write native code only where .NET has a real gap: text (similarity, vectorization, semantic search). All of it with no Python at runtime.

Why

Python dominates data analysis through its ecosystem and its exploratory notebook workflow, not through the language itself. Its performance comes from C/Fortran kernels.

C# brings static typing, real parallelism without a global interpreter lock, safe refactoring, and simple deployment. The only objective reason to stay on Python for this domain was the lack of an equivalent .NET library. Lodestar removes that reason.

What has no .NET equivalent

The claim this project is judged on, strongest first.

1. Sparse text vectorization. CountVectorizer, TfidfVectorizer and HashingVectorizer at scikit-learn semantics, over a CsrMatrix written here. ML.NET's FeaturizeText produces a dense vector coupled to IDataView; for the same thousand documents it materializes 81 million floats where the sparse matrix stores 39 974 values. There is no third option in .NET.
2. Framework-free classification metrics. 54 metric classes against ML.NET's six result types — and no IDataView, no schema, no pipeline object between the caller and a double. ML.NET has no call that returns one metric: asking it for accuracy costs the whole evaluation bundle.
3. Loading the tokenizer files people actually have. tokenizer.json — normalizer, pre-tokenizer, model, decoder, added tokens — is what Llama-2 and Mistral v0.1 ship, and Microsoft.ML.Tokenizers has no entry point that reads one. Every one of its factories takes a vocabulary, a merges file or a spiece.model.
4. Split conformal prediction. An interval instead of a point, a set instead of a class, with a finite-sample coverage guarantee — MAPIE's job, and the survey behind #441 found no C# implementation at all, maintained or otherwise. The guarantee assumes exchangeable calibration and test data, which docs/guides/conformal.md leads with rather than footnotes.
5. Distances, embeddings and fuzzy matching, bundled for pipeline coherence rather than because .NET is empty here — it is not, and the table below says by how much.

All of it with no Python at runtime, on .NET 10 and .NET Standard 2.0 from a single package (also .NET Framework 4.6.1+, Mono, Xamarin, Unity — see docs/decisions/0001).

The second deliverable is the migration guides for people arriving from Python: docs/migration/ points each need (NumPy, pandas, scikit-learn, statsmodels, PyTorch, matplotlib, seaborn) at the right .NET building block and its pitfalls, marks the libraries that are no longer maintained with the dates that prove it, and says when calling Python is still the right answer. Its four-column inventory is the project map — use, build, decide.

Measured against the .NET incumbents

A claim nobody checked is a claim nobody believes, so every package is benchmarked against the .NET library a reader would otherwise reach for — not only against Python. Both sides are checked to return the same answers before either is timed; bench/README.md's section 15 has the harness and the agreement checks.

Numbers with the machine that produced them are in docs/guides/performance.md; a shared runner's absolutes are not comparable and are deliberately not published there.

Why not just call Python?

CSnakes and Python.NET both work, both are maintained, and for a model that only exists as a Python package they are the right answer — docs/migration/ says so. What they cost is a Python runtime to deploy and version alongside the application, no ahead-of-time compilation to a single artifact, and the GIL between your threads and theirs. Where a .NET library will do, that is a poor trade, and this project exists to make it an avoidable one.

What is delivered

The five lots of the original brief are complete, and the repository has since grown past that framing — docs/reference/ documents considerably more than this table lists.

Every building block is oracle-validated against rapidfuzz / jellyfish / textdistance / difflib / scikit-learn / nltk / HuggingFace tokenizers / sentencepiece / numpy / ONNX Runtime (see docs/equivalence.md).

Getting started

dotnet add package Lodestar.Text

using Lodestar.Text.Distances;

Levenshtein.Distance("kitten", "sitting");             // 3
Levenshtein.NormalizedSimilarity("kitten", "sitting"); // 0.5714…

A runnable version of the above, consuming the packages exactly as you would:

for p in src/Lodestar.Abstractions src/Lodestar.Text src/Lodestar.Embeddings \
        src/Lodestar.Fuzzy src/Lodestar.Metrics src/Lodestar.Conformal \
        src/Lodestar.Decomposition src/Lodestar.Onnx src/Lodestar.Extensions.AI \
        src/Lodestar.Extensions.MathNet src/Lodestar.Cluster src/Lodestar.Preprocessing \
        src/Lodestar.Stats src/Lodestar.Stats.Regression src/Lodestar.Survival \
        src/Lodestar.Gpu; do
  dotnet pack "$p" -c Release -o ./artifacts
done
NUGET_PACKAGES=$(mktemp -d) dotnet run -c Release --project samples/Lodestar.Sample

The isolated NUGET_PACKAGES is not decoration: the global packages folder is consulted ahead of any source, so a machine that has ever restored a published Lodestar.* at one of these versions runs the sample against that rather than against what pack just produced — see docs/decisions/0009. On PowerShell the same isolation is two lines, $env:NUGET_PACKAGES = (New-Item -ItemType Directory -Path (Join-Path $env:TEMP (New-Guid))).FullName before the dotnet run, and Remove-Item Env:NUGET_PACKAGES after it.

Full guide: docs/guides/quickstart.md. See also the vectorization, embeddings, fuzzy-matching, decomposition, dictionary-lookup and metrics guides — the last one answers which metric to reach for, which the per-member reference pages deliberately cannot.

Function by function, the reference pages under docs/reference/ say what each member is for, when to prefer it to its neighbour and what the trap is — start with the distances. The same pages are published to the wiki, where each package's channel follows main and every release is archived under its own version.

Developing

dotnet build                                   # build the solution
dotnet test                                    # replay oracles + property tests
dotnet run -c Release --project bench/Lodestar.Text.Benchmarks -- --filter '*Levenshtein*'

The project follows GitHub flow: main is always releasable, and every change arrives through a short-lived branch and a pull request. Branch conventions, the definition of done, the oracle-validation procedure and the analyzer-suppression policy are in CONTRIBUTING.md; release history is in CHANGELOG.md.

Oracle validation

Conformance to Python behavior is proven, not assumed (§4 of the brief). tools/generate_oracles.py freezes a few thousand reference cases from rapidfuzz/jellyfish/etc. into tests/oracles/*.json (versioned). The C# suite replays them with a 1e-9 tolerance. Python is a development-only dependency. See tools/README.md.

Structure

Lodestar.slnx
├── src/Lodestar.Abstractions/              CsrMatrix and SparseNorm — the sparse primitive the others share (no dependencies)
├── src/Lodestar.Text/                      distances, similarity, tokenizers, vectorizers, stemmers
├── src/Lodestar.Embeddings/                sub-word tokenizers, pooling, SIMD kNN (no dependencies)
├── src/Lodestar.Fuzzy/                     fuzz.*, process.extract, deduplication
├── src/Lodestar.Metrics/                   confusion matrix, precision/recall/F1, report, ROC-AUC
├── src/Lodestar.Conformal/                 split conformal intervals and prediction sets (no dependencies)
├── src/Lodestar.Decomposition/             truncated SVD and non-negative matrix factorization over a CsrMatrix
├── src/Lodestar.Onnx/                      ONNX inference — the one package with an external dependency (decision 0076)
├── src/Lodestar.Stats/                     classical hypothesis tests, at scipy.stats parity (no dependencies)
├── tests/                                  xUnit: two projects per package — net10.0, and a mirror linking the same sources against netstandard2.0
├── tests/oracles/                          frozen JSON corpora (generated from Python) + a synthetic ONNX model
├── bench/Lodestar.Text.Benchmarks/         BenchmarkDotNet: every non-netstandard benchmark, whatever package it measures
├── bench/Lodestar.NetStandard.Benchmarks/  the netstandard2.0 assemblies, measured on the same host
├── tools/generate_oracles.py               reference generation
├── Directory.Build.props                   (root); src|tests/Directory.Packages.props (central package management)
├── src/*/Version.props                     one version per publishable package (decision 0012)
├── docs/                                   guides, equivalence table, decision log
├── docs/reference/<package>/               one reference entry per exported type and public method
└── docs/wiki-map.json                      which page ships with which package, and which namespaces the reference gate enforces

Where a fact belongs

Each document below has one subject; content whose subject is another document's belongs there instead, with a link left behind. The source column is what tells you whether to correct the document itself or something upstream of it.

Publishing

Nine NuGet packages are produced: Lodestar.Abstractions, Lodestar.Text, Lodestar.Embeddings, Lodestar.Fuzzy, Lodestar.Metrics, Lodestar.Conformal, Lodestar.Decomposition, Lodestar.Onnx and Lodestar.Stats. Eight of them are core tier and carry no external dependency; Lodestar.Onnx is the satellite that carries ONNX Runtime, which is the whole reason it is a package — decisions/0076. Each versions and releases on its own: shared metadata (license, README, repository) lives in Directory.Build.props, while the version is declared per project in src/<Package>/Version.props. Lodestar.Fuzzy depends on Lodestar.Text as a published package, not as a project reference — see docs/decisions/0012.

GitHub Packages (no nuget.org account needed — uses GitHub's automatic token). Bump the version, then tag it with the package name. The release workflow packs and publishes that package alone:

# 1. edit src/Lodestar.Fuzzy/Version.props, commit, merge to main
# 2. tag the released version — <PackageId>/v<Version>
git tag Lodestar.Fuzzy/v0.3.0
git push origin Lodestar.Fuzzy/v0.3.0

The tag does not set the version; it names which declared version to release. The workflow refuses the job if the tag and Version.props disagree. Repository-wide v* tags are retired — there is no single version left for one to designate.

Step 1 is not optional. Because the tag only confirms the declared version, tagging without bumping first is a tag that agrees with Version.props and names a version the feed already has. The push is then rejected rather than absorbed. The workflows do not pass --skip-duplicate, which used to report that case as a successful release that shipped nothing. Keeping a declared version off the feed is also checked directly in CI by tools/check_version_floor.py.

To consume them, add a source pointing at the owner's feed (with a GitHub token that has read:packages):

dotnet nuget add source "https://nuget.pkg.github.com/CyrilB1531/index.json" \
  --name github --username CyrilB1531 --password <GITHUB_TOKEN>
dotnet add package Lodestar.Text

nuget.org uses Trusted Publishing (OIDC, no stored key): run the Publish to nuget.org workflow from the Actions tab, choosing the package and confirming its version. By hand, with an API key, one package at a time:

dotnet pack src/Lodestar.Text -c Release -o artifacts
dotnet nuget push "artifacts/Lodestar.Text.*.nupkg" \
  --source https://api.nuget.org/v3/index.json --api-key <KEY>

License

Apache-2.0. See NOTICE and THIRD-PARTY-NOTICES.md for attributions. The license choice and the code-provenance rule are documented in docs/decisions/0003-provenance-and-licensing.md.

This repository is not legal advice.