NumSharp.Interop.MLNet 0.80.0

NumSharp.Interop.MLNet

Interop between NumSharp NDArray and ML.NET (Microsoft.ML) data — the IDataView pipeline currency and the VBuffer<T> vector primitive.

Expose an NDArray to a trained ML.NET pipeline as an IDataView over the very same unmanaged buffer — read lazily through the array's strides, no per-row POCO fill, no densifying copy — and read a transformer's output column straight back into an NDArray. The whole pipeline bridge is two verbs:

using NumSharp;
using NumSharp.Interop.MLNet;

// A (rows, features) matrix in NumSharp -> feed a trained model -> read the scores back
using var featuresView = features.AsDataView("Features");     // no copy; any layout
IDataView output = transformer.Transform(featuresView);       // your ML.NET pipeline
using NDArray scores = output.ToNDArray("Score");             // straight into NumSharp

This is a conversion library in the mould of NumSharp.Interop.OnnxRuntime / NumSharp.Interop.pythonnet: no engine backend, no [ModuleInitializer], no native dependency. It depends only on the standalone Microsoft.ML.DataView contract package (IDataView, VBuffer<T>, DataViewSchema, DataViewType) and composes with whatever Microsoft.ML (MLContext, trainers, transforms) you already reference — NuGet unifies the DataView dependency upward.

The verbs

As… shares memory (a lazy, pinned view); To… copies.

Zero-copy and VBuffer

vbuffer.AsNDArray<T>() is a genuine zero-copy view — it reaches the private T[] backing a dense VBuffer<T> through a compiled-expression accessor, pins it, and wraps it write-through (mutations visible both ways), exactly like the pythonnet bridge. Caveat: the view shares the VBuffer's array, so it is valid only while that array is not refilled — a VBuffer from a cursor getter reuses one buffer across rows, so use ToNDArray() (a copy) there; zero-copy is safe for a standalone VBuffer you own.

The reverse direction (NDArray → VBuffer) cannot be zero-copy: VBuffer<T> is backed by a managed T[] at every ML.NET version (verified 2.0.0 / 3.0.0 / 4.0.2 / 5.0.1 and the 6.0 previews — no ReadOnlyMemory<T>, no non-public constructor), and a managed array cannot alias NumSharp's unmanaged buffer. ToVBuffer<T> copies. (The bulk NumSharp→ML.NET path — AsDataView — is already zero-copy on its source; the VBuffer is a per-vector convenience.)

Postprocess.{Softmax, LogSoftmax, Sigmoid, Argmax, TopK} turn a score column into a prediction with np.* compositions instead of hand-written loops. The class is NumSharp.Interop.OnnxRuntime.Postprocess: this package references the ONNX Runtime bridge for it (using NumSharp.Interop.OnnxRuntime;), so an app that uses both bridges has one Postprocess, and calling it never loads ONNX Runtime.

Dtypes

Eleven dtypes map directly to an ML.NET column type — bool → BooleanDataViewType; the eight integers, float and double → the matching NumberDataViewType. Char crosses as NumberDataViewType.UInt16 (UTF-16 code units), one-way. ML.NET has no half / decimal / complex column type: Half is converted to Single (lossless) and Decimal to Double by the view-producing verbs (in a temporary the view owns); Complex is refused (split into real / imaginary planes with np.real / np.imag), and so are text arrays (U, S, StringDType: the bridge maps no ML.NET text column). ToNDArray refuses a text column too (string labels after MapKeyToValue("PredictedLabel")): ML.NET's own reader does it in a line, np.array(view.GetColumn<string>("PredictedLabel").ToArray()) (a U array; GetColumn<string[]> for a vector-of-text column).

ML.NET's trainers and most transforms read Single features, while NumSharp's default float dtype is float64, as NumPy's: convert with x.astype(np.float32) before AsDataView, or a pipeline stops with ML.NET's Schema mismatch ... expected Vector<Single>, got Vector<Double>. A key column (PredictedLabel of a multiclass model or k-means) reads back as its uint32 keys, 1-based.

Lifetime

AsDataView returns an NDArrayDataView (which, unlike a bare IDataView, is IDisposable): dispose it — a using, or wire it into your MLContext lifetime — to release the buffer pin. The pin takes its own ARC reference, so it is even safe to dispose the source array while the view (or its cursors) still read it. Not disposing leaks the pin until finalization.

Reachable means valid. Whatever can reach the view — you, a transform's output, a cursor, a getter delegate — keeps the pins and the buffer alive, and a getter keeps its cursor reachable until its read is done, so a collection landing mid-copy cannot release the row under it. The finalizers of the view and its cursors only give their reference back (never free eagerly), so an alias of the buffer you still hold stays valid after they run, in any order.

A source whose buffer has already been released is refused (ObjectDisposedException) by every export verb: each takes its reference before its first read (the Half / Decimal conversion, ToDataView's snapshot, ToVBuffer's copy), so it never reads memory the pool may have handed to another array.

Cursors keep ML.NET's own contract (checked against ML.NET's Cache, FilterRowsByColumn, TakeRows and LoadFromEnumerable cursors): Position is -1 before the first row, after the last and after Dispose; a getter called off a row throws ML.NET's own InvalidOperationException ("Values cannot be fetched at this time…") and reads nothing; MoveNext stays false once it returned false or the cursor was disposed; and Schema still answers after the view is disposed. Off-row getters used to read the element before the first row or past the last one — after Dispose, from a buffer the cursor no longer kept alive.

Each cursor holds its own pin and releases it on Dispose — or, for a cursor nobody disposes, from a finalizer. ML.NET abandons cursors itself (ShuffleRows, used by training pipelines, never disposes the cursor it opens on its input; the time-series package's DetectEntireAnomalyBySrCnn leaves two per call), and before that net a single shuffle kept the NumSharp buffer referenced for the life of the process. Until a collection finalizes such a cursor its pin still holds the buffer, so the memory comes back only then.

Imports have no managed-array cap. view.ToNDArray(column) stages rows straight into NumSharp memory: one exact allocation when the view reports its row count, growing chunks (copied together once) when it does not, so a column of more than Array.MaxLength elements in total imports (it used to stage through a managed T[] and fail there). A sparse VBuffer densifies at any size (the clear counted bytes in an int and threw at 2 GB).

ML.NET's buffers hold at most 2,146,435,071 elements (its internal ArrayUtils.ArrayMaxSize, a little below Array.MaxLength): a vector column wider than that, or a ToVBuffer longer than that, is refused at export, before anything is allocated — ML.NET could not read such a row or copy such a buffer.

NDArrayMLNetInterop.LiveExports counts live views, for leak detection in tests.

Dependency

The package references Microsoft.ML.DataView only. There is no standalone Microsoft.ML.Data NuGet package, and ITransformer is not needed here: IDataView is ML.NET's universal currency, so feeding a transformer and reading its output are both just conversions. Bring the Microsoft.ML you already use for training / transforms.

License

Apache-2.0, same as NumSharp.

What's new in 0.80.0

We are at 95.2% NumPy API coverage, and everything except datetime, complex64 and other exotic dtypes matches NumPy byte for byte. This release brings NumPy's text dtypes with numpy.strings and numpy.char, numpy.ma, numpy.polynomial, np.emath and 48 new np.* functions. Integration rather than competition goes on, this time inside .NET: an NDArray now runs through ONNX Runtime models, serves as an ML.NET IDataView and shares its memory with System.Numerics.Tensors.

🧭 Overview

• NumPy 2.x API coverage around 95.2%, everything except datetime, complex64 and other exotic dtypes is fully supported, with byte-to-byte equality and compatibility to NumPy.
  • Text - NumPy's fixed-width bytes_ (S) and str_ (U) dtypes and NEP 55's variable-width StringDType, with every computation NumPy runs on text, all 46 numpy.strings functions and numpy.char with chararray.
  • np.polynomial.* - the whole numpy.polynomial package, all 193 functions and the six series classes for all six bases, bit-exact.
  • np.* - +48 functions, the ten most used on GitHub: np.hypot, np.histogram, np.logspace, np.gradient, np.vectorize, np.hanning, np.geomspace, np.apply_along_axis, np.trapezoid and np.histogram2d.
  • np.random.* - every NumPy bit generator, the whole Generator distribution surface, and a legacy RandomState that is NumPy's own sampler code line for line, all byte-identical streams.
  • np.ma.* - the masked-array module; np.emath.* - the automatic-domain math module; np.matrix - numpy.matrix as the new NDMatrix type.
• DType is NumPy's two-level dtype model - NDArray.dtype returns a numpy.dtype-shaped descriptor, with np.dtypes, byte order and datetime64 / timedelta64 descriptors, and NumPy's scalar classes (np.integer, np.floating, np.number, ...) with np.typing.
• Reductions on NumPy's own schedules - max, min, ptp, nanmax, nanmin, argmax, the flat sum / mean and the fused np.evaluate reductions match NumPy bit for bit, down to the sign of a ±0 tie and a NaN's payload; mean, var, std and their NaN-aware forms are NumPy's own statements; and most of them run faster.
• NumPy's structure for the engine - every computing np.* function is a TensorEngine member named after its NumPy source file, numpy.linalg and numpy.fft are replaceable sub-engines, and the default engine is settable.
• The 2026-11-02 parity QA - 53 of its 55 findings fixed on the owner's decisions (one kept as a documented divergence, one deferred), and NumPy's own dot functions behind np.dot / np.vdot / np.vecdot.
• 86 NumPy-parity, memory-safety and tooling fixes, 40 measured speedups and 44 breaking changes, across the release's 1,670 commits.
• 3 new NuGet packages - NumSharp.Interop.System.Numerics.Tensors, NumSharp.Interop.OnnxRuntime and NumSharp.Interop.MLNet.
• 1,119,305 NumPy 2.4.2 oracle test cases (0.70.0: 119,517) and 20,506 test methods (0.70.0: 14,416).

📦 New NuGet Packages

• NumSharp.Interop.System.Numerics.Tensors - zero-copy interop between NDArray and the BCL's Tensor<T> / TensorSpan<T> (System.Numerics.Tensors), all 15 dtypes.
• NumSharp.Interop.OnnxRuntime - run ONNX Runtime models on NDArrays, fed from the arrays' own memory, with the outputs read back as NDArrays.
• NumSharp.Interop.MLNet - an NDArray as an ML.NET IDataView, and a transformer's output column back as an NDArray.

Full NumSharp 0.80.0 release notes