NumSharp.Interop.OnnxRuntime 0.80.0

NumSharp.Interop.OnnxRuntime

Zero-copy interop between NumSharp NDArray and ONNX Runtime tensors. Feed an NDArray to InferenceSession.Run as an OrtValue built over the very same unmanaged buffer — no DenseTensor fill loop, no copy — read outputs back as NDArrays (an owning copy, or a zero-copy view), and do the pre-/post-processing (normalize, transpose, softmax, argmax, top-k, clip) as vectorized NumSharp ops instead of the per-pixel for loops every ORT C# tutorial hand-writes.

using NumSharp;
using NumSharp.Interop.OnnxRuntime;
using Microsoft.ML.OnnxRuntime;

using var session = new InferenceSession("resnet50-v2-7.onnx");

NDArray input = /* (1,3,224,224) float32, C-contiguous */;
using NDArray logits = session.Run(input);                 // zero-copy in, owning NDArray out
using NDArray probs  = Postprocess.Softmax(logits);        // (1,1000)
var (top5, classes) = Postprocess.TopK(probs, 5);          // values + int64 indices

This is a conversion library, in the mould of NumSharp.Interop.pythonnet, not an engine backend: ONNX Runtime does not compute NumSharp operations, it consumes tensors NumSharp produces and produces tensors NumSharp reads. So there is no TensorEngine seam, no [ModuleInitializer], and referencing the package changes nothing until a verb is called.

Verified against ONNX Runtime 1.16.0 (the package floor) and 1.31.0 (current) on net8.0 / net10.0, by test/NumSharp.Tests.Interop.OnnxRuntime: committed .onnx models that ONNX Runtime really executes, with an every-dtype × every-layout matrix and property-based, metamorphic, bit-pattern-fuzz, concurrency, boundary, reference-accounting, lifetime and API-approval suites. The package's first release is 0.80.0.

Install

dotnet add package NumSharp.Interop.OnnxRuntime
dotnet add package Microsoft.ML.OnnxRuntime          # or .Gpu / .DirectML / .QNN — exactly ONE native flavour

The package depends only on Microsoft.ML.OnnxRuntime.Managed (the C# API). ORT ships its native runtime as mutually exclusive flavours — Microsoft.ML.OnnxRuntime (CPU), .Gpu, .DirectML, ... — and two of them side by side is a documented misconfiguration, so the flavour is yours to pick: the one you already reference to run inference at all. The dependency is 1.16.0 or later with no upper bound (a cap below a consumer's ORT would be a restore error, NU1107, the day they upgrade); a consumer's own ORT reference (say 1.31.1) carries the same .Managed version and NuGet unifies upward to it. The NuGet range is the only thing that sets the floor: the 1.16.0 managed assembly reports assembly version 0.0.0.0, so the runtime's binder refuses nothing.

On Windows, forgetting the native flavour does not fail at once: the loader finds the onnxruntime.dll that Windows ships in C:\Windows\System32 for Windows ML (1.17 on a current Windows 11), and the app runs on that build: another version than the one you reference, whatever the machine has, and absent on Linux and macOS (DllNotFoundException). Reference the native flavour explicitly.

Views, copies and OnnxCodecMode

An ORT tensor is a pointer, a row-major shape and a fixed-size element type. It has no strides. So ORT can read a NumSharp buffer in place only when the array is C-contiguous and its dtype crosses bit for bit (bool, the eight integers, float16, float32, float64, and char as UInt16). Every entry point that could share memory takes an OnnxCodecMode, with the member names and meanings of the pythonnet bridge's NumpyCodecMode:

A copy is needed for a Fortran-ordered, stepped, reversed, transposed or broadcast view (a C-order copy), for a Decimal array (a float64 conversion) and for text. No mode repairs a dtype with no ONNX element type (Complex), a released buffer, or a size the target cannot hold.

The verbs

As… gives a view when it can (and a handle or an array that says which it gave), To… always copies.

Plus the dtype maps ToTensorElementType / FromTensorElementType (+ Try… forms, ToTensorElementClrType), IsCpuAccessible, and two counters, NDArrayOnnxInterop.LiveExports / LiveImports, that make every live crossing observable.

The ergonomic tier — session.Run(NDArray)

InferenceSessionExtensions hides every OrtValue lifetime:

NDArray y = session.Run(x);                                   // single input/output: names from metadata
NDArray y2 = session.Run("X", x, "Y2");                       // named
IReadOnlyDictionary<string, NDArray> outs = session.Run(      // several inputs, all (or some) outputs
    new Dictionary<string, NDArray> { { "A", a }, { "B", b } });
var views = session.Run(inputs, zeroCopyOutputs: true);       // views that OWN their OrtValue
session.RunInto(inputs, new Dictionary<string, NDArray> { { "Y", preallocated } });   // ORT writes INTO your array
await session.RunIntoAsync(inputs, outputs);                  // the same, without blocking a thread (ORT 1.30+)

• Inputs. An input that is C-contiguous and of the declared dtype is shared. Any other is fed as a temporary under Auto (a cast, a C-order copy of a view), refused under View, and every input is copied under Copy. A dtype that differs from the declaration is cast to it when NumPy's casting rule allows it ("safe" by default — so an int32 array into an int64 BERT input or float32 into float64 just works; a lossy float64 → float16, and a Decimal array into a float64 input, need casting: "same_kind" or an astype at the call site). Char into a UInt16 input is safe and stays zero-copy. The rule name is checked on every call. Declared fixed extents and rank are checked before ORT sees the tensor, with a message naming the input (input 'data' expects shape (1, 3, 4, 4), but the array has shape (1, 3, 4, 5): dimension 3 must be 4, not 5); symbolic dims ('batch', 'seq') are free.
• Names. An input name is looked up among the model's inputs, then among its overridable initializers. An input declared optional(tensor) takes a plain array.
• The feed and the fetch are checked before the run. Every input that is not optional must be fed (Python onnxruntime's rule and message); an unknown output name is refused with the list of outputs; and an output Run cannot return (a sequence, a map, a sparse tensor) is refused with the readers that take it, before the model runs.
• Outputs are copied into owning arrays by default and every ORT value is released before the call returns. zeroCopyOutputs: true returns views that own their OrtValue — ORT's memory is freed when the array (and every slice derived from it) dies — for large outputs you consume in place.
• RunInto writes into arrays you allocated: zero-copy in both directions, the idiom for a hot loop that reuses its buffers (Stable Diffusion's UNet step). An output array must be C-contiguous, writeable and of the declared dtype (it cannot be cast). ORT may write an output while it still reads an input, so an output that shares memory with an input is computed from a copy of that input under Auto (NumPy's rule for out=) and refused under View; two outputs that overlap are always refused.
• An output that is a fed input. ONNX permits a model output that no node produces (a model input passed through, or an overridable initializer you feed). ORT does not write a pre-allocated array for it and returns a value over the fed memory. The tier handles it: RunInto writes the fed values into your array, the copy form copies them, and the zero-copy form returns a counted view of the fed buffer.
• Text. A text array (U or S) feeds a string input, and a string output comes back as a text array; see below.

Asynchronous runs

RunIntoAsync and RunAsync build on ORT's own native asynchronous run: the model executes on the session's intra-op thread pool and no .NET thread waits.

• They need ONNX Runtime 1.30.0 or newer (InferenceSessionExtensions.IsRunAsyncSupported). Older managed assemblies do not pin the arrays the native run keeps reading (microsoft/onnxruntime#32015), so both forms throw NotSupportedException there.
• The session needs 2 or more intra-op threads (SessionOptions.IntraOpNumThreads is not 1). With one, ORT refuses and the task is faulted.
• RunIntoAsync is RunInto for every model. RunAsync allocates the outputs itself, which it can do only before the run: for a numeric tensor with a fixed declared shape, and for an output that is a fed input. For a dimension fixed only at run time, a declaration without dimensions, a string tensor or a non-tensor output it throws NotSupportedException with what to do instead.
• Usage errors throw from the call. A failure inside ORT faults the task. To cancel, pass a RunOptions and set Terminate = true.
• ORT completes its own task on an intra-op thread of the session. Code that continues there and disposes the session never returns. The bridge's task runs your continuation off that thread, and a thread that blocks on it under a synchronization context is released (ORT's own task waits forever there).
• Until the task completes, ORT reads the shared inputs and writes the outputs: do not write an input (pass OnnxCodecMode.Copy to feed a snapshot). Disposing an array in flight is safe; the run holds a reference.
• An NDScope does not cross the await: a scope belongs to the thread that opened it, and the continuation resumes on another. Held across await session.RunAsync(...), it reaches its end on the continuation's thread, where Dispose throws InvalidOperationException and leaves the scope's arrays to the finalizer: closing it there would free the opening thread's later arrays (one filled with 7.0 would read another array's 9.0). Scope the code before the await and after it, or mark the async method [NDScopedAsync].
• RunAsync allocates its outputs when the call starts, so a scope open then tracks them: hand the task out with scope.ReturnsTask(task) and close the scope with NDScope.CloseUnlessDeferred(scope) (the form [NDScopedAsync] emits); the result dictionary carries its outputs, so the scope yields them to you when the task completes. A plain using closes the scope at once and the task returns arrays over freed memory. Under the synchronous Run, yield each output you keep with scope.Returns(outputs[name]).

Text

NumSharp's fixed-width text dtypes (U: UCS-4 code points; S: bytes) cross as ORT string tensors, in both directions and through every route (ToOrtValue, AsOrtValue, ToNDArray, AsNDArray, the readers, Run).

• Everything is a copy. ORT keeps each string as a separate object in its own memory. Auto and Copy do the same; View throws.
• A U element crosses as its text (ORT stores UTF-8). An S element crosses as its bytes, unchecked, as Python's onnxruntime does. A string tensor reads back as a U array as wide as its longest string.
• The fixed width costs two things, because the bridge reads string tensors as U, not as NumSharp's variable-width StringDType: a string that ENDS in NUL characters loses them (ReadStringTensor keeps them), and one long string widens every element. A StringDType array is not a bridge input (Python's onnxruntime refuses it too): cast it to a sized U first, wide enough for its longest string (nd.astype(np.dtype("U16")); an unsized "U" target is refused for StringDType, as in NumPy).
• The bridge does not cast between text and numbers: a numeric array for a string input, or a text array for a numeric input, is refused under every casting rule. Convert first with NumSharp's own casts, nd.astype(np.dtype("U")) (each number as its str()) or nd.astype(np.float32) (NumPy's text-to-number parse).
• RunInto and RunAsync cannot take a string output (ORT cannot write its strings into NumSharp memory); the synchronous Run returns it.

Dtypes

The maps answer for every descriptor: TryToTensorElementType returns false and ToTensorElementClrType returns null for a dtype without storage (a datetime64 descriptor).

Post-processing

Postprocess is the output half of every tutorial as np compositions — no new kernels, kept apart from the verbs so those stay conversion-only:

They agree with ORT's own Softmax / ArgMax / TopK operators run on the same data (the tests do exactly that). Float32 in gives float32 out. A bool or integer input is promoted to NumPy's float tier before the computation (bool, int8, uint8 → float16; int16, uint16, char → float32; wider → float64), so the result is the mathematical one, not a wrapped integer. LogSoftmax follows scipy.special.log_softmax for a row whose maximum is not finite. Argmax and TopK check their axis.

The #512 example — a BGRA image to an NCHW tensor, no pixel loop

The issue fills a DenseTensor<float> one pixel at a time (/255, channel reorder). With NumSharp it is four vectorized ops feeding ORT zero-copy:

NDArray bgra  = np.frombuffer(data, NPTypeCode.Byte).reshape(H, W, 4);                      // interleaved B,G,R,A
NDArray chw   = np.stack(new[] { bgra[":,:,2"], bgra[":,:,1"], bgra[":,:,0"] }, axis: 0)    // (3,H,W) RGB
                  .astype(NPTypeCode.Single) / 255f;                                        // fresh C-contiguous f32
NDArray input = chw.reshape(1, 3, H, W);                                                    // NCHW

using OrtTensor t = input.AsOrtValue(OnnxCodecMode.View);                                   // ZERO COPY, or it throws
using var results = session.Run(runOptions, new[] { "data" }, new[] { t.Value }, session.OutputNames);
NDArray output = results[0].ToNDArray();

The one copy left — stack + astype/divide producing the planar normalized array — is genuine work (layout + dtype + scale), not marshaling. Or, one line: NDArray output = session.Run(input);.

Lifetime & memory safety

• Exports (AsOrtValue / AsDenseTensor<T>) take their own atomic reference on the NumSharp buffer (NumSharp's ARC), so the memory survives even if the source NDArray is disposed or collected while ORT holds the tensor. Release is deterministic and owned by the returned handle (OrtTensor.Dispose / OrtTensor<T>.Dispose), which also disposes the temporary copy when the handle holds one. Keep the handle (or its Value) reachable across Run — pass handle.Value, dispose after the run (using): ORT's own Run holds no reference to the managed inputs while native code reads them, and neither do OrtIoBinding, SessionOptions.AddInitializer or OrtValue.CreateSequence. While exported, ndarray.resize(refcheck: true) on the source refuses (cannot resize an array that references or is referenced by another array), exactly NumPy's rule.
• ToOrtValue returns ORT's own value — dispose it. ORT allocates and frees the copy, so the value stays valid for every native holder (an OrtIoBinding it was bound to, a run's feeds) after the managed OrtValue is disposed or collected. NumSharp books the copy's bytes with its own memory accounting until the caller disposes the value (the booking ends when ONNX Runtime releases it: at once, or after the last view of it) or the GC collects it: 5,000 forgotten values of 602 KB peaked at about +70 MB of private memory (unbooked: +2.9 GB until an unrelated collection ran), and a value that is disposed never triggers a collection. The booking is a token stored in ORT's private OrtValue._memHandle slot, whose MemoryHandle ORT's Dispose releases; on an ONNX Runtime without that field (1.16.0 through 1.31.1 have it) the value is returned unbooked, and a string tensor is never booked.
• Reachable means valid. Holding only handle.Value, or only Tensor / Memory, keeps the handle — and its reference on the buffer — alive, so a collection never disposes an OrtValue or disables a tensor you still hold. A forgotten handle is released by a finalizer safety net once nothing reaches any of them; it only gives the reference back (never frees eagerly), so an alias of the buffer you still hold stays valid, and it never disposes the OrtValue or a Source conversion you can still reach.
• A value of a disposed handle is refused, not passed to ORT. After handle.Dispose(), ONNX Runtime has released handle.Value, and ORT does not guard a released value (a native call on one ends the process). Every verb that takes an OrtValue throws ObjectDisposedException for such a value. So does a value you disposed yourself, and every Run form on a disposed InferenceSession (both were access violations): the bridge reads ONNX Runtime's own private disposal flag, and skips the check on a version without it.
• A Dispose on another thread is safe for the bridge's verbs. ONNX Runtime's own Dispose takes no lock (two disposers corrupted the heap; a Dispose during a verb was an access violation). Every value the bridge touches carries an ARC: a verb holds a lease while it calls ORT on the value, a view for its whole life, and a Dispose - from any thread, any number of times at once - waits for the verbs in flight and releases the value once, after its last view. A ToOrtValue result and an AsOrtValue value have it from birth; a value ORT allocated (a Run output of ORT's own API) gets it at its first bridge verb. For sessions, use NDInferenceSession (a drop-in InferenceSession with ORT's eight constructors): its Dispose releases the native session once, after the bridge's runs on it returned (the asynchronous release runs on the thread pool), and refuses new runs. ORT's own calls (its Run overloads, GetTensorDataAsSpan, an OrtIoBinding) take no lease: dispose only after those returned.
• Pins of an AsDenseTensor<T> handle's Memory<T> outlive the handle. Whoever pins that memory — ORT's CreateTensorValueFromMemory, any Memory<T>.Pin() — takes a reference of its own on the NumSharp buffer, given back exactly once: when the pin is disposed (two copies of one MemoryHandle count once) or when it is collected undisposed (ORT's OrtValue finalizer never disposes the pin it holds). A disposed handle only refuses NEW access: its tensor's Memory<T> throws ObjectDisposedException for a new span or pin, and so do ToNDArray / AsNDArray of that tensor (before allocating anything).
• A released source is refused with ObjectDisposedException by every verb that reads it (the exports, the copies, Run's cast and C-order copy): each takes its reference on the buffer before its first read, so it never reads memory the pool may have handed to another array. Dtype and size refusals come first, so nothing is converted, copied or allocated for an array that cannot cross.
• Imports of a buffer ORT owns (AsNDArray of a run output) lease it through NumSharp's memory-block reference counting: the lease fires when the last NumSharp view over the memory — derived slices included — is disposed or collected. A non-owning view holds a lease on the OrtValue: the caller may dispose it at any time and the memory stays valid until the last view is gone (a value over memory ORT does not own, or built from members, keeps ORT's rule for GetTensorDataAsSpan: do not dispose it while a view is in use). ownsValue: true transfers ownership: the last view disposes the OrtValue, and the caller must not. A call that throws never disposes the value. An owning view books the buffer's size with NumSharp's memory accounting (not for a ToOrtValue result, which the value has booked already), and so does a DenseTensor<T> view of a managed array, whose pin keeps the array alive until the last view is gone (a forgotten view is then collected like a forgotten NumSharp array); a view of a legacy run's output books nothing. Views do not own their data (flags.owndata == False): a size-changing resize refuses instead of detaching.
• Imports of NumSharp memory are ordinary NumSharp views. A value this package made (nd.AsOrtValue().Value.AsNDArray()), a tensor.AsNDArray() of an AsDenseTensor<T> handle and an output that is a fed input are views of a NumSharp buffer. They hold their own counted reference, stay valid after the handle and the source are disposed, and are read-only when the exported array was. ORT itself has no read-only tensors: a read-only array is a legal model input, and ORT could write a tensor it was given.
• A value ORT returned over memory that is not ORT's. When you run the session with ORT's own API and feed an AsOrtValue handle, ORT returns a value over the handle's memory for an output that is a fed input. AsNDArray cannot know that: such a view is valid only while the handle, or the array you fed, is alive. session.Run(…, zeroCopyOutputs: true) of this package knows what it fed and returns a counted view.
• A DenseTensor<T> view keeps its tensor reachable. denseTensor.AsNDArray() over a legacy session.Run output (DisposableNamedOnnxValue.AsTensor<T>()) roots the tensor — and through it the OrtValue — so results that are merely dropped are not finalized under the view. Disposing the results explicitly while a view lives still frees the memory under it: keep them alive, or copy with ToNDArray.
• Only CPU-addressable tensors can be read or viewed. A CUDA / DirectML / TensorRT-resident output is refused (the tensor lives in 'Cuda' memory ... which the CPU cannot address) — bind a CPU output or copy to CPU first. Host-accessible device memory (CudaPinned, HipPinned, QnnHtpShared) and CpuAligned4K are fine: the bridge asks ORT for the device memory kind where the loaded ORT can tell, and uses the allocator name otherwise.
• No GIL, no interpreter lifetime, no thread-affinity: ORT is thread-safe and the verbs are plain calls.

Limits (stated up front)

• Complex (ORT has no complex tensors), BFloat16 / float8 / 4-bit (no NumSharp dtype). A BFloat16 input goes through ORT's own OrtValue.CreateTensorValueFromMemory (new BFloat16(bits)) and session.Run; a float8 or 4-bit model does not load in ORT's C# API itself (Unregistered TensorElementType value).
• Sparse tensors. ORT's C# API cannot create or read one. A model input declared sparse passes the checks of Run and fails inside ORT.
• Non-tensor outputs through Run. Run returns tensors only (text included). Fetch a sequence or a map with ORT's own session.Run and read it with ToNDArrays / ToMap / ToMaps. A string-keyed map's keys come back sorted: ORT keeps a map ordered by key.
• GPU-resident tensors cannot be viewed as NDArrays.
• ORT's managed data API (GetTensorMutableRawData and every span over it) is int-indexed — it hands out a tensor's memory only as a Span of bytes, whose length it casts to int unchecked. Export: an array over 2 GB is refused by ToOrtValue before anything is copied or allocated — share it zero-copy with AsOrtValue, which hands ORT a native pointer (no limit). Import: a tensor of 2 GB or more that ORT owns can be neither copied (ToNDArray) nor viewed (AsNDArray) out of an OrtValue; both refuse up front, before anything is allocated. Let ORT write such an output into NumSharp's memory instead: pre-allocate it and call RunInto. A value this package made, and a DenseTensor<T> (up to int.MaxValue elements), cross at any byte size.
• ToDenseTensor<T> copies into a managed T[], which holds at most Array.MaxLength (2,147,483,591) elements; a larger array is refused up front. AsDenseTensor<T> takes up to int.MaxValue elements (a Memory<T> cannot index more), AsOrtValue has no element limit.
• Run(...) shape validation can only see what NodeMetadata reports: a declaration without dimensions is either a scalar or an unknown-rank tensor (indistinguishable there) and is left to ORT's own check. For the same reason RunAsync does not allocate such an output.
• The plan and the record of this surface: docs/plans/onnxruntime-rework.md.

What's new in 0.80.0

We are at 95.2% NumPy API coverage, and everything except datetime, complex64 and other exotic dtypes matches NumPy byte for byte. This release brings NumPy's text dtypes with numpy.strings and numpy.char, numpy.ma, numpy.polynomial, np.emath and 48 new np.* functions. Integration rather than competition goes on, this time inside .NET: an NDArray now runs through ONNX Runtime models, serves as an ML.NET IDataView and shares its memory with System.Numerics.Tensors.

🧭 Overview

• NumPy 2.x API coverage around 95.2%, everything except datetime, complex64 and other exotic dtypes is fully supported, with byte-to-byte equality and compatibility to NumPy.
  • Text - NumPy's fixed-width bytes_ (S) and str_ (U) dtypes and NEP 55's variable-width StringDType, with every computation NumPy runs on text, all 46 numpy.strings functions and numpy.char with chararray.
  • np.polynomial.* - the whole numpy.polynomial package, all 193 functions and the six series classes for all six bases, bit-exact.
  • np.* - +48 functions, the ten most used on GitHub: np.hypot, np.histogram, np.logspace, np.gradient, np.vectorize, np.hanning, np.geomspace, np.apply_along_axis, np.trapezoid and np.histogram2d.
  • np.random.* - every NumPy bit generator, the whole Generator distribution surface, and a legacy RandomState that is NumPy's own sampler code line for line, all byte-identical streams.
  • np.ma.* - the masked-array module; np.emath.* - the automatic-domain math module; np.matrix - numpy.matrix as the new NDMatrix type.
• DType is NumPy's two-level dtype model - NDArray.dtype returns a numpy.dtype-shaped descriptor, with np.dtypes, byte order and datetime64 / timedelta64 descriptors, and NumPy's scalar classes (np.integer, np.floating, np.number, ...) with np.typing.
• Reductions on NumPy's own schedules - max, min, ptp, nanmax, nanmin, argmax, the flat sum / mean and the fused np.evaluate reductions match NumPy bit for bit, down to the sign of a ±0 tie and a NaN's payload; mean, var, std and their NaN-aware forms are NumPy's own statements; and most of them run faster.
• NumPy's structure for the engine - every computing np.* function is a TensorEngine member named after its NumPy source file, numpy.linalg and numpy.fft are replaceable sub-engines, and the default engine is settable.
• The 2026-11-02 parity QA - 53 of its 55 findings fixed on the owner's decisions (one kept as a documented divergence, one deferred), and NumPy's own dot functions behind np.dot / np.vdot / np.vecdot.
• 86 NumPy-parity, memory-safety and tooling fixes, 40 measured speedups and 44 breaking changes, across the release's 1,670 commits.
• 3 new NuGet packages - NumSharp.Interop.System.Numerics.Tensors, NumSharp.Interop.OnnxRuntime and NumSharp.Interop.MLNet.
• 1,119,305 NumPy 2.4.2 oracle test cases (0.70.0: 119,517) and 20,506 test methods (0.70.0: 14,416).

📦 New NuGet Packages

• NumSharp.Interop.System.Numerics.Tensors - zero-copy interop between NDArray and the BCL's Tensor<T> / TensorSpan<T> (System.Numerics.Tensors), all 15 dtypes.
• NumSharp.Interop.OnnxRuntime - run ONNX Runtime models on NDArrays, fed from the arrays' own memory, with the outputs read back as NDArrays.
• NumSharp.Interop.MLNet - an NDArray as an ML.NET IDataView, and a transformer's output column back as an NDArray.

Full NumSharp 0.80.0 release notes