NumSharp.Interop.pythonnet 0.80.0

NumSharp.Interop.pythonnet

Zero-copy interop between NumSharp NDArray and the Python ecosystem via Python.NET (pythonnet) — with no Numpy.NET dependency. NumPy and PEP 3118 buffers enter the memory bridge directly; registered IPythonArrayAdapter instances feed library objects such as torch.Tensor and Pandas containers through that same bridge. PyTorch and Pandas adapters are built in; neither duplicates the memory implementation.

using NumSharp;
using NumSharp.Interop.PythonNet;
using Python.Runtime;

Runtime.PythonDLL = "python312.dll";     // or the PYTHONNET_PYDLL env var
PythonEngine.Initialize();

Runnable tour: examples/NumSharp.Interop.pythonnet.Examples — twelve single-file dotnet run tutorials covering every feature below, written for reading (a dynamic Python namespace, one NDScope per script, the outcome of each line in a comment); their asserted twins live in test/NumSharp.Tests.Interop/Examples.

Two projects to copy: examples/NumSharpPythonnetLocalExample is a console app that finds the Python on your machine, starts the engine inside a virtual environment and runs NumPy, pandas, SciPy, scikit-learn, Matplotlib and PyTorch over NumSharp arrays. examples/NumSharpPythonnetGhActionsCicd is its asserted twin: an MSTest project with the GitHub Actions workflow that runs it on Windows, Linux and macOS.

The four verbs

Everything is packaging over four operations on the static NDArrayPythonInterop engine:

The lease buffer is always acquired through the exporter's memoryview, never the raw object: the memoryview is CPython's canonical, uniformly-behaved buffer exporter, so this sidesteps pythonnet 3.0.x's per-exporter GetBuffer bugs — a raw ctypes array hard-crashes obj.GetBuffer on every flag, while the memoryview over the same memory leases cleanly. Measured coverage across 55 exporter varieties: 50 view, 4 copy (complex64 widens, sub-item strides linearize, and a StringDType array or an object array of str copies as a NumSharp StringDType array; fixed-width S / U text views). Big-endian multi-byte data is a further copy — the view path declines it (a native-endian shared view is impossible), but ToNDArray byte-reverses each element on copy — so the only refusals left are element types with no NumSharp dtype at all (datetime64, structured, void) and object arrays that hold anything but str.

Plus ToMemoryView(nd) (a writable Python memoryview of raw bytes for non-numpy consumers) and the dtype maps ToNumpyDtypeStr / FromNumpyDtypeStr / ToBufferFormat / FromBufferFormat.

PyTorch through the existing bridge

The built-in TorchPythonArrayAdapter makes the ordinary verbs work directly; there is no TorchSharp or compile-time PyTorch dependency:

using (Py.GIL())
{
    using PyObject tensor = nd.ToTorch();                  // shared CPU tensor
    using PyObject owned = nd.ToTorch(copy: true);         // independent tensor
    using NDArray view = tensor.AsNDArray();               // existing shared bridge
    using NDArray copy = tensor.ToNDArray();               // existing copy bridge (force permitted)

    // Torch-specific aliases/policy controls remain available:
    using NDArray sameView = tensor.AsTorchNDArray();
    using NDArray strictCopy = tensor.ToTorchNDArray();     // no detach/device transfer
    using NDArray forced = tensor.ToTorchNDArray(force: true); // detach + CPU transfer, then copy
}

The sharing path preserves positive strides and roots the owner on both sides. Decimal is the documented exception: neither NumPy nor PyTorch has that dtype, so it converts to float64. Negative-stride and non-writeable NumSharp arrays require copy:true; this avoids PyTorch's documented undefined behavior for tensors over read-only NumPy memory. A Tensor requiring gradients or living on CUDA/MPS cannot be shared directly with CPU NumSharp memory — use an explicit detach or force:true copy.

Rare tensor families follow PyTorch's NumPy compatibility boundary: complex64 copy-widens to NumSharp Complex; bfloat16, float8, sparse and quantized tensors cannot cross through NumPy; meta tensors have no data even for force:true. PyTorch expand() tensors import safely as non-writeable NumSharp broadcasts. Scalars, empty arrays, storage offsets, F-order/transposes, conjugate/negative view bits, derived-view lifetimes, resize refusal, all dtype boundary values, caller-owned GIL mode and concurrent churn are pinned by the live interop test matrix.

Pandas through the existing bridge

The built-in PandasPythonArrayAdapter recognizes DataFrame, Series, Index, ExtensionArray, and their subclasses. Pandas → NumSharp uses the ordinary verbs:

using NDArray view = series.AsNDArray(allowReadonly: true); // verified shared projection
using NDArray copy = frame.ToNDArray();                     // owning C-contiguous snapshot

NDArrayPythonInterop.RegisterCodec();
using NDArray automatic = frame.As<NDArray>();              // Auto: view, then copy fallback

Pandas documents that to_numpy(copy:false) does not guarantee a view. View mode therefore checks that two independent official projections overlap before the existing bridge leases one. Homogeneous NumPy-backed frames, numeric Series/Indexes, and some extension arrays share; mixed numeric blocks, nullable arrays with missing values, categorical, and sparse arrays copy through Auto/Copy. Pandas 3 usually exposes shared projections read-only, so direct view callers pass allowReadonly:true and receive a non-writeable NDArray. A text column of str (an object projection) copies as a NumSharp StringDType() array, and fixed-width S / U projections cross like any NumPy text array; a text projection holding a missing value (None, NaN, pd.NA) is refused until it is converted to a NumPy StringDType with a representable NA object. Mixed object, datetime and timedelta outputs remain unsupported because NumSharp has no corresponding dtype. Axis labels are metadata and do not enter the value array.

NumSharp → Pandas is the existing NumPy encoder; request Pandas copy:false when sharing is intended:

NDArrayPythonInterop.RegisterCodec();
scope.Set("values", nd);
scope.Exec("series = pd.Series(values, copy=False)");

Adapter-aware pythonnet codec

After NDArrayPythonInterop.RegisterCodec(), pythonnet uses the same adapter registry implicitly:

dynamic torch = Py.Import("torch");
using PyObject tensor = (PyObject)torch.from_numpy(nd); // registered encoder: NDArray → NumPy
using NDArray a = tensor.As<NDArray>();                 // registered decoder: Torch → shared bridge
using NDArray b = (NDArray)tensor.AsManagedObject(typeof(NDArray));

NumpyCodecOptions.DecodeArrayAdapters is on by default and independent of DecodeAnyBuffer and DecodeArrayLike. View permits only share-preserving adaptation; Copy permits library-specific materialization (including Pandas coercion or Torch detach/device transfer); Auto tries View and falls back to Copy.

nd.ToPython() itself produces the bridge's NumPy view because pythonnet's encoder sees the managed source type, not the eventual Python library target. That is also why torch.from_numpy(nd) works: the callable receives the implicitly encoded NumPy view. On return, As<NDArray>() and AsManagedObject(typeof(NDArray)) do carry an explicit CLR target and select the adapter-aware decoder.

Other libraries can register the same kind of front door without writing memory code:

PythonArrayAdapterRegistry.Register(myAdapter); // implements IPythonArrayAdapter

The adapter returns a canonical buffer/NumPy object. NDArrayPythonInterop still owns dtype, shape, strides, writeability, leases, copy fallback, GIL and shutdown behavior. Adapters may decline an instance by returning null; the registry then tries the next adapter. Disabling codec adapters does not suppress a recognized object's independent NumPy/buffer capability.

Extension methods (the ergonomic default)

The verbs above double as extension methods — NDArrayPythonInterop provides them, so importing its namespace lets you call them fluently on NDArray / PyObject as well as statically:

using NumSharp.Interop.PythonNet;

PyObject a = nd.ToNumpy();          // zero-copy view (also: nd.ToPython())
PyObject c = nd.ToNumpyCopy();      // independent copy
NDArray  b = py.ToNDArray();        // copy   (To… = copy)
NDArray  v = py.AsNDArray();        // view   (As… = share, like numpy array/asarray)

Codec (auto-marshaling)

NDArrayPythonInterop.RegisterCodec();     // once per engine session; idempotent
Py.RegisterNumSharpCodec();               // identical alias, reads beside Py.Import / Py.GIL

using (Py.GIL())
{
    scope.Set("x", nd);                          // NDArray auto-encoded to a numpy view
    dynamic np = Py.Import("numpy");
    NDArray r = ((PyObject)np.matmul(a, b)).As<NDArray>();   // decoded back, no explicit calls
}

Py.RegisterNumSharpCodec() is a static extension member (C# 14) on pythonnet's Py, so it reads beside Py.Import(...) / Py.GIL() at the call site. It forwards verbatim to NDArrayPythonInterop.RegisterCodec() (both the no-arg and the NumpyCodecOptions overload) — an alias, not a second registration.

This one call enables ALL of NumSharp's support; NumpyCodecOptions is the single control surface and every support is a flag defaulting ON — NDArray⇄numpy (EncodeMode/DecodeMode), PEP 3118 buffers (DecodeAnyBuffer), the built-in torch/pandas (and any registered) adapters (DecodeArrayAdapters), list/tuple/scalar array-likes (DecodeArrayLike), and C# tuples ⇄ Python tuples (ConvertTuples). So torch and pandas support is part of this call — there is no separate "register the adapters" step. Adding your OWN third-party adapter is the one thing outside it, a rare extension point on NDArrayPythonInterop.RegisterArrayAdapter(IPythonArrayAdapter) (it takes an adapter instance; it is not a support toggle).

Your project must compile with C# 14 to use the Py. spelling. Static extension members are resolved at the call site by the C# 14+ compiler (the .NET 10 SDK); a consumer on C# 13 or earlier (the .NET 8/9 SDK default) gets CS9202 and should call NDArrayPythonInterop.RegisterCodec() instead — it is identical and compiles on every C# version.

NumpyCodecOptions controls the policies via NumpyCodecMode — one enum for both directions (EncodeMode / DecodeMode):

Plus DecodeArrayAdapters (default true, including PyTorch and Pandas), DecodeAnyBuffer (default true) and DecodeArrayLike (default true for numeric builtin containers). NumPy ndarray subclasses decode via an __mro__ walk.

Tuples. pythonnet has no tuple conversion of its own — through dynamic, numpy.zeros((2, 3)) used to hand numpy an opaque wrapped System.ValueTuple. RegisterCodec() therefore also registers TupleCodec (NumpyCodecOptions.ConvertTuples, default true): a C# ValueTuple/Tuple of any arity — nested, mixed, an NDArray element becoming a numpy view, null becoming None — crosses as a Python tuple, and a Python tuple (or a subclass such as a namedtuple or torch.Size) decodes into a C# tuple of the same arity; a length mismatch or a list is declined. So shapes, axes and multi-indices read the numpy way:

dynamic z = numpy.zeros((2, 3));            // a C# tuple is the shape
(long, long) strides = z.strides;           // (24, 8)
double corner = z.item((1, 2));             // a tuple multi-index

Implicit encodes do not linger. pythonnet takes its own reference from the wrapper an encoder returns and never disposes that wrapper, which would leave every scope.Set("x", nd) / py.x = nd / numpy.sum(nd) export pinned until the CLR finalizer ran (a 150-iteration loop meant 150 live exports). The codec therefore hands its wrapper to a per-thread slot that is disposed at the next encode on that thread and at the next conversion verb, so a rebinding loop keeps exactly one export live. A dynamic read of a numpy array (py.x.sum()) is an ordinary pythonnet wrapper and still lives until the GC; where a reference count matters, read the object as using PyObject or read a scalar through scope.Eval(...).

Auto decode shares memory. Under the default, pyObj.As<NDArray>() on a contiguous/strided-numpy source is a zero-copy view: mutations flow both ways, read-only sources decode as non-writeable views, and the view holds a Py_buffer lock on the source (a bytearray cannot be resized while it lives). Use DecodeMode = Copy when you want a detached snapshot that never touches the Python object.

Lifetime & memory safety (the design)

Exports take their own atomic reference on the NumSharp buffer (NumSharp's ARC), so the memory survives even if every C# reference — including the returned PyObject wrapper — is disposed or collected. Release is owned by Python: a weakref.finalize on the exported array's base buffer object (which every derived numpy view chains to) fires when the last Python-side view dies. If the engine shuts down while Python still holds views, the pin is swept right after PythonEngine.Shutdown() completes — pythonnet runs no Python atexit pass, so the finalize callbacks cannot fire during shutdown (probed; the sweep waits until the interpreter is provably gone, so it can never race Python reads). While exported, ndarray.resize(refcheck: true) on the source refuses to reallocate — the same guard NumPy applies to referenced arrays.

Imports lease the Python buffer through NumSharp's memory-block reference counting: the lease is released when the last NumSharp view over the memory — including derived slices like nd["2:"] — is disposed or collected. C-contiguous leases hold a real Py_buffer, so resizable exporters (bytearray) are locked against reallocation, and numpy's resize refcheck refuses while the lease lives. The Python-side release is marshaled to the GIL safely (never raw on a finalizer thread); NDArrayPythonInterop.LiveExports / LiveImports expose the counters.

Read-only sources (bytes, non-writeable numpy arrays) are refused for views by default — writing through them would corrupt immutable Python objects. Pass allowReadonly: true to opt in: the view comes back non-writeable (Shape.IsWriteable == false, exactly numpy's writeable=False), so guarded write paths raise assignment destination is read-only instead of corrupting the source. Or use ToNDArray to copy. An empty read-only source (np.frombuffer(b''), an empty broadcast) is refused or non-writeable the same way, and an empty NumSharp broadcast exports as a read-only numpy array.

Import views also do not own their data (like np.frombuffer(...), whose flags.owndata is False): a size-changing ndarray.resize refuses with NumPy's cannot resize this array: it does not own its data instead of silently reallocating away from the shared Python memory, and np.require(..., "O") produces an owning copy.

Rules & limits

• GIL: every method acquires the GIL itself (re-entrant). Your own PyObject usage still follows pythonnet's rules: dispose a PyObject (a ToNumpy result included) inside Py.GIL(). pythonnet's Dispose has no guard of its own, so two threads disposing one without the GIL can both release it (1 crash in 397 races of a probe; 0 under the GIL).

• GIL opt-out: every conversion verb takes a nullable requireGIL parameter; null (the default) follows the process-wide NDArrayPythonInterop.RequireGIL (default true). An effective false replaces Py.GIL() with a shared no-op guard — the calling thread must already hold the GIL (an enclosing Py.GIL() block); converting GIL-less without holding it is an immediate access violation. Trap: a .NET method/delegate body invoked from Python does not hold the GIL — pythonnet's binder releases it around managed bodies (probed on 3.0.5/3.1.0: PyGILState_Check() == 0 inside the body) — so keep GIL management on inside Python→.NET callbacks. The interop's background machinery (deferred lease disposal, shutdown drain) always manages the GIL itself, regardless of the policy.

  using (Py.GIL())                                     // ONE acquisition...
      for (int i = 0; i < n; i++)
          using (PyObject p = batch[i].ToNumpy(requireGIL: false))   // ...N conversions inside
              consumer.Invoke(p);
  
  NDArrayPythonInterop.RequireGIL = false;                   // or process-wide, when EVERY call site holds the GIL
  

• Engine lifetime: import views die with the interpreter. PythonEngine.Shutdown() releases all outstanding leases crash-free (a registered shutdown handler), but the NDArrays over that memory must not be touched afterwards (disposing them stays safe). Exports still held by Python are swept right after shutdown completes — no leak survives the engine.

• PythonEngine.Shutdown on .NET 8+: pythonnet crashes in its own BinaryFormatter state-stashing. Opt out first: RuntimeData.FormatterType = typeof(NoopFormatter); (NoopFormatter ships from pythonnet 3.0.4 onward — guaranteed by the 3.0.5 floor below).

• dtypes: all NumSharp numeric dtypes map except Decimal (text: next bullet) — it has no numpy dtype, so the array-producing verbs (ToNumpy / ToNumpyCopy, and the codec) auto-convert it to float64 (the astype(Double) guidance, automated — lossy beyond ~16 digits), while the low-level maps (ToNumpyDtypeStr / ToBufferFormat / ToMemoryView) stay honest and still refuse it. Char is exported as uint16 (UTF-16 code units) and imported from 2-byte wchar buffers (PEP 3118 'u': array.array('u') on Windows, ctypes.c_wchar) as a zero-copy view. Big-endian multi-byte buffers cannot be viewed (a native-endian shared view is impossible), but ToNDArray byteswaps them on copy (>c16 reverses each 8-byte half; >c8 reverses then widens; single-byte >i1/|u1/|b1 view regardless). complex64 imports copy-widen to complex128 — it has no zero-copy view.

• text: fixed-width S<n> / U<n> arrays cross as views in both directions, the same bytes on both sides (numpy <U1 is NumSharp's U1, the same 4-byte code points, astral characters included; a 4-byte wchar buffer, 'w', views as U1); big-endian >U<n> copies byte-reversed; np.str_ / np.bytes_ scalars are 0-d U<n> / S<n>. A NEP 55 StringDType array has no buffer — numpy keeps each element's text in an allocator its descriptor owns — so ToNDArray and ToNumpyCopy copy it exactly through numpy's own NpyString C API (every NA value, NULs, non-BMP text, any layout; the copy keeps na_object and coerce), and the view verbs refuse it. The missing-value object crosses when it has a C# form (None, bool, int, float, complex, str, bytes, np.float16, and numpy's np.float64 / np.complex128 / np.str_ / np.bytes_ as the Python type they subclass); any other (pd.NA, np.int64) is refused. An object array whose every element is a str (a pandas text column) copies to StringDType() (a str subclass element as str(x), what numpy's own StringDType() cast stores); any other object array is refused. PyTorch has no string tensors, so ToTorch refuses text.

• array-like decode (on by default): py.FromArrayLike() materializes a Python list / tuple / nested sequence / scalar (int/float/bool/complex) into an owning NDArray via numpy.asarray, and the registered codec does it automatically (NumpyCodecOptions.DecodeArrayLike, default true) so a numpy call or plain Python that returns a list crosses without a manual conversion. It fires ONLY for genuine non-buffer builtins — a numpy array the view/copy paths declined is never rescued into a silent copy. Set DecodeArrayLike = false to restrict decode to buffer exporters (fully numpy-agnostic).

• pythonnet versions: the dependency is [3.0.5, 4.0.0). PyBuffer is broken for shape/strides/format flags, so metadata is read through Python's memoryview and only the crash-free PyBUF.SIMPLE/PyBUF.WRITABLE are used — the same code path works on every v3. The source compiles clean against all of 3.0.0–3.1.0; 3.0.5 is the floor because it is the oldest release that covers the Python versions people actually run (see below).

Choosing a pythonnet version

Every pythonnet release hard-caps the Python it can drive. MinSupportedVersion has been 3.7 for the entire v3 line, so the ceiling is the only thing that moves — which makes 3.0.5 a strict superset of every earlier release. Read from each package's own PythonEngine.MaxSupportedVersion:

NuGet installs the floor, not the latest. Resolution is lowest-applicable, so a plain dotnet add package NumSharp.Interop.pythonnet gives you pythonnet 3.0.5 even though newer releases exist. On Python 3.14, ask for it explicitly:

<PackageReference Include="pythonnet" Version="3.1.0" />

Overshooting a release's ceiling usually kills PythonEngine.Initialize() with a bare MissingMethodException: Failed to load symbol PyUnicode_AsUnicode (or similar). If the engine initializes anyway, the interop checks the range itself on first use and raises an actionable error naming the version to install.

What's new in 0.80.0

We are at 95.2% NumPy API coverage, and everything except datetime, complex64 and other exotic dtypes matches NumPy byte for byte. This release brings NumPy's text dtypes with numpy.strings and numpy.char, numpy.ma, numpy.polynomial, np.emath and 48 new np.* functions. Integration rather than competition goes on, this time inside .NET: an NDArray now runs through ONNX Runtime models, serves as an ML.NET IDataView and shares its memory with System.Numerics.Tensors.

🧭 Overview

• NumPy 2.x API coverage around 95.2%, everything except datetime, complex64 and other exotic dtypes is fully supported, with byte-to-byte equality and compatibility to NumPy.
  • Text - NumPy's fixed-width bytes_ (S) and str_ (U) dtypes and NEP 55's variable-width StringDType, with every computation NumPy runs on text, all 46 numpy.strings functions and numpy.char with chararray.
  • np.polynomial.* - the whole numpy.polynomial package, all 193 functions and the six series classes for all six bases, bit-exact.
  • np.* - +48 functions, the ten most used on GitHub: np.hypot, np.histogram, np.logspace, np.gradient, np.vectorize, np.hanning, np.geomspace, np.apply_along_axis, np.trapezoid and np.histogram2d.
  • np.random.* - every NumPy bit generator, the whole Generator distribution surface, and a legacy RandomState that is NumPy's own sampler code line for line, all byte-identical streams.
  • np.ma.* - the masked-array module; np.emath.* - the automatic-domain math module; np.matrix - numpy.matrix as the new NDMatrix type.
• DType is NumPy's two-level dtype model - NDArray.dtype returns a numpy.dtype-shaped descriptor, with np.dtypes, byte order and datetime64 / timedelta64 descriptors, and NumPy's scalar classes (np.integer, np.floating, np.number, ...) with np.typing.
• Reductions on NumPy's own schedules - max, min, ptp, nanmax, nanmin, argmax, the flat sum / mean and the fused np.evaluate reductions match NumPy bit for bit, down to the sign of a ±0 tie and a NaN's payload; mean, var, std and their NaN-aware forms are NumPy's own statements; and most of them run faster.
• NumPy's structure for the engine - every computing np.* function is a TensorEngine member named after its NumPy source file, numpy.linalg and numpy.fft are replaceable sub-engines, and the default engine is settable.
• The 2026-11-02 parity QA - 53 of its 55 findings fixed on the owner's decisions (one kept as a documented divergence, one deferred), and NumPy's own dot functions behind np.dot / np.vdot / np.vecdot.
• 86 NumPy-parity, memory-safety and tooling fixes, 40 measured speedups and 44 breaking changes, across the release's 1,670 commits.
• 3 new NuGet packages - NumSharp.Interop.System.Numerics.Tensors, NumSharp.Interop.OnnxRuntime and NumSharp.Interop.MLNet.
• 1,119,305 NumPy 2.4.2 oracle test cases (0.70.0: 119,517) and 20,506 test methods (0.70.0: 14,416).

📦 New NuGet Packages

• NumSharp.Interop.System.Numerics.Tensors - zero-copy interop between NDArray and the BCL's Tensor<T> / TensorSpan<T> (System.Numerics.Tensors), all 15 dtypes.
• NumSharp.Interop.OnnxRuntime - run ONNX Runtime models on NDArrays, fed from the arrays' own memory, with the outputs read back as NDArrays.
• NumSharp.Interop.MLNet - an NDArray as an ML.NET IDataView, and a transformer's output column back as an NDArray.

Full NumSharp 0.80.0 release notes