Lokad.Utf8Regex.PythonRe 0.4.0

Lokad.Utf8Regex.PythonRe

0.4.0, .NET 10. A managed Python re-style adapter over strict UTF-8 Unicode scalars, requiring exactly Lokad.Utf8Regex [0.4.0]. It can coexist with the PCRE2 adapter at the same version. No Python runtime or BCL regex matcher is a shipped implementation dependency.

Supported scope

The profile targets the documented CPython 3.13.2 str grammar with owned Unicode 15.1.0 identifier/class/case data. Patterns are parsed before scalar lowering. Captures are numbered in opening order; references/conditionals, fixed-width lookbehind, greedy/lazy repetition, scoped flags and Python empty-match retry use the owned engine. LastIndex/LastGroup identify the last capture to close on the successful path; an unnamed last capture has a null name. Unmatched captures keep the documented null/empty shaping for each API.

Search, Match and FullMatch expose value, detailed, string and UTF-8 forms. FindIterDetailed returns owned capture snapshots. FindAllToStrings and FindAllToUtf8 provide Python-shaped scalar/tuple outputs; FindAll is the structural full-match collector. Split and template/callback replacement include capture-aware output shaping and substitution counts. Primary ranges are byte coordinates; derived UTF-16 coordinates do not define matching width.

Bytes-pattern semantics, LOCALE and \N{...} Unicode-name lookup are excluded. There is no endpos parameter. Start offsets follow the documented Python pos contract without resetting prefix/suffix assertion context. This is a selected profile, not complete Python re equivalence. See the profile specification.

Controls and lifetimes

Instances accept positive nullable Utf8ResourcePolicy.MaxWorkSteps, MaxDepth, MaxWorkingBytes and MaxResultBytes. Null is unlimited; zero is invalid. Defaults remain unlimited. Work includes covered validation, matching and required shaping cumulatively. Depth admits live frames before acquisition; working/result controls admit modeled major capacities and growth overlap. Result/staging charges do not refund and may conservatively count storage again. Denials use Utf8ResourceLimitExceededException with the corresponding Execution* resource kind. This is not a total heap/RSS or caller-retention bound.

Replacement evaluator delegates must be non-null, even on misses; otherwise ArgumentNullException names evaluator. An already-cancelled explicit token still takes precedence. Return-value rules for valid callbacks are unchanged.

Strict full-subject validation, original synchronous cancellation and one absolute deadline apply through required shaping and persistent cursor advances, including waits between moves. Caller callbacks cannot be preempted; copying/encoding their returned output remains covered. Timeouts use RegexMatchTimeoutException; its Input is empty when working or result bytes are bounded to avoid decoding the subject for diagnostics after expiry.

Explicit Compile factories accept separate construction limits, timeout and token, and eagerly prepare FullMatch. Constructors keep fixed construction backstops and lazy FullMatch preparation. Cold FullMatch/replacement preparation, cache retention, incidental owners, standalone getters, callback allocations and host containers retain separate documented scope. The execution token must be passed on each operation. See resource contract and MIGRATION.md.

Named captures remain distinct when Unicode identifiers and names such as g1 are mixed. Public names and Python opening-order numbering remain unchanged. Fixed finite policies omit optional unmetered Core runtime/emission storage and native prefix buffers those instances cannot execute. Useful search/scalar preparation remains; lower construction charges can admit earlier-denied patterns.

Optional Core preparations support native iteration, canonical literal FindAll string counting, literal values/results and scalar cursor start hints. The current grammar-level iteration flag can also retain Core for patterns whose operations decline it. Tightening that flag is deferred after a bounded cost experiment; backend presence alone does not establish execution. A Core preparation may still be built transiently to discover eligibility, under the original construction owner. Discarding a root reduces retention, not all construction traffic.

Default and unsuccessful detailed results expose empty capture/name arrays and empty value text. Default match/group data also expose empty text. Successful snapshots keep their supplied arrays and retain their values across later calls.

Public API

using System;
using Lokad.Utf8Regex;
using Lokad.Utf8Regex.PythonRe;

var regex = new Utf8PythonRegex(
    pattern: @"(?P<word>foo)-(?P=word)",
    options: PythonReCompileOptions.None,
    matchTimeout: TimeSpan.FromSeconds(1),
    resourcePolicy: new Utf8ResourcePolicy
    {
        MaxWorkSteps = 1_000_000,
        MaxDepth = 1024,
        MaxWorkingBytes = 16 * 1024 * 1024,
        MaxResultBytes = 16 * 1024 * 1024
    });

bool isMatch = regex.IsMatch("foo-foo"u8);
var first = regex.Search("xx foo-foo yy"u8);
int count = regex.Count("foo-foo x foo-foo"u8);
byte[] replaced = regex.Replace("foo x foo"u8, "<\\g<word>>");
Utf8PythonMatchData[] matches = regex.FindAll("foo-foo xx foo-foo"u8);
var iter = regex.FindIterDetailed("foo-foo xx foo-foo"u8);
string?[] parts = regex.SplitToStrings("foo-foo xx foo-foo"u8);

String-returning helpers such as SearchToString(...), ReplaceToString(...), and FindAllToStrings(...) are also available as host-friendly conveniences when a runtime needs scalar string results.

CreateMatchCursor(ReadOnlyMemory<byte>) discovers one detailed snapshot per MoveNext instead of materializing the complete FindIterDetailed array:

ReadOnlyMemory<byte> subject = System.Text.Encoding.UTF8.GetBytes("é😀");
using var cursor = new Utf8PythonRegex("(?P<scalar>.)").CreateMatchCursor(subject);
while (cursor.MoveNext())
{
    Utf8PythonDetailedMatchData match = cursor.Current;
    // The supplementary scalar occupies four bytes and is one match.
    Console.WriteLine(match.Value.ValueText);
}

Keep the subject unchanged and its memory manager alive until exhaustion or disposal. Dispose when stopping early. The cursor has one shared position and lifetime across reference aliases, rejects concurrent/reentrant access, and has no finalizer. EOF and failure release scratch and subject references immediately. Returned snapshots own their arrays and text and remain valid after advance or disposal. Sequential advances may cross host calls or asynchronous waits; those waits count against the original absolute match deadline. The offset/token overloads preserve scalar-aligned byte offsets and the original invocation token. Existing span APIs retain their existing contracts.

A cursor has cumulative work/result allowance across all moves. Scratch remains charged while positioned and releases at EOF/failure/Dispose. Published snapshots never borrow reusable scratch. Faults preserve earlier snapshots; a new call gets a new allowance and needs new host admission while earlier results survive. Core value-cursor copies have independent positions/shared allowance; PythonRe reference aliases share position/disposal. No finalizer-based lease is required.

Read release notes. Repository benchmark results retain their recorded source/package/runtime identities and qualification status; older rows do not automatically qualify this release.