Bbieniek.Uax29 1.0.3

Bbieniek.Uax29

A managed .NET word tokenizer implementing Unicode UAX #29 word boundary segmentation (Unicode 15.0). Splits text into words, numbers, punctuation, and whitespace tokens — correctly handling multilingual text, emoji, contractions, and numeric formatting.

No native ICU required. Zero dependencies on netstandard2.0. SQL CLR compatible.

Why use this?

Splitting on spaces or regex gives inconsistent results with real-world text. Unicode UAX #29 is the standard used by ICU, Lucene, quanteda, and most NLP tooling. This package gives you the same tokenization in pure managed .NET:

• "don't" stays as one token (apostrophe is MidLetter)
• "1,000,000.50" stays as one token (comma/period are MidNum)
• "self-aware" stays as one token (hyphen preservation)
• Emoji ZWJ sequences like 👩‍❤️‍💋‍👨 stay together
• Works with Latin, Greek, Cyrillic, Hebrew, Katakana, CJK, and all Unicode scripts

Installation

dotnet add package Bbieniek.Uax29

Quick start

using Bbieniek.Uax29;

// Tokenize into strings
var tokens = WordBreakTokenizer.TokenizeToStrings("Hello, world! Price: $1,000.50");
// → ["Hello", ",", " ", "world", "!", " ", "Price", ":", " ", "$", "1,000.50"]

// Tokenize into spans with metadata
var spans = WordBreakTokenizer.Tokenize("self-aware robot");
foreach (var span in spans)
{
    var text = "self-aware robot".Substring(span.Start, span.Length);
    Console.WriteLine($"{text} (IsWord: {span.IsWord})");
}
// → self-aware (IsWord: True)
// →   (IsWord: False)
// → robot (IsWord: True)

Zero-allocation API (net8.0+)

On .NET 8 and later, use the zero-allocation enumerator for high-throughput scenarios:

foreach (var token in WordBreakTokenizer.EnumerateWords("hello, world".AsSpan()))
{
    // token.Span is ReadOnlySpan<char> — no heap allocation
    // token.IsWord, token.Start, token.Length also available
}

Quanteda/ICU-compatible mode

By default, colon is treated as MidLetter per the Unicode spec ("key:value" → one token). Use WordBreakOptions.Quanteda to match ICU/quanteda behavior where colon splits words:

var tokens = WordBreakTokenizer.TokenizeToStrings("key:value", WordBreakOptions.Quanteda);
// → ["key", ":", "value"]

API

TokenSpan

WordBreakOptions

UAX #29 rules implemented

Compatibility

Targets netstandard2.0 and net8.0:

• .NET 8, 9, 10+
• .NET 6, 7
• .NET Framework 4.6.1+
• SQL CLR (SAFE mode)

Performance

• Zero-allocation enumerator (net8.0+): ref struct via EnumerateWords() — no heap allocation per token
• Bitwise property matching: [Flags] enum with single-op combined checks
• ASCII fast path: pre-computed lookup table for characters 0-127
• Latin-1 fast path: direct classification for U+00C0-U+024F
• Inlined hot paths: AggressiveInlining on frequently called predicates

Unicode conformance

Validated against the official Unicode 15.0 WordBreakTest suite (1823 test cases), plus quanteda 4.3.1 and ICU4N verification.

Versioning

Package versions are generated from Git tags using MinVer. Release by pushing a tag like v1.2.3.

License

MIT