TinyTokenizer 0.10.0

.NET 8.0 This package targets .NET 8.0. The package is compatible with this framework or higher.
Suggested Alternatives

TinyAst

Additional Details

Package is being renamed TinyAst

dotnet add package TinyTokenizer --version 0.10.0
                    


                    

                
NuGet\Install-Package TinyTokenizer -Version 0.10.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="TinyTokenizer" Version="0.10.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="TinyTokenizer" Version="0.10.0" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="TinyTokenizer" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add TinyTokenizer --version 0.10.0
                    


                    

                
#r "nuget: TinyTokenizer, 0.10.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package TinyTokenizer@0.10.0
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=TinyTokenizer&version=0.10.0
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=TinyTokenizer&version=0.10.0
                    


                            Install as a Cake Tool

TinyTokenizer

A high-performance syntax tree library for .NET 8+ with fluent queries, pattern matching, and undo/redo editing — built on a zero-allocation tokenizer with SIMD optimization.

Features

  • TinyAst — Red-green syntax tree with fluent queries, editing, and undo/redo
  • Query API — CSS-like selectors with combinators, lookahead, and repetition
  • SyntaxEditor — Batch mutations with atomic commit and full undo/redo support
  • Syntax Nodes — Pattern-based AST matching (function calls, property access, etc.)
  • Schema System — Unified configuration for tokenization + syntax node definitions
  • TreeWalker — DOM-style filtered tree traversal
  • High Performance — Zero-allocation parsing with SIMD-optimized SearchValues<char>
  • Error Recovery — Gracefully handles malformed input and continues parsing

Installation

dotnet add package TinyTokenizer

Quick Start

TinyAst

using TinyTokenizer.Ast;

// Parse source into a syntax tree
var tree = SyntaxTree.Parse("function foo() { return 1; }");

// Query nodes with CSS-like selectors
var idents = tree.Select(Query.AnyIdent);  // [Ident("function"), Ident("foo"), Ident("return")]

// Fluent mutations with undo support
tree.CreateEditor()
    .Replace(Query.Ident("foo"), "bar")
    .Insert(Query.BraceBlock.First().InnerStart(), "console.log('enter');")
    .Commit();

// Undo/redo
tree.Undo();
tree.Redo();

Pattern Matching with Schema

var tree = SyntaxTree.Parse("obj.method(x)", Schema.Default);

var methodCalls = tree.Match<MethodCallSyntax>().ToList();  // [MethodCallSyntax { Object="obj", Method="method" }]

Low-Level Tokenization

For scenarios where you don't need a syntax tree:

using TinyTokenizer;

var tokens = "func(a, b)".TokenizeToTokens();

// tokens contains:
// - IdentToken("func")
// - BlockToken("(a, b)") with children:
//   - IdentToken("a"), SymbolToken(","), WhitespaceToken(" "), IdentToken("b")

📚 Documentation

Full documentation on the Wiki →

TinyAst

Topic Description
TinyAst Guide Syntax tree API
Schema Unified configuration
Query API CSS-like node selectors
SyntaxEditor Batch mutations, undo/redo
Syntax Nodes Pattern-based matching
TreeWalker DOM-style traversal
Trivia Whitespace preservation

Low-Level Tokenization

Topic Description
Token Types SimpleToken vs Token, all types
Configuration Operators, comments, symbols
Async Streaming Stream/PipeReader APIs
Error Handling ErrorToken and recovery

Reference

Topic Description
Getting Started Installation and basic usage
Architecture Two-level design, red-green trees
API Reference Types, enums, methods

Requirements

  • .NET 8.0 or later
  • CommunityToolkit.HighPerformance (automatically included)

License

MIT

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 was computed.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
0.10.0 118 1/7/2026 0.10.0 is deprecated.
0.9.0 105 1/5/2026
0.8.0 109 1/4/2026
0.7.0 107 1/3/2026
0.6.8 103 1/2/2026
0.6.7 104 1/2/2026
0.6.6 101 1/2/2026
0.6.5 108 1/1/2026
0.6.4 105 1/1/2026
0.6.3 103 1/1/2026
0.6.2 107 1/1/2026
0.6.1 101 12/31/2025
0.6.0 106 12/31/2025
0.5.1 103 12/31/2025
0.5.0 107 12/30/2025
0.4.1 101 12/29/2025
0.4.0 97 12/29/2025
0.3.0 107 12/27/2025
0.2.0 173 12/26/2025
0.1.0 187 12/25/2025