Reluxer 0.1.1

reluxer

A token-based pattern matching and visitor framework for C#, inspired by Babel's AST traversal. Designed for transforming TSX/JSX to C# or other source-to-source transformations.

Features

• Token Pattern DSL - Regex-like syntax for matching token sequences
• Attribute-Based Visitors - Decorate methods with [TokenPattern] to handle matches
• Automatic Parameter Injection - Captured groups are injected as method parameters
• Nested Traversal - Use Traverse() with nameof() for Babel-style nested visiting
• Scoped Visitors - Restrict which visitors can call other visitors with From
• Depth-Aware Matching - Handle balanced structures like JSX with </\1@0>
• TSX/JSX Lexer - Built-in lexer for TypeScript/JavaScript with JSX support
• TypeScript Type Support - Full lexing of type annotations, generics, tuples, conditional types
• Lookahead/Lookbehind - Zero-width assertions: (?=...), (?!...), (?<=...), (?<!...)
• Named Captures - Named groups (?<name>...) and backreferences \k<name>
• Balanced Matching - Match balanced brackets with \Bp, \Bb, \Bk, \Ba
• Token Manipulation - Insert, replace, remove tokens with GetModifiedSource() preserving formatting

Installation

dotnet add reference path/to/TokenLexer.csproj

Quick Start

using TokenLexer.Attributes;
using TokenLexer.Lexer;
using TokenLexer.Matching;
using TokenLexer.Tokens;
using TokenLexer.Visitor;

// 1. Lex your source code
var lexer = new TsxLexer(sourceCode);
var tokens = lexer.Tokenize();

// 2. Create a visitor and process tokens
var visitor = new MyVisitor();
visitor.Visit(tokens);

Pattern Syntax

Token Type Shorthands

TypeScript Type Shorthands

Literal Values

Match specific token values using quotes:

// Match keyword with specific value
[TokenPattern(@"\k""const""")]

// Match any token with specific value
[TokenPattern(@"""function""")]

// Match punctuation with specific value
[TokenPattern(@"""{""")]

Quantifiers

Add ? for non-greedy: *?, +?

Groups and Captures

// Capture group - value injected into method parameter
[TokenPattern(@"\k""function"" (\i)")]
public void Visit(string functionName) { }

// Multiple captures - injected in order
[TokenPattern(@"\k""const"" (\i) ""="" (\n)")]
public void Visit(string varName, string value) { }

// Non-capturing group
[TokenPattern(@"(?:\k""async"")? \k""function""")]

// Named capture group
[TokenPattern(@"\k""function"" (?<funcName>\i)")]
public void Visit(TokenMatch match)
{
    var name = match.NamedCaptures["funcName"].Value;
}

// Named backreference - matches same value
[TokenPattern(@"\k""const"" (?<name>\i) ""="" \k<name>")]

Alternation

// Match keyword OR identifier
[TokenPattern(@"[\k|\i]")]

// Match const OR let OR var
[TokenPattern(@"[\k""const""|\k""let""|\k""var""]")]

Lookahead and Lookbehind Assertions

Zero-width assertions that match without consuming tokens:

// Positive lookahead: match identifier only if followed by (
[TokenPattern(@"(\i)(?=""("")")]  // Matches function name in foo(x)

// Negative lookahead: match identifier NOT followed by (
[TokenPattern(@"(\i)(?!""("")")]  // Matches variable, not function call

// Positive lookbehind: match if preceded by pattern
[TokenPattern(@"\k""const"" (?<=\k""const"")(\i)")]

// Negative lookbehind: match if NOT preceded by pattern
[TokenPattern(@"(\i)(?<!""."")")]  // Identifier not after dot

Balanced Bracket Matching

Match balanced pairs including nested content:

// \Bp - balanced parentheses ( ... )
[TokenPattern(@"\i \Bp")]  // Matches foo(a, (b + c))

// \Bb - balanced braces { ... }
[TokenPattern(@"\k""if"" \Bp \Bb")]  // Matches if (cond) { body }

// \Bk - balanced brackets [ ... ]
[TokenPattern(@"\i \Bk")]  // Matches arr[i + j]

// \Ba - balanced angle brackets < ... >
[TokenPattern(@"\i \Ba")]  // Matches generic<T, U>

Backreferences (Depth-Aware)

// Match closing tag with same name as captured opening tag
[TokenPattern(@"<(\i)> .* </\1>")]

// Match only when depth returns to 0 (balanced)
[TokenPattern(@"<(\i)> .* </\1@0>")]

Visitor Methods

Basic Visitor

public class MyVisitor : TokenVisitor
{
    [TokenPattern(@"\k""function"" (\i) ""(""")]
    public void VisitFunction(TokenMatch match, string name)
    {
        Console.WriteLine($"Found function: {name}");
    }
}

Parameter Types

The framework automatically injects captured values based on parameter type:

[TokenPattern(@"\k""const"" (\i) ""="" (.*?) "";""")]
public void VisitConst(
    TokenMatch match,      // Full match info
    string varName,        // First capture as string
    Token[] initializer)   // Second capture as token array
{
}

Attribute Properties

[TokenPattern(
    @"\k""function"" (\i)",
    Name = "FuncDecl",     // Explicit name for Traverse lookup
    Priority = 10,          // Higher priority matches first
    Consumes = true,        // Whether to consume matched tokens
    From = new[] { nameof(VisitClass) }  // Restrict callers
)]

Nested Traversal

Use Traverse() with nameof() to process nested structures, similar to Babel's path.traverse().

public class MyVisitor : TokenVisitor
{
    // Top-level visitor
    [TokenPattern(@"\k""function"" (\i) ""(""")]
    public void VisitFunction(string name)
    {
        Console.WriteLine($"Function: {name}");

        // Extract the function body
        var body = ExtractFunctionBody();

        // Traverse body with specific visitors
        Traverse(body,
            nameof(VisitNestedFunction),
            nameof(VisitStatement));
    }

    // Only called via Traverse, not during Visit()
    [TokenPattern(@"\k""function"" (\i)", Name = "Nested")]
    public void VisitNestedFunction(string name)
    {
        Console.WriteLine($"  Nested function: {name}");
    }

    [TokenPattern(@"\k""const"" (\i)", Name = "Stmt")]
    public void VisitStatement(string varName)
    {
        Console.WriteLine($"  Variable: {varName}");
    }
}

Scoped Visitors with From

Restrict which visitors can call other visitors:

public class ScopedVisitor : TokenVisitor
{
    [TokenPattern(@"\k""class"" (\i)")]
    public void VisitClass(string name)
    {
        var body = ExtractFunctionBody();
        Traverse(body, nameof(VisitMethod), nameof(VisitField));
    }

    [TokenPattern(@"\k""function"" (\i)")]
    public void VisitFunction(string name)
    {
        var body = ExtractFunctionBody();
        // VisitField will NOT be called here - restricted to VisitClass
        Traverse(body, nameof(VisitField), nameof(VisitStatement));
    }

    // Can only be called from VisitClass
    [TokenPattern(@"(\i) ""=""", From = new[] { nameof(VisitClass) })]
    public void VisitField(string name)
    {
        Console.WriteLine($"Field: {name}");
    }

    // Can only be called from VisitClass or VisitFunction
    [TokenPattern(@"(\i) ""(""", From = new[] { nameof(VisitClass), nameof(VisitFunction) })]
    public void VisitMethod(string name) { }

    // No restriction - can be called from anywhere
    [TokenPattern(@"\k""const"" (\i)", Name = "Stmt")]
    public void VisitStatement(string name) { }
}

Extracting Balanced Blocks

The TokenVisitor base class provides helpers for extracting balanced content:

public class MyVisitor : TokenVisitor
{
    [TokenPattern(@"\k""function"" (\i) ""(""")]
    public void VisitFunction(string name)
    {
        // Extract content between { }
        Token[] body = ExtractFunctionBody();

        // Extract content between ( )
        Token[] params = ExtractParenthesized();

        // Extract content between [ ]
        Token[] items = ExtractBracketed();

        // Custom delimiters
        Token[] custom = ExtractBalanced(0, "<", ">");
    }
}

Token Manipulation

Transform source code by inserting, replacing, or removing tokens while preserving original formatting:

public class AddAttributeVisitor : TokenVisitor
{
    private int _counter = 0;

    [TokenPattern(@".")]
    public void VisitToken(TokenMatch match)
    {
        var token = match.MatchedTokens[0];

        // Only process JSX opening tags
        if (token.Type != TokenType.JsxTagOpen) return;

        // Insert hexPath attribute after the tag name
        InsertAfter(token,
            Token.Whitespace(" "),
            Token.JsxAttrName("hexPath"),
            Token.Operator("="),
            Token.JsxAttrValue($"\"{_counter++:X4}\"")
        );
    }
}

// Usage
var lexer = new TsxLexer(source);
var tokens = lexer.Tokenize();

var visitor = new AddAttributeVisitor();
visitor.Visit(tokens, source);  // Pass original source for formatting preservation

var modified = visitor.GetModifiedSource();  // Reconstructed source with edits

Manipulation Methods

Token Factory Methods

Create synthetic tokens for insertion:

Token.Keyword("const")
Token.Identifier("myVar")
Token.String("\"hello\"")
Token.Number("42")
Token.Operator("=")
Token.Punctuation(";")
Token.JsxAttrName("className")
Token.JsxAttrValue("\"container\"")
Token.Whitespace(" ")
Token.Create(TokenType.Comment, "// comment")

Skipping / Fast-Forward

When you manually explore tokens (e.g., extracting a function body), you can skip ahead so subsequent visitors don't re-process those tokens:

public class MyVisitor : TokenVisitor
{
    [TokenPattern(@"\k""function"" (\i) ""(""")]
    public void VisitFunction(string name)
    {
        // Extract and process the function body
        var body = ExtractFunctionBody();
        ProcessBody(body);

        // Skip past the body so next iteration starts after the closing }
        SkipTo(body[^1]);  // Skip to last token in body
    }

    [TokenPattern(@"\k""if"" ""(""")]
    public void VisitIf(TokenMatch match)
    {
        // Skip the entire if block including balanced braces
        SkipBalanced("{", "}");
    }
}

Skip Methods

Lifecycle Hooks

Override these methods to hook into the visitor lifecycle:

public class MyVisitor : TokenVisitor
{
    public override void OnBegin(IReadOnlyList<Token> tokens)
    {
        // Called before visiting starts
    }

    public override void OnEnd()
    {
        // Called after visiting completes
    }

    public override void OnUnmatched(Token token)
    {
        // Called for each token not matched by any pattern
    }
}

TSX/JSX Lexer

The built-in TsxLexer tokenizes TypeScript/JavaScript with JSX support:

var lexer = new TsxLexer(source);

// Basic tokenization (excludes whitespace and comments)
var tokens = lexer.Tokenize();

// Include whitespace and/or comments
var allTokens = lexer.Tokenize(
    includeWhitespace: true,
    includeComments: true);

Token Types

public enum TokenType
{
    // JavaScript/TypeScript
    Keyword,        // function, const, if, etc.
    Identifier,     // variable names
    String,         // "..." or '...'
    Number,         // 42, 3.14
    Operator,       // +, -, =, =>, etc.
    Punctuation,    // (, ), {, }, etc.
    TemplateString, // `...${...}...`
    Comment,        // // or /* */
    Regex,          // /pattern/flags
    Decorator,      // @Component, @Injectable

    // TypeScript Type Annotations
    Colon,          // : (in type context)
    GenericOpen,    // < (generic type)
    GenericClose,   // > (generic type)
    TypeName,       // string, number, MyClass
    QuestionMark,   // ? (optional)
    Arrow,          // => (in type context)
    TypeOperator,   // typeof, keyof, infer, readonly
    Extends,        // extends (in types)
    TupleOpen,      // [ (tuple type)
    TupleClose,     // ] (tuple type)
    MappedIn,       // in (mapped types)
    AsConst,        // as const

    // JSX-specific
    JsxTagOpen,     // <div
    JsxTagClose,    // </div>
    JsxTagSelfClose,// />
    JsxTagEnd,      // >
    JsxAttrName,    // className
    JsxAttrValue,   // "value"
    JsxText,        // text content
    JsxExprStart,   // {
    JsxExprEnd,     // }

    // Special
    Whitespace,
    Eof,
    Unknown
}

Complete Example

Transform TSX useState hooks to C# properties:

public class TsxToCSharpVisitor : TokenVisitor
{
    private readonly StringBuilder _output = new();

    public override void OnBegin(IReadOnlyList<Token> tokens)
    {
        _output.AppendLine("public partial class Component {");
    }

    public override void OnEnd()
    {
        _output.AppendLine("}");
    }

    // Match: const [value, setValue] = useState(initial)
    [TokenPattern(@"\k""const"" ""\["" (\i) "","" (\i) ""\]"" ""="" \i""useState"" ""("" (\n) "")""")]
    public void VisitUseState(string valueName, string setterName, string initial)
    {
        _output.AppendLine($"    private int _{valueName} = {initial};");
        _output.AppendLine($"    public int {Capitalize(valueName)}");
        _output.AppendLine( "    {");
        _output.AppendLine($"        get => _{valueName};");
        _output.AppendLine($"        set {{ _{valueName} = value; OnPropertyChanged(); }}");
        _output.AppendLine( "    }");
    }

    // Match: function ComponentName()
    [TokenPattern(@"\k""function"" (\i) ""(""")]
    public void VisitComponent(string name)
    {
        var body = ExtractFunctionBody();
        Traverse(body, nameof(VisitUseState));
    }

    private static string Capitalize(string s) =>
        char.ToUpper(s[0]) + s[1..];

    public string GetOutput() => _output.ToString();
}

// Usage
var lexer = new TsxLexer(tsxSource);
var visitor = new TsxToCSharpVisitor();
visitor.Visit(lexer.Tokenize());
Console.WriteLine(visitor.GetOutput());

Debugging

Access the visitor call stack for debugging:

public class MyVisitor : TokenVisitor
{
    [TokenPattern(@"(\i)")]
    public void VisitIdentifier(string name)
    {
        // See who called us via Traverse
        Console.WriteLine($"Caller: {CurrentCaller}");
        Console.WriteLine($"Stack: {string.Join(" -> ", CallStack)}");
    }
}

Project Structure

src/TokenLexer/
├── Tokens/
│   ├── Token.cs           # Token class
│   └── TokenType.cs       # Token type enum
├── Lexer/
│   └── TsxLexer.cs        # TSX/JSX tokenizer
├── Pattern/
│   ├── PatternNode.cs     # Pattern AST nodes
│   └── PatternParser.cs   # Pattern DSL parser
├── Matching/
│   ├── TokenMatch.cs      # Match result
│   └── PatternMatcher.cs  # Pattern execution
├── Attributes/
│   └── TokenPatternAttribute.cs
└── Visitor/
    └── TokenVisitor.cs    # Base visitor class

API Reference

TokenVisitor

TokenPatternAttribute

TokenMatch

See Also

• ENHANCEMENTS.md - Planned improvements and roadmap