ToonLib 1.4.2

🔄 ToonSharp (C#)

A production-grade C# library and CLI that converts data between JSON, YAML, TOML, and TOON (Token-Oriented Object Notation) while fully conforming to TOON SPEC v2.0. Perfect for .NET developers and data engineers who need efficient, token-optimized data serialization.

✅ Full TOON SPEC v2.0 Compliance - This library implements all examples from the official TOON specification repository, ensuring complete compatibility with the standard.

✨ Features

The ToonSharp library provides comprehensive JSON ↔ TOON ↔ YAML ↔ TOML conversion capabilities:

🔧 1. Lossless Conversion

• Bidirectional conversion between JSON, YAML, TOML, and TOON formats
• Round-trip preservation - data integrity guaranteed
• Supports all JSON/YAML/TOML data types (objects, arrays, scalars)
• Handles nested structures of any depth
• YAML support - Convert YAML ↔ TOON seamlessly
• TOML support - Convert TOML ↔ TOON for configuration files (Rust Cargo.toml, Python pyproject.toml)

📊 2. Advanced Parser & Lexer

• Recursive descent parser with indentation tracking
• Comment support - inline (#, //) and block (/* */) comments
• ABNF-backed grammar - fully compliant with TOON SPEC v2.0
• Error reporting with line and column numbers

🚀 3. Automatic Tabular Detection

• Smart detection of uniform-object arrays
• Automatic emission of efficient tabular mode (key[N]{fields}:)
• Token savings estimation
• Configurable modes: auto, compact, readable

⚡ 4. Performance Optimizations

• Direct JSON string support via JsonElement serialization (v1.4.0) 🚀
• POCO object serialization via reflection (v1.4.0, improved in v1.4.1) 🚀
• 60-85% faster YAML serialization (v1.2.0)
• 40-46% faster for large table operations (v1.1.0)
• 16-20% faster for large array deserialization
• Parallel processing for large tables (50+ rows) and arrays (200+ items)
• Span<T> optimizations for zero-allocation string operations
• Static instance reuse for YAML serializers/deserializers
• Memory-efficient parsing and serialization
• Automatic threshold tuning based on data size

🛠️ 5. CLI & Utilities

• Command-line interface (toonsharp) for file conversion
• Validation API for syntax checking
• Streaming helpers for large files
• Formatting tools for code style consistency

📦 Installation

Via NuGet Package Manager (Recommended)

dotnet add package ToonLib

Or using Package Manager Console in Visual Studio:

Install-Package ToonLib

Or using NuGet CLI:

nuget install ToonLib

Package: ToonLib on NuGet.org

From Source

# Clone the repository
git clone https://github.com/shinjiDev/toonsharp.git
cd toonsharp

# Build the solution
dotnet build

# Run tests
dotnet test

Requirements: .NET 9.0 or later

🚀 Quick Start

using ToonSharp;

// Convert .NET object to TOON
var data = new Dictionary<string, object?>
{
    ["crew"] = new List<Dictionary<string, object?>>
    {
        new() { ["id"] = 1, ["name"] = "Luz", ["role"] = "Light glyph" },
        new() { ["id"] = 2, ["name"] = "Amity", ["role"] = "Abomination strategist" }
    },
    ["active"] = true,
    ["ship"] = new Dictionary<string, object?>
    {
        ["name"] = "Owl House",
        ["location"] = "Bonesborough"
    }
};

var toonText = Api.ToToon(data, mode: "auto");
Console.WriteLine(toonText);
// Output:
// crew[2]{id,name,role}:
//   1,Luz,"Light glyph"
//   2,Amity,"Abomination strategist"
// active: true
// ship:
//   name: "Owl House"
//   location: Bonesborough

// Convert TOON back to .NET object
var roundTrip = Api.FromToon(toonText);
// ✅ Perfect round-trip!

📖 Usage

C# API

Basic Conversion

using ToonSharp;

// JSON → TOON
var data = new Dictionary<string, object?>
{
    ["name"] = "Luz",
    ["age"] = 16,
    ["active"] = true
};
var toon = Api.ToToon(data, indent: 2, mode: "auto");

// TOON → JSON
var parsed = Api.FromToon(toon);

YAML Conversion

// YAML → TOON
var yamlText = @"
name: Luz
age: 16
active: true
";
var toon = Api.YamlToToon(yamlText, indent: 2, mode: "auto");

// TOON → YAML
var toonText = @"
name: Luz
age: 16
active: true
";
var yaml = Api.ToonToYaml(toonText);

// Direct YAML serialization/deserialization
var data = new Dictionary<string, object?> { ["name"] = "Luz" };
var yamlOutput = Api.ToYaml(data);
var parsedData = Api.FromYaml(yamlOutput);

TOML Conversion

// TOML → TOON (Perfect for Rust Cargo.toml, Python pyproject.toml)
var tomlText = @"
[package]
name = ""my-project""
version = ""0.1.0""

[dependencies]
serde = ""1.0""
tokio = ""1.0""
";
var toon = Api.TomlToToon(tomlText, indent: 2, mode: "auto");

// TOON → TOML
var toonText = @"
package:
  name: my-project
  version: 0.1.0
dependencies:
  serde: 1.0
  tokio: 1.0
";
var toml = Api.ToonToToml(toonText);

// Direct TOML serialization/deserialization
var data = new Dictionary<string, object?> 
{ 
    ["name"] = "my-project",
    ["version"] = "0.1.0"
};
var tomlOutput = Api.ToToml(data);
var parsedData = Api.FromToml(tomlOutput);

JSON String Conversion (v1.4.0)

using System.Text.Json;

// Direct JSON string → TOON (via JsonElement deserialization)
var jsonString = @"{""DocumentId"":""DOC-2024-001"",""Content"":""Analysis report..."",""MaxTokens"":500,""Metrics"":[""sentiment"",""topics""]}";

// Deserialize JSON to object (returns JsonElement) and convert to TOON
var obj = JsonSerializer.Deserialize<object>(jsonString);
var toon = Api.ToToon(obj, indent: 2, mode: "auto");
// Output:
// DocumentId: DOC-2024-001
// Content: "Analysis report..."
// MaxTokens: 500
// Metrics:
//   - sentiment
//   - topics

POCO Object Serialization (v1.4.0, v1.4.1)

// Serialize any C# class directly to TOON
// Works with or without namespace (v1.4.1 fix)
public class Person
{
    public string Name { get; set; }
    public int Age { get; set; }
    public List<string> Tags { get; set; }
}

var person = new Person 
{ 
    Name = "Alice", 
    Age = 30, 
    Tags = new List<string> { "developer", "blogger" } 
};
var toon = Api.ToToon(person);
// Output:
// Name: Alice
// Age: 30
// Tags:
//   - developer
//   - blogger

// Also works with anonymous types and classes without namespace
var anon = new { Title = "Report", Value = 123 };
var toonAnon = Api.ToToon(anon);

Validation

var toonText = @"
crew[2]{id,name}:
  1,Luz
  2,Amity
";

var (isValid, errors) = Api.ValidateToon(toonText, strict: true);
if (!isValid)
{
    foreach (var error in errors)
    {
        Console.WriteLine(error);
    }
}

Command-Line Interface

Convert JSON to TOON

dotnet run --project src/ToonSharp.CLI -- to --in data.json --out data.toon --mode readable --indent 2

Convert TOON to JSON

dotnet run --project src/ToonSharp.CLI -- from --in data.toon --out data.json --permissive

Format a TOON File

dotnet run --project src/ToonSharp.CLI -- fmt --in data.toon --out data.formatted.toon --mode readable

Convert YAML to TOON

dotnet run --project src/ToonSharp.CLI -- yaml-to-toon --in data.yaml --out data.toon --mode readable --indent 2

Convert TOON to YAML

dotnet run --project src/ToonSharp.CLI -- toon-to-yaml --in data.toon --out data.yaml --permissive

Convert TOML to TOON

dotnet run --project src/ToonSharp.CLI -- toml-to-toon --in Cargo.toml --out config.toon --mode readable --indent 2

Convert TOON to TOML

dotnet run --project src/ToonSharp.CLI -- toon-to-toml --in config.toon --out output.toml

Exit Codes:

• 0 - Success
• 2 - TOON syntax error
• 3 - General error
• 4 - I/O error

🧪 Testing

# Run all tests
dotnet test

# Run with coverage
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

⚡ Performance

ToonSharp is optimized for high performance using parallel processing and Span<T> optimizations. The library automatically uses parallel processing for large datasets (tables with 50+ rows, arrays with 200+ items) and leverages Span<T> for zero-allocation string operations, significantly reducing memory allocations and improving throughput.

🚀 Version 1.1.0 Performance Improvements:

• 40-46% faster for large table operations (deserialization and round-trip)
• 16-20% faster for large array deserialization
• Optimized IterLines with Span-based line processing
• Optimized ParseValue with simplified comparisons and caching

The following benchmarks were executed on .NET 9.0 with BenchmarkDotNet (v1.1.0):

Performance Results

Notes:

• Benchmarks run in Release mode with full optimizations
• Times include GC overhead and memory allocation
• Results may vary based on hardware and system load
• Large Table and Large Array benchmarks use parallel processing (50+ rows, 200+ items)
• Performance improvements in v1.1.0: 40-46% faster for large table operations compared to v1.0.0

YAML Conversion Performance

🚀 Version 1.2.0 YAML Performance Improvements:

ToonSharp v1.2.0 introduces significant YAML performance optimizations through static serializer/deserializer instance reuse:

YAML Performance Notes:

• v1.2.0 achieves 60-85% faster YAML serialization through static instance reuse
• YAML conversion leverages the YamlDotNet library for robust YAML support
• Deserialization improvements are more modest (4-36%) due to parser overhead
• Memory allocation significantly reduced across all operations

TOML Conversion Performance

🎯 Version 1.3.0 TOML Performance:

ToonSharp v1.3.0 introduces TOML support with excellent performance characteristics using the Tomlyn library:

TOML Performance Notes:

• Excellent serialization speed: TOON → TOML is 2-4x faster than YAML serialization
• Efficient memory usage: Lower memory allocation compared to YAML operations
• Ideal for configuration files: Perfect for Rust Cargo.toml, Python pyproject.toml
• Leverages the high-performance Tomlyn library for TOML parsing
• Optimized for typical configuration file sizes (small to medium)

JSON & Object Serialization Performance

🚀 Version 1.4.0 JsonElement & POCO Support:

ToonSharp v1.4.0 introduces direct serialization support for JsonElement (from System.Text.Json) and POCO objects via reflection:

JsonElement Serialization (JSON String → TOON)

POCO Object Serialization (C# Class → TOON)

JsonElement & POCO Performance Notes:

• JsonElement near-native speed: Only 1.27x overhead vs pre-built dictionaries for small objects
• POCO serialization via reflection: Serialize any C# class directly to TOON
• Anonymous types support: Works with new { ... } syntax
• 2-4x overhead for typical use cases: Excellent performance for most scenarios

Running Benchmarks

# Run all benchmarks
dotnet run --project benchmarks/ToonSharp.Benchmarks -c Release

# Run specific benchmarks
dotnet run --project benchmarks/ToonSharp.Benchmarks -c Release -- --filter "*Small*"

Complete results are automatically exported to BenchmarkDotNet.Artifacts/results/ in Markdown, HTML, and CSV formats.

📚 Documentation

Comprehensive documentation is available in the docs/ directory:

• docs/spec_summary.md – Concise TOON SPEC v2.0 overview with ABNF notes
• docs/examples.md – JSON⇄TOON conversion examples
• docs/assumptions.md – Documented gaps/assumptions + strict vs. permissive behavior

🌟 Use Cases

• Data Serialization: Efficient storage and transmission of structured data
• API Development: Lightweight data format for REST APIs
• Configuration Files: Human-readable config format with comments support
• Data Pipelines: Stream processing of large JSON/YAML/TOML datasets
• ML/AI Projects: Token-optimized format for LLM training data
• Format Migration: Convert between JSON, YAML, TOML, and TOON seamlessly
• DevOps & Infrastructure: Transform configuration files between different formats
• Cross-Ecosystem Development: Convert Rust Cargo.toml ↔ Python pyproject.toml ↔ .NET configs

📖 Examples

The examples/spec_v2/ directory contains all material from the official toon-format/spec repository:

• conversions/ – JSON ↔ TOON pairs published by the specification.
• valid/ – all canonical examples (key folding, custom delimiters, primitive arrays, etc.).
• invalid/ – edge cases that must fail in strict mode.
• basic_object, tabular_array, mixed_structures – ToonSharp-specific examples designed for quick documentation.

The ExamplesComplianceTests test suite iterates through every official TOON file and verifies that:

1. Valid examples can be parsed, validated, and round-tripped without loss.
2. JSON ↔ TOON pairs remain equivalent after serializing/deserializing with ToonSharp.
3. Invalid examples throw ToonSyntaxError in strict mode.

dotnet test --filter ExamplesComplianceTests

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Guidelines:

• Follow C# coding conventions
• Add tests for new features
• Update documentation as needed
• Ensure all tests pass: dotnet test
• Keep additions aligned with TOON SPEC v2.0

📋 Release Notes

v1.4.1 (2024-12-05)

• Bugfix: Fixed POCO serialization for classes without namespace
• Classes defined without a namespace declaration now serialize correctly to TOON
• Improved reflection logic for detecting serializable types

v1.4.0 (2024-12-04)

• New: Direct JsonElement serialization support
• New: POCO object serialization via reflection
• New: Anonymous type serialization support

v1.3.0 (2024-12-03)

• New: TOML ↔ TOON bidirectional conversion
• New: CLI commands for TOML conversion

v1.2.0 (2024-12-02)

• New: YAML ↔ TOON bidirectional conversion
• Performance: 60-85% faster YAML serialization with static instance reuse

v1.1.0 (2024-12-01)

• Performance: 40-46% faster large table operations
• Performance: Parallel processing for large datasets

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👨‍💻 Author

Christian Palomares - @shinjidev

☕ Support

If you find this project helpful, consider supporting my work:

🙏 Acknowledgments

• Built following TOON SPEC v2.0
• Inspired by the need for efficient, token-optimized data serialization
• C# implementation inspired by the original TOON reference tooling

⭐ Star this repository if you find it useful! ⭐