TokenKit 1.0.0

🧠 TokenKit

TokenKit — A lightweight .NET 8.0 library and CLI tool for unified tokenization, validation, and model registry management across multiple LLM providers (OpenAI, Anthropic, Gemini, etc.)

✨ Features

⚙️ Installation

📦 NuGet (Library use)

dotnet add package TokenKit

🧰 Global CLI Tool

dotnet tool install -g TokenKit

🚀 Quick Start (CLI)

Analyze, validate, or update model data directly from your terminal.

1️⃣ Analyze Inline Text

tokenkit analyze "Hello from TokenKit!" --model gpt-4o

Output:

{
  "Model": "gpt-4o",
  "Provider": "OpenAI",
  "TokenCount": 4,
  "EstimatedCost": 0.00002,
  "Valid": true
}

2️⃣ Analyze File Input

tokenkit analyze prompt.txt --model gpt-4o

3️⃣ Pipe Input (stdin)

echo "This is piped text input" | tokenkit analyze --model gpt-4o

4️⃣ Validate Prompt

tokenkit validate "A very long prompt to validate" --model gpt-4o

Output:

{
  "IsValid": true,
  "Message": "OK"
}

5️⃣ Update Model Data

Default Update (No API Key)

Fetch the latest built-in model metadata:

tokenkit update-models

Update Using OpenAI API Key

Use your OpenAI key to fetch live model data from the /v1/models endpoint:

tokenkit update-models --openai-key sk-xxxx

Update from JSON (stdin)

Pipe a JSON file with model specs:

cat newmodels.json | tokenkit update-models

Example newmodels.json:

[
  {
    "Id": "gpt-4o-mini",
    "Provider": "OpenAI",
    "MaxTokens": 64000,
    "InputPricePer1K": 0.002,
    "OutputPricePer1K": 0.01,
    "Encoding": "cl100k_base"
  }
]

6️⃣ Scrape Model Data (Preview Only)

Fetch latest OpenAI model data (does not overwrite your registry):

tokenkit scrape-models --openai-key sk-xxxx

If no key is provided, TokenKit falls back to its offline model list.

Example Output:

🔍 Fetching latest OpenAI model data...
✅ Retrieved 3 models:
  - OpenAI: gpt-4o (128000 tokens)
  - OpenAI: gpt-4o-mini (64000 tokens)
  - OpenAI: gpt-3.5-turbo (4096 tokens)

🧩 Model Registry

TokenKit stores all known model information in:

src/TokenKit/Registry/models.data.json

Each entry contains:

{
  "Id": "gpt-4o",
  "Provider": "OpenAI",
  "MaxTokens": 128000,
  "InputPricePer1K": 0.005,
  "OutputPricePer1K": 0.015,
  "Encoding": "cl100k_base"
}

🧮 CLI Command Reference

🧠 Programmatic Use (SDK)

using TokenKit.Registry;
using TokenKit.Services;

var model = ModelRegistry.Get("gpt-4o");
var tokenizer = new TokenizerService();

var result = tokenizer.Analyze("Hello from TokenKit!", model!.Id);
var cost = CostEstimator.Estimate(model, result.TokenCount);

Console.WriteLine($"Tokens: {result.TokenCount}, Cost: ${cost}");

🧪 Testing

TokenKit includes an xUnit test suite with coverage for tokenization, cost estimation, and registry loading.

dotnet test

🛠 Project Structure

TokenKit/
├── src/
│   └── TokenKit/
│       ├── Models/
│       ├── Services/
│       ├── Registry/
│       ├── CLI/
│       └── Program.cs
└── tests/
    └── TokenKit.Tests/

⚙️ Phase 6 Additions (Advanced Tokenization & CLI UX)

🧠 Example Usage

List all models

tokenkit models list

Filter by provider (case-insensitive)

tokenkit models list --provider openai
tokenkit models list --provider Anthropic

Analyze with a specific encoder

tokenkit analyze "Hello from TokenKit" --model gpt-4o --engine sharptoken

🗺️ Roadmap

• Tokenization, cost, and validation services
• CLI for analyze, validate, and update-models
• Stdin + file + inline input support
• Model registry auto-load and safe paths
• Live model scraping from OpenAI (optional API key)
• Add tokenkit models list command
• Optional SharpToken / Microsoft.ML.Tokenizers integration
• Publish stable v1.0.0 to NuGet + dotnet tool feed

💡 License

Licensed under the MIT License.
© 2025 Andrew Clements

🎨 Phase 8 — CLI Polish, Logging & Automation Support

🧪 Extended CLI Examples

🔹 Standard Analysis

tokenkit analyze "Hello from TokenKit!" --model gpt-4o

✅ Produces colorized JSON summary + log entry.

🔹 JSON Mode (Automation / CI)

tokenkit analyze "Hello world" --model gpt-4o --json

Outputs pure JSON only, suppressing banner and emojis:

{
  "Model": "gpt-4o",
  "Provider": "OpenAI",
  "TokenCount": 7,
  "EstimatedCost": 0.000105,
  "Engine": "simple",
  "Valid": true
}

🔹 Quiet Mode (Log Only)

tokenkit analyze "Silent test" --model gpt-4o --quiet

No console output. Log file receives entries like:

2025-10-17 22:43:15 [INFO] Analyze started with model=gpt-4o
2025-10-17 22:43:15 [SUCCESS] Analyzed 7 tokens using simple (gpt-4o)

🔹 Model Listing with JSON

tokenkit models list --json

📜 Logs

All CLI runs write to tokenkit.log (auto-rotated at 1 MB).
You can find it under your TokenKit working directory, e.g.:

src/TokenKit/bin/Debug/net8.0/tokenkit.log

📈 Code Coverage

TokenKit targets 100% test coverage with xUnit and Codecov integration.
Run coverage locally:

dotnet test --collect:"XPlat Code Coverage"

View detailed results in Codecov:

🗺️ Updated Roadmap (as of 2025-10-17)

© 2025 Andrew Clements — MIT License
Flow Labs / TokenKit — https://github.com/AndrewClements84/TokenKit