Microsoft.DeepDev.TokenizerLib
1.3.0
Prefix Reserved
See the version list below for details.
dotnet add package Microsoft.DeepDev.TokenizerLib --version 1.3.0
NuGet\Install-Package Microsoft.DeepDev.TokenizerLib -Version 1.3.0
<PackageReference Include="Microsoft.DeepDev.TokenizerLib" Version="1.3.0" />
paket add Microsoft.DeepDev.TokenizerLib --version 1.3.0
#r "nuget: Microsoft.DeepDev.TokenizerLib, 1.3.0"
// Install Microsoft.DeepDev.TokenizerLib as a Cake Addin
#addin nuget:?package=Microsoft.DeepDev.TokenizerLib&version=1.3.0
// Install Microsoft.DeepDev.TokenizerLib as a Cake Tool
#tool nuget:?package=Microsoft.DeepDev.TokenizerLib&version=1.3.0
Tokenizer
This repo contains a C# implementation of byte pair encoding(BPE) tokenizer for OpenAI LLMs, it's based on open sourced rust implementation in the OpenAI tiktoken. The library is valuable to run tokenization on prompt in .NET environment before feeding prompt into a LLM.
The TokenizerLib is built in .NET Standard 2.0, which can be consumed in projects on any version of .NET later than .NET Core 2.0 or .NET Framework 4.6.1.
You can download and install the nuget package of TokenizerLib here.
Example C# code to use TokenizerLib in your code:
using System.Collections.Generic;
using Microsoft.DeepDev;
var IM_START = "<|im_start|>";
var IM_END = "<|im_end|>";
var specialTokens = new Dictionary<string, int>{
{ IM_START, 100264},
{ IM_END, 100265},
};
tokenizer = TokenizerBuilder.CreateByModelName("gpt-4", specialTokens);
var text = "<|im_start|>Hello World<|im_end|>";
var encoded = tokenizer.Encode(text, new HashSet<string>(specialTokens.Keys));
Console.WriteLine(encoded.Count);
var decoded = tokenizer.Decode(encoded.ToArray());
Console.WriteLine(decoded);
Performance
PerfBenchmark result based on PerfBenchmark.csproj:
BenchmarkDotNet=v0.13.3, OS=Windows 11 (10.0.22621.1413)
Intel Core i7-1065G7 CPU 1.30GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=7.0.300-preview.23165.23
[Host] : .NET 6.0.15 (6.0.1523.11507), X64 RyuJIT AVX2
DefaultJob : .NET 6.0.15 (6.0.1523.11507), X64 RyuJIT AVX2
Method | Mean | Error | StdDev |
---|---|---|---|
Encode | 2.376 s | 0.0446 s | 0.0639 s |
Contributing
We welcome contributions. Please follow this guideline.
Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 was computed. net5.0-windows was computed. net6.0 was computed. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
.NET Core | netcoreapp2.0 was computed. netcoreapp2.1 was computed. netcoreapp2.2 was computed. netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
.NET Standard | netstandard2.0 is compatible. netstandard2.1 was computed. |
.NET Framework | net461 was computed. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen40 was computed. tizen60 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETStandard 2.0
- No dependencies.
NuGet packages (4)
Showing the top 4 NuGet packages that depend on Microsoft.DeepDev.TokenizerLib:
Package | Downloads |
---|---|
Microsoft.DotNet.Interactive.AIUtilities
Utilities for AI workload in .NET Interactive and Polyglot Notebooks |
|
LangChain.NET
LangChain.NET provides the ability to build applications with LLMs through composability |
|
Cnblogs.KernelMemory.AI.DashScope
Provide access to DashScope LLM models in Kernel Memory to generate embeddings and text |
|
ContextFlow
Package Description |
GitHub repositories (3)
Showing the top 3 popular GitHub repositories that depend on Microsoft.DeepDev.TokenizerLib:
Repository | Stars |
---|---|
microsoft/semantic-kernel
Integrate cutting-edge LLM technology quickly and easily into your apps
|
|
dotnet/interactive
.NET Interactive combines the power of .NET with many other languages to create notebooks, REPLs, and embedded coding experiences. Share code, explore data, write, and learn across your apps in ways you couldn't before.
|
|
dotnet/ResXResourceManager
Manage localization of all ResX-Based resources in one central place.
|