BpeTokenizer 1.0.1
Heavy refactoring of the internal types. Breaking changes from previous versions. This deprecates the prior versions for that reason.
See the version list below for details.
dotnet add package BpeTokenizer --version 1.0.1
NuGet\Install-Package BpeTokenizer -Version 1.0.1
<PackageReference Include="BpeTokenizer" Version="1.0.1" />
paket add BpeTokenizer --version 1.0.1
#r "nuget: BpeTokenizer, 1.0.1"
// Install BpeTokenizer as a Cake Addin #addin nuget:?package=BpeTokenizer&version=1.0.1 // Install BpeTokenizer as a Cake Tool #tool nuget:?package=BpeTokenizer&version=1.0.1
BpeTokenizer
BpeTokenizer is a C# implementation of tiktoken written by OpenAI. It is a byte pair encoding tokenizer that can be used to tokenize text into subword units.
As a BpeTokenizer derived from tiktoken, it can be used as a token counter. Useful to ensure that when streaming tokens from the OpenAI API for GPT Chat Completions, you could keep track of the cost related to the software calling the API.
This library is built for x64 architectures.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net7.0 is compatible. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
-
net7.0
- Newtonsoft.Json (>= 13.0.3)
NuGet packages (1)
Showing the top 1 NuGet packages that depend on BpeTokenizer:
Package | Downloads |
---|---|
BpeChatAI
Package Description |
GitHub repositories
This package is not used by any popular GitHub repositories.
Correcting ReadMe.md to include information about architecture, and removed CoPilot generated text that was not intended to be there.