BpeTokenizer 1.0.0

.NET 7.0 This package targets .NET 7.0. The package is compatible with this framework or higher.
Additional Details

ReadMe.md file was incorrect, lacking information about the architecture and an accidental snippet of text generated by CoPilot.

There is a newer version of this package available.
See the version list below for details. 
dotnet add package BpeTokenizer --version 1.0.0                

                    
                

            
NuGet\Install-Package BpeTokenizer -Version 1.0.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="BpeTokenizer" Version="1.0.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
paket add BpeTokenizer --version 1.0.0                

                    
                

            
#r "nuget: BpeTokenizer, 1.0.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
// Install BpeTokenizer as a Cake Addin
#addin nuget:?package=BpeTokenizer&version=1.0.0

// Install BpeTokenizer as a Cake Tool
#tool nuget:?package=BpeTokenizer&version=1.0.0

BpeTokenizer

NuGet Last Commit GitHub Issues Used by Contributors License

BpeTokenizer is a C# implementation of tiktoken written by OpenAI. It is a byte pair encoding tokenizer that can be used to tokenize text into subword units. It is a very useful tool for NLP tasks such as machine translation, text generation, and text classification.

As a BpeTokenizer derived from tiktoken, it can be used as a token counter. Useful to ensure that when streaming tokens from the OpenAI API for GPT Chat Completions, you could keep track of the cost related to the software calling the API.

Product Compatible and additional computed target framework versions.
.NET net7.0 is compatible.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on BpeTokenizer:

Package Downloads
BpeChatAI

Package Description

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.0.5 203 7/21/2023
1.0.4 154 7/15/2023 1.0.4 is deprecated.
1.0.3 185 7/15/2023 1.0.3 is deprecated.
1.0.2 198 7/14/2023 1.0.2 is deprecated.
1.0.1 162 7/13/2023 1.0.1 is deprecated.
1.0.0 148 7/13/2023 1.0.0 is deprecated.

First release.