Tokenizers.NET 1.1.1-alpha.64-2d85cb3

.NET 9.0 This package targets .NET 9.0. The package is compatible with this framework or higher.
This is a prerelease version of Tokenizers.NET. 
dotnet add package Tokenizers.NET --version 1.1.1-alpha.64-2d85cb3
                    


                    

                
NuGet\Install-Package Tokenizers.NET -Version 1.1.1-alpha.64-2d85cb3
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="Tokenizers.NET" Version="1.1.1-alpha.64-2d85cb3" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="Tokenizers.NET" Version="1.1.1-alpha.64-2d85cb3" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="Tokenizers.NET" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add Tokenizers.NET --version 1.1.1-alpha.64-2d85cb3
                    


                    

                
#r "nuget: Tokenizers.NET, 1.1.1-alpha.64-2d85cb3"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package Tokenizers.NET@1.1.1-alpha.64-2d85cb3
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=Tokenizers.NET&version=1.1.1-alpha.64-2d85cb3&prerelease
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=Tokenizers.NET&version=1.1.1-alpha.64-2d85cb3&prerelease
                    


                            Install as a Cake Tool

Tokenizers.NET

High-performance .NET wrapper for https://github.com/huggingface/tokenizers

How to install

dotnet add package Tokenizers.NET

Why Tokenizers.NET?

  • Run HuggingFace tokenizers on .NET
  • High performance, zero GC-allocations
  • I don't know, TrumpMcDonaldz rock!

Supported Archs

  • OSX ( ARM64 / x86_64 )
  • Linux ( ARM64 / x86_64 )
  • Windows ( ARM64 / x86_64 )

( Who uses Windows ARM lol )

View test status

https://budgetdevv.github.io/Tokenizers.NET/

How to build

  • Navigate to project's root directory
dotnet clean
dotnet pack -c Release -v:m /p:DownloadNativeFiles=true
Product Compatible and additional computed target framework versions.
.NET net9.0 is compatible.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
1.1.1-alpha.64-2d85cb3 147 6/21/2025
1.1.1-alpha.63-7b06e04 104 6/21/2025
1.1.1-alpha.62-553b5a7 76 6/21/2025
1.1.1-alpha.61-2ee35b8 87 6/20/2025
1.1.1-alpha.60-25259e9 90 6/20/2025
1.1.0-alpha.59-336bc56 91 6/20/2025
1.1.0-alpha.58-f7bfbf2 95 6/20/2025
1.1.0-alpha.57-4cd7510 93 6/20/2025
1.1.0-alpha.56-ffa6248 114 6/20/2025
1.1.0-alpha.55-d924b96 114 6/20/2025
1.1.0-alpha.54-cccd2a3 153 6/18/2025
1.1.0-alpha.53-82bad4d 148 6/18/2025
1.1.0-alpha.52-3b5a4ed 146 6/18/2025
1.1.0-alpha.51-acaee15 143 6/18/2025
1.1.0-alpha.50-b9fd3da 146 6/15/2025
1.1.0-alpha.47-d0bebbd 151 6/15/2025
1.1.0-alpha.46-dfaa5d8 150 6/15/2025
1.1.0-alpha.45-0457746 146 6/15/2025
1.1.0-alpha.44-215f432 151 6/15/2025
1.1.0-alpha.43-374fd51 158 10/19/2024
Loading failed