Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
47 packages returned for Tags:"tokenizer"
Sort by
Relevance
Downloads
Recently updated
TokenCs
by:
zeplar_exe
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
2,209 total downloads
last updated
8/10/2022
Latest version:
1.0.5
token
tokenizer
lexer
parse
parser
A simple library to abstract the gritty process of lexical parsing.
LLMSharp.
OpenAi.
Tokenizer
by:
veerash-ayyagari
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
4,210 total downloads
last updated
10/4/2023
Latest version:
2.0.3
openai
gpt
3.5
gpt
4
tokenizer
dotnet
tiktoken
Open AI Chat Completion Models (GPT 3.5/GPT 4) BPE Tokenizer unofficial implementation
Brainfuck-
Runner
by:
nikolayresh
.NET 5.0
9,708 total downloads
last updated
11/9/2021
Latest version:
1.0.14
brainfuck
brain
brainfuck
interpreter
brainfuck
tokenizer
brainfuck
validator
brainfuck
engine
More tags
Powerful Brainfuck language interpreter with ability to tokenize & validate the Brainfuck code
NReco.
NLQuery
by:
nreco
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.5
9,835 total downloads
last updated
8/29/2022
Latest version:
1.1.1
NLP
NER
NLQ
search
search-interface
natural
language
query
named
entity
More tags
NLQuery: natural language query parser recognizes entities in context of structured sources (like tabular dataset). Can be used for building natural language interface to SQL database or OLAP cube, implementing...
More information
XLemmatizer
by:
rchristen
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
842 total downloads
last updated
5/29/2020
Latest version:
0.1.0
nlp
lemmatizer
tokenizer
Pre-release version. API might change later. A lemma is the canonical form of the word. For example, the words "run", "runs", "ran" and "running" can be lemmatized to "run" XLemmatizer tokenizes and lemmatizes...
More information
GParse
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
2,208 total downloads
last updated
6/15/2021
Latest version:
5.0.0-alpha.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
Parsing and lexing utilities to create your own parser and lexer
VisualFA.
SourceGenerator
by:
honey.the.codewitch
512 total downloads
last updated
4/25/2024
Latest version:
1.3.1
lexer
tokenizer
dfa
regex
Generates fast DFA lexers and matchers in C# during the build process
Virastyar
by:
senobari
.NET Framework
5,408 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library. This package includes both library and it's required data files.
GParse.
Extensions.
StateMachines
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
1,033 total downloads
last updated
6/7/2021
Latest version:
5.0.0-alpha09
parser
lexer
parsing
lexing
tokenizer
tokenizing
Extension methods to integrate GParse with Tsu.StateMachines
Tweekenizer
by:
cristipufu
.NET 5.0
440 total downloads
last updated
11/30/2020
Latest version:
1.0.0
tokenizer
hashtag
nlp
emoji
Tokenizer for social media posts and comments
Cynic-
Magnit.
Tokenization
by:
catapart
.NET 6.0
1,042 total downloads
last updated
10/20/2022
Latest version:
1.0.1
string
parse
parsing
token
tokenizer
tokenization
Tokenize strings into custom tokens using ordered regex operations.
GBertTokenizer
by:
GeorgeTer
.NET 6.0
286 total downloads
last updated
2/4/2024
Latest version:
1.1.1
BERT
Tokenizer
charp
dotnet
Package Description
Noise
by:
m-wantia
.NET 6.0
251 total downloads
last updated
6/19/2022
Latest version:
0.0.0.1-alpha
lexer
lexical
tokenizer
Package Description
APCSharp
by:
Dotch
.NET 5.0
.NET Core 3.1
3,571 total downloads
last updated
4/11/2021
Latest version:
1.0.8
lexer
parser
parsing
language
tokenizer
lexical
semantic
analysis
compiler
combinator
More tags
Another Parser Combinator for C#. A library for building optimized and flexible parsers.
PrismSharp
by:
tkubec
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
193 total downloads
last updated
2/18/2022
Latest version:
1.0.0-beta
syntax
highlighting
highlighter
tokenizer
PrismSharp is a syntax highlighting library based on an excellent javascript library PrismJS, fully written in C#. It currently supports over 270 programming languages and has 44 built-in visual themes, also...
More information
Virastyar.
Data
by:
senobari
3,818 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
Required data files for Virastyar library.
Leeax.
Parsing.
CSS
by:
leeax
.NET 5.0
202 total downloads
last updated
1/12/2021
Latest version:
1.0.0-beta
css
css3
tokenizer
parser
A lightweight CSS tokenizer/parser with no dependencies.
Virastyar.
Lib
by:
senobari
.NET Framework
3,755 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library.
TKN
by:
Gulg
.NET Framework 4.7.2
193 total downloads
last updated
7/22/2022
Latest version:
1.0.0
Tokenizer
Lexer
Parser
Regex
An advanced tokenizer by GulgDevs
AllMiniLmL6V2Sharp
by:
ksanman
.NET 5.0
.NET Core 3.0
.NET Standard 2.1
543 total downloads
last updated
4/15/2024
Latest version:
0.0.2
embeddings
all-mini-lm-l6-v2
tokenizer
BERT
Sentence
Transfomers
onnx
NET Standard 2.1 library to produces embeddings using C# Bert Tokenizer and Onnx All-Mini-LM-L6-v2 model.
Previous
Next