Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
13 packages returned for Tags:"Tokenization"
Sort by
Relevance
Downloads
Recently updated
Lexepars.
Grammars.
Yaml
by:
Dimigur
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
69,799 total downloads
last updated
9/11/2019
Latest version:
0.1.0
yaml
parsing
lexer
tokenization
grammar
validation
YAML grammar based on Lexepars parser lib.
Lexepars.
Grammars.
Json
by:
Dimigur
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
33,084 total downloads
last updated
7/27/2019
Latest version:
1.0.0
json
parsing
lexer
tokenization
JSON grammar based on Lexepars parser lib.
Lexepars.
TestFixtures
by:
Dimigur
.NET 5.0
.NET Core 3.1
69,602 total downloads
last updated
12/29/2021
Latest version:
1.3.0
monadic
parser
combinator
parsing
lexer
tokenization
parsec
fparsec
Package Description
CyberSource.
Flex.
Server
by:
CyberSource
Deprecated
.NET Framework
36,365 total downloads
last updated
1/15/2019
Latest version:
0.2.2
CyberSource
tokenization
tokenisation
tokens
token
SDK for server-side integration with CyberSource Flex API
Stanford.
NLP.
Segmenter
by:
sergey_tihon
Deprecated
.NET 5.0
.NET Core 3.1
.NET Framework 4.6.1
22,772 total downloads
last updated
8/25/2022
Latest version:
4.2.0.2
nlp
stanford
segmenter
tokenization
splitting
IKVM
Tokenization of raw text is a standard pre-processing step for many NLP tasks. For English, tokenization usually involves punctuation splitting and separation of some affixes like possessives. Other languages...
More information
McBits.
Tokenization
by:
McBits
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
5,859 total downloads
last updated
12/22/2018
Latest version:
2.0.0
mcbits
tokenization
Extract tokens from a string of text for use with NLP tools or statistical analysis.
Lexepars
by:
Dimigur
.NET 5.0
.NET Core 3.0
.NET Standard 2.1
18,272 total downloads
last updated
12/29/2021
Latest version:
1.3.0
monadic
parser
combinator
parsing
lexer
tokenization
parsec
fparsec
Concise monadic parser combinator library with separate lexer/parser phases and big-size input support.
Alumis.
Text.
Tokenization
by:
alumis
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
5,844 total downloads
last updated
4/24/2019
Latest version:
1.0.9
text
tokenization
unicode
Text tokenization based on Unicode grapheme clustering, and the XID_Start and XID_Continue binary properties.
TextMatch
by:
cris-almodovar
.NET Framework
11,408 total downloads
last updated
11/22/2015
Latest version:
0.6.5
text
search
match
fuzzy
wildcard
tokenization
stemming
TextMatch is a library for searching inside texts using Lucene query expressions. Supports all types of Lucene query expressions - boolean, wildcard, fuzzy. Options are available for tweaking tokenization, such...
More information
MetaParser
by:
dsisco11
3,790 total downloads
last updated
2/23/2023
Latest version:
1.0.7
parsing
tokenization
tokenizer
text
parser
Package Description
Universal.
Common.
Language
by:
ong.andrew
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
13,479 total downloads
last updated
8/3/2022
Latest version:
1.2.1
Universal
Common
Language
Sentence
Component
Extensions
Bag
Of
Words
Tokenization
More tags
Class library for working with general language constructs.
Cynic-
Magnit.
Tokenization
by:
catapart
.NET 6.0
1,037 total downloads
last updated
10/20/2022
Latest version:
1.0.1
string
parse
parsing
token
tokenizer
tokenization
Tokenize strings into custom tokens using ordered regex operations.
UralicNLP
by:
mikahama
.NET 8.0
86 total downloads
last updated
2/21/2024
Latest version:
1.0.0
nlp
tokenization
lemmatization
English
Finnish
Spanish
French
Italian
German
Russian
An NLP library for many languages.