43 packages returned for Tags:"tokenizer"

Tokenizer extracts structured information from blocks of text and reflects them onto .NET objects

C# Expression parser and evaluator, inspired from jokenizer project.

OpenAI GPT utils, e.g. GPT3 Tokenizer

The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech... More information

This package contains tokenizers for following models: · BERT Base · BERT Large · BERT German · BERT Multilingual · BERT Base Uncased · BERT Large Uncased

A .NET class library that makes it easier to parse text. The library tracks the current position within the text, ensures your code never attempts to access a character at an invalid index, and includes many... More information

Tokenizador (Xamarin) para Conekta. Necesitas tener alguna libreria de servidor para usar el token.

A shared package used by Loretta. Do not install this package manually, it will be added as a prerequisite by other packages that require it.

NLTK python library wrapper for .NET

A GLua/Lua lexer, parser, code analysis, transformation and generation library.

VBF.Compilers.Scanners is a scanner builder. It contains a regular expression to DFA engine, can generate high performance scanners for unicode source text.

Experimental code that might become part of Loretta.CodeAnalysis.Lua.

Package Description

Package Description

Trl.PegParser contains a tokenizer and a parser. The tokenizer uses regular expressions to define tokens, and exposes both matched and unmatched character ranges. The PEG Parser uses parsing expression grammers... More information

The Stringe is a wrapper for the .NET String object that tracks line, column, offset, and other metadata for substrings.

A html preprocessor which allows you to write html code using a beautiful syntax

Classe qui gère le découpage d'une chaine en paires de clés/valeurs et permet de la reconstruire avec des valeurs de remplacement ou des valeurs additionnelles

A simple library to abstract the gritty process of lexical parsing.

Anthropic Claude BPE Tokenizer unofficial implementation