DiffLibLLM 1.0.1

DiffLibLLM

Semantic diff library (using embeddings): Compare texts with their translation.

DiffLibLLM utilizes only the embedding stage characteristic of large language models (LLMs), that is, the vectorization of for example words within a sentence. Vectorization is already part of the semantic aspect of LLMs; it is one of the very first steps in their operation.

Available embedding providers

• SimpleHashEmbeddingProvider Simple local provider with no external AI dependencies
• OllamaEmbeddingsProvider local using Ollama server
• OpenAiEmbeddingsProvider distant using API

Quick start with Ollama

1. Download an install Ollama.
2. Download some Ollama embedding models:

ollama pull all-minilm
ollama pull nomic-embed-text
ollama pull mxbai-embed-large
ollama pull embeddinggemma
ollama pull granite-embedding:278m
ollama pull bge-m3
ollama pull qllama/multilingual-e5-base
ollama pull qwen3-embedding:0.6b
ollama pull nomic-embed-text-v2-moe
ollama pull paraphrase-multilingual:278m-mpnet-base-v2-fp16
ollama pull paraphrase-multilingual
ollama pull zylonai/multilingual-e5-large
ollama pull snowflake-arctic-embed2
ollama pull qwen3-embedding
ollama pull ...

Vectorization samples

Vectorization is the step that follows tokenization: words (or word fragments, or even multi-word expressions) are organized within a multidimensional space, which already allows for the calculation of a form of basic semantics. Here is an example: "King - Man + Woman" yields a semantic vector that should be very close to the vector corresponding to the word "Queen".

Here are other samples.

See the samples for some models.

Depending on the model's capabilities, some examples also work in languages ​​other than English—for instance, in French.

The embedding models used in this library are very simple and are not bidirectional: they only allow for the calculation of the vector corresponding to an expression (the final token determined by the tokenization algorithm), without including an index (such as HNSW—Hierarchical Navigable Small World) that aggregates the vectors of all the model's tokens. Such an HNSW index would be useful, for instance, for searching for words with vectors close to another—that is, for finding synonyms. Here, however, one can only calculate the vectors for a set of tokens and rank them by similarity, without being able to discover new synonyms.