Iciclecreek.Azure.Storage.Lucene
2.1.0-rc001
dotnet add package Iciclecreek.Azure.Storage.Lucene --version 2.1.0-rc001
NuGet\Install-Package Iciclecreek.Azure.Storage.Lucene -Version 2.1.0-rc001
<PackageReference Include="Iciclecreek.Azure.Storage.Lucene" Version="2.1.0-rc001" />
<PackageVersion Include="Iciclecreek.Azure.Storage.Lucene" Version="2.1.0-rc001" />
<PackageReference Include="Iciclecreek.Azure.Storage.Lucene" />
paket add Iciclecreek.Azure.Storage.Lucene --version 2.1.0-rc001
#r "nuget: Iciclecreek.Azure.Storage.Lucene, 2.1.0-rc001"
#:package Iciclecreek.Azure.Storage.Lucene@2.1.0-rc001
#addin nuget:?package=Iciclecreek.Azure.Storage.Lucene&version=2.1.0-rc001&prerelease
#tool nuget:?package=Iciclecreek.Azure.Storage.Lucene&version=2.1.0-rc001&prerelease
![]()
Iciclecreek.Azure.Storage.Lucene
A Lucene.NET-backed drop-in replacement for Azure.Data.Tables and Azure.Storage.Queues clients, and for the core of Azure.Storage.Blobs. Use the same Azure SDK types without Azurite or a live Azure account.
Data is kept through the Lucene.NET Directory abstraction (Lucene.Net.Store.Directory): in memory (RAMDirectory), in a folder (FSDirectory), or in any Lucene Directory you supply. Each table, container and queue is a Lucene Directory of its own.
Installation
dotnet add package Iciclecreek.Azure.Storage.Lucene
Usage
Blobs
using Iciclecreek.Azure.Storage.Lucene.Blobs;
var blobService = new LuceneBlobServiceClient(@"C:\temp\storage");
BlobContainerClient container = blobService.GetBlobContainerClient("my-container");
await container.CreateIfNotExistsAsync();
BlobClient blob = container.GetBlobClient("hello.txt");
await blob.UploadAsync(BinaryData.FromString("Hello, World!"), overwrite: true);
var result = (await blob.DownloadContentAsync()).Value;
Console.WriteLine(result.Content.ToString()); // "Hello, World!"
Tables
using Iciclecreek.Azure.Storage.Lucene.Tables;
var tableService = new LuceneTableServiceClient(@"C:\temp\storage");
TableClient table = tableService.GetTableClient("people");
await table.CreateIfNotExistsAsync();
await table.AddEntityAsync(new TableEntity("users", "alice") { ["Name"] = "Alice" });
var entity = (await table.GetEntityAsync<TableEntity>("users", "alice")).Value;
Queues
using Iciclecreek.Azure.Storage.Lucene.Queues;
var queueService = new LuceneQueueServiceClient(@"C:\temp\storage");
QueueClient queue = queueService.GetQueueClient("tasks");
queue.Create();
queue.SendMessage("do the thing");
var msg = queue.ReceiveMessage().Value;
Console.WriteLine(msg.Body.ToString()); // "do the thing"
In memory, or in any Lucene Directory
// In memory: gone when the client is
var queueService = new LuceneQueueServiceClient();
// Anywhere: the factory is given a path such as "queues/tasks" and returns the Lucene Directory for it
var tableService = new LuceneTableServiceClient(path => OpenDirectoryFor(path));
Swap in via dependency injection
Every Lucene* client inherits from its Azure SDK base type:
// Production
services.AddSingleton<QueueServiceClient>(
new QueueServiceClient(connectionString));
// Test
services.AddSingleton<QueueServiceClient>(
new LuceneQueueServiceClient());
Layout
Each table, container and queue is a Lucene Directory of its own. Given a folder, each is an FSDirectory on a folder of its own inside it, and the three clients can share that folder.
{directory}
tables
people
orders
__registry__ the list of tables
blobs
photos
documents
queues
work
__registry__ the list of queues and their metadata
Names
A name made of lowercase letters, digits and hyphens is its own directory name. Anything else is made safe first, so that two names never share a directory and no name can point outside the folder:
- Table names are not case sensitive, as in Azure:
Peopleandpeopleare one table, listed under the name it was created with, in the directorypeople. - Queue and container names are case sensitive.
Workandworkare two queues, on any file system. - Any other character is written in the directory name as
_and its four-digit hexadecimal code: the queueWorkis the directory_0057ork, and a name such as../xis the single directory_002e_002e_002fx. The Azure naming rules are not otherwise enforced. __registry__and the empty name are refused with400 InvalidResourceName. So is a name whose directory name would be longer than 255 characters, the most a folder name can be.
Behaviour
Tables
- One Lucene index per table, in a Lucene
Directoryof its own, with its own index writer and a document per entity. - The whole table API: add, get, upsert, update and delete with ETag checks, merge and replace, queries by OData filter or expression, and transactions.
- A change is one step. The index's write lock is held from the first read to the commit, so an ETag check and the write it guards cannot be separated. A transaction that fails leaves the table as it was.
- A query reads every entity of the table and then filters. That suits tables of modest size; it is not an index lookup.
Blobs
- One Lucene
Directoryper container, with a file per blob, written and read through theDirectory's own file API. There is no index. - Supported: upload (with overwrite, If-Match or If-None-Match), download to memory or as a stream, delete, exists, properties with content type and metadata, and listing flat or by hierarchy. Download and delete honour If-Match.
- Containers are not stored. A container is only the place its blobs are in: every container exists, creating one always succeeds, and deleting one deletes its blobs. There is no list of containers, so
GetBlobContainersthrowsNotSupportedException. - Not supported: block, append and page blob operations, leases, tags, snapshots and copies.
OpenRead,DownloadandDownloadToto a file path or with options throwNotSupportedException; useDownloadContent,DownloadStreamingorDownloadTo(Stream). - Dispose the service client to close the directories it opened, as with the table and queue service clients.
Queues
- One Lucene
Directoryper queue, with a file per message and no index. A queue uses only what theDirectoryabstraction gives (list, create, read, delete and sync a file, and its lock), so it works on any LuceneDirectory, not only a folder. Everything about a message that changes is in the name of its file, so the messages to receive are found by listing theDirectory, with no file opened, and a pop receipt names its message's file outright. - First in, first out, in the order the messages were sent by the clock. A message whose visibility timeout runs out goes back to the place it had.
- Receiving lists the queue's Lucene
Directory, so one receive costs more the more messages are waiting. Receiving several messages at a time spreads that cost. - Receiving hides a message from every other receiver until its visibility timeout runs out. Two receivers are never given the same message.
- Pop receipts are checked on delete and update, and change on every receive and update. The receipt returned by a send deletes the message for as long as nobody has received it.
- Time to live defaults to seven days;
TimeSpan.FromSeconds(-1)means never.
Durability and sharing
Every table operation opens its index's writer, does its work, commits and closes the writer. Every queue operation that changes a queue takes the write lock of the queue's Lucene Directory, writes the files it changes through to disk, and lets the lock go. Nothing is lost if the process dies between operations, and several clients and processes can share the same folder.
- A process that dies while changing a message leaves the message as it was, or as it was to become. What it left unfinished is never given out, and is cleared away by the first receive that comes to it; until then it can be counted in
ApproximateMessagesCount. - A process that dies while making a queue can leave it half made: it exists and takes messages, but is not in
GetQueues. The nextCreate,CreateIfNotExistsorSetMetadatafor it finishes making it, and keeps its messages. - Clients in one process take turns. All clients on the same folder write to a table or queue one at a time, in the order they asked, so a busy client cannot shut another out.
- Processes do not queue; they retry. A table or queue can only be written by one process at a time, and Lucene's write lock cannot be waited on, only tried. An operation that finds the table or queue held by another process tries again every few tens of milliseconds for up to ten seconds. That works well when processes write now and then. Under sustained writing from one process, another can lose every try; its call then fails with
RequestFailedException, status503, error codeServerBusy(for a table, theLockObtainFailedExceptionis the inner exception), and can be retried like the same error from Azure. - Receiving from an empty queue takes no lock, so idle receivers do not hold up senders.
- Blobs are not coordinated between clients: two clients or processes writing the same blob at the same moment are not serialised.
Differences from Azure Storage
- No size limits are enforced on entities, blobs or messages, and
maxMessagesis not capped at 32. - For queues, a wrong pop receipt and a missing message both give
404 MessageNotFound. - Message text is stored as given;
QueueClientOptions.MessageEncodingis not applied.
Related Packages
| Package | Description |
|---|---|
| Iciclecreek.Azure.Storage.Memory | Thread-safe in-memory implementation (fastest, no I/O) |
| Iciclecreek.Azure.Storage.FileSystem | Filesystem-backed implementation (files on disk) |
| Iciclecreek.Azure.Storage.SQLite | SQLite-backed implementation (one portable .db file) |
| Iciclecreek.Azure.Storage.Server | ASP.NET Core REST API server on top of any provider |
License
MIT
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 was computed. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Azure.Data.Tables (>= 12.13.0)
- Azure.Storage.Blobs (>= 12.30.1)
- Azure.Storage.Queues (>= 12.28.1)
- Iciclecreek.Azure.Storage.OData (>= 2.1.0-rc001)
- Lucene.Net (>= 4.8.0-beta00018)
- Lucene.Net.Analysis.Common (>= 4.8.0-beta00018)
-
net8.0
- Azure.Data.Tables (>= 12.13.0)
- Azure.Storage.Blobs (>= 12.30.1)
- Azure.Storage.Queues (>= 12.28.1)
- Iciclecreek.Azure.Storage.OData (>= 2.1.0-rc001)
- Lucene.Net (>= 4.8.0-beta00018)
- Lucene.Net.Analysis.Common (>= 4.8.0-beta00018)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 2.1.0-rc001 | 0 | 11/16/2026 |