ReadOnlyDataStorage 2.0.1

ReadOnlyDataStorage (RODS)

A write-once, read-many indexed binary data store for .NET 8 with optional compression and encryption.

Features

• Binary format with magic header RODS for fast validation
• MessagePack serialization — compact binary format, ~3–5× smaller and faster than JSON
• Single-property indexes via [Indexable] for fast exact/range lookups
• Composite indexes via [Indexable(CompositeGroup = "...", Order = N)] for multi-property lookups
• Compression: GZip or Brotli at four configurable levels (None / Fastest / Optimal / SmallestSize)
• Encryption: AES-256-GCM with PBKDF2 key derivation (100,000 iterations, SHA-256)
• MemoryMappedFile data section — OS page-cache-backed, zero-copy record reads for uncompressed files
• Parallel serialization / deserialization — RodsWriter serializes records in parallel; RodsReader deserializes GetAllAsync/FindAsync/FindCompositeAsync results in parallel using all available CPU cores
• Index fully loaded into RAM on open — O(1) hash-map lookup per indexed query
• Async API with CancellationToken support throughout

Quickstart

1. Define your model

using ReadOnlyDataStorage;

// Single-property indexes
public class Product
{
    [Indexable(Order = 0)]
    public string Category { get; set; } = string.Empty;

    [Indexable(Order = 1)]
    public string Name { get; set; } = string.Empty;

    [Indexable(Order = 2)]
    public decimal Price { get; set; }

    public string? Description { get; set; }
}

// Composite index: Category+Name together, plus an individual Price index
public class ProductComposite
{
    [Indexable(CompositeGroup = "CategoryName", Order = 0)]
    public string Category { get; set; } = string.Empty;

    [Indexable(CompositeGroup = "CategoryName", Order = 1)]
    public string Name { get; set; } = string.Empty;

    [Indexable]   // single-property index
    public decimal Price { get; set; }
}

2. Write

var writer = new RodsWriter<Product>(new WriteOptions
{
    Compression = RodsCompressionLevel.Optimal,   // None / Fastest / Optimal / SmallestSize
    Provider    = CompressionProvider.GZip,        // GZip (default) or Brotli
    Password    = "optional-password"              // null = no encryption
});

await writer.WriteAsync(products, "products.rods");

3. Read

await using var reader = new RodsReader<Product>("products.rods", password: "optional-password");

// Get all records
var all = await reader.GetAllAsync();

// Find by exact single-property index value
var electronics = await reader.FindAsync(p => p.Category, "Electronics");

// Range query on a single-property index
var midRange = await reader.FindRangeAsync(p => p.Price, 10.0m, 100.0m);

// Count records without loading them
var count = await reader.CountAsync();

4. Composite index query

await using var reader = new RodsReader<ProductComposite>("products.rods");

var result = await reader.FindCompositeAsync(
    groupName: "CategoryName",
    components:
    [
        ("Category", "Electronics"),
        ("Name",     "TV"),
    ]);

[Indexable] Attribute Reference

Note: A property in a CompositeGroup is not available via FindAsync / FindRangeAsync. Use FindCompositeAsync instead.

API Reference

RodsWriter<T>

public sealed class RodsWriter<T>
{
    public RodsWriter(WriteOptions? options = null);

    /// Serializes, compresses, and optionally encrypts all records, then writes the .rods file.
    public Task WriteAsync(IEnumerable<T> records, string filePath, CancellationToken ct = default);
}

WriteOptions

public sealed class WriteOptions
{
    public RodsCompressionLevel Compression { get; set; } = RodsCompressionLevel.Optimal;
    public CompressionProvider  Provider    { get; set; } = CompressionProvider.GZip;
    public string?              Password    { get; set; }   // null = no encryption
}

RodsReader<T> (implements IDisposable, IAsyncDisposable)

public sealed class RodsReader<T> : IDisposable, IAsyncDisposable
{
    public RodsReader(string filePath, string? password = null);

    /// Returns all records in write order.
    public Task<IReadOnlyList<T>> GetAllAsync(CancellationToken ct = default);

    /// Exact match on a single-property index.
    public Task<IReadOnlyList<T>> FindAsync<TKey>(
        Expression<Func<T, TKey>> keySelector, TKey value, CancellationToken ct = default);

    /// Range query [min, max] on a single-property index (TKey must implement IComparable<TKey>).
    public Task<IReadOnlyList<T>> FindRangeAsync<TKey>(
        Expression<Func<T, TKey>> keySelector, TKey min, TKey max, CancellationToken ct = default)
        where TKey : IComparable<TKey>;

    /// Exact match on a composite index by group name.
    public Task<IReadOnlyList<T>> FindCompositeAsync(
        string groupName,
        IEnumerable<(string PropertyName, object? Value)> components,
        CancellationToken ct = default);

    /// Returns the total record count without loading record data.
    public Task<long> CountAsync(CancellationToken ct = default);
}

File Format Specification (v2)

[Header — 68 bytes, little-endian]
  4 bytes  – magic "RODS"
  2 bytes  – version (2)
  1 byte   – flags: bit0=encrypted, bit1=compressed, bit2=brotli(0=gzip)
  1 byte   – compression level (0=none, 1=fastest, 2=optimal, 3=smallest)
  16 bytes – PBKDF2 salt (zeros when not encrypted)
  12 bytes – reserved (zeros)
  8 bytes  – index section absolute offset
  8 bytes  – index section byte length
  8 bytes  – data section absolute offset
  8 bytes  – data section byte length

[Data Section]
  For each record: raw bytes (MessagePack → optional GZip/Brotli → optional AES-256-GCM)
  Encrypted record layout: nonce(12) + GCM-tag(16) + ciphertext

[Index Section]
  4 bytes  – number of index definitions (D)
  For each definition:
    2 bytes  – index name length
    N bytes  – index name (UTF-8)
    1 byte   – number of component property names (C)
    For each component:
      2 bytes  – property name length
      N bytes  – property name (UTF-8)
  4 bytes  – number of records (R)
  For each record:
    8 bytes  – data section absolute offset
    4 bytes  – data byte length
    For each definition (D key values):
      2 bytes  – key value byte length (0 = null)
      N bytes  – key value (UTF-8); composite keys join components with U+001F

Encryption & Security

• Algorithm: AES-256-GCM (authenticated encryption)
• Key derivation: PBKDF2-SHA-256, 100,000 iterations; 32-byte key
• Each record gets a fresh random 96-bit nonce prepended to its ciphertext
• The index section is stored in plaintext — index key values (not field names) are visible without a password
• Wrong password → CryptographicException on first record read (GCM authentication fails)

Performance Comparison: RODS vs SQLite

Measured with 10,000 records in a CI environment (single-core container).
Run dotnet run --project src/ReadOnlyDataStorage.Perf -c Release to reproduce on your hardware,
or run the SqliteComparisonTests.CompareRodsVsSqlite xUnit test for an in-process comparison.

Note: Parallel deserialization (used in GetAllAsync / FindAsync / FindCompositeAsync) provides the most benefit on multi-core hardware. In constrained environments the gains are smaller.

All RODS variants vs all SQLite variants

All times in milliseconds · File KB = on-disk size

Head-to-head: RODS (No Compression) vs SQLite (default)

Head-to-head: RODS (GZip Optimal) vs SQLite (WAL, 16 KB pages)

When to choose RODS over SQLite

Performance Guidelines

• The entire index is loaded into RAM on RodsReader open. For very large files (millions of records) pre-size appropriately.
• Record data is read via MemoryMappedFile — the OS page cache handles repeated reads efficiently; uncompressed/unencrypted reads use a zero-copy MemoryManager path.
• Deserialization is parallelised across all CPU cores — the speedup scales with Environment.ProcessorCount.
• For highest write throughput use RodsCompressionLevel.Fastest; for smallest files use SmallestSize.
• Brotli generally achieves better compression ratios than GZip at the cost of higher CPU.

Limitations

• Write-once: files cannot be updated after creation. Write a new file and swap atomically.
• No partial index reads: the full index is read into RAM; plan accordingly for huge datasets.
• Encrypted index key values are visible in plain text (only record payloads are encrypted).

Projects

EF Core Provider

ReadOnlyDataStorage.EntityFrameworkCore wraps a RODS file in an Entity Framework Core in-memory database, letting you query your read-only data with the full LINQ power of EF Core.

How it works

1. RodsDbContext<TEntity>.OpenAsync() opens the RODS file and deserialises every record.
2. The records are bulk-inserted into an isolated EF Core in-memory database.
3. The returned context exposes a DbSet<TEntity> that supports all EF Core LINQ operators — Where, Select, OrderBy, GroupBy, Count, FirstOrDefault, etc.

Note on performance: opening the context is slower than opening a bare RodsReader<T> because it includes the full RODS-to-EF-Core load step. Once loaded, individual LINQ queries are evaluated in-memory and are very fast. Use AsNoTracking() on read-only queries to skip EF Core's change-tracker overhead.

Installation

Add a reference to the ReadOnlyDataStorage.EntityFrameworkCore project (or NuGet package):

<PackageReference Include="ReadOnlyDataStorage.EntityFrameworkCore" Version="1.0.0" />

Usage

Basic example

using ReadOnlyDataStorage.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore;

// Open the RODS file and load all records into an EF Core in-memory DB
await using var ctx = await RodsDbContext<Product>.OpenAsync("products.rods");

// Use standard EF Core LINQ
var all = await ctx.Entities.AsNoTracking().ToListAsync();

var electronics = await ctx.Entities
    .AsNoTracking()
    .Where(p => p.Category == "Electronics")
    .OrderBy(p => p.Price)
    .ToListAsync();

var count = await ctx.Entities.CountAsync();
var cheapest = await ctx.Entities.MinAsync(p => p.Price);

Encrypted files

await using var ctx = await RodsDbContext<Product>.OpenAsync(
    "products.rods",
    password: "my-secret-password");

Custom model configuration

If your entity type has computed alias properties (e.g. proxies that write to the same backing field as another property), use configureModel to tell EF Core to ignore them:

await using var ctx = await RodsDbContext<MyEntity>.OpenAsync(
    "data.rods",
    configureModel: mb => mb.Entity<MyEntity>()
        .Ignore(e => e.AliasProperty));

Injecting via DI

Because RodsDbContext<TEntity> is an ordinary DbContext, it can participate in .NET's DI container. The recommended pattern is to use the OpenAsync factory method in a hosted service or application startup, then register the resulting context as a singleton:

// In your startup / background service
var ctx = await RodsDbContext<Product>.OpenAsync("products.rods");
builder.Services.AddSingleton(ctx);

// Elsewhere in the app
public class ProductService(RodsDbContext<Product> db)
{
    public Task<List<Product>> GetElectronicsAsync()
        => db.Entities.AsNoTracking()
               .Where(p => p.Category == "Electronics")
               .ToListAsync();
}

License

MIT