WALnutDB 1.0.6

WALnutDB

A simple, safe embedded database for .NET 8+. Built for devices with little RAM and slow flash (SD/eMMC). It uses Write-Ahead Logging (WAL) for power-loss durability, compacts data into sorted segment files (SST), and provides secondary indexes and time-series scans. Fully managed (no native deps), cross-platform, and async by default.

Highlights

• Power-loss safety: WAL with per-frame CRC, truncation-tolerant replay.
• Async I/O everywhere: designed for slow media and small RAM.
• Document tables: pluggable serialization (e.g., System.Text.Json); GUID/string/byte[] keys.
• Secondary indexes: range scans over string/bytes/ints/floats/decimal (with scale).
• Time-series mode: UTC-ordered keys, efficient range scans by time.
• Checkpoint & SST: flush memtables to sorted segments, fast reads, WAL truncation.
• Cross-platform: Linux/Windows/macOS; file-per-table for fault isolation.
• No native dependencies: pure C# for easy deployment on IoT.

Roadmap: online compaction/defrag stats & swap, unique index enforcement, optional encryption-at-rest, richer query push-downs.

Install

dotnet add package WALnutDB

Target framework: .NET 8.0+

Quick Start

using System.Text.Json;
using WalnutDb;
using WalnutDb.Core;
using WalnutDb.Wal;

// Sample POCO
public sealed class User
{
    [DatabaseObjectId] public string Id { get; set; } = "";
    [DbIndex("Age")]   public int Age { get; set; }
    public string Name { get; set; } = "";
}

// Create database directory
var dir = Path.Combine(Path.GetTempPath(), "walnut-demo");
Directory.CreateDirectory(dir);

// WAL writer (group-commit, fsync per batch)
await using var wal = new WalWriter(Path.Combine(dir, "wal.log"));

// DB (managed, no native deps)
await using var db = new WalnutDatabase(
    directory: dir,
    options: new DatabaseOptions(),
    manifest: new FileSystemManifestStore(dir),
    wal: wal
);

// Open a table (default serializer is used unless you override it)
var users = await db.OpenTableAsync(new TableOptions<User>
{
    GetId = u => u.Id, // string / Guid / byte[] supported
    // Optional custom serializer:
    //Serialize   = u => JsonSerializer.SerializeToUtf8Bytes(u),
    //Deserialize = b => JsonSerializer.Deserialize<User>(b.Span)!,
    StoreGuidStringsAsBinary = true // optional optimization
});

// Upsert + Get
await users.UpsertAsync(new User { Id = "u1", Name = "Ada", Age = 37 });
var ada = await users.GetAsync("u1");
Console.WriteLine($"{ada?.Id}: {ada?.Name} ({ada?.Age})");

// Enumerate all
await foreach (var u in users.GetAllAsync())
    Console.WriteLine($"{u.Id}: {u.Name} ({u.Age})");

// Durability checkpoint (flush mem→SST, truncate WAL)
await db.CheckpointAsync();

Secondary Indexes

Declare indexes:

public sealed class Product
{
    [DatabaseObjectId] public string Id { get; set; } = "";

    [DbIndex("Category")]
    public string Category { get; set; } = "";

    [DbIndex("Price", decimalScale: 2)] // decimals require a fixed scale
    public decimal Price { get; set; }
}

Open table and scan ranges (decimal + string prefix):

using System.Text.Json;
using WalnutDb.Indexing;

var products = await db.OpenTableAsync(new TableOptions<Product> {
    GetId = p => p.Id,
    // Optional custom serializer:
    //Serialize   = p => JsonSerializer.SerializeToUtf8Bytes(p),
    //Deserialize = b => JsonSerializer.Deserialize<Product>(b.Span)!
});

// Insert a few
await products.UpsertAsync(new Product { Id = "A", Category = "sensors",   Price = 12.50m });
await products.UpsertAsync(new Product { Id = "B", Category = "sensors",   Price = 18.75m });
await products.UpsertAsync(new Product { Id = "C", Category = "actuators", Price = 33.40m });

// Range by decimal index (inclusive/exclusive semantics)
var from = IndexKeyCodec.Encode(10m, decimalScale: 2);
var to   = IndexKeyCodec.Encode(20m, decimalScale: 2);

await foreach (var p in products.ScanByIndexAsync("Price", from, to))
    Console.WriteLine($"{p.Id} {p.Price:0.00}");

// Range by string prefix (e.g., category "sensors")
var start = IndexKeyCodec.Encode("sensors");
var end   = IndexKeyCodec.PrefixUpperBound(start);

await foreach (var p in products.ScanByIndexAsync("Category", start, end))
    Console.WriteLine($"{p.Id} {p.Category}");

Index hints

You can steer scans using IndexHint factories. (Descending Asc=false is planned; current implementation returns ascending.)

using WalnutDb.Indexing;

// Price in [10.00, 20.00), ascending, skip first 5, take 10
var hint = IndexHint.FromValues("Price", 10.00m, 20.00m, decimalScale: 2, asc: true, skip: 5, take: 10);
await foreach (var p in products.QueryAsync(x => true, hint))
    Console.WriteLine($"{p.Id} {p.Price:0.00}");

// Prefix by category:
var hint2 = IndexHint.FromPrefix("Category", "sensors");
await foreach (var p in products.QueryAsync(_ => true, hint2))
    Console.WriteLine($"{p.Id} {p.Category}");

Unique indexes

Mark a property with [DbIndex(Name, Unique = true)]. null values do not participate in uniqueness (SQL-like behavior).

public sealed class User
{
    [DatabaseObjectId] public string Id { get; set; } = "";
    [DbIndex("Email", Unique = true)]
    public string? Email { get; set; }
}

// Insert A with email X, then checkpoint
await users.UpsertAsync(new User { Id = "A", Email = "x@example.com" });
await db.CheckpointAsync();

// Inserting B with the same email throws
await Assert.ThrowsAsync<InvalidOperationException>(() =>
    users.UpsertAsync(new User { Id = "B", Email = "x@example.com" }));

// Deleting A releases the unique constraint immediately (in mem)
// and after checkpoint (for SST-backed entries)
await users.DeleteAsync("A");
await db.CheckpointAsync();
await users.UpsertAsync(new User { Id = "B", Email = "x@example.com" });

Time-Series

Provide a series id and a UTC timestamp. Keys are encoded so that lexicographic byte order matches chronological order.

public sealed class SensorSample
{
    [DatabaseObjectId] public string Id { get; set; } = ""; // optional
    public string  DeviceId { get; set; } = "";
    public DateTime Utc     { get; set; }       // must be UTC
    public double  Temperature { get; set; }
}

var ts = await db.OpenTimeSeriesAsync(new TimeSeriesOptions<SensorSample>
{
    GetSeriesId     = s => s.DeviceId,
    GetUtcTimestamp = s => s.Utc,
    // Optional custom serializer:
    //Serialize       = s => JsonSerializer.SerializeToUtf8Bytes(s),
    //Deserialize     = b => JsonSerializer.Deserialize<SensorSample>(b.Span)!,
});

// Append
await ts.AppendAsync(new SensorSample { DeviceId = "dev-1", Utc = DateTime.UtcNow, Temperature = 22.3 });

// Query by time range
var fromUtc = DateTime.UtcNow.AddMinutes(-10);
var toUtc   = DateTime.UtcNow;

await foreach (var s in ts.QueryAsync("dev-1", fromUtc, toUtc))
    Console.WriteLine($"{s.Utc:o} {s.Temperature:0.0}");

Transactions

Both automatic (implicit) and manual (explicit) transactions are supported.

// Auto: each call is an individual transaction
await users.UpsertAsync(new User { Id = "u2", Name = "Linus", Age = 49 });

// Manual: batch multiple ops → single WAL fsync
await using (var tx = await db.BeginTransactionAsync())
{
    await users.UpsertAsync(new User { Id = "u3", Name = "Grace", Age = 44 }, tx);
    await users.DeleteAsync("u1", tx);
    await tx.CommitAsync(Durability.Group);
}

Encryption at rest

WALnutDB can encrypt values at rest (WAL + SST) with AES-GCM(256). Keys and index keys remain plaintext for sortability.

Encrypted: table values written to WAL and SST.
Not encrypted: primary keys, index keys (they include the index value), filenames/metadata.
Threat model: protects against offline reads of WAL/SST files. In-memory data is plaintext.

var dir = Path.Combine(Path.GetTempPath(), "walnut-enc");
await using var wal = new WalWriter(Path.Combine(dir, "wal.log"));

var key = Convert.FromHexString("00112233445566778899AABBCCDDEEFF00112233445566778899AABBCCDDEEFF");

await using var db = new WalnutDatabase(
    dir,
    new DatabaseOptions { Encryption = new AesGcmEncryption(key) },
    new FileSystemManifestStore(dir),
    wal);

var tbl = await db.OpenTableAsync(new TableOptions<MyDoc> { GetId = d => d.Id });
await tbl.UpsertAsync(new MyDoc { Id = "x", Secret = "hello" });
await db.CheckpointAsync(); // values written to SST are ciphertext

Crash-recovery with encryption: recovery verifies crc32 and replays committed frames, decrypting values on the fly before applying them to memtables.

Durability & Checkpoints

• await db.FlushAsync() — fsync WAL (fast).
• await db.CheckpointAsync() — flush memtables → SST and truncate the WAL.
• Recovery reads wal.log, verifies the CRC of each frame, replays committed transactions, and safely stops at a torn tail.

For safe shutdown on embedded devices:

await db.CheckpointAsync();  // shortens recovery time and shrinks the WAL

Preflight Checks

Verify permissions, free space, and ability to acquire exclusive locks:

var report = await db.PreflightAsync(dir, reserveBytes: 4 * 1024 * 1024);
Console.WriteLine($"{report.FileSystem} free={report.FreeBytes} CanExclusive={report.CanExclusiveLock}");

Benchmarks

In this repo there is BenchmarkDotNet (WalnutDb.Bench). Run in Release:

dotnet run -c Release --project WalnutDb.Bench

On-disk formats (v1)

WAL (v1)

Stream of frames:

[len:U32_LE] [payload:len bytes] [crc32:U32_LE]

• crc32 is computed over payload (polynomial 0xEDB88320).
• payload layout varies by op:
  • Begin: [op:1][txId:U64][seqNo:U64]
  • Put: [op:1][txId:U64][tlen:U16][klen:U32][vlen:U32][table:tlen][key:klen][value:vlen]
  • Del: [op:1][txId:U64][tlen:U16][klen:U32][table:tlen][key:klen]
  • Commit:[op:1][txId:U64][opsCount:U32] (reserved)
• Replay stops on the first malformed or truncated frame (“crash tail”).
• If encryption is enabled, values inside WAL frames are encrypted (AES‑GCM v1) with AAD=[table|pk].

SST (v1)

Flat sorted segment file:

Header:  "SSTv1\0\0\0"  (8 bytes)
Records: repeated [klen:U32_LE][vlen:U32_LE][key:klen][value:vlen]
Trailer: [count:U32_LE] (optional in future versions)

• Keys are sorted lexicographically; merges assume sorted input.
• If encryption is enabled, values stored in SST are ciphertext (AES‑GCM v1, same AAD).

Filenames

Each logical table name is encoded to a filesystem‑safe base64‑url (without padding) when stored on disk. Example: orders → b3JkZXJz.

Notes & Limitations

• IndexHint.Asc=false (descending scans) is planned; current implementation returns ascending.
• Unique indexes: null value does not participate in uniqueness.
• One SST per table in v1 (simple model suitable for embedded; background compaction will come later).

License

MIT © WALnutDB contributors