LumDbEngine 4.0.0

LumDb 4.0.0

A single-file, thread-safe embedded database for .NET 10. 100% C#, AOT-friendly, no native dependencies.

Package 4.0.0. The on-disk page format is still 1.3.9 (DbHeader.VERSION = 1_003_009, packed as major * 1_000_000 + minor * 1_000 + patch). DbEngine.Version / VersionString report that page format, not the package version. The page layout is unchanged, so existing 1.3.9 files open as before. Vacuum rebuilds pages in the same format and does not rewrite the format number.

Engines

using var mem = new DbEngine();              // in-memory; same instance shares one committed image
using var disk = new DbEngine(@"d:\app.db"); // file + WAL

using (var ts = mem.StartTransaction())
{
    ts.Create<Student>("students");
    ts.Insert("students", new Student { Name = "lj", Age = 20 });
} // Dispose commits. The next StartTransaction on this engine sees the data.

mem.SaveTo(@"d:\app.db");                    // dump committed image (not uncommitted dirty pages)

SaveChanges / Dispose commit. Discard rolls back the current transaction and reloads from the committed image. A following Dispose does not overwrite the image with an empty cache.

Transactions, errors and limits

• A statement that throws aborts the whole transaction: later statements and SaveChanges() throw TransactionAborted (the message keeps the first failure), and Dispose writes nothing. Discard() rolls back to the committed image and makes the same transaction usable again.
• A failed commit (disk full, .log locked, handle lost) does not abort the transaction. Dirty pages stay in memory, and SaveChanges() can be retried once the disk problem is gone. The .log is kept when replay fails, so the next open can replay it.
• One writer at a time plus concurrent readonly transactions. A thread that already holds any live transaction cannot start a write transaction (IllegaTransaction); readonly transactions may nest.
• At most 32 transactions per engine. The 33rd fails immediately with TooManyOpenTransactions (limit and current count in the message). Set TransactionPoolWaitMilliseconds to wait for a slot instead.
• Serializable is not implemented: assigning it to TransactionPolicy is accepted, starting a transaction then throws NotSupportedException. Unknown enum values throw ArgumentOutOfRangeException.
• DbEngine.Dispose waits TimeoutMilliseconds (default 3000) for live transactions; on timeout it throws and the engine stays usable. Disposing a transaction from another thread is supported.

Files and versions

Opening a database validates the header: length at least 100 bytes, the LumDBS signature, and page format exactly 1.3.9. Files from other page formats are rejected with UnsupportedFileVersion instead of being read with the wrong layout; foreign or truncated files get InvalidDatabaseFile. Export pre-1.3.9 data with the matching older package.

Upgrading from 3.0.1

4.0.0 is a major bump: one public member was removed, and two failure paths now throw where they used to return.

• Failed statements now abort the transaction (TransactionAborted, Discard() to reuse) instead of letting SaveChanges() commit an empty/partial image.
• Header validation rejects foreign files and page formats other than 1.3.9.
• DbEngine.ReadWriteLock was removed (it was an implementation detail); use StartTransaction / StartTransactionReadonly for concurrency.

LumEntity (source generator)

Mark a partial class. Build generates ILumEntity<T> (works without PublishAOT — generation is compile-time).

[LumEntity]
public partial class Student
{
    [Id] public uint Id { get; set; }          // engine auto row id, not a column
    [Key] [Str32B] public string Name { get; set; } = "";
    public int Age { get; set; }
    public bool Active { get; set; }
}

Unsupported nested members on [LumEntity] are skipped with warning LUMDB002. [Ignore] is silent. DateTime columns are UTC (DateTimeKind.Utc).

Tuple / DbCell API (no entity type):

ts.Create("kv", [("k", DbValueType.Int, true), ("v", DbValueType.StrVar, false)]);
ts.Insert("kv", [("k", 1), ("v", "hello")]);
var row = ts.Find("kv", "k", 1);
Console.WriteLine(row.Row.GetString(1));

Query (LINQ)

Where / Join / Skip / Take only build a plan (no I/O). ToList / Count / Delete / Exists run the whole plan under one transaction lock. The thread that called Query() must consume that instance; other threads call Query() themselves.

Scan uses RowView (no entity new until the result window). Count and WhereExists do not materialize inner rows.

using (var ts = eng.StartTransaction())
{
    var page = ts.Query<Student>("students")
        .Where(s => s.Active && s.Name.Trim().ToUpper().StartsWith("LJ"))
        .OrderByDescending(s => s.Age)
        .Skip(0)
        .Take(20)
        .ToList();

    var n = ts.Query<Student>("students").Where(s => s.Age >= 18).Count();

    var pairs = ts.Query<Student>("students")
        .Join<Order>("orders", (s, o) => s.Id == o.StudentId && o.Status == 1)
        .Where((s, o) => o.Amount >= 10)
        .ToList(); // List<(Student Outer, Order Inner)>

    var withOrders = ts.Query<Student>("students")
        .WhereExists<Order>("orders", (s, o) => s.Id == o.StudentId)
        .ToList();
}

Hand-written IDbEntity: Query_Entity<T> / JoinEntity. Three-table: .Join<A>(...).Join<B>(...) → List<(T1 A, T2 B, T3 C)>.

Expressions

Translated to a RowView tree (AOT: no Expression.Compile of the whole lambda):

• Comparisons == != < <= > >=, logic && || !, arithmetic + - * / %
• String (Ordinal / Invariant, one-argument overloads): StartsWith / EndsWith / Contains / Equals, ToUpper / ToLower (and *Invariant), Trim / TrimStart / TrimEnd, Substring, Replace, IndexOf / LastIndexOf, Length, IsNullOrEmpty / IsNullOrWhiteSpace, ToString()
• Math: Abs Sign Min Max Clamp Floor Ceiling Truncate Round Sqrt Pow Sin Cos Tan Log Log10 Exp
• Nesting is allowed: s.Name.Trim().ToUpper().Contains("LJ"), Math.Abs(Add(s.Age, 0))

Your own methods and captured Func (any TResult, not void / Action) are invoked per matching row:

static int Add(int a, int b) => a + b;
static string Fold(string s) => s.Trim().ToUpperInvariant();

ts.Query<Student>("students").Where(s => Add(s.Age, 1) > 18);
ts.Query<Student>("students").Where(s => Fold(s.Name).Contains("LJ"));

Func<string, bool> rx = n => System.Text.RegularExpressions.Regex.IsMatch(n, "^lj");
ts.Query<Student>("students").Where(s => rx(s.Name));

C# rejects Action in Where (predicate must be bool). Local functions inside the method cannot appear in expression trees (CS8110). System.* instance APIs that are not in the list (Split, Contains(..., StringComparison), ToUpper(culture), ternary ?:) compile, then throw LumException at ToList / Count.

Zero-alloc predicate: Where((ref RowView row) => row.GetInt(1) > 90). Entity callback (allocates every scanned row): WhereCallback(s => ...).

Performance notes

• Count / Take / Join / WhereExists vs loading every entity (Find all, or skipping by new T()): scan stays on RowView; only the result window is boxed. That is the large win (see BenchLumDb GoThrough vs object scan, entity page vs Find all).
• Whitelist string/Math nesting is the same RowOp path as a simple comparison.
• Captured Func and user MethodInfo.Invoke run managed IL per row — correct, not faster than RowOp.
• Equi-join is hash; inequality join is nested-loop over inner ids (O(N×M) CPU, still no full-table entity alloc). OrderBy on a join materializes all matching ids before Skip/Take.

Features

• Relational tables with optional multi-key secondary indexes
• Types: bool, byte, int/uint, long/ulong, float/double, decimal, DateTime UTC, fixed/var string, fixed/var bytes
• Fluent Query / Join / WhereExists (one lock per terminal)
• Read-committed: a readonly transaction sees the last commit, not another writer’s dirty pages
• Thread-safe: one writer (upgradeable) + concurrent readers
• Source-generated entities; GoThrough / RowView scan without allocating entities
• AOT publish supported (PublishAot); daily dotnet build still runs the generator
• Tools: LumDbExplorer (WinForms) to create/copy databases, create tables, CRUD rows, inspect schema and stats
• AddColumn / DropColumn rewrite one table inside the current transaction (up to 100 columns, the slots already reserved on the table page)
• Vacuum rebuilds the committed image and drops free pages, still in page format 1.3.9. The old file is replaced only after the new image is fully committed

Columns and vacuum

AddColumn / DropColumn run inside a transaction. SaveChanges / Dispose commits them through the same .log as any other write. Discard, or a crash before the log is marked done, leaves the previous committed image. A failed alter reloads that image so a later Dispose cannot store a half-rewritten table.

Column identity is the order on the table page. Inserting or deleting in the middle changes later ordinals. Appending does not. Existing rows take defaultValue, or the zero value of the new type when it is omitted (0, false, "", empty bytes). A new key column is rejected when more than one row would share that default. At least one column must remain. There is no spare room in the row itself: the table is rewritten at the new width, and row ids stay the same.

Vacuum writes a complete new image in page format 1.3.9 while other transactions keep running, then swaps it in when no commit has landed since that snapshot. A commit in between is kept and the rebuild is repeated. A crash before the swap leaves the previous file and its .log untouched. Call it from a thread that does not already hold a transaction.

using (var ts = eng.StartTransaction())
{
    ts.Create("kv", [("k", DbValueType.Int, true), ("v", DbValueType.StrVar, false)]);
    ts.Insert("kv", [("k", 1), ("v", "hello")]);

    ts.AddColumn("kv", "n", DbValueType.Int, false, defaultValue: 0); // append
    ts.AddColumn("kv", "flag", DbValueType.Bool, false, defaultValue: true, ordinal: 1);
    ts.DropColumn("kv", "flag");
    ts.SaveChanges();
}

long before;
using (var ts = eng.StartTransaction())
    before = ts.GetSpace().FileBytes;

eng.Vacuum();

using (var ts = eng.StartTransaction())
{
    var row = ts.Find("kv", "k", 1);
    Console.WriteLine(row.Row.GetInt(0));    // 1
    Console.WriteLine(row.Row.GetString(1)); // hello
    Console.WriteLine(row.Row.GetInt(2));    // 0, the appended column
    Console.WriteLine(ts.GetSpace().FileBytes <= before);
}

Getting started

1. Reference LumDbEngine (net10). For [LumEntity], the analyzer is packed with the NuGet.
2. new DbEngine() or new DbEngine(path).
3. using var ts = engine.StartTransaction(); — dispose to commit, or Discard().

License

MIT. See LICENSE.txt.