BLite 1.4.0

⚡ BLite

High-Performance BSON Database Engine for .NET 10

BLite is an embedded, ACID-compliant, document-oriented database built from scratch for maximum performance and zero allocation. It leverages modern .NET features like Span<T>, Memory<T>, and Source Generators to eliminate runtime overhead.

Note: Currently targets .NET 10 to maximize performance with Span<T> and modern hardware intrinsics. Future support for .netstandard2.1 is being evaluated.

🚀 Why BLite?

Most embedded databases for .NET are either wrappers around C libraries (SQLite, RocksDB) or legacy C# codebases burdened by heavy GC pressure.

BLite is different:

• Zero Allocation: I/O and interaction paths use Span<byte> and stackalloc. No heap allocations for reads/writes.
• Type-Safe: No reflection. All serialization code is generated at compile-time.
• Developer Experience: Full LINQ provider (IQueryable) that feels like Entity Framework but runs on bare metal.
• Reliable: Full ACID transactions with Write-Ahead Logging (WAL) and Snapshot Isolation.

✨ Key Features

🚄 Zero-Allocation Architecture

• Span-based I/O: The entire pipeline, from disk to user objects, utilizes Span<T> to avoid copying memory.
• Memory-Mapped Files: OS-level paging and caching for blazing fast access.

🧠 Powerful Query Engine (LINQ)

Write queries naturally using LINQ. The engine automatically translates them to optimized B-Tree lookups.

// Automatic Index Usage
var users = collection.AsQueryable()
    .Where(x => x.Age > 25 && x.Name.StartsWith("A"))
    .OrderBy(x => x.Age)
    .Take(10)
    .AsEnumerable(); // Executed efficiently on the engine

• Optimized: Uses B-Tree indexes for =, >, <, Between, and StartsWith.
• Hybrid Execution: Combines storage-level optimization with in-memory LINQ to Objects.
• Advanced Features: Full support for GroupBy, Join, Select (including anonymous types), and Aggregations (Count, Sum, Min, Max, Average).

🔍 Advanced Indexing

• B-Tree Indexes: Logarithmic time complexity for lookups.
• Composite Indexes: Support for multi-column keys.
• Vector Search (HNSW): Fast similarity search for AI embeddings using Hierarchical Navigable Small World algorithm.

🤖 AI-Ready Vector Search

BLite natively supports vector embeddings and fast similarity search.

// 1. Configure vector index on float[] property
modelBuilder.Entity<VectorItem>()
    .HasVectorIndex(x => x.Embedding, dimensions: 1536, metric: VectorMetric.Cosine);

// 2. Perform fast similarity search
var results = db.Items.AsQueryable()
    .VectorSearch(x => x.Embedding, queryVector, k: 5)
    .ToList();

🌍 High-Performance Geospatial Indexing

BLite features a built-in R-Tree implementation for lightning-fast proximity and bounding box searches.

• Zero-Allocation: Uses coordinate tuples (double, double) and Span-based BSON arrays.
• LINQ Integrated: Search naturally using .Near() and .Within().

// 1. Configure spatial index (uses R-Tree internally)
modelBuilder.Entity<Store>()
    .HasSpatialIndex(x => x.Location);

// 2. Proximity Search (Find stores within 5km)
var stores = db.Stores.AsQueryable()
    .Where(s => s.Location.Near((45.4642, 9.1899), 5.0))
    .ToList();

// 3. Bounding Box Search
var area = db.Stores.AsQueryable()
    .Where(s => s.Location.Within((45.0, 9.0), (46.0, 10.0)))
    .ToList();

🆔 Custom ID Converters (ValueObjects)

Native support for custom primary key types using ValueConverter<TModel, TProvider>. Configure them easily via the Fluent API.

// 1. Define your ValueObject and Converter
public record OrderId(string Value);
public class OrderIdConverter : ValueConverter<OrderId, string> { ... }

// 2. Configure in OnModelCreating
modelBuilder.Entity<Order>()
    .Property(x => x.Id)
    .HasConversion<OrderIdConverter>();

// 3. Use it naturally
var order = collection.FindById(new OrderId("ORD-123"));

📡 Change Data Capture (CDC)

Real-time event streaming for database changes with transactional consistency.

• Zero-Allocation: Events are only captured when watchers exist; no overhead when disabled.
• Transactional: Events fire only after successful commit, never on rollback.
• Scalable: Uses Channel-per-subscriber architecture to support thousands of concurrent listeners.

// Watch for changes in a collection
using var subscription = db.People.Watch(capturePayload: true)
    .Subscribe(e => 
    {
        Console.WriteLine($"{e.Type}: {e.DocumentId}");
        if (e.Entity != null) 
            Console.WriteLine($"  Name: {e.Entity.Name}");
    });

// Perform operations - events fire after commit
db.People.Insert(new Person { Id = 1, Name = "Alice" });

🛡️ Transactions & ACID

• Atomic: Multi-document transactions.
• Durable: WAL ensures data safety even in power loss.
• Isolated: Snapshot isolation allowing concurrent readers and writers.
• Thread-Safe: Protected with SemaphoreSlim to prevent race conditions in concurrent scenarios.
• Async-First: Full async/await support across reads, writes, and transactions — with proper CancellationToken propagation throughout the entire stack (B-Tree traversal → page I/O → RandomAccess.ReadAsync on OS level).
• Implicit Transactions: Use SaveChanges() / SaveChangesAsync() for automatic transaction management (like EF Core).

⚡ Async Read Operations

All read paths have a true async counterpart — cancellation is propagated all the way down to OS-level RandomAccess.ReadAsync (IOCP on Windows).

// FindById — async primary-key lookup via B-Tree
var order = await db.Orders.FindByIdAsync(id, ct);

// FindAll — async streaming (IAsyncEnumerable)
await foreach (var order in db.Orders.FindAllAsync(ct))
    Process(order);

// LINQ — full async materialisation
var shipped = await db.Orders
    .AsQueryable()
    .Where(o => o.Status == "shipped")
    .ToListAsync(ct);

// Async aggregates
int count = await db.Orders.AsQueryable().CountAsync(ct);
bool any  = await db.Orders.AsQueryable().AnyAsync(o => o.Total > 500, ct);
bool all  = await db.Orders.AsQueryable().AllAsync(o => o.Currency == "EUR", ct);

// First/Single helpers
var first  = await db.Orders.AsQueryable().FirstOrDefaultAsync(o => o.Status == "pending", ct);
var single = await db.Orders.AsQueryable().SingleOrDefaultAsync(o => o.Id == id, ct);

// Materialise to array
var arr = await db.Orders.AsQueryable().ToArrayAsync(ct);

// SaveChanges is also async
await db.SaveChangesAsync(ct);

Available async read methods on DocumentCollection<TId, T>:

🔌 Intelligent Source Generation

• Zero Reflection: Mappers are generated at compile-time for zero overhead.
• Nested Objects & Collections: Full support for complex graphs, deep nesting, and ref struct handling.
• Robust Serialization: Correctly handles nested objects, collections, and complex type hierarchies.
• Lowercase Policy: BSON keys are automatically persisted as lowercase for consistency.
• Custom Overrides: Use [BsonProperty] or [JsonPropertyName] for manual field naming.

✅ Supported Scenarios

The source generator handles a wide range of modern C# patterns:

❌ Limitations & Design Choices

💡 Best Practice: For relationships between entities, prefer referencing (storing ObjectIds) over embedding (full nested objects) to avoid data duplication and maintain consistency. See tests in CircularReferenceTests.cs for implementation patterns.

🏷️ Supported Attributes

BLite supports standard .NET Data Annotations for mapping and validation:

📚 Documentation

For in-depth technical details, see the complete specification documents:

• RFC.md - Full architectural specification covering storage engine, indexing, transactions, WAL protocol, and query processing
• C-BSON.md - Detailed wire format specification for BLite's Compressed BSON format, including hex dumps and performance analysis

📦 Quick Start

1. Installation

dotnet add package BLite

2. Basic Usage

// 1. Define your Entities
public class User 
{ 
    public ObjectId Id { get; set; } 
    public string Name { get; set; } 
}

// 2. Define your DbContext (Source Generator will produce InitializeCollections)
public partial class MyDbContext : DocumentDbContext
{
    public DocumentCollection<ObjectId, User> Users { get; set; } = null!;

    public MyDbContext(string path) : base(path) 
    {
        InitializeCollections();
    }
}

// 3. Use with Implicit Transactions (Recommended)
using var db = new MyDbContext("mydb.db");

// Operations are tracked automatically
db.Users.Insert(new User { Name = "Alice" });
db.Users.Insert(new User { Name = "Bob" });

// Commit all changes at once
db.SaveChanges();

// 4. Query naturally with LINQ
var results = db.Users.AsQueryable()
    .Where(u => u.Name.StartsWith("A"))
    .AsEnumerable();

// 5. Or use explicit transactions for fine-grained control
using (var txn = db.BeginTransaction())
{
    db.Users.Insert(new User { Name = "Charlie" });
    txn.Commit(); // Explicit commit
}

� Schema-less API (BLiteEngine / DynamicCollection)

When compile-time types are not available — server-side query processing, scripting, migrations, or interop scenarios — BLite exposes a fully schema-less BSON API via BLiteEngine and DynamicCollection.

Both paths share the same kernel: StorageEngine, B-Tree, WAL, Vector / Spatial indexes.

Entry Point

using var engine = new BLiteEngine("data.db");

// Open (or create) a schema-less collection
var orders = engine.GetOrCreateCollection("orders", BsonIdType.ObjectId);

// List all collections
IReadOnlyList<string> names = engine.ListCollections();

// Drop a collection
engine.DropCollection("orders");

Insert

// Build a BsonDocument using the engine's field-name dictionary
var doc = orders.CreateDocument(
    ["status", "total", "currency"],
    b => b
        .Set("status",   "pending")
        .Set("total",    199.99)
        .Set("currency", "EUR"));

BsonId id = orders.Insert(doc);

// Async variant
BsonId id = await engine.InsertAsync("orders", doc, ct);

Read

// Primary-key lookup
BsonDocument? doc = orders.FindById(id);
BsonDocument? doc = await orders.FindByIdAsync(id, ct);

// Full scan
foreach (var d in orders.FindAll()) { ... }
await foreach (var d in orders.FindAllAsync(ct)) { ... }

// Zero-copy predicate scan (BsonSpanReader — no heap allocation per document)
var pending = orders.Scan(reader =>
{
    // Read "status" field directly from the BSON bytes
    if (reader.TryReadString("status", out var status))
        return status == "shipped";
    return false;
});

// B-Tree range query on a secondary index
var recent = orders.QueryIndex("idx_placed_at", minDate, maxDate);

// Vector similarity search
var similar = orders.VectorSearch("idx_embedding", queryVector, k: 10);

// Geospatial proximity / bounding box
var nearby = orders.Near("idx_location", (45.46, 9.18), radiusKm: 5.0);
var inArea  = orders.Within("idx_location", (45.0, 9.0), (46.0, 10.0));

// Count
int total = orders.Count();

Update & Delete

bool updated = orders.Update(id, newDoc);
bool deleted = orders.Delete(id);

// or via engine shortcuts (async)
await engine.UpdateAsync("orders", id, newDoc, ct);
await engine.DeleteAsync("orders", id, ct);

Index Management

// B-Tree secondary index
orders.CreateIndex("status");                         // default name = "idx_status"
orders.CreateIndex("placed_at", unique: false);

// Unique index
orders.CreateIndex("order_number", unique: true);

// Vector index (HNSW)
orders.CreateVectorIndex("embedding", dimensions: 1536, metric: VectorMetric.Cosine);

// Spatial index (R-Tree)
orders.CreateSpatialIndex("location");

// Introspect
IReadOnlyList<string> indexes = orders.ListIndexes();

// Drop
orders.DropIndex("idx_status");

Reading BsonDocument fields

BsonDocument? doc = orders.FindById(id);
if (doc is not null)
{
    string status   = doc.GetString("status");
    double total    = doc.GetDouble("total");
    BsonId docId    = doc.Id;
}

When to use which API

�🗺️ Roadmap & Status

We are actively building the core. Here is where we stand:

• ✅ Core Storage: Paged I/O, WAL, Transactions with thread-safe concurrent access.
• ✅ BSON Engine: Zero-copy Reader/Writer with lowercase policy.
• ✅ Indexing: B-Tree implementation.
• ✅ Vector Search: HNSW implementation for Similarity Search.
• ✅ Geospatial Indexing: Optimized R-Tree with zero-allocation tuple API.
• ✅ Query Engine: Hybrid execution (Index/Scan + LINQ to Objects).
• ✅ Advanced LINQ: GroupBy, Joins, Aggregations, Complex Projections.
• ✅ Async I/O: True async reads and writes — FindByIdAsync, FindAllAsync (IAsyncEnumerable<T>), ToListAsync/ToArrayAsync/CountAsync/AnyAsync/AllAsync/FirstOrDefaultAsync/SingleOrDefaultAsync for LINQ pipelines, SaveChangesAsync. CancellationToken propagates to RandomAccess.ReadAsync (IOCP on Windows).
• ✅ Source Generators: Auto-map POCO/DDD classes with robust nested objects, collections, and ref struct support.

🔮 Future Vision

1. Advanced Querying & Specialized Indices

• Graph Traversals:
  • Specialized index for "links" (Document IDs) for $O(1)$ navigation without full scans.

2. CDC & Event Integration

• BSON Change Stream: "Log Miner" that decodes WAL entries and emits structured events.
• Internal Dispatcher: Keeps specialized indices updated automatically via CDC.

3. Performance & Optimization

• Projection Engine: Read only specific fields from disk (via BSON offsets) without full document deserialization.
• Portability: Evaluate .netstandard2.1 support for broader compatibility (Unity, MAUI, etc.).

🤝 Contributing

We welcome contributions! This is a great project to learn about database internals, B-Trees, and high-performance .NET.

How to Build

1. Clone: git clone https://github.com/mrdevrobot/BLite.git
2. Build: dotnet build
3. Test: dotnet test (We have comprehensive tests for Storage, Indexing, and LINQ).

Areas to Contribute

• Missing LINQ Operators: Help us implement additional IQueryable functions.
• Benchmarks: Help us prove BLite is faster than the competition.
• Documentation: Examples, Guides, and Wiki.

📝 License

Licensed under the MIT License. Use it freely in personal and commercial projects.