Sop4CS.Demo 2.0.40

SOP for C# (Sop4CS)

Scalable Objects Persistence (SOP) is a high-performance, transactional storage engine for C#, powered by a robust Go backend. It combines the raw speed of direct disk I/O with the reliability of ACID transactions and the flexibility of modern AI data management.

Documentation

• API Cookbook: Common recipes and patterns (Key-Value, Transactions, AI).
• Examples: Complete runnable examples.

Installation

Install the library via NuGet:

dotnet add package Sop4CS

To run the examples and launch the Data Browser, install the global tool:

dotnet tool install -g Sop4CS.Demo

Key Features

• Unified Database: Single entry point for managing Vector, Model, and Key-Value stores.
• Transactional B-Tree Store: Unlimited, persistent B-Tree storage for key-value data.
• Complex Keys: Support for composite keys (structs/classes) with custom index specifications.
• Metadata "Ride-on" Keys: Store metadata directly in the B-Tree key (e.g., timestamps, status flags) to enable high-speed scanning and filtering of millions of records without fetching the heavy value payload. Ideal for "Big Data" management and analytics.
• Vector Database: Built-in vector search (k-NN) for AI embeddings and similarity search.
• Text Search: Transactional, embedded text search engine (BM25).
• AI Model Store: Versioned storage for machine learning models (B-Tree backed).
• ACID Compliance: Full transaction support (Begin, Commit, Rollback) with isolation.
• High Performance: Written in Go with a lightweight C# wrapper (P/Invoke).
• Caching: Integrated Redis-backed L1/L2 caching for speed.
• Replication: Optional Erasure Coding (EC) for fault-tolerant storage across drives.
• Multi-Tenancy: Native support for Cassandra Keyspaces or Directory-based isolation.
• Flexible Deployment: Supports both Standalone (local) and Clustered (distributed) modes.

Performance & Big Data Management

SOP is designed for high-throughput, low-latency scenarios, making it suitable for "Big Data" management on commodity hardware.

• "Ride-on" Metadata: By embedding metadata (like IsDeleted, LastUpdated, Category) directly into the Key struct but excluding it from the index (using IndexSpecification), you can scan millions of keys per second to filter data. This avoids the I/O penalty of fetching the full Value (which might be a large JSON blob or binary file) just to check a status flag.
• Direct I/O: SOP bypasses OS page caches where appropriate to offer consistent, raw disk performance.
• Parallelism: The underlying Go engine utilizes highly concurrent goroutines for managing B-Tree nodes and vector indexes.

Running the Examples

The Sop4CS.Demo tool provides a comprehensive suite of examples covering B-Trees, Vector Search, Model Store, and more.

Once installed as a global tool:

# Run interactive menu
sop-demo

# Run a specific example (e.g., Complex Keys)
sop-demo run 2

# Launch the SOP Data Browser
sop-demo browser

The suite includes:

1. Basic B-Tree: CRUD operations.
2. Complex Keys: Composite keys and anonymous type lookups.
3. Metadata: "Ride-on" keys for high-performance updates.
4. Paging: Forward/Backward navigation.
5. Vector Search: Simulated AI/RAG embedding search.
6. Model Store: Large binary object storage.
7. Logging: Demonstration of the logging capabilities.
8. Batched Operations: High-performance batched inserts/updates.
9. Cassandra Init: Demo of Cassandra-backed initialization.
10. Text Search: Full-text search capabilities.
11. Clustered Database: Distributed database operations (requires Redis).
12. Concurrent Transactions: Multi-threaded transaction handling (requires Redis).
13. Concurrent Transactions (Standalone): Multi-threaded transaction handling (local only).

Data Browser & Full Data Management

SOP includes a powerful Data Browser that provides full data management capabilities for your B-Tree stores. It goes beyond simple viewing, offering a complete GUI for inspecting, searching, and manage your data at scale.

To launch it using the global tool:

sop-demo browser

Key Capabilities

• Full Data Management: Perform comprehensive CRUD (Create, Read, Update, Delete) operations on any record directly from the UI.
• High-Performance Search: Utilizes B-Tree positioning for instant lookups, even in datasets with millions of records. Supports both simple keys and complex composite keys (e.g., searching by Country + City).
• Efficient Navigation: Smart pagination and traversal controls (First, Previous, Next, Last) allow you to browse massive datasets without performance penalties.
• Bulk Operations: Designed for rapid-fire management of records with a clean, non-distracting interface.
• Responsive & Cross-Platform: Works seamlessly across diverse monitor sizes and devices.
• Automatic Setup: The tool automatically downloads the correct binary for your OS/Architecture upon first run.

Usage: By default, it opens on http://localhost:8080. Arguments: You can pass standard flags, e.g., sop-demo browser -- -port 9090 -registry ./my_data.

Generating Sample Data

To see the Data Browser in action, you can generate a sample database with complex keys using the included example:

1. Run the generator:

   sop-demo run 14
   

   This will create a database in sop_data_complex (or similar path defined in the example) with two stores: people (Complex Key) and products (Composite Key).

2. Open in Browser:

   sop-demo browser -- -registry sop_data_complex
   

Prerequisites

• Redis: Required for caching and transaction coordination (especially in Clustered mode). Note: Redis is NOT used for data storage, just for coordination & to offer built-in caching.
• Storage: Local disk space (supports multiple drives/folders).
• OS: macOS, Linux, or Windows.
  • Architectures: x64 (AMD64/Intel64) and ARM64 (Apple Silicon/Linux aarch64).
• .NET SDK: .NET 10.0 or later.

Installation

1. Build the Go Bridge: From the repository root:

   go build -buildmode=c-shared -o bindings/csharp/Sop.Examples/bin/Debug/net10.0/libjsondb.dylib ./bindings/main/...
   # Note: Adjust the output path and extension (.so for Linux, .dll for Windows) as needed.
   

2. Add Reference: Add the Sop project to your solution or reference the compiled assembly.

3. Native Library: Ensure the compiled libjsondb (dylib/so/dll) is in your application's output directory (e.g., bin/Debug/net10.0/).

Quick Start Guide

SOP uses a unified Database object to manage all types of stores. All operations are performed within a Transaction.

1. Initialize Database & Context

First, create a Context and open a Database connection.

using Sop;
using System.Collections.Generic;

// Initialize Context
using var ctx = new Context();

// Open Database (Standalone Mode)
var dbOpts = new DatabaseOptions 
{ 
    StoresFolders = new List<string> { "./sop_data" },
    Type = (int)DatabaseType.Standalone
};

var db = new Database(dbOpts);

2. Start a Transaction

All data operations (Create, Read, Update, Delete) must happen within a transaction.

// Begin a transaction
var trans = db.BeginTransaction(ctx);
try
{
    // --- 3. Vector Store (AI) ---
    // Open a Vector Store named "products"
    var vectorStore = db.OpenVectorStore(ctx, "products", trans);
    
    // Upsert a Vector Item
    vectorStore.Upsert(new VectorItem 
    { 
        Id = "prod_101", 
        Vector = new float[] { 0.1f, 0.5f, 0.9f },
        Payload = new Dictionary<string, object> { { "name", "Laptop" }, { "price", 999 } }
    });

    // --- 4. Model Store (AI) ---
    // Open a Model Store named "classifiers"
    var modelStore = db.OpenModelStore(ctx, "classifiers", trans);
    
    // Save a Model (any serializable object)
    modelStore.Save("churn", "v1.0", new { Algorithm = "random_forest", Trees = 100 });

    // --- 5. B-Tree Store (Key-Value) ---
    // Open/Create a B-Tree named "users"
    var btree = db.NewBtree<string, string>(ctx, "users", trans);
    
    // Add a Key-Value pair
    btree.Add(ctx, new Item<string, string>("user_123", "John Doe"));
    
    // Find a value
    if (btree.Find(ctx, "user_123"))
    {
        var items = btree.GetValues(ctx, "user_123");
        Console.WriteLine($"Found User: {items[0].Value}");
    }

    // --- 6. Complex Keys & Index Specification ---
    // Define a composite key structure
    public class EmployeeKey
    {
        public string Region { get; set; }
        public string Department { get; set; }
        public int Id { get; set; }
    }

    // Define Index Specification
    // This enables fast prefix scans (e.g., "Get all employees in US")
    var indexSpec = new IndexSpecification
    {
        IndexFields = new List<IndexFieldSpecification>
        {
            new IndexFieldSpecification { FieldName = "Region", AscendingSortOrder = true },
            new IndexFieldSpecification { FieldName = "Department", AscendingSortOrder = true },
            new IndexFieldSpecification { FieldName = "Id", AscendingSortOrder = true }
        }
    };

    var empOpts = new BtreeOptions("employees") { IndexSpecification = indexSpec };
    var employees = db.NewBtree<EmployeeKey, string>(ctx, "employees", trans, empOpts);

    employees.Add(ctx, new Item<EmployeeKey, string>(
        new EmployeeKey { Region = "US", Department = "Sales", Id = 101 }, 
        "Alice"
    ));

    // --- 7. Metadata "Ride-on" Keys (UpdateCurrentKey) ---
    // Efficiently update metadata embedded in the key without fetching/writing the value.
    if (employees.Find(ctx, new EmployeeKey { Region = "US", Department = "Sales", Id = 101 }))
    {
        var currentItem = employees.GetCurrentKey(ctx);
        // Update metadata (e.g. promote employee, change status)
        // Note: In a real scenario, you'd likely have a mutable field in the key.
        // This operation is very fast as it avoids value I/O.
        employees.UpdateCurrentKey(ctx, currentItem);
    }

    // --- 8. Simplified Lookup (Anonymous Types) ---
    // You can search using an anonymous object that matches the key structure.
    // This is useful if you don't have the original Key class definition.
    
    // Open existing B-Tree using 'object' as the key type
    var employeesSimple = db.OpenBtree<object, string>(ctx, "employees", trans);
    
    // Search using an anonymous type
    var searchKey = new { Region = "US", Department = "Sales", Id = 101 };
    
    if (employeesSimple.Find(ctx, searchKey))
    {
        var values = employeesSimple.GetValues(ctx, searchKey);
        Console.WriteLine($"Found Alice using anonymous object: {values[0].Value}");
    }

    // --- 9. Paging Navigation ---
    // Efficiently page through keys (metadata) without fetching values.
    var pagingInfo = new PagingInfo 
    { 
        PageSize = 50, 
        PageOffset = 0 
    };
    
    // Get first page of keys
    var keys = employees.GetKeys(ctx, pagingInfo);
    foreach (var item in keys)
    {
        Console.WriteLine($"Employee: {item.Key.Region}/{item.Key.Department}/{item.Key.Id}");
    }

    // --- 10. Text Search ---
    var idx = db.OpenSearch(ctx, "articles", trans);
    idx.Add("doc1", "The quick brown fox");

    // --- 11. Batched Operations ---
    // Add multiple items in a single call for better performance
    var batchItems = new List<Item<string, string>>
    {
        new Item<string, string>("k1", "v1"),
        new Item<string, string>("k2", "v2")
    };
    btree.Add(ctx, batchItems);

    // Commit the transaction
    trans.Commit();
}
catch
{
    trans.Rollback();
    throw;
}

3. Querying Data (Read-Only)

using var trans = db.BeginTransaction(ctx, mode: TransactionMode.ForReading);
try
{
    // --- Vector Search ---
    var vs = db.OpenVectorStore(ctx, "products", trans);
    var hits = vs.Query(new float[] { 0.1f, 0.5f, 0.8f }, k: 5);
    foreach (var hit in hits)
    {
        Console.WriteLine($"Match: {hit.Id}, Score: {hit.Score}");
    }

    // --- Text Search ---
    var idx = db.OpenSearch(ctx, "articles", trans);
    var results = idx.SearchQuery("fox");
    foreach (var res in results)
    {
        Console.WriteLine($"Doc: {res.DocID}, Score: {res.Score}");
    }

    // --- Model Retrieval ---
    var ms = db.OpenModelStore(ctx, "classifiers", trans);
    var model = ms.Load<dynamic>("churn", "v1.0");
    
    trans.Commit();
}
catch
{
    trans.Rollback();
}

Advanced Configuration

Logging

Configure the global logger to output to a file or stderr.

// Log to a file
Logger.Configure(LogLevel.Debug, "sop.log");

// Log to stderr (default)
Logger.Configure(LogLevel.Info, "");

Redis Connection

Initialize the shared Redis connection for caching and coordination.

Redis.Initialize("redis://localhost:6379");

// ... perform operations ...

Redis.Close();

Cassandra Connection

Initialize the shared Cassandra connection for multi-tenant storage.

var config = new CassandraConfig
{
    ClusterHosts = new List<string> { "localhost" },
    Consistency = 1,
    ReplicationClause = "{'class':'SimpleStrategy', 'replication_factor':1}"
};

Cassandra.Initialize(config);

// ... perform operations ...

Cassandra.Close();

Clustered Database

Create a clustered database with Erasure Coding support and Cassandra Keyspace.

var dbOpts = new DatabaseOptions 
{ 
    StoresFolders = new List<string> { "/mnt/data1", "/mnt/data2" },
    Type = (int)DatabaseType.Clustered,
    Keyspace = "my_tenant_keyspace",
    ErasureConfig = new Dictionary<string, ErasureCodingConfig>
    {
        { "default", new ErasureCodingConfig { DataShards = 2, ParityShards = 1 } }
    }
};

var db = new Database(dbOpts);

Concurrent Transactions Example

SOP supports concurrent access from multiple threads or processes. The library handles conflict detection and merging automatically.

Important: Pre-seed the B-Tree with at least one item in a separate transaction before launching concurrent workers.

Note: This requirement is simply to have at least one item in the tree. It can be a real application item or a dummy seed item.

using Sop;
using System.Threading;
using System.Threading.Tasks;

// 1. Setup & Pre-seed
using var ctx = new Context();

// Option A: Standalone (Local Disk, In-Memory Cache)
var db = new Database(new DatabaseOptions { 
    StoresFolders = new List<string> { "./sop_data" },
    Type = (int)DatabaseType.Standalone 
});

// Option B: Clustered (Redis Cache) - Required for distributed swarm
// var db = new Database(new DatabaseOptions { 
//     StoresFolders = new List<string> { "./sop_data" },
//     Type = (int)DatabaseType.Clustered 
// });

using (var trans = db.BeginTransaction(ctx))
{
    var btree = db.NewBtree<int, string>(ctx, "concurrent_tree", trans);
    btree.Add(ctx, new Item<int, string> { Key = -1, Value = "Root Seed" });
    trans.Commit();
}

// 2. Launch Threads
Parallel.For(0, 5, i => 
{
    int threadId = i;
    int retryCount = 0;
    bool committed = false;
    
    while (!committed && retryCount < 10)
    {
        try 
        {
            using var trans = db.BeginTransaction(ctx);
            var btree = db.OpenBtree<int, string>(ctx, "concurrent_tree", trans);

            for (int j = 0; j < 100; j++)
            {
                int key = (threadId * 100) + j;
                btree.Add(ctx, new Item<int, string> { Key = key, Value = $"Thread {threadId} - Item {j}" });
            }
            trans.Commit();
            committed = true;
        }
        catch
        {
            retryCount++;
            Thread.Sleep(100 * retryCount);
        }
    }
});