Rayneforge.Stratus.RemoteIndexer 1.0.1

Rayneforge.Stratus.RemoteIndexer

A .NET library for semantic code indexing with hybrid search capabilities, supporting OpenAI and Azure OpenAI embeddings.

Features

• Text Extraction: Intelligent file classification and text extraction with configurable size limits
• Smart Chunking: Token-aware chunking with overlap policies for optimal context preservation
• Embeddings: Support for OpenAI and Azure OpenAI embedding models with caching and batching
• Hybrid Search: Combined vector and lexical (FTS) search with configurable scoring
• Draft/Commit Workflow: Transactional indexing with rollback support via SQLite
• Clustering: K-means clustering with AI-powered labeling for semantic grouping

Installation

dotnet add package Rayneforge.Stratus.RemoteIndexer

Quick Start

Configuration

{
  "Indexer": {
    "WorkspaceDirectory": ".workspace",
    "Sqlite": {
      "ConnectionString": "Data Source=indexer.db"
    },
    "Indexing": {
      "MaxFileBytes": 1000000,
      "Chunking": {
        "MaxCharsPerChunk": 2000,
        "OverlapChars": 200
      }
    },
    "AI": {
      "Profile": "Lite",
      "Provider": "OpenAI",
      "ApiKey": "your-api-key"
    }
  }
}

Basic Usage

using Rayneforge.Stratus.RemoteIndexer.Storage;
using Rayneforge.Stratus.RemoteIndexer.Search;
using Rayneforge.Stratus.RemoteIndexer.Options;

// Configure services
services.Configure<IndexerOptions>(configuration.GetSection("Indexer"));
services.AddSingleton<IIndexStore, SqliteIndexStore>();
services.AddSingleton<HybridSearch>();

// Index files
var store = serviceProvider.GetRequiredService<IIndexStore>();
await store.UpsertFileAsync(new IndexRecord
{
    RelativePath = "src/Program.cs",
    Content = fileContent,
    Chunks = chunks,
    State = "draft"
});

// Commit changes
var commitId = await store.CommitAllAsync();

// Search
var search = serviceProvider.GetRequiredService<HybridSearch>();
var results = await search.SearchAsync("authentication logic", 
    new HybridSearchOptions { MaxResults = 10 });

Architecture

The library is deterministic by design - the same file always produces the same chunks, embeddings, and hashes. This ensures consistency between local and remote indexing operations.

Core Components

• Extraction: File type detection, text extraction with size limits
• Chunking: Token-aware chunkers with overlap for context preservation
• Embeddings: OpenAI/Azure OpenAI clients with batching and caching
• Storage: SQLite-backed persistence with draft/commit/rollback support
• Search: Hybrid scoring combining vector similarity and lexical matching
• Clustering: K-means with LLM-powered cluster labeling

AI Profiles

The library supports three AI profiles for different use cases:

• Lite: Fast, cost-effective (small embedding models, basic LLMs)
• Basic: Balanced performance and cost (standard models)
• Premium: Maximum quality (largest models, advanced features)

Storage Model

The SQLite schema supports:

• Draft and committed states for all records
• Commit log with rollback capabilities
• Vector storage for semantic search
• Full-text search indexes for hybrid queries

Security

• All file paths are validated against the workspace root
• File and chunk sizes are capped to prevent resource exhaustion
• SQL operations use parameterized queries
• AI credentials are never logged or exposed

Testing

The library includes comprehensive tests:

• Unit tests for chunking, extraction, and scoring
• Integration tests with SQLite harness
• Deterministic seeding for reproducible tests

License

See the repository root for license information.

Contributing

Contributions are welcome! Please see the main repository for contribution guidelines.