HistoryVault 1.0.3

HistoryVault.Net

A high-performance market data storage library for .NET HFT/HPC systems. Store, load, and aggregate OHLCV candlestick data with optimal memory efficiency and minimal allocations.

Features

• High-Performance Binary Serialization: Optimized for HFT/HPC with Span<byte>, ArrayPool, and zero-allocation patterns
• GZip Compression: Optional compression with configurable levels
• Cross-Platform Storage: Automatic platform-specific path resolution for Windows, macOS, and Linux
• Safe Symbol Paths: Encodes path-unsafe symbol names while preserving the original symbol for reloads
• Timeframe Aggregation: Aggregate candlesticks from smaller to larger timeframes (M1 → M5 → H1 → D1)
• Partial Overwrite: Merge new data with existing data, preserving non-overlapping candles
• Wildcard Symbol Matching: Glob pattern support for symbol lookup (e.g., BTC.*, *.USD)
• Data Availability Reporting: Check data coverage and identify gaps
• Warmup Candles: Load extra candles before requested time range for indicator calculations
• Thread-Safe Operations: Built-in synchronization for concurrent access

Installation

dotnet add package HistoryVault

Or via NuGet Package Manager:

Install-Package HistoryVault

Quick Start

Basic Usage

using HistoryVault;
using HistoryVault.Configuration;
using HistoryVault.Models;
using HistoryVault.Storage;

// Configure the vault (paths are auto-detected based on OS and scope)
var options = new HistoryVaultOptions
{
    DefaultScope = StorageScope.Local  // Data stored in ./data/history-vault/
};

await using var vault = new HistoryVaultStorage(options);

// Save candlestick data
var symbolData = new SymbolDataV2
{
    Symbol = "BTCUSDT",
    Timeframes = new List<TimeframeV2>
    {
        new TimeframeV2
        {
            Timeframe = CandlestickInterval.M1,
            Candlesticks = candlesticks // Your candlestick list
        }
    }
};

await vault.SaveAsync(symbolData, new SaveOptions
{
    UseCompression = true,
    AllowPartialOverwrite = true
});

// Load candlestick data
var loadOptions = LoadOptions.ForSymbol(
    "BTCUSDT",
    new DateTime(2025, 1, 1),
    new DateTime(2025, 1, 31),
    CandlestickInterval.M1
);

var data = await vault.LoadAsync(loadOptions);

Timeframe Aggregation

// Load M1 data and aggregate to H1 on-the-fly
var loadOptions = new LoadOptions
{
    Symbol = "BTCUSDT",
    StartDate = new DateTime(2025, 1, 1),
    EndDate = new DateTime(2025, 1, 31),
    Timeframes = new[] { CandlestickInterval.H1 },
    AllowAggregation = true  // Aggregates from M1 if H1 not available
};

var data = await vault.LoadAsync(loadOptions);

Wildcard Symbol Matching

// Load all BTC pairs
var loadOptions = new LoadOptions
{
    Symbol = "BTC.*",  // Matches BTC.USD, BTC.EUR, etc.
    Timeframes = new[] { CandlestickInterval.M1 }
};

var allBtcData = await vault.LoadMultipleAsync(loadOptions);

// Get matching symbol names
var symbols = await vault.GetMatchingSymbolsAsync("*.USD");

Data Availability Check

var report = await vault.CheckAvailabilityAsync(
    "BTCUSDT",
    CandlestickInterval.M1,
    new DateTime(2025, 1, 1),
    new DateTime(2025, 12, 31)
);

Console.WriteLine($"Coverage: {report.CoveragePercentage:P2}");
Console.WriteLine($"Available: {report.TotalCandlesAvailable} / {report.ExpectedCandlesCount} candles");

foreach (var gap in report.Gaps)
{
    Console.WriteLine($"Gap: {gap.Start} - {gap.End}");
}

Warmup Candles for Indicators

var loadOptions = new LoadOptions
{
    Symbol = "BTCUSDT",
    StartDate = new DateTime(2025, 1, 15),
    EndDate = new DateTime(2025, 1, 31),
    Timeframes = new[] { CandlestickInterval.H1 },
    WarmupCandlesCount = 50  // Extra 50 candles before StartDate
};

var data = await vault.LoadAsync(loadOptions);
// Use warmup candles to initialize moving averages, etc.

API Reference

IHistoryVault Interface

Configuration Classes

HistoryVaultOptions

Automatic Path Resolution:

• Local scope: ./data/history-vault/ (relative to current working directory)
• Global scope: OS-specific application data directory
  • Windows: %LOCALAPPDATA%\HistoryVault
  • macOS: ~/Library/Application Support/HistoryVault
  • Linux: $XDG_DATA_HOME/HistoryVault or ~/.local/share/HistoryVault

SaveOptions

LoadOptions

CandlestickInterval Enum

Storage Format

Directory Structure

{BasePath}/
  {Symbol}/
    {Timeframe}/
      {Year}/
        {Month}.bin[.gz]

Example:

/data/
  BTCUSDT/
    1m/
      2025/
        01.bin.gz
        02.bin.gz
    1h/
      2025/
        01.bin.gz

Binary Format

• Header: 64 bytes (magic number, version, flags, timeframe, record count, date range)
• Records: 96 bytes per candlestick (timestamps and OHLCV decimals)

Performance Benchmarks

BenchmarkDotNet v0.13.12, macOS 26.2 (25C56) [Darwin 25.2.0]
Apple M3 Max, 1 CPU, 14 logical and 14 physical cores
.NET SDK 10.0.100
  [Host]   : .NET 10.0.0, Arm64 RyuJIT AdvSIMD
  .NET 10.0 : .NET 10.0.0, Arm64 RyuJIT AdvSIMD

Aggregation Rules

When aggregating candlesticks to larger timeframes:

• Open: First candle's open price
• High: Maximum high across all candles
• Low: Minimum low across all candles
• Close: Last candle's close price
• Volume: Sum of all volumes
• OpenTime: First candle's open time
• CloseTime: Last candle's close time

Valid aggregation paths:

• M1 → M3, M5, M15, M30, H1, H2, H4, H6, H8, H12, D1
• M5 → M15, M30, H1, etc.
• H1 → H2, H4, H6, H8, H12, D1

Thread Safety

HistoryVaultStorage is thread-safe for concurrent read/write operations. Internal locking ensures data consistency when multiple threads access the same symbol/timeframe.

Integration with High-Performance-Backtest.Net

This library works seamlessly with High-Performance-Backtest.Net - a high-performance backtesting engine. Both libraries use identical data structures (SymbolDataV2, TimeframeV2, CandlestickV2) but with different namespaces:

• HistoryVault.Net: HistoryVault.Models
• High-Performance-Backtest.Net: Backtest.Net.SymbolsData

Converting Between Types

The easiest way to convert between these types is using JSON serialization:

using System.Text.Json;
using HistoryVault.Models;

// Load data from HistoryVault
var vault = new HistoryVaultStorage(options);
var historyVaultData = await vault.LoadAsync(loadOptions);

// Convert HistoryVault.Models.SymbolDataV2 -> Backtest.Net.SymbolsData.SymbolDataV2
var json = JsonSerializer.Serialize(historyVaultData);
var backtestData = JsonSerializer.Deserialize<Backtest.Net.SymbolsData.SymbolDataV2>(json);

// Now use backtestData with High-Performance-Backtest.Net
var backtester = new Backtester();
backtester.Run(backtestData);

For better performance with large datasets, consider creating an extension method:

public static class SymbolDataConverter
{
    private static readonly JsonSerializerOptions JsonOptions = new()
    {
        PropertyNameCaseInsensitive = true
    };

    public static Backtest.Net.SymbolsData.SymbolDataV2 ToBacktestFormat(
        this HistoryVault.Models.SymbolDataV2 source)
    {
        var json = JsonSerializer.Serialize(source, JsonOptions);
        return JsonSerializer.Deserialize<Backtest.Net.SymbolsData.SymbolDataV2>(json, JsonOptions)!;
    }

    public static HistoryVault.Models.SymbolDataV2 ToHistoryVaultFormat(
        this Backtest.Net.SymbolsData.SymbolDataV2 source)
    {
        var json = JsonSerializer.Serialize(source, JsonOptions);
        return JsonSerializer.Deserialize<HistoryVault.Models.SymbolDataV2>(json, JsonOptions)!;
    }
}

// Usage
var backtestData = historyVaultData.ToBacktestFormat();

Contributing

See CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Changelog

See CHANGELOG.md for version history.