Bitvantage.HyperMetrics 0.9.0-rc1

Bitvantage.HyperMetrics

A high-performance .NET 8.0 TimescaleDB client library that transforms your C# classes into optimized time-series database schemas with minimal configuration and maximum performance.

What It Does

Bitvantage.HyperMetrics solves the complexity of efficiently storing and managing time-series data in PostgreSQL/TimescaleDB by providing:

🏗️ Code-First Schema Generation Define your data models using simple C# classes with attributes, and the library automatically generates optimized database schemas, tables, indexes, and hypertables.

⚡ Ultra-High Performance Ingestion Achieve 100,000+ records/second throughput using binary COPY protocol with temporary tables, and compiled expression trees.

🧠 Smart Data Organization Choose between denormalized (fast writes) or normalized (storage efficient) models based on your data patterns.

Key Features

• 🚀 High Performance: Binary COPY protocol for optimal throughput
• 🏗️ Automatic Schema Generation: Creates database tables, indices, and constraints from .NET attributes
• 🔄 Dual Data Models: Support for both denormalized (single table) and normalized (lookup tables) approaches
• 🛡️ Type Safety: Compile-time verification with runtime validation
• 📊 Grafana Compatible: Grafana friendly table structure
• 🔧 Smart Duplicate Handling: Configurable overwrite or ignore behavior for duplicate measurements
• 🧵 Thread-Safe: Concurrent insertions supported in some cases

Quick Start****

Installation

dotnet add package Bitvantage.HyperMetrics

Basic Usage

using Bitvantage.HyperMetrics;
using Bitvantage.HyperMetrics.Attributes;

// Define your measurement model
[Measurement(Table = "sensor_data")]
public class SensorReading
{
    [Timestamp]
    public DateTimeOffset Time { get; set; }

    [Label]
    public string DeviceName { get; set; }

    [Metric]
    public double Temperature { get; set; }

    [Metric]
    public double? Humidity { get; set; }
}

// Open a time series connection
await using var timeSeries = await TimeSeries<SensorReading>.OpenAsync(connectionString);

// Insert measurements
var readings = new[]
{
    new SensorReading
    {
        Time = DateTimeOffset.Now,
        DeviceName = "sensor-1",
        Temperature = 23.5,
        Humidity = 65.2
    },
    new SensorReading
    {
        Time = DateTimeOffset.Now,
        DeviceName = "sensor-2",
        Temperature = 24.1,
        Humidity = null // Metrics can be nullable
    }
};

await timeSeries.InsertAsync(readings);

Core Concepts

Attributes System

Bitvantage.HyperMetrics uses attributes to define how .NET types map to time-series database tables:

[Measurement] - Class Configuration

Applied to classes to configure database mapping:

[Measurement(
    Table = "server_metrics",    // Table name (defaults to snake_case class name)
    Schema = "monitoring",       // Schema name (defaults to user's default schema)
    OptIn = false               // Include all public members unless marked [Ignore]
)]

[Timestamp] - Time Column

Marks the timestamp field (exactly one per model):

[Timestamp]                               // Defaults to 'time' column for Grafana compatibility
public DateTimeOffset Time { get; set; }

[Timestamp(Column = "measurement_time")]  // or use a custom column name
public DateTimeOffset MeasurementTime { get; set; }

[Label] - Dimensional Data

Defines categorical data for filtering and grouping:

[Label]                           // Stored in main table (denormalized)
public string ServerName { get; set; }

[Label(LookupTable = "servers")]  // Stored in separate lookup table (normalized)
public string ServerHostname { get; set; }

[Metric] - Measurement Values

Defines measurable numeric values:

[Metric]
public double CpuPercent { get; set; }

[Metric(Column = "memory_bytes")]
public long MemoryUsageBytes { get; set; }

[Ignore] - Exclusion

Excludes fields from database persistence:

[Ignore]
public string InternalNote { get; set; }

Data Models

Denormalized Model (Single Table)

Best for high-performance scenarios with simple schemas:

[Measurement(Table = "simple_metrics")]
public class DenormalizedMetrics
{
    [Timestamp]
    public DateTimeOffset Time { get; set; }

    [Label]  // Stored directly in main table
    public string DeviceName { get; set; }

    [Label]  // Stored directly in main table
    public string Location { get; set; }

    [Metric]
    public double Temperature { get; set; }
}

Normalized Model (Lookup Tables)

Best for storage efficiency with low-cardinality labels:

[Measurement(Table = "normalized_metrics")]
public class NormalizedMetrics
{
    [Timestamp]
    public DateTimeOffset Time { get; set; }

    [Label(LookupTable = "devices")]   // Separate lookup table
    public string DeviceName { get; set; }

    [Label(LookupTable = "devices")]   // Same lookup table as DeviceName
    public string DeviceType { get; set; }

    [Label(LookupTable = "locations")] // Different lookup table
    public string Location { get; set; }

    [Metric]
    public double Temperature { get; set; }
}

Configuration Options

using Bitvantage.HyperMetrics.Types;

var config = new TimeSeriesConfig(
    Mode: TimeSeriesMode.OpenOrCreate,                       // Schema handling mode
    DuplicateHandling: DuplicateHandlingBehavior.Overwrite,  // When the measurement contains an idenitical timestamp and set of labels, overwrite the values
    CreateView: true                                         // Create convenience view for normalized data
);

await using var timeSeries = await TimeSeries<SensorReading>.OpenAsync(connectionString, config);

TimeSeriesMode Options

• OpenOrCreate (Default): Create schema if missing, validate if exists
• Open: Schema must exist and be correct
• Use: Assume schema exists, skip validation (highest performance)

Duplicate Handling

• Overwrite (Default): Replace existing measurements with same timestamp + labels
• Ignore: Skip duplicate measurements silently

Advanced Features

Schema Generation

Generate database schema as SQL for review or deployment:

await using var timeSeries = await TimeSeries<SensorReading>.OpenAsync(connectionString);
string schemaSql = timeSeries.GenerateSchema();
Console.WriteLine(schemaSql);

Output example:

CREATE TABLE sensor_data (
    time TIMESTAMPTZ NOT NULL,
    device_id TEXT NOT NULL,
    temperature DOUBLE PRECISION NOT NULL,
    humidity DOUBLE PRECISION,
    PRIMARY KEY (time, device_id)
);

SELECT create_hypertable('sensor_data', 'time');

Runtime Model Transformation

Modify the model at runtime for dynamic configurations:

var config = TimeSeriesConfig.WithModelTransform(model =>
{
    model.Schema = $"tenant_{tenantId}";
    model.Table = $"{model.Table}_{DateTime.UtcNow:yyyyMM}";
    model["DeviceId"].Column = "device_identifier";
});

Thread Safety and Concurrency

Thread Safety Guarantees

TimeSeries<T> instances are thread-safe when created using:

• Connection string constructor: TimeSeries<T>.OpenAsync(connectionString, ...)
• NpgsqlDataSource constructor: TimeSeries<T>.OpenAsync(dataSource, ...)

Multiple threads can safely call InsertAsync() concurrently on the same TimeSeries<T> instance.

Note: TimeSeries<T> instances created with an existing NpgsqlConnection are not thread-safe and should only be used from a single thread.

Database Concurrency Considerations

While the client library is thread-safe, concurrent insertions can encounter database-level contention depending on your configuration.

Denormalized Data with Duplicate Handling Set to Ignore

When using denormalized data models with DuplicateHandling.Ignore, the library leverages PostgreSQL's ON CONFLICT DO NOTHING clause which does not acquire row locks. This means concurrent threads can insert duplicate records without creating duplicate records blocking each other, achieving near-linear scaling with multiple threads. This configuration is ideal for high-throughput scenarios where duplicates are either acceptable or rare, as there is no database contention even under heavy concurrent load.

Denormalized Data with Duplicate Handling Set to Overwrite

When using DuplicateHandling.Overwrite, concurrent threads inserting duplicate records with the same timestamp and labels will cause temporary database contention as PostgreSQL must serialize conflicting row updates. To mitigate this, you can route duplicate records to the same thread using timestamp-based hashing or partitioning, ensure your data pipeline eliminates duplicates before insertion, or decrease batch sizes to reduce the duration that locks are held.

Normalized Data with Lookup Tables

Normalized data models introduce additional complexity due to concurrent lookup table insertions. When multiple threads insert new lookup values simultaneously, PostgreSQL may encounter deadlocks while serializing unique constraint checks and resolving foreign key references. To address this, you can pre-insert lookup values by populating lookup tables with known label values before beginning high-concurrency insertion, implement retry logic with deadlock detection and exponential backoff, or serialize new lookup insertions by using a dedicated thread or queue for records containing previously unseen label values.

Supported Data Types

Timestamp Types

• DateTime → timestamp without time zone
• DateTimeOffset → timestamp with time zone (recommended)

Label Types (Always Non-Nullable)

• string → text
• Guid → uuid
• IPAddress → inet
• int, long, short → integer, bigint, smallint
• float, double, decimal → real, double precision, numeric
• bool → boolean

Metric Types (Nullable Supported)

• int, int? → integer
• long, long? → bigint
• short, short? → smallint
• float, float? → real
• double, double? → double precision
• decimal, decimal? → numeric
• bool, bool? → boolean
• string? → text (nullable reference type for metrics only)

Examples

IoT Sensor Monitoring

[Measurement(Table = "iot_sensors", Schema = "monitoring")]
public class IoTSensorReading
{
    [Timestamp]
    public DateTimeOffset Timestamp { get; set; }

    [Label]
    public string DeviceId { get; set; }

    [Label]
    public string SensorType { get; set; }

    [Label]
    public string Location { get; set; }

    [Metric]
    public double Value { get; set; }

    [Metric]
    public string? Unit { get; set; }

    [Metric]
    public double? Quality { get; set; }
}

Application Performance Monitoring

[Measurement(Table = "request_metrics")]
public class RequestMetric
{
    [Timestamp]
    public DateTimeOffset Time { get; set; }

    [Label(LookupTable = "services")]
    public string ServiceName { get; set; }

    [Label(LookupTable = "services")]
    public string Environment { get; set; }

    [Label]
    public string Endpoint { get; set; }

    [Label]
    public int StatusCode { get; set; }

    [Metric]
    public long ResponseTimeMs { get; set; }

    [Metric]
    public int RequestSizeBytes { get; set; }

    [Metric]
    public int? ResponseSizeBytes { get; set; }
}

Background Service Integration

public class MetricsIngestionService : BackgroundService
{
    private readonly TimeSeries<RequestMetric> _timeSeries;

    public MetricsIngestionService(string connectionString)
    {
        var config = new TimeSeriesConfig(
            DuplicateHandling: DuplicateHandlingBehavior.Ignore
        );
        _timeSeries = await TimeSeries<RequestMetric>.OpenAsync(connectionString, config);
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var batch in GetMetricBatches(stoppingToken))
        {
            await _timeSeries.InsertAsync(batch, stoppingToken);
        }
    }

    public override async ValueTask DisposeAsync()
    {
        await _timeSeries.DisposeAsync();
        await base.DisposeAsync();
    }
}

Best Practices

Model Design

• Use DateTimeOffset for timestamps to handle time zones properly
• Keep labels non-nullable for data integrity
• Use nullable metrics (double?) only when genuinely optional
• Choose denormalized models for high-cardinality data
• Choose normalized models for low-cardinality data with high repetition

Schema Management

• Use TimeSeriesMode.OpenOrCreate for development
• Use TimeSeriesMode.Open for production environments
• Generate and review schemas before deployment
• Plan for data retention and partitioning strategies

Database Integration

PostgreSQL/TimescaleDB Setup

The library automatically creates:

• Main tables with composite primary keys
• Lookup tables for normalized data (when specified)
• TimescaleDB hypertables with time-based partitioning
• Appropriate indexes and constraints

Grafana Integration

Default 'time' column naming ensures automatic Grafana compatibility:

• Time-series data source detection
• Time-based visualizations
• Optimized time-range filtering

Requirements

• .NET 8.0 or later
• PostgreSQL 12+ with TimescaleDB extension (recommended)
• Npgsql 8.0+

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

Support

For issues, questions, or feature requests, please visit our GitHub issues page.