FsstSharp 1.0.0

FsstSharp

A high-performance C# implementation of the FSST (Fast Static Symbol Table) compression algorithm, optimized for .NET 10 with modern APIs and SIMD support.

What is FSST?

FSST is a compression algorithm specifically designed for string/text data that allows random access to compressed data. Unlike block-based algorithms like GZip or LZ4, FSST can decompress individual strings without touching surrounding data.

Key Features:

• 🚀 Fast compression and decompression - Optimized for text, JSON, and structured data
• 🎯 Random access - Decompress individual strings without full block decompression
• 📦 Stream-based API - Compatible with standard .NET compression streams
• ⚡ Zero-allocation techniques - Uses Span<byte>, Memory<byte>, and unsafe code
• 🔧 SIMD optimizations - Automatic CPU feature detection (AVX2, SSE2, ARM NEON)
• 🎨 Modern C# APIs - Built for .NET 10 with latest language features

Installation

Install via NuGet:

dotnet add package FsstSharp

Or via Package Manager:

Install-Package FsstSharp

Quick Start

Basic Compression/Decompression

using FsstSharp;

// Sample data for training the encoder
var samples = new[]
{
    "user_id: 123, name: John"u8.ToArray(),
    "user_id: 456, name: Jane"u8.ToArray()
};

// Create encoder and decoder
var encoder = FsstEncoder.Create(samples);
var decoder = encoder.GetDecoder();

// Compress data
var input = "user_id: 789, name: Bob"u8.ToArray();
var compressed = new byte[input.Length * 2];
int compressedSize = encoder.Compress(input, compressed);

// Decompress data
var decompressed = new byte[input.Length * 2];
int decompressedSize = decoder.Decompress(
    compressed.AsSpan(0, compressedSize), 
    decompressed);

Stream-based Compression

using System.IO;
using System.IO.Compression;
using FsstSharp;

var originalData = "user_id: 789, name: Bob"u8.ToArray();

// Compress with on-the-fly learning
var compressedStream = new MemoryStream();
using (var fsstStream = new FsstStream(compressedStream, CompressionMode.Compress, leaveOpen: true))
{
    fsstStream.Write(originalData, 0, originalData.Length);
}

// Decompress
compressedStream.Position = 0;
var decompressedStream = new MemoryStream();
using (var fsstStream = new FsstStream(compressedStream, CompressionMode.Decompress))
{
    fsstStream.CopyTo(decompressedStream);
}

Use Cases

FSST is ideal for:

• Database systems - Compress string columns with random access
• Data file formats - Parquet, ORC, or custom formats
• JSON/XML processing - Better compression than general-purpose algorithms
• Log aggregation - Compress structured log data
• Network protocols - Reduce bandwidth for text-heavy APIs

Performance

Benchmarks on realistic JSON data (~12KB):

Building from Source

Prerequisites

• .NET 10 SDK or later

Build

git clone https://github.com/reedz/FsstSharp.git
cd FsstSharp/src
dotnet build -c Release

Run Tests

cd src
dotnet test -c Release

Run Benchmarks

cd src/FsstSharp.Benchmarks

# Standard compression benchmarks (FSST vs GZip vs LZ4)
dotnet run -c Release

# Sample data effect analysis
dotnet run -c Release -- sample-effect

# Sample count analysis (0, 1, 5, 10, 20, 50, 100, 500 samples)
dotnet run -c Release -- sample-count

See SAMPLE_DATA_ANALYSIS.md for detailed analysis of how training data affects compression.

Create NuGet Package

cd src/FsstSharp
dotnet pack -c Release

The package will be created in bin/Release/FsstSharp.{version}.nupkg

Projects

• FsstSharp - Core compression library
• FsstSharp.Cli - Command-line tool for file compression
• FsstSharp.Tests - NUnit test suite
• FsstSharp.Benchmarks - BenchmarkDotNet performance tests

Documentation

For detailed documentation, see:

• src/README.md - Comprehensive usage guide
• SAMPLE_DATA_ANALYSIS.md - Sample data effect on compression
• PERFORMANCE_OPTIMIZATIONS.md - Performance details
• BENCHMARK_RESULTS.md - Benchmark results

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

This project maintains the MIT License from the original FSST implementation:

Copyright (c) 2018-2020, CWI, TU Munich, FSU Jena

See LICENSE for full details.

Original FSST

This is a C# port of the original FSST algorithm:

• Paper: FSST: Fast Random Access String Compression
• Original C++ Implementation: https://github.com/cwida/fsst
• Authors: Peter Boncz (CWI), Viktor Leis (FSU Jena), Thomas Neumann (TU Munich)

Acknowledgments

Special thanks to the original FSST authors for creating this innovative compression algorithm.