PanoramicData.Chunker 1.0.19

PanoramicData.Chunker

A powerful, flexible .NET library for hierarchical document chunking, optimized for Retrieval-Augmented Generation (RAG) systems.

?? Overview

PanoramicData.Chunker transforms documents into semantically coherent chunks while preserving their hierarchical structure. It supports multiple document formats and provides rich metadata for each chunk, making it ideal for RAG systems, vector databases, and document analysis pipelines.

? Features

• Multi-Format Support: DOCX, PPTX, XLSX, HTML, PDF, Markdown, Plain Text, CSV, and more
• Hierarchical Chunking: Maintains parent-child relationships for context preservation
• RAG-Optimized: Built-in support for token counting, embedding models, and vector databases
• Type-Safe: Strongly-typed chunk models with rich metadata
• Async-First: Modern async/await patterns throughout
• Extensible: Plugin architecture for custom document formats and processors
• Quality Metrics: Automatic calculation of chunk quality and completeness scores

?? Installation

dotnet add package PanoramicData.Chunker

?? Quick Start

Basic Usage

using PanoramicData.Chunker;
using PanoramicData.Chunker.Configuration;

// Chunk a Markdown file
var result = await DocumentChunker.ChunkFileAsync(
    "document.md",
    ChunkingPresets.ForOpenAIEmbeddings()
);

// Access chunks
foreach (var chunk in result.Chunks)
{
    if (chunk is ContentChunk contentChunk)
    {
        Console.WriteLine($"{contentChunk.SpecificType}: {contentChunk.Content}");
    }
}

Fluent API

var result = await DocumentChunker.CreateBuilder()
    .WithDocument("document.md")
    .WithMaxTokens(512)
    .WithOverlap(50)
    .ExtractImages()
    .IncludeQualityMetrics()
    .ChunkAsync();

Custom Options

var options = new ChunkingOptions
{
    MaxTokens = 1024,
    OverlapTokens = 100,
    Strategy = ChunkingStrategy.Structural,
    ExtractImages = true,
    IncludeQualityMetrics = true,
    Tags = new List<string> { "documentation", "v1.0" },
    ExternalHierarchy = "Docs/UserGuide"
};

await using var stream = File.OpenRead("document.docx");
var result = await DocumentChunker.ChunkAsync(stream, DocumentType.Docx, options);

?? Chunk Types

StructuralChunk

Represents grouping containers (sections, pages, slides) that provide high-level context.

ContentChunk

Represents embeddable text content (paragraphs, list items, table rows).

VisualChunk

Represents non-textual objects (images, charts) with metadata and binary references.

TableChunk

Specialized chunk for tables with enhanced metadata and serialization options.

?? Supported Formats

?? Configuration Presets

// Optimized for OpenAI embeddings
ChunkingPresets.ForOpenAIEmbeddings()

// Optimized for Cohere embeddings
ChunkingPresets.ForCohereEmbeddings()

// Optimized for Azure AI Search
ChunkingPresets.ForAzureAISearch()

// For large documents with streaming
ChunkingPresets.ForLargeDocuments()

// Maximum context preservation for RAG
ChunkingPresets.ForRAG()

?? Roadmap

• Phase 0 (Current): Foundation and core infrastructure ?
• Phase 1: Markdown chunking (MVP)
• Phase 2: HTML chunking
• Phase 3: Advanced token counting (OpenAI, Claude, Cohere)
• Phase 4: Plain text chunking
• Phase 5-7: Office formats (DOCX, PPTX, XLSX)
• Phase 8: CSV chunking
• Phase 9: PDF chunking
• Phase 10-12: AI features (image descriptions, LLM integration, semantic chunking)
• Phase 13: Performance optimization and streaming
• Phase 14: Serialization and vector database integration
• Phase 15: Validation and quality assurance

See MasterPlan.md for complete roadmap.

?? Documentation

• Technical Specification - Complete technical specification
• Master Plan - Implementation roadmap
• API Reference (Coming soon)
• Cookbook (Coming soon)

?? Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

?? License

This project is licensed under the MIT License - see the LICENSE file for details.

?? Acknowledgments

Built with:

• DocumentFormat.OpenXml for Office formats
• Markdig for Markdown parsing (planned)
• AngleSharp for HTML parsing (planned)

?? Support

• ?? Issues: GitHub Issues
• ?? Discussions: GitHub Discussions

Status: ?? Active Development (Phase 0 Complete)

Made with ?? for the .NET and RAG communities