FlinkDotnet 1.0.1

FlinkDotNet

FlinkDotNet is a .NET framework that enables developers to write Apache Flink 2.1 streaming jobs in C# and submit them to production Flink clusters. This repo also provides a comprehensive distributed message-oriented architecture that enables developers to build production-grade stream processing applications using Apache Flink 2.1, Apache Kafka, Temporal workflows, and Microsoft Aspire orchestration - all accessible through a native .NET SDK.

What is Kafka and Flink? Why Do We Need Them?

The Problems You'll Face as Your Application Grows

The Solutions: Kafka, Flink, and Temporal

FlinkDotNet brings billion-scale architecture to .NET developers - combining these three technologies to handle routing billions of messages per second across distributed systems, processing them in real-time with global context awareness, and coordinating millions of complex workflows simultaneously.

What is FlinkDotNet?

FlinkDotNet lets you write Apache Flink 2.1 streaming jobs in C# and submit them to Flink clusters. No Java required. A complete distributed message-oriented architecture for building enterprise stream processing applications in .NET, it combines:

• Apache Flink 2.1 - distributed stream processing engine with state management.
• Apache Kafka - Distributed message streaming/queue
• Temporal - Durable workflow orchestration/Durable execution solution.
• Microsoft Aspire - Local development containerised orchestration.
• FlinkDotNet SDK - Native .NET API for writing Flink jobs in C#

var env = Flink.GetExecutionEnvironment();
var stream = env.FromKafka("orders")
    .Filter(order => order.Amount > 100)
    .Map(order => order.ToUpperCase())
    .SinkToKafka("processed-orders");

await env.ExecuteAsync("order-processor");

Distributed Architecture with Aspire Orchestration

.NET Aspire orchestrates the complete distributed message-oriented architecture locally, providing production-parity development environments.

Full Architecture Stack

Stream Processing Layer:

• Apache Flink 2.1 - JobManager, TaskManager, and SQL Gateway for real-time stream processing
• FlinkDotNet SDK - Native .NET DataStream API with fluent C# DSL
• Job Gateway - ASP.NET Core service for job submission and management

Messaging & Orchestration Layer:

• Apache Kafka - KRaft-mode message broker with JMX metrics export
• Temporal - Durable workflow orchestration with PostgreSQL backend

Observability Stack (LearningCourse mode):

• Prometheus - Metrics collection from Flink, Kafka, and custom applications
• Grafana - Visualization dashboards for performance monitoring
• JMX Exporters - Metrics bridge for Java components

With one command, Aspire starts all containers, configures service discovery, and provides a unified dashboard - enabling you to develop and test complete distributed streaming applications locally.

Component Roles in High-Throughput Processing

When processing millions of messages per second, each component plays a critical role:

Apache Kafka - Message ingestion and buffering

• Handles message ingestion at scale (millions of messages/second)
• Provides durable message storage with partitioning for parallel consumption
• Acts as buffer between producers and stream processors
• Enables replay and reprocessing of historical data

Apache Flink 2.1 - Distributed stream processing engine

• Processes messages in parallel across multiple TaskManager instances
• Provides stateful computations with exactly-once processing guarantees
• Scales horizontally by adding more TaskManager slots
• Handles backpressure to prevent system overload

FlinkDotNet SDK - .NET development interface

• Enables writing stream processing logic in C# with type safety
• Compiles to Flink's native execution model
• Provides fluent API for common streaming patterns (map, filter, window, join)
• Eliminates need for Java expertise while maintaining full Flink performance

Temporal - Durable workflow orchestration

• Manages long-running workflows across distributed job submissions
• Provides guaranteed execution with automatic retries and compensation
• Maintains workflow state even during infrastructure failures
• Coordinates complex multi-step processing pipelines

Microsoft Aspire - Local development orchestration

• Simulates production environment locally with container orchestration
• Manages service discovery between Kafka, Flink, Temporal, and custom services
• Provides unified dashboard for monitoring all components
• Enables rapid iteration and testing before production deployment

Together, these components form a production-grade streaming architecture capable of processing high-volume event streams with reliability and fault tolerance.

Try It Out

# Prerequisites: .NET 9.0 SDK, Docker Desktop (or Podman)

# 1. Clone and run LocalTesting
git clone https://github.com/devstress/FlinkDotnet.git
cd FlinkDotnet/LocalTesting
dotnet run --project LocalTesting.FlinkSqlAppHost

# 2. Aspire Dashboard opens at http://localhost:15888
# 3. Navigate to LearningCourse folder and follow the instructions there

# Or run integration tests to validate everything works
dotnet test LocalTesting.IntegrationTests

LocalTesting includes integration tests that validate the complete pipeline: Kafka → Flink → Processing → Output.

Apache Flink Versions Coverage

FlinkDotNet provides 100% feature parity with Apache Flink 1.0-2.1, implementing all major features across every version release.

Version-by-Version Coverage

Feature Categories (21/21 Complete - 100% 🎉)

P0 Features (Critical - ALL COMPLETE):

• ✅ Unified Sink API v2 (WI6, Flink 1.20) - Modern sink pattern replacing SinkFunction
• ✅ Materialized Tables (WI7, Flink 1.20) - Declarative ETL with auto-refresh
• ✅ AI/ML Integration (WI8-WI9, Flink 2.1) - CREATE MODEL, ML_PREDICT, AI providers (OpenAI, Azure OpenAI, Amazon Bedrock, Google Vertex AI, Hugging Face)

P1 Features (High Priority - ALL COMPLETE):

• ✅ VARIANT Data Type (WI10, Flink 2.1) - Semi-structured JSON data handling
• ✅ Table API & Advanced SQL (WI10, Flink 2.1) - All 7 sub-features complete
• ✅ Process Table Functions (PTFs) (WI10, Flink 2.1) - Advanced table processing
• ✅ Apache Paimon (WI13, Flink 1.15-1.18) - Lakehouse storage integration
• ✅ Observability Testing (WI11) - Comprehensive test coverage and monitoring
• ✅ Catalog API (WI14, Flink 1.10) - Hive/JDBC/GenericInMemory catalog management
• ✅ Unified Source API (WI15, Flink 1.12) - FLIP-27 modern source connector framework

P2 Features (Medium Priority - ALL COMPLETE):

• ✅ Performance & Format (WI12, WI16, Flink 2.1) - All 4 sub-features complete:
  • ✅ Custom Async Sink Batching (WI12)
  • ✅ State Backend Configuration (WI16)
  • ✅ Smile Format for Compiled Plans (WI16)
  • ✅ MultiJoin Optimization (WI16)

Learn FlinkDotNet

LearningCourse provides a 15-day hands-on course covering:

• Day 01: Kafka-Flink Data Pipeline
• Day 02: Flink 2.1 Fundamentals - Complete Apache Flink 1.0-2.1 Version Coverage
• Day 03-15: Advanced topics (AI integration, backpressure, observability, workflows, stress testing, and more)

Each day includes working examples and integration tests you can run locally.

Documentation

• Getting Started Guide - Setup and first job
• API Reference - Complete DataStream API
• Features - Apache Flink 2.1 capabilities
• Architecture & Use Cases - System design

Installation

NuGet Package

dotnet add package FlinkDotNet

Docker Image

docker pull devstress/flinkdotnet:latest
docker run -p 8086:8086 \
  -e FLINK_CLUSTER_HOST=your-flink-host \
  -e FLINK_CLUSTER_PORT=8081 \
  devstress/flinkdotnet:latest

Standalone Executables

Download from GitHub Releases - includes Windows and Linux packages.

Requirements

• .NET 9.0 SDK - For development
• Docker Desktop or Podman - For local testing with Aspire
• Apache Flink 2.1 cluster - For production deployments

Community & Support

• 💬 GitHub Issues - Bug reports and feature requests
• 📧 Discussions - Questions and best practices
• 🤝 Contributing - Development guidelines

License

MIT License - see LICENSE for details.

Get Started: Try the LocalTesting environment or explore the 15-Day Learning Course.