GothiaAI.DNFileRAG.Core 1.3.2

DNFileRAG

A .NET 9 real-time, file-driven RAG (Retrieval-Augmented Generation) engine that watches a folder, ingests documents, and serves fast answers over an HTTP API (with Qdrant vector search).

<a href="https://youtu.be/6O5fafYHkAc"> <img src="https://img.youtube.com/vi/6O5fafYHkAc/maxresdefault.jpg" alt="Watch the demo" width="600"> </a>

Click to watch the demo video

What you get

• Real-time ingestion: watches a folder and keeps your index up to date
• Formats: .pdf, .docx, .txt, .md, .html, .png, .jpg, .jpeg, .webp
• Providers: OpenAI / Azure OpenAI / Anthropic / Ollama (local)
• Vector store: Qdrant
• API: /api/query, /api/documents, /api/health
• Example UI: mock company landing page + popup help chat (examples/HelpChat)

Choose your path

• Tutorial 1: Local dev (recommended): Ollama + Qdrant + dotnet run (fastest to try)
• Tutorial 2: HelpChat demo UI: run a static page that calls your local API
• Tutorial 3: Docker deploy: docker-compose up -d (self-contained stack)
• Tutorial 4: Testing: fast vs integration tests
• Tutorial 5: Production tips: hardening checklist

Tutorial 1 — Local dev (Ollama + Qdrant)

This uses the defaults in src/DNFileRAG/appsettings.Development.json:

• API on http://localhost:8181
• ApiSecurity:RequireApiKey = false (no key needed)
• Embeddings + LLM via Ollama
• Qdrant vector size 1024 (matches mxbai-embed-large)

Step 1) Start Qdrant

docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant

Step 2) Install Ollama + pull models

ollama pull mxbai-embed-large
ollama pull llama3.2:3b
ollama pull llava

Images: Vision extraction is enabled by default in Development via Vision:Enabled=true and uses the Ollama model llava (configurable).

Step 3) Run DNFileRAG

dotnet run --project ./src/DNFileRAG

Step 4) Add documents

Put files into:

• src/DNFileRAG/data/documents/

If you want to watch a different folder, change FileWatcher:WatchPath in:

• src/DNFileRAG/appsettings.Development.json

Step 5) Verify indexing

curl http://localhost:8181/api/documents

Step 6) Query

• cURL

curl -X POST http://localhost:8181/api/query \
  -H "Content-Type: application/json" \
  -d "{\"query\":\"What are our support hours?\"}"

• PowerShell

Invoke-RestMethod http://localhost:8181/api/query -Method Post -ContentType "application/json" -Body (@{ query = "What are our support hours?" } | ConvertTo-Json)

Tutorial 2 — HelpChat demo UI (landing page + popup chat)

HelpChat is a static mock company page under examples/HelpChat/ that opens a popup chat and calls your local DNFileRAG API.

Step 1) Start DNFileRAG

Follow Tutorial 1 so the API is running at http://localhost:8181.

Step 2) Serve the static files

cd examples/HelpChat
python -m http.server 3000

Step 3) Open it

Open http://localhost:3000 and click Help.

You can open examples/HelpChat/index.html directly, but some browsers restrict file:// pages from calling http://localhost.

Tutorial 3 — Docker deploy (self-contained stack)

This uses docker-compose.yml to run:

• Qdrant
• Ollama
• DNFileRAG API on http://localhost:8080

Step 1) Start the stack

docker-compose up -d

Step 2) Add documents

Files in ./documents are mounted into the container at /app/data/documents:

mkdir -p documents
cp /path/to/your/files/* documents/

Step 3) Query

curl -X POST http://localhost:8080/api/query \
  -H "Content-Type: application/json" \
  -d '{"query":"What are our support hours?"}'

Stop / reset

docker-compose down
docker-compose down -v   # also removes Qdrant + Ollama volumes

Tutorial 4 — Testing

Fast tests (unit tests + fast checks)

• Windows (PowerShell)

dotnet test .\DNFileRAG.sln -c Release --filter "Category!=Integration"

• macOS/Linux (bash/zsh)

dotnet test ./DNFileRAG.sln -c Release --filter "Category!=Integration"

Full suite (includes integration tests)

Integration tests may start Docker containers (Testcontainers) and will run slower.

• Windows (PowerShell)

dotnet test .\DNFileRAG.sln -c Release

• macOS/Linux (bash/zsh)

dotnet test ./DNFileRAG.sln -c Release

Note on FluentAssertions licensing

Tests use FluentAssertions. If you plan commercial use, you may need a commercial license (see the warning emitted during test runs).

Tutorial 5 — Production tips (checklist)

Security

• Enable API keys: set ApiSecurity:RequireApiKey = true and configure ApiSecurity:ApiKeys.
• Run behind HTTPS: terminate TLS at a reverse proxy (or configure Kestrel HTTPS). Ensure forwarded headers are configured if applicable.
• CORS: lock down origins (avoid AllowAnyOrigin() for production).

Reliability

• Persist Qdrant: store Qdrant data on durable storage (volumes/backups).
• Health checks: use /api/health and /api/health/detailed for monitoring.
• Resource sizing: embeddings + parsing can be CPU/RAM heavy; size accordingly.

Performance & quality

• Vector size must match your embedding model (e.g., mxbai-embed-large → 1024).
• Tune chunking: Chunking:ChunkSize and Chunking:ChunkOverlap.
• Tune retrieval: Rag:DefaultTopK and Rag:MinRelevanceScore.

Ops

• Logging: keep Production log levels at Info/Warn (Debug is noisy).
• Documents path: ensure your FileWatcher:WatchPath points to the mounted directory in your environment.

API reference

Using as a NuGet package

DNFileRAG is published as GothiaAI.DNFileRAG on NuGet:

dotnet add package GothiaAI.DNFileRAG

Then register services in your ASP.NET Core application:

using DNFileRAG.Infrastructure;

var builder = WebApplication.CreateBuilder(args);

// Add DNFileRAG services (includeFileWatcher: false for library usage)
builder.Services.AddDNFileRAGServices(builder.Configuration, includeFileWatcher: false);

Multi-tenant HTTP consumers

DNFileRAG is designed to be called as an external HTTP service (not just embedded as a NuGet package) by a multi-tenant application that needs to keep each tenant's documents and RAG answers isolated from every other tenant's. (An earlier "GothiaAI Portal" integration — HelperBot — used to be that consumer; it has since been removed from that codebase, so nothing in this repo currently depends on it. Whichever service calls DNFileRAG next should follow the pattern below from the start.)

• POST /api/ingest — upload documents with folder={tenantId} and collectionName={tenantId} for tenant isolation
• POST /api/query / POST /api/query/stream — pass the same collectionName
• GET /api/documents / POST /api/documents/reindex / DELETE /api/documents — pass the same collectionName
• GET /api/crawl?url=... — fetch a single web page
• POST /api/crawl — recursive crawl + auto-index for web sources

Tenant isolation is collection-based, not filter-based. Each tenant's files live under ./data/documents/{collectionName}/ and are stored in their own Qdrant collection (_factory.Create(collectionName)), never the shared default collection. ApiSecurity:RequireTenantIsolation (default true) makes collectionName mandatory and validates it against ^[a-zA-Z0-9_-]{1,128}$ on every ingest/query/document-list/reindex/delete request — a request that omits it is rejected with 400 rather than silently falling back to the shared collection. FileWatcherService's real-time watcher also derives the collection from a file's immediate subfolder, so an edit picked up by the file-system watcher (not the HTTP API) stays scoped to the same tenant.

Filtering matched results by filePaths over a single shared collection is not tenant isolation — it is a soft filter over a shared pool of vectors, and a missing/bypassed filter leaks across tenants. Use a per-tenant collectionName instead; set ApiSecurity:RequireTenantIsolation: false only for a genuinely single-tenant deployment (e.g. the gothiaai-docuchat demo widget, which has no tenant concept at all).

Embedding as a library

If you want to embed the RAG engine directly in your own .NET app (instead of calling the HTTP API), use the NuGet packages and register services:

using DNFileRAG.Infrastructure;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddDNFileRAGServices(builder.Configuration, includeFileWatcher: false);

Required configuration sections in appsettings.json:

• Qdrant - Vector store connection
• Embedding - Embedding provider (OpenAI, Ollama, etc.)
• Llm - LLM provider for answer generation
• Rag - Query parameters (topK, temperature, etc.)
• Chunking - Document chunking settings

Project structure

DNFileRAG/
├── src/
│   ├── DNFileRAG/                 # Web API host
│   ├── DNFileRAG.Core/            # Domain models & interfaces
│   └── DNFileRAG.Infrastructure/  # External service implementations
├── tests/
│   └── DNFileRAG.Tests/           # Test suite
└── examples/
    └── HelpChat/                  # Mock landing page + popup support chat

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License — see LICENSE.

Acknowledgments

• Qdrant - Vector database
• Serilog - Structured logging
• PdfPig - PDF parsing
• DocumentFormat.OpenXml - DOCX parsing