ScribeAi 1.0.0

ScribeAi — .NET Content Pipeline & Publishing NuGet Package

Table of Contents

1. Architecture Overview
2. Project Structure
3. Core Concepts
   • Content Pipeline
   • Pipeline Stages
   • Content Payload (DTO)
   • AI Providers
   • Content Sources
   • Publishers
   • Events
   • Extensions
   • Run Tracking & Resume
4. Configuration Reference
5. Data Flow Walkthrough
6. Extending ScribeAi

Architecture Overview

ScribeAi is a .NET 8/9 NuGet package that implements a modular content pipeline for automated content processing and publishing. It follows the Strategy + Manager pattern throughout, making every component replaceable.

┌─────────────────────────────────────────────────────────────────────┐
│                        IContentPipeline                            │
│  ProcessAsync(payload) / ResumeAsync(runId)                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────┐  ┌──────────────┐  ┌───────────────┐  ┌───────────┐ │
│  │  Scrape   │→│  AI Rewrite   │→│ Generate Image │→│ Optimize  │ │
│  │  Stage    │  │  Stage        │  │  Stage         │  │ Image    │ │
│  └──────────┘  └──────────────┘  └───────────────┘  └───────────┘ │
│       ↓                                                      ↓     │
│  ┌───────────────┐                            ┌──────────────┐     │
│  │ Create Article │←──────────────────────────│  Publish      │    │
│  │ Stage          │                            │  Stage        │    │
│  └───────────────┘                            └──────────────┘     │
│                                                                     │
│  ─── Powered by ───                                                │
│  IContentSourceManager  │  IAiService  │  IPublisherManager        │
│  IImageGenerator        │  IImageOptimizer  │  IScribeEventBus     │
└─────────────────────────────────────────────────────────────────────┘

Key design decisions:

• Immutable payload: ContentPayload is a C# record — stages produce new instances via with {} expressions instead of mutating state.
• Strategy + Manager: AI providers, content sources, and publishers are all managed through factory-based managers that resolve drivers by name.
• DI-first: Everything is registered in the DI container via services.AddScribeAi(configuration).
• Event-driven: A lightweight event bus (IScribeEventBus) fires events at each pipeline stage transition.

Project Structure

src/ScribeAi/
├── Abstractions/           17 interfaces (IAiService, IContentPipeline, etc.)
├── Ai/                     AI service + providers
│   ├── AiService.cs             Core AI orchestrator with fallback
│   ├── AiProviderManager.cs     Strategy manager for AI providers
│   ├── ContentRewriter.cs       Default IContentRewriter (prompt-based)
│   ├── ImageGenerator.cs        Default IImageGenerator (prompt-based)
│   ├── SeoSuggester.cs          Default ISeoSuggester (prompt-based)
│   └── Providers/               5 built-in: OpenAI, Claude, Gemini, Ollama, PiApi
├── Configuration/          Option classes (bound from appsettings.json)
│   ├── ScribeAiOptions.cs       Root options container
│   ├── AiOptions.cs             AI provider settings
│   ├── PipelineOptions.cs       Pipeline behavior (stages, tracking, halt-on-error)
│   ├── PublishingOptions.cs     Active channels list
│   ├── ImageOptions.cs          Image generation/optimization settings
│   ├── SourceOptions.cs         Content source settings
│   └── ExtensionOptions.cs      Extension toggles
├── Data/                   DTOs (records)
│   ├── ContentPayload.cs        Immutable pipeline state carrier
│   ├── PublishResult.cs          Per-channel publish outcome
│   ├── AiChatResponse.cs        AI provider response wrapper
│   ├── ContentFetchResult.cs    Source fetch outcome
│   └── SeoSuggestion.cs         SEO metadata from AI
├── DependencyInjection/    Registration + builder
│   ├── ServiceCollectionExtensions.cs   AddScribeAi() entry point
│   ├── ScribeAiBuilder.cs               Fluent customization builder
│   └── ApplicationBuilderExtensions.cs  UseScribeAi() middleware hook
├── Enums/                  ArticleStatus, PipelineRunStatus, PublishStatus, SourceType
├── Events/                 ScribeEventBus + 9 event records
├── Extensions/             IExtension implementations (TelegramApproval built-in)
├── Models/                 EF Core entities (Article, Tag, Category, etc.)
├── Persistence/            IScribeAiDbContext + entity configurations
├── Pipeline/               ContentPipeline + 6 stages
├── Publishing/             PublisherManager + 5 drivers (Log, Telegram, Facebook, etc.)
├── Services/               ImageOptimizer, WebScraper utility services
└── Sources/                ContentSourceManager + 3 drivers (Web, RSS, Text)

Core Concepts

Content Pipeline

The IContentPipeline (ContentPipeline) is the orchestrator. It sends an immutable ContentPayload through an ordered list of IPipelineStage implementations.

public interface IContentPipeline
{
    Task<ContentPayload> ProcessAsync(ContentPayload payload, CancellationToken ct = default);
    Task<ContentPayload> ResumeAsync(int runId, CancellationToken ct = default);
    IContentPipeline Through(params Type[] stages);
    IContentPipeline WithoutTracking();
    IContentPipeline OnProgress(Action<string, string> callback);
}

Default stage order (configured in ScribeAiOptions.Pipeline.Stages):

ScrapeStage → AiRewriteStage → GenerateImageStage → OptimizeImageStage → CreateArticleStage → PublishStage

Each stage receives the payload + a next delegate. It can:

• Transform & continue: return await next(payload with { Title = "New" }, ct);
• Reject & halt: return payload with { Rejected = true, RejectionReason = "Bad content" }; (don't call next)
• Skip silently: return await next(payload, ct);

Pipeline Stages

Content Payload (DTO)

ContentPayload is an immutable C# record that carries all state through the pipeline:

public record ContentPayload
{
    // Input
    public string? SourceUrl { get; init; }
    public StagedContent? StagedContent { get; init; }
    public IReadOnlyList<string> Categories { get; init; }

    // Scraping output
    public string? RawContent { get; init; }
    public string? CleanedContent { get; init; }

    // AI rewriting output
    public string? Title { get; init; }
    public string? Slug { get; init; }
    public string? Content { get; init; }
    public string? Description { get; init; }
    public string? MetaTitle { get; init; }
    public string? MetaDescription { get; init; }
    public string? ImagePrompt { get; init; }
    public int? CategoryId { get; init; }
    public IReadOnlyList<string> Tags { get; init; }

    // Image output
    public string? ImagePath { get; init; }

    // Article output
    public Article? Article { get; init; }

    // Publish output
    public IReadOnlyList<PublishResult> PublishResults { get; init; }

    // Control flow
    public bool Rejected { get; init; }
    public string? RejectionReason { get; init; }

    // Factory methods
    public static ContentPayload FromUrl(string url) => ...;
    public static ContentPayload FromStagedContent(StagedContent staged) => ...;

    // Snapshot for resume
    public Dictionary<string, string?> ToSnapshot() => ...;
    public static ContentPayload FromSnapshot(Dictionary<string, string?> snap) => ...;
}

AI Providers

IAiProviderManager manages multiple AI providers. IAiService is the high-level API used by stages.

Built-in providers: OpenAI, Claude (Anthropic), Gemini (Google), Ollama (local), PiApi.

public interface IAiService
{
    Task<AiChatResponse> ChatAsync(IReadOnlyList<ChatMessage> messages, ...);
    Task<string> CompleteAsync(string systemPrompt, string userPrompt, ...);
    Task<T?> CompleteJsonAsync<T>(string systemPrompt, string userPrompt, ...);
}

AiService handles fallback: if the primary model fails, it retries with AiOptions.FallbackModel.

Content Sources

IContentSourceManager manages content source drivers.

Built-in drivers:

• web — scrapes a URL via IWebScraper
• rss — fetches RSS/Atom feed entries
• text — wraps raw text input

Auto-detection order: rss → custom sources → web → text

Publishers

IPublisherManager manages publishing drivers and persists PublishLog entries.

Built-in drivers:

• log — writes to ILogger (development-only, no external calls)
• telegram — posts to Telegram via Bot API
• facebook — posts to Facebook Page API
• blogger — posts to Blogger API
• wordpress — posts to WordPress REST API

Each publisher implements:

public interface IPublisher
{
    string Channel { get; }
    bool Supports(Article article);
    Task<PublishResult> PublishAsync(Article article, PublishOptions? options, CancellationToken ct);
}

Events

IScribeEventBus is a lightweight in-process pub/sub system. Subscribe to typed events; stages fire them automatically.

9 event types:

// Subscribe
eventBus.Subscribe<ArticleCreatedEvent>(async (evt, ct) =>
{
    Console.WriteLine($"Article created: {evt.Article.Title}");
});

// Events are fired automatically by stages — you don't call Publish manually

Extensions

Extensions hook into the pipeline lifecycle via events. The built-in TelegramApprovalExtension sends staged content to a Telegram chat for approval before processing.

public interface IExtension
{
    string Name { get; }
    Task InitializeAsync(IServiceProvider services, CancellationToken ct);
}

Run Tracking & Resume

When Pipeline.TrackRuns = true, each ProcessAsync() call creates a PipelineRun record:

• Tracks Status (Pending → Running → Completed/Failed/Rejected)
• Snapshots payload after each stage → enables resume from failure
• ResumeAsync(runId) rehydrates the payload and continues from CurrentStageIndex

Configuration Reference

All configuration lives under the "ScribeAi" section in appsettings.json:

{
  "ScribeAi": {
    "Pipeline": {
      "HaltOnError": true,
      "TrackRuns": true
    },
    "Ai": {
      "DefaultProvider": "openai",
      "DefaultModel": "gpt-4o",
      "FallbackModel": "gpt-4o-mini",
      "OutputLanguage": "English",
      "MaxTokens": 4096,
      "Providers": {
        "openai": { "ApiKey": "sk-..." },
        "claude": { "ApiKey": "sk-ant-..." },
        "gemini": { "ApiKey": "AIza..." },
        "ollama": { "BaseUrl": "http://localhost:11434" },
        "piapi":  { "ApiKey": "..." }
      }
    },
    "Publishing": {
      "Channels": ["telegram", "log"],
      "Drivers": {
        "telegram": { "BotToken": "...", "ChatId": "..." },
        "facebook": { "PageAccessToken": "...", "PageId": "..." },
        "blogger":  { "BlogId": "...", "ApiKey": "..." },
        "wordpress": { "SiteUrl": "...", "Username": "...", "Password": "..." }
      }
    },
    "Image": {
      "Optimize": true,
      "DefaultModel": "dall-e-3",
      "DefaultSize": "1024x1024",
      "DefaultQuality": "standard"
    },
    "Sources": {
      "DefaultDriver": "web"
    },
    "Categories": {
      "0": "General",
      "1": "Technology",
      "2": "Science",
      "3": "Health"
    }
  }
}

Data Flow Walkthrough

Here's the complete journey of a URL through the pipeline:

Input: ContentPayload.FromUrl("https://example.com/article")
  │
  ▼
┌─── ScrapeStage ─────────────────────────────────────────────────┐
│ 1. Check: RawContent already set? → Skip                       │
│ 2. IContentSourceManager.FetchAsync(sourceUrl)                  │
│    → auto-detects driver (web/rss/text)                         │
│    → returns ContentFetchResult { Content, Title }              │
│ 3. Sets: RawContent, CleanedContent, Title (if available)       │
│ 4. Fires: ContentScrapedEvent                                   │
└──────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─── AiRewriteStage ──────────────────────────────────────────────┐
│ 1. Check: no content? → Skip                                   │
│ 2. Builds system prompt with categories + target language       │
│ 3. IAiService.CompleteJsonAsync<AiRewriteResult>(prompt, content)│
│    → AI returns: title, content, description, meta, tags,       │
│      image_prompt, category_index, rejected, rejection_reasons  │
│ 4. If rejected: payload.Rejected = true, return (halt pipeline) │
│ 5. Sets: Title, Slug, Content, Description, MetaTitle,          │
│          MetaDescription, ImagePrompt, CategoryId, Tags         │
│ 6. Fires: ContentRewrittenEvent                                 │
└──────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─── GenerateImageStage ──────────────────────────────────────────┐
│ 1. Check: no ImagePrompt or ImagePath set? → Skip               │
│ 2. IImageGenerator.GenerateAsync(prompt, model, size, quality)  │
│    → delegates to AI provider's image API                       │
│ 3. Sets: ImagePath (URL or local path)                          │
│ 4. Fires: ImageGeneratedEvent                                   │
└──────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─── OptimizeImageStage ──────────────────────────────────────────┐
│ 1. Check: no ImagePath or optimization disabled? → Skip         │
│ 2. IImageOptimizer.OptimizeAsync(imagePath)                     │
│    → compresses/converts to webp                                │
│ 3. Sets: ImagePath (optimized path)                             │
│ 4. Fires: ImageOptimizedEvent                                   │
└──────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─── CreateArticleStage ──────────────────────────────────────────┐
│ 1. Check: payload rejected? → Skip                              │
│ 2. Creates Article entity from payload fields                   │
│ 3. Persists to DB via IScribeAiDbContext.SaveChangesAsync()     │
│ 4. Creates/attaches Tag entities                                │
│ 5. Marks StagedContent as published (if from staged)            │
│ 6. Sets: Article (saved with ID)                                │
│ 7. Fires: ArticleCreatedEvent                                   │
└──────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─── PublishStage ────────────────────────────────────────────────┐
│ 1. Check: no Article? → Skip                                   │
│ 2. IPublisherManager.PublishToChannelsAsync(article, channels)  │
│    → iterates configured channels (telegram, facebook, etc.)    │
│    → calls each driver's PublishAsync()                         │
│    → persists PublishLog per channel                            │
│ 3. Sets: PublishResults (list of per-channel outcomes)          │
│ 4. Fires: ArticlePublishedEvent (once per channel)             │
│ 5. On exception + HaltOnError: rejects payload                 │
└──────────────────────────────────────────────────────────────────┘
  │
  ▼
Output: ContentPayload with Article, PublishResults, all metadata

Extending ScribeAi

Custom AI Provider

services.AddScribeAi(configuration)
    .AddAiProvider("my-provider", sp => new MyAiProvider(sp.GetRequiredService<HttpClient>()));

Implement IAiProvider:

public class MyAiProvider : IAiProvider
{
    public string Name => "my-provider";
    public Task<AiChatResponse> ChatAsync(IReadOnlyList<ChatMessage> messages, string? model, int? maxTokens, bool jsonMode, CancellationToken ct);
    public Task<string?> GenerateImageAsync(string prompt, string model, string size, string quality, CancellationToken ct);
}

Custom Publisher

services.AddScribeAi(configuration)
    .AddPublisher("medium", sp => new MediumDriver(sp.GetRequiredService<HttpClient>()));

Custom Content Source

services.AddScribeAi(configuration)
    .AddContentSource("api", sp => new ApiContentSource(sp.GetRequiredService<HttpClient>()));

Custom Pipeline Stage

Create a class implementing IPipelineStage:

public class TranslationStage : IPipelineStage
{
    public async Task<ContentPayload> HandleAsync(
        ContentPayload payload,
        PipelineStageDelegate next,
        CancellationToken ct = default)
    {
        // Transform
        var translated = await TranslateContent(payload.Content, ct);
        var updated = payload with { Content = translated };

        // Continue to next stage
        return await next(updated, ct);
    }
}

Register it:

services.AddScribeAi(configuration)
    .WithStages(
        typeof(ScrapeStage),
        typeof(AiRewriteStage),
        typeof(TranslationStage),  // <-- inserted
        typeof(GenerateImageStage),
        typeof(OptimizeImageStage),
        typeof(CreateArticleStage),
        typeof(PublishStage));

Custom Extension

public class SlackNotificationExtension : IExtension
{
    public string Name => "slack-notification";

    public Task InitializeAsync(IServiceProvider services, CancellationToken ct)
    {
        var eventBus = services.GetRequiredService<IScribeEventBus>();
        eventBus.Subscribe<ArticleCreatedEvent>(async (evt, ct) =>
        {
            // Send Slack notification
        });
        return Task.CompletedTask;
    }
}

// Register
services.AddScribeAi(configuration).AddExtension(new SlackNotificationExtension());