GhostCrawlApi 2.3.5

ghostcrawl — C# SDK

The official C# client for the GhostCrawl API. Collect web data at scale — scrape, crawl, search, and extract structured data.

Install

Minimum runtime: .NET 8.

dotnet add package GhostCrawlApi --version 2.3.0

Or add to your .csproj:

<PackageReference Include="GhostCrawlApi" Version="2.3.0" />

Quickstart

using GhostCrawlSdk;

// Token from constructor or GHOSTCRAWL_API_KEY environment variable
var client = new GhostCrawlClient("gck_live_YOUR_KEY");

// Scrape a URL
var result = await client.ScrapeAsync("https://example.com");
Console.WriteLine(result["content"]);

// Start a crawl run and WAIT for it to finish — event-driven, no poll loop.
// The wait is server-blocking (the API long-polls on the run's completion
// event); it returns when the run is terminal or waitTimeout elapses.
var run = await client.CrawlRuns.StartAsync(
    "https://example.com",
    wait: true,
    waitTimeout: TimeSpan.FromMinutes(5));
Console.WriteLine($"Status: {run["status"]}");  // completed | failed | cancelled

// Already have a run_id? Wait on it directly (honours CancellationToken):
var final = await client.CrawlRuns.WaitForCompletionAsync((string)run["run_id"]!);

// Prefer fire-and-forget? Start without wait and register a webhook instead:
//   var started = await client.CrawlRuns.StartAsync("https://example.com");
//   await client.Webhooks.CreateAsync("https://you.example.com/hook",
//       new[] { "crawl.completed" });

// Search
var search = await client.SearchAsync("latest C# releases");
Console.WriteLine(search["results"]);

Authentication

Pass your API key directly to the constructor:

var client = new GhostCrawlClient("gck_live_YOUR_KEY");

Or set the GHOSTCRAWL_API_KEY environment variable and use the no-arg constructor:

var client = new GhostCrawlClient();

All requests are authenticated with Authorization: Bearer <token>. No other auth scheme is supported.

For self-hosted deployments, override the base URL:

var client = new GhostCrawlClient("gck_live_YOUR_KEY", "https://your-instance.example.com");

Or set GHOSTCRAWL_BASE_URL.

Timeouts

Scrape and crawl jobs run through the rendering fleet and can take longer than the framework's default 100s. The client defaults to a 180s per-request timeout; override it (or disable it) via the constructor:

using System;

// Custom 5-minute timeout for heavy crawls
var client = new GhostCrawlClient("gck_live_YOUR_KEY", timeout: TimeSpan.FromMinutes(5));

// Disable the timeout entirely
var noTimeout = new GhostCrawlClient("gck_live_YOUR_KEY", timeout: System.Threading.Timeout.InfiniteTimeSpan);

Extract structured data

var schema = new Dictionary<string, object?>
{
    ["type"] = "object",
    ["properties"] = new Dictionary<string, object?>
    {
        ["title"]       = new Dictionary<string, object?> { ["type"] = "string" },
        ["price"]       = new Dictionary<string, object?> { ["type"] = "number" },
        ["description"] = new Dictionary<string, object?> { ["type"] = "string" },
    }
};

var result = await client.ExtractAsync("https://example.com/product/123", schema);
Console.WriteLine(result["data"]);

Browser utilities — content, screenshot, PDF

// Rendered content as a JSON envelope: {url, status, format, status_code, content, bytes}
var page = await client.ContentAsync("https://example.com");
Console.WriteLine($"{page["status_code"]} — {page["bytes"]} bytes");

// Screenshot — returns raw PNG bytes
byte[] png = await client.ScreenshotAsync("https://example.com", fullPage: true);
await File.WriteAllBytesAsync("page.png", png);

// PDF — returns raw application/pdf bytes (Chrome-only; a Firefox/WebKit identity
// is rejected with a 400 pdf_engine_unsupported)
byte[] pdf = await client.PdfAsync("https://example.com", paperFormat: "a4");
await File.WriteAllBytesAsync("page.pdf", pdf);

Agent (BYO model, account-gated)

The agent lane runs a natural-language browser task. It is bring-your-own-model — supply your own providerConfig — and account-gated: the API returns 404 not_found unless the capability is enabled for your account. AgentAsync does not throw on that 404 — it returns the problem+json body as a dictionary, so branch on result.ContainsKey("detail").

// provider_config is BYO — reference your provider key by env-var name, never a literal.
var providerConfig = new Dictionary<string, object?>
{
    ["provider"] = "openai",
    ["api_key"]  = "OPENAI_API_KEY",
    ["model"]    = "gpt-4o",
};

var result = await client.AgentAsync(
    "https://books.toscrape.com",
    "click the 'Books to Scrape' link",
    providerConfig);

if (result.ContainsKey("detail"))
    Console.WriteLine("agent lane not enabled for this account (BYO/gated)");
else
    Console.WriteLine(result);

Error handling

using GhostCrawlSdk;

try
{
    var result = await client.ScrapeAsync("https://example.com");
}
catch (AuthenticationException e)
{
    Console.Error.WriteLine($"Check your API key: {e.Message}");
}
catch (RateLimitException e)
{
    Console.Error.WriteLine($"Rate limit — back off and retry: {e.Message}");
}
catch (PaymentRequiredException e)
{
    Console.Error.WriteLine($"Spend limit reached: {e.Message}");
}
catch (InvalidRequestException e)
{
    Console.Error.WriteLine($"Bad request: {e.Message}");
}
catch (ApiException e)
{
    Console.Error.WriteLine($"Server error {e.StatusCode}: {e.Message}");
}
catch (GhostCrawlException e)
{
    Console.Error.WriteLine($"API error: {e.Message}");
}

All resources

Architecture

The facade delegates all HTTP transport, URL routing, serialization, and authentication to the Kiota-generated canonical core (_generated/). The facade is the shipped API surface; the Kiota core regenerates automatically when the spec changes.

License

Proprietary — GhostCrawl Software License. See LICENSE.