WebScrapingAI 4.0.0

WebScraping.AI .NET Client

Official .NET client for the WebScraping.AI API. Async, zero runtime dependencies, multi-targets netstandard2.0 and net8.0.

Install

dotnet add package WebScrapingAI

Or in your .csproj:

<PackageReference Include="WebScrapingAI" Version="4.0.0" />

Quickstart

using WebScrapingAI;

using var client = new WebScrapingAIClient(new WebScrapingAIClientOptions
{
    ApiKey = "YOUR_API_KEY",
});

// Or read from the WEBSCRAPING_AI_API_KEY environment variable:
using var client2 = new WebScrapingAIClient();

// Get the HTML of a page
string html = await client.HtmlAsync(new HtmlRequest
{
    Url = "https://example.com",
    Js = true,
});

// Ask a question about a page
string answer = await client.QuestionAsync(new QuestionRequest
{
    Url = "https://example.com",
    Question = "What is this page about?",
});

// Extract structured fields
FieldsResult result = await client.FieldsAsync(new FieldsRequest
{
    Url = "https://example.com/product/123",
    Fields = new Dictionary<string, string>
    {
        ["title"] = "Main product title",
        ["price"] = "Current price",
    },
});
Console.WriteLine(result.Result?["price"]);

API

All methods are async and accept an optional CancellationToken.

Common request options

Every request type extends CommonRequest, which exposes shared options like Js, Country, Proxy, Timeout, WaitFor, Headers, Device, JsScript, etc. See the API reference for the full list.

Errors

The SDK throws unchecked exceptions:

All of the above extend WebScrapingAIException (a RuntimeException). The transport-level pair (ApiTimeoutException, ApiConnectionException) do not extend ApiException, so catch (ApiException) reliably means "we got an HTTP response back".

try
{
    var html = await client.HtmlAsync(new HtmlRequest { Url = "..." });
}
catch (RateLimitException ex)
{
    // back off
}
catch (ApiException ex)
{
    // any other non-2xx; ex.Status has the HTTP code
}
catch (WebScrapingAIException ex)
{
    // transport failure (timeout or connection)
}

Response-shape notes

Two endpoints return shapes that differ slightly from the OpenAPI spec examples; the SDK preserves them verbatim so the wire layout stays observable:

• FieldsAsync returns FieldsResult whose .Result holds the extracted fields under the API's result key.
• SelectedMultipleAsync returns SelectedMultipleResult whose .Results is IReadOnlyList<IReadOnlyList<string>> (the API wraps matches in an outer array).

Configuration

new WebScrapingAIClientOptions
{
    ApiKey = "...",                                          // or WEBSCRAPING_AI_API_KEY env var
    BaseUrl = "https://api.webscraping.ai",                  // override for staging/test
    Timeout = TimeSpan.FromSeconds(60),                      // per-request
    UserAgent = "webscraping-ai-dotnet/4.0.0",               // overridable
    HttpHandler = null,                                      // plug in a custom HttpMessageHandler
};

Targets

• netstandard2.0 — covers .NET Framework 4.6.1+, Mono, Xamarin, Unity, every modern .NET.
• net8.0 — current LTS; enables nullable annotations and modern BCL features at the call site.

Smoke test

The samples/Smoke console app exercises all 7 endpoints against the live API. Costs ~17 credits per run.

WEBSCRAPING_AI_API_KEY=... dotnet run --project samples/Smoke

Contributing

dotnet build
dotnet test
dotnet format

The repo pins .NET 10 via mise.toml; if you use mise, cd into the repo and run mise install to pick it up.

License

MIT — see LICENSE.