SuperScrapers.Client 1.4.14

SuperScrapers.Client configuration and usage

SuperScrapers.Client is the typed .NET client for the SuperScrapers server. It submits scraping jobs to the server and retrieves their rendered HTML. A running server and at least one eligible browser agent are required to process jobs; installing the package does not start either component.

This guide describes SuperScrapers.Client version 1.3.12.1 and is included as the README of the NuGet package. Server-side limits below describe the current repository implementation.

1. Install the package

Use a .NET 10 project:

dotnet add package SuperScrapers.Client --version 1.3.12.1 --source https://api.nuget.org/v3/index.json

Equivalent project reference:

<ItemGroup>
    <PackageReference Include="SuperScrapers.Client" Version="1.3.12.1" />
</ItemGroup>

SuperScrapers.Contracts is installed transitively. It provides ScrapeRequest, ScrapeResult, JobState, QueryResult<T> and the other public transport types. You do not need references to the Host, Core, Sqlite or agent projects.

2. Obtain an application API key

In the SuperScrapers dashboard, sign in with administrator access:

1. Open Applications (/applications).
2. Select Add application and enter the consuming application's name.
3. Select Create API key.
4. Save the displayed key immediately. The dashboard does not show it again.

Use this application key as SuperScrapers:ApiKey. The client sends it automatically in the X-SuperScrapers-Key HTTP header. An application ID, dashboard TOTP code, agent identity key or enrollment token cannot replace it.

Each key identifies one application. Its jobs and profiles are isolated from other applications. Revoking a key prevents further access and cancels that application's queued or running jobs. If the key is lost, create a new application key; profiles belonging to the previous application cannot be used with the new application.

3. Configure the consuming application

Minimal Program.cs registration

The client uses https://superscrapers.appliman.com automatically when ServerUrl is not assigned. Only the application API key is required in Program.cs; no server URL or appsettings.json entry is needed for the default server.

Add the following registration after creating the builder and before builder.Build(). Keep the consuming application's existing endpoint and Blazor configuration.

using SuperScrapers.Client;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSuperScrapers(_options =>
{
    _options.ApiKey = "REPLACE_WITH_YOUR_APPLICATION_KEY";
});

The placeholder shows the single value to supply. To keep the actual key out of source control, use the configuration-based variant below.

Secrets and environment variables

For local ASP.NET Core development, run these commands in the consuming project directory:

dotnet user-secrets init
dotnet user-secrets set "SuperScrapers:ApiKey" "REPLACE_WITH_YOUR_APPLICATION_KEY"

For production or Docker, supply the key through your environment or secret manager:

SuperScrapers__ApiKey=REPLACE_WITH_YOUR_APPLICATION_KEY

WebApplication.CreateBuilder loads JSON configuration, development user secrets and environment variables. Environment variables override JSON values. A plain ServiceCollection does not load configuration by itself.

Keep the key in the consuming server process. Do not place it in a Blazor WebAssembly application, browser JavaScript, query string or log. For Blazor Server and SSR, register and call the client on the server.

Read the key from configuration in Program.cs, still leaving ServerUrl at its default:

using SuperScrapers.Client;

var builder = WebApplication.CreateBuilder(args);

var _apiKey = builder.Configuration["SuperScrapers:ApiKey"];

if (string.IsNullOrWhiteSpace(_apiKey))
{
    throw new InvalidOperationException(
        "SuperScrapers:ApiKey must contain an application API key.");
}

builder.Services.AddSuperScrapers(_options =>
{
    _options.ApiKey = _apiKey;
});

The key check above belongs to your application startup. The library does not validate these settings eagerly and does not bind the SuperScrapers configuration section automatically. Registering Configure<SuperScrapersOptions> alone does not configure this client: AddSuperScrapers uses the values supplied to its callback.

Optional custom server

Assign ServerUrl only when you need another server:

builder.Services.AddSuperScrapers(_options =>
{
    _options.ApiKey = _apiKey;
    _options.ServerUrl = new Uri("https://scrapers.example.com/");
});

Use this registration instead of the default-server registration. Supply an absolute HTTP(S) base URL, not the target website URL or a URL ending in /api/v1/jobs. A trailing / matters when the base URL contains a path prefix: the client appends relative paths such as api/v1/jobs. Any path prefix must also be supported by the server and reverse proxy.

Available client settings

AddSuperScrapers registers SuperScrapersClient as an IHttpClientFactory typed client and sets a 30-second timeout per HTTP request. There are no Timeout, polling-interval, retry-count, profile or session properties on SuperScrapersOptions in this version.

Resolve the typed client through dependency injection; do not capture it in a singleton. Settings are captured during registration. Restart the consuming application after changing the key or server URL; changing a configuration file does not update an already registered client automatically.

4. Fetch rendered HTML

The following service can be added to your consuming project as PageScraper.cs:

using System.Net.Http;
using System.Text.Json;

using Microsoft.Extensions.Logging;

using SuperScrapers.Client;
using SuperScrapers.Contracts.Scraping;

public sealed class PageScraper(
    SuperScrapersClient client,
    ILogger<PageScraper> logger)
{
    public async Task<string?> GetHtml(
        string url,
        CancellationToken cancellationToken)
    {
        try
        {
            var _request = new ScrapeRequest(
                Url: url,
                SessionKey: "public-pages",
                TimeoutSeconds: 120);

            var _result = await client.Scrape(_request, cancellationToken);

            if (_result.HasError)
            {
                foreach (var _rule in _result.BrokenRules)
                {
                    logger.LogWarning(
                        "SuperScrapers API error {Code}: {Message}",
                        _rule.Code,
                        _rule.Message);
                }

                return null;
            }

            if (_result.Data is not { } _data)
            {
                logger.LogWarning("SuperScrapers returned no job data.");

                return null;
            }

            if (_data.Job.State != JobState.Succeeded
                || _data.Html == null)
            {
                logger.LogWarning(
                    "Scrape job {JobId} ended in state {State} with error {ErrorCode}.",
                    _data.Job.Id,
                    _data.Job.State,
                    _data.Job.ErrorCode);

                return null;
            }

            return _data.Html;
        }
        catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
        {
            throw;
        }
        catch (OperationCanceledException)
        {
            logger.LogWarning("The SuperScrapers operation timed out.");

            return null;
        }
        catch (HttpRequestException)
        {
            logger.LogWarning("The SuperScrapers server could not be reached.");

            return null;
        }
        catch (JsonException)
        {
            logger.LogWarning("The SuperScrapers server returned invalid JSON.");

            return null;
        }
    }
}

Register this consuming service before building your application:

builder.Services.AddTransient<PageScraper>();

Inject PageScraper into a server-side component, endpoint or scoped service. For a Blazor Server/SSR component:

@inject PageScraper Scraper

Call await Scraper.GetHtml("https://example.com/", cancellationToken) with a token tied to the request or component lifetime. The example uses a public page; replace it with the intended target URL.

Scrape submits the job, polls its status every 500 ms, and retrieves the result when the job leaves Queued or Running. A completed API call does not necessarily mean a successful scrape. Check both HasError and Data.Job.State; the server can return a failed job without any BrokenRules.

Returned HTML is page source. Treat it as untrusted and potentially sensitive content. Display it as encoded text or process it with an HTML parser instead of inserting it into your application's page as executable markup.

5. Configure request options, profiles and sessions

ScrapeRequest has the following constructor parameters:

These request values are independent of the client registration. Placing ProfileId, SessionKey or TimeoutSeconds in JSON configuration has no effect unless your application reads them and passes them to ScrapeRequest.

Public pages

Leave ProfileId as null. Use a stable session key if successive pages should share browser state. Use distinct keys to isolate accounts or workflows; generating a new session key on every call prevents session reuse.

Authenticated sites or content checks

1. In the dashboard, create a Site profile for the application identified by your API key.
2. Set Host to the exact target hostname, without scheme or path. Subdomains are distinct hosts.
3. For authenticated sites, configure the login URL, credentials, signed-in selector and declarative login steps.
4. Configure a content-ready CSS selector and required text when HTTP 200 alone does not prove the expected content is loaded.
5. Save the profile. Its GUID is present in the edit-page URL /profiles/{id}.
6. Pass that GUID as ProfileId and use a stable SessionKey for the account.

Example inside a method where _profileId is the saved profile GUID and client is the injected client:

var _request = new ScrapeRequest(
    Url: "https://example.com/account",
    SessionKey: "supplier-account-42",
    ProfileId: _profileId,
    TimeoutSeconds: 180);

var _result = await client.Scrape(_request, cancellationToken);

Credentials and login steps are managed on the server. The client does not accept site usernames, passwords, CSS selectors or arbitrary JavaScript as ScrapeRequest properties. See Site profiles and sessions for profile configuration.

Browser sessions are isolated by application + exact host + profile ID/revision + session key. Jobs sharing a session execute sequentially and reuse the assigned agent while it remains eligible. An agent being busy queues the work. A blocked or unavailable agent can cause a fresh session on another agent, without copying the previous cookies. Editing a profile changes its revision and creates a fresh session for subsequent submissions.

Public-address validation applies to target navigation, redirects and browser subrequests. Loopback, private networks, metadata addresses and unsupported ports are rejected. This restriction concerns the scraped website; it does not prohibit configuring a local SuperScrapers server URL during development.

6. Manage jobs explicitly

Use the methods below when your application needs to store a job ID, resume polling later or cancel a specific job:

Example inside a method with an injected client and a cancellationToken:

var _request = new ScrapeRequest(
    Url: "https://example.com/",
    SessionKey: "public-pages",
    IdempotencyKey: "catalog-refresh-20261008-001");

var _submitted = await client.Submit(_request, cancellationToken);

if (_submitted.HasError || _submitted.Data == null)
{
    return;
}

var _jobId = _submitted.Data.Id;

// Persist _jobId in your application if polling must survive a restart.
var _status = await client.GetStatus(_jobId, cancellationToken);

if (!_status.HasError
    && _status.Data?.Job.State == JobState.Succeeded)
{
    var _result = await client.GetResult(_jobId, cancellationToken);

    if (!_result.HasError && _result.Data?.Html is { } _html)
    {
        // Process or store _html in your application.
    }
}

This example checks the status once. Your own polling loop or background worker must repeat GetStatus while the state is Queued or Running, with a delay and a cancellation token. GetResult can return Html == null before completion or after HTML retention expires. Current server retention is one hour for HTML and 30 days for job metadata.

To cancel explicitly:

var _cancelled = await client.Cancel(_jobId, cancellationToken);

if (_cancelled.HasError)
{
    // Handle _cancelled.BrokenRules.
}

Use the same application key to retrieve or cancel a job. Cancellation updates the server-side job state; it does not promise that the browser has already stopped at the instant the call returns.

Idempotency and retries

The client does not automatically retry failed HTTP calls. Scrape generates an idempotency key only when IdempotencyKey is null; Submit does not generate one.

For application-managed retries, create and persist a key for the logical operation before the first call, then reuse the same request values and key. A new Scrape call without an explicit key generates a new key and can create another job.

With the same application and idempotency key, an identical request returns the existing job while its metadata is retained. Reusing the key with different request values is rejected with not_available. Use a new key for a new scrape; reusing a completed or cancelled job's key does not restart that job.

7. Understand deadlines and errors

Three different time limits

Increasing TimeoutSeconds does not increase the HTTP timeout. It gives the server more time to queue and execute the job. Your calling endpoint, reverse proxy or component lifetime can still cancel the overall operation sooner.

If polling/result retrieval in Scrape throws OperationCanceledException after successful submission, the client attempts to cancel the known job with an independent five-second token, then rethrows. Cleanup is best effort. A transport error during submission can leave acceptance uncertain; an explicit idempotency key allows your application to resubmit the same logical operation safely. Cancelling Submit, GetStatus or GetResult directly does not automatically call Cancel.

API errors, job failures and exceptions

Check these separately:

1. API rejection: HasError == true; inspect every BrokenRule.Code and .Message.
2. Scraping failure: HasError == false, but Data.Job.State is Failed, Cancelled or Expired; inspect Job.ErrorCode, Job.HttpStatus, Job.FinalUrl, Attempts and Session.
3. Transport/format failure: HttpRequestException, OperationCanceledException or JsonException can be thrown; they are not converted into BrokenRules by the client.

Terminal states are Succeeded, Failed, Cancelled and Expired. Job.HttpStatus is the target website's HTTP status, which is different from the SuperScrapers API HTTP response status. An HTTP 200 challenge page can still produce a blocked job.

For non-success API HTTP responses, the client returns a rule named http_<status> with the message The server rejected the request. It does not expose that response body. For a successful response whose JSON value is null, it returns invalid_response; malformed JSON or an empty body can instead throw JsonException.

8. Troubleshooting

The client API covers job submission, status, results and cancellation. Application-key creation, profile editing and agent enrollment are dashboard/server operations. See Installation for server and agent setup.