Piloterr 0.1.0

Piloterr .NET SDK

The official .NET SDK for the Piloterr web scraping API.

Scrape 300+ websites (Amazon, LinkedIn, Google, Zillow, Crunchbase…) as structured JSON with one API.

The SDK exposes 392 typed async methods, one per documented endpoint (with required properties checked at compile time), plus a generic RequestAsync.

Requirements

• .NET 8 or newer

Installation

dotnet add package Piloterr

Get your API key

Sign up for Piloterr or sign in, then copy your API key from the dashboard. new PiloterrClient() reads the PILOTERR_API_KEY environment variable.

Quick start

using Piloterr;

using var client = new PiloterrClient();
var response = await client.AmazonProductAsync(new AmazonProductParams { Query = "B09DFCB66S" });
Console.WriteLine(response.Json);

Parameter classes mirror the API reference. Use response.Deserialize<MyType>() to map the JSON onto your own types.

Binary responses

Endpoints that return a file (image, PDF…) give you the bytes; IsJson tells both cases apart:

var response = await client.WebsiteScreenshotAsync(new WebsiteScreenshotParams { Url = "https://example.com" });
if (!response.IsJson)
{
    await File.WriteAllBytesAsync("screenshot.png", response.Bytes);
}

Async jobs

Any call accepts is_async: true (useful past the 60-second server limit): the response is then a job_id. WaitForJobAsync polls GET /v1/jobs/{id} until a terminal status (CompletedJob, Error, NotFound, BadRequest):

var submitted = await client.AmazonProductAsync(new AmazonProductParams { Query = "B09DFCB66S", IsAsync = true });
var job = await client.WaitForJobAsync(
    submitted.Json.GetProperty("job_id").GetString()!,
    onProgress: j => Console.WriteLine($"{j.GetProperty("status")} {j.GetProperty("progress")}%"));

GET or POST

Methods use GET when the endpoint supports it (query string). POST sends the same parameters as JSON:

await client.AboutyouProductAsync(new AboutyouProductParams { Query = "test" }, new RequestOptions { Method = "POST" });

Any endpoint

var response = await client.RequestAsync("GET", "/v1/usage");

Errors

try
{
    await client.AmazonProductAsync(new AmazonProductParams { Query = "B09DFCB66S" });
}
catch (PiloterrApiException exception)
{
    Console.Error.WriteLine($"{exception.StatusCode}: {exception.Message}");
}

All of them derive from PiloterrException. Caller cancellation surfaces as OperationCanceledException.

Retries

Requests are retried up to 3 times after a network error, a 408, 500, 502, 503 or 504, or a rate limit — which the API returns as 401 (or 429) with a Rate limit exceeded message. Per-key quota errors (…: quota daily) are not retried, nor are 400, 402, 404 or client-side timeouts. The delay starts at 1 second and grows 1.5× per retry (capped at 30 seconds); a Retry-After header wins.

using var client = new PiloterrClient(new PiloterrClientOptions { MaxRetries = 5, RetryDelay = TimeSpan.FromMilliseconds(500) });
await client.AmazonProductAsync(new AmazonProductParams { Query = "B09DFCB66S" }, new RequestOptions { MaxRetries = 0 }); // one request without retries

Batch requests

BatchAsync runs a function per item with at most 10 in flight (concurrency) and returns one BatchResult per item, in input order. A failing item never stops the others.

string[] inputs = { "B09DFCB66S", "B0BSHF7WHW", "B08N5WRWNW" };
var results = await client.BatchAsync(inputs, (value, ct) =>
    client.AmazonProductAsync(new AmazonProductParams { Query = value }, cancellationToken: ct), concurrency: 5);
foreach (var result in results)
{
    if (result.Ok)
    {
        Console.WriteLine(result.Value!.Json);
    }
    else
    {
        Console.Error.WriteLine($"{inputs[result.Index]}: {result.Error!.Message}");
    }
}

Configuration

using var client = new PiloterrClient(new PiloterrClientOptions
{
    ApiKey = Environment.GetEnvironmentVariable("PILOTERR_API_KEY"),
    Timeout = TimeSpan.FromSeconds(60), // default 120 s
    HttpClient = myHttpClient,           // optional, you keep ownership
});

Endpoints

392 methods, generated from the OpenAPI document.

License

MIT — see LICENSE.