Berrycrawl 0.2.0

Berrycrawl .NET SDK

The official .NET SDK for scraping, crawling, searching, mapping, structured extraction, screenshots, and brand profiles.

Documentation · Dashboard · GitHub

Table of Contents

• Requirements
• Installation
• Reference
• Usage
• Environments
• Exception Handling
• Advanced
  • Retries
  • Timeouts
  • Raw Response
  • Additional Headers
  • Additional Query Parameters
  • Additional Body Properties
  • Forward Compatible Enums
• Contributing

Requirements

This SDK supports .NET Framework 4.6.2, .NET Standard 2.0, .NET 8, and .NET 9.

Installation

dotnet add package Berrycrawl

Reference

A full reference for this library is available here.

Usage

Set BERRYCRAWL_API_KEY to an API key from the Berrycrawl dashboard.

using Berrycrawl;

var client = new global::Berrycrawl.Berrycrawl(
    Environment.GetEnvironmentVariable("BERRYCRAWL_API_KEY")!
);

var page = await client.ScrapeAsync(new ScrapeDto
{
    Url = "https://example.com/pricing"
});

Crawl and search

var job = await client.CrawlAsync(new CrawlDto
{
    Url = "https://example.com/docs",
    Limit = 50
});

var results = await client.SearchAsync(new SearchDto
{
    Query = "best headless browser libraries",
    Limit = 10
});

Retrieve a brand profile

var brand = await client.Brand.RetrieveAsync(new BrandDto
{
    Url = "https://stripe.com"
});

Brand design system

Brand responses include an optional branding object for compatibility with older API deployments. When available, it contains the rendered light/dark scheme, semantic colors, typography, spacing, representative input and button styles, and semantic image roles. Use branding.images.favicon for the square icon and branding.images.logo for the wordmark.

Environments

This SDK allows you to configure different environments for API requests.

using Berrycrawl;

var client = new Berrycrawl(new ClientOptions
{
    BaseUrl = BerrycrawlEnvironment.Production
});

Exception Handling

When the API returns a non-success status code (4xx or 5xx response), a subclass of the following error will be thrown.

using Berrycrawl;

try {
    var response = await client.Brand.RetrieveAsync(...);
} catch (BerrycrawlApiException e) {
    System.Console.WriteLine(e.Body);
    System.Console.WriteLine(e.StatusCode);

    // Access the raw HTTP response (status code, URL, headers) off the exception
    var rawResponse = e.RawResponse;
    if (rawResponse != null)
    {
        System.Console.WriteLine(rawResponse.Url);
        if (rawResponse.Headers.TryGetValue("X-Request-Id", out var requestId))
        {
            System.Console.WriteLine($"Request ID: {requestId}");
        }
    }
}

Advanced

Retries

The SDK is instrumented with automatic retries with exponential backoff. A request will be retried as long as the request is deemed retryable and the number of retry attempts has not grown larger than the configured retry limit (default: 2).

Which status codes are retried depends on the retryStatusCodes generator configuration:

legacy (current default): retries on

• 408 (Timeout)
• 429 (Too Many Requests)
• 5XX (All server errors, including 500)

recommended: retries on

• 408 (Timeout)
• 429 (Too Many Requests)
• 502 (Bad Gateway)
• 503 (Service Unavailable)
• 504 (Gateway Timeout)

Use the MaxRetries request option to configure this behavior.

var response = await client.Brand.RetrieveAsync(
    ...,
    new RequestOptions {
        MaxRetries: 0 // Override MaxRetries at the request level
    }
);

Timeouts

The SDK defaults to a 30 second timeout. Use the Timeout option to configure this behavior.

var response = await client.Brand.RetrieveAsync(
    ...,
    new RequestOptions {
        Timeout: TimeSpan.FromSeconds(3) // Override timeout to 3s
    }
);

Raw Response

Access raw HTTP response data (status code, headers, URL) alongside parsed response data using the .WithRawResponse() method.

using Berrycrawl;

// Access raw response data (status code, headers, etc.) alongside the parsed response
var result = await client.Brand.RetrieveAsync(...).WithRawResponse();

// Access the parsed data
var data = result.Data;

// Access raw response metadata
var statusCode = result.RawResponse.StatusCode;
var headers = result.RawResponse.Headers;
var url = result.RawResponse.Url;

// Access specific headers (case-insensitive)
if (headers.TryGetValue("X-Request-Id", out var requestId))
{
    System.Console.WriteLine($"Request ID: {requestId}");
}

// For the default behavior, simply await without .WithRawResponse()
var data = await client.Brand.RetrieveAsync(...);

// .WithRawResponse() also works on streaming endpoints (returns IAsyncEnumerable<T> + RawResponse)
// and on endpoints with no response body (returns RawResponse only).

Additional Headers

If you would like to send additional headers as part of the request, use the AdditionalHeaders request option.

var response = await client.Brand.RetrieveAsync(
    ...,
    new RequestOptions {
        AdditionalHeaders = new Dictionary<string, string?>
        {
            { "X-Custom-Header", "custom-value" }
        }
    }
);

Additional Query Parameters

If you would like to send additional query parameters as part of the request, use the AdditionalQueryParameters request option.

var response = await client.Brand.RetrieveAsync(
    ...,
    new RequestOptions {
        AdditionalQueryParameters = new Dictionary<string, string>
        {
            { "custom_param", "custom-value" }
        }
    }
);

Additional Body Properties

If you would like to send additional body properties as part of the request, use the AdditionalBodyProperties request option. This is only applied to JSON requests.

var response = await client.Brand.RetrieveAsync(
    ...,
    new RequestOptions {
        AdditionalBodyProperties = new Dictionary<string, object>
        {
            { "custom_field", "custom-value" }
        }
    }
);

Forward Compatible Enums

This SDK uses forward-compatible enums that can handle unknown values gracefully.

using Berrycrawl;

// Using a built-in value
var listJobsRequestType = ListJobsRequestType.Crawl;

// Using a custom value
var customListJobsRequestType = ListJobsRequestType.FromCustom("custom-value");

// Using in a switch statement
switch (listJobsRequestType.Value)
{
    case ListJobsRequestType.Values.Crawl:
        Console.WriteLine("Crawl");
        break;
    default:
        Console.WriteLine($"Unknown value: {listJobsRequestType.Value}");
        break;
}

// Explicit casting
string listJobsRequestTypeString = (string)ListJobsRequestType.Crawl;
ListJobsRequestType listJobsRequestTypeFromString = (ListJobsRequestType)"crawl";

Contributing

While we value open-source contributions to this SDK, this library is generated programmatically. Additions made directly to this library would have to be moved over to our generation code, otherwise they would be overwritten upon the next generated release. Feel free to open a PR as a proof of concept, but know that we will not be able to merge it as-is. We suggest opening an issue first to discuss with us!

On the other hand, contributions to the README are always very welcome!