WebSpark.HttpClientUtility 2.1.2

WebSpark.HttpClientUtility

Drop-in HttpClient wrapper with Polly resilience, response caching, and OpenTelemetry for .NET 8-10+ APIs—configured in one line

Stop writing 50+ lines of HttpClient setup. Get enterprise-grade resilience (retries, circuit breakers), intelligent caching, structured logging with correlation IDs, and OpenTelemetry tracing in a single AddHttpClientUtility() call. Perfect for microservices, background workers, and web scrapers.

🚀 Why Choose WebSpark.HttpClientUtility?

Your HTTP setup in 1 line vs. 50+

When to use WebSpark:

• ✅ Building microservices with distributed tracing requirements
• ✅ Need resilience patterns without writing Polly boilerplate
• ✅ Want intelligent caching for API rate-limit compliance
• ✅ Building web scrapers or crawlers (with Crawler package)

When NOT to use WebSpark:

• ❌ You need declarative, type-safe API clients (use Refit)
• ❌ You want maximum control and minimal magic (use raw HttpClient)
• ❌ Legacy .NET Framework 4.x projects (WebSpark requires .NET 8+)

🛡️ Production Trust

Battle-Tested & Production-Ready

• ✅ 252+ unit tests with 100% passing - tested on .NET 8, 9, and 10
• ✅ Continuous Integration via GitHub Actions - every commit tested
• ✅ Semantic Versioning - predictable, safe upgrades
• ✅ Zero breaking changes within major versions - backward compatibility guaranteed
• ✅ Framework Support: .NET 8 LTS (supported until Nov 2026), .NET 9, .NET 10 (Preview)
• ✅ MIT Licensed - free for commercial use

Support & Maintenance

• 🔄 Active development - regular updates and improvements
• 📅 Long-term support - each major version supported for 18+ months
• 💬 Community support - GitHub Discussions for questions and best practices
• 📖 Comprehensive documentation - Full docs site

Breaking Change Commitment

We follow semantic versioning strictly:

• Patch versions (2.0.x): Bug fixes only, zero breaking changes
• Minor versions (2.x.0): New features, backward compatible
• Major versions (x.0.0): Breaking changes with detailed migration guides

📦 v2.0 - Now in Two Focused Packages!

Starting with v2.0, the library is split into two packages:

Upgrading from v1.x? Most users need no code changes! See Migration Guide.

📚 Documentation

View Full Documentation →

The complete documentation site includes:

• Getting started guide
• Feature documentation
• API reference
• Code examples
• Best practices

⚡ 30-Second Quick Start

Install

dotnet add package WebSpark.HttpClientUtility

Minimal Example (Absolute Minimum)

// Program.cs
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpClientUtility();
var app = builder.Build();

app.MapGet("/weather", async (IHttpRequestResultService http) =>
{
    var request = new HttpRequestResult<WeatherData>
    {
        RequestPath = "https://api.weather.com/forecast?city=Seattle",
        RequestMethod = HttpMethod.Get
    };
    var result = await http.HttpSendRequestResultAsync(request);
    return result.IsSuccessStatusCode ? Results.Ok(result.ResponseResults) : Results.Problem();
});

app.Run();

record WeatherData(string City, int Temp);

That's it! You now have:

• ✅ Automatic correlation IDs for tracing
• ✅ Structured logging with request/response details
• ✅ Request timing telemetry
• ✅ Proper error handling and exception management
• ✅ Support for .NET 8 LTS, .NET 9, and .NET 10 (Preview)

<details> <summary>📖 Show more: Service-based pattern with error handling</summary>

// Program.cs
builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;      // Cache responses
    options.EnableResilience = true;   // Retry on failure
});

// WeatherService.cs
public class WeatherService
{
    private readonly IHttpRequestResultService _http;
    private readonly ILogger<WeatherService> _logger;

    public WeatherService(
        IHttpRequestResultService http,
        ILogger<WeatherService> logger)
    {
        _http = http;
        _logger = logger;
    }

    public async Task<WeatherData?> GetWeatherAsync(string city)
    {
        var request = new HttpRequestResult<WeatherData>
        {
            RequestPath = $"https://api.weather.com/forecast?city={city}",
            RequestMethod = HttpMethod.Get,
            CacheDurationMinutes = 10  // Cache for 10 minutes
        };

        var result = await _http.HttpSendRequestResultAsync(request);

        if (!result.IsSuccessStatusCode)
        {
            _logger.LogError(
                "Weather API failed: {StatusCode} - {Error}",
                result.StatusCode,
                result.ErrorDetails
            );
            return null;
        }

        return result.ResponseResults;
    }
}

</details>

<details> <summary>📖 Show more: Full-featured with auth and observability</summary>

// Program.cs - Advanced configuration
builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;
    options.EnableResilience = true;
    options.ResilienceOptions.MaxRetryAttempts = 3;
    options.ResilienceOptions.RetryDelay = TimeSpan.FromSeconds(2);
    options.DefaultTimeout = TimeSpan.FromSeconds(30);
});

// WeatherService.cs - Advanced usage
public async Task<WeatherData?> GetWeatherWithAuthAsync(string city, string apiKey)
{
    var request = new HttpRequestResult<WeatherData>
    {
        RequestPath = $"https://api.weather.com/forecast?city={city}",
        RequestMethod = HttpMethod.Get,
        CacheDurationMinutes = 10,
        Headers = new Dictionary<string, string>
        {
            ["X-API-Key"] = apiKey,
            ["Accept"] = "application/json"
        }
    };

    var result = await _http.HttpSendRequestResultAsync(request);

    // Correlation ID is automatically logged and propagated
    _logger.LogInformation(
        "Weather request completed in {Duration}ms with correlation {CorrelationId}",
        result.RequestDuration,
        result.CorrelationId
    );

    return result.IsSuccessStatusCode ? result.ResponseResults : null;
}

</details>

Web Crawling Features (Crawler Package)

Install Both Packages

dotnet add package WebSpark.HttpClientUtility
dotnet add package WebSpark.HttpClientUtility.Crawler

Register Services

// Program.cs
builder.Services.AddHttpClientUtility();
builder.Services.AddHttpClientCrawler();  // Adds crawler features

Use Crawler

public class SiteAnalyzer
{
    private readonly ISiteCrawler _crawler;
    
    public SiteAnalyzer(ISiteCrawler crawler) => _crawler = crawler;

    public async Task<CrawlResult> AnalyzeSiteAsync(string url)
    {
        var options = new CrawlerOptions
        {
            MaxDepth = 3,
            MaxPages = 100,
            RespectRobotsTxt = true
        };
        
        return await _crawler.CrawlAsync(url, options);
    }
}

🚀 Features

Base Package Features

• Simple API - Intuitive request/response model
• Authentication - Bearer token, Basic auth, API key providers
• Correlation IDs - Automatic tracking across distributed systems
• Structured Logging - Rich context in all log messages
• Telemetry - Request timing and performance metrics
• Error Handling - Standardized exception processing
• Type-Safe - Strongly-typed request and response models
• Caching - In-memory response caching (optional)
• Resilience - Polly retry and circuit breaker policies (optional)
• Concurrent Requests - Parallel request processing
• Fire-and-Forget - Background request execution
• Streaming - Efficient handling of large responses
• OpenTelemetry - Full observability integration (optional)
• CURL Export - Generate CURL commands for debugging

Crawler Package Features

• Site Crawling - Full website crawling with depth control
• Robots.txt - Automatic compliance with robots.txt rules
• Sitemap Generation - Create XML sitemaps from crawl results
• HTML Parsing - Extract links and metadata with HtmlAgilityPack
• SignalR Progress - Real-time crawl progress updates
• CSV Export - Export crawl results to CSV files
• Performance Tracking - Monitor crawl speed and efficiency

📚 Common Scenarios

Enable Caching

builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;
});

// In your service
var request = new HttpRequestResult<Product>
{
    RequestPath = "https://api.example.com/products/123",
    RequestMethod = HttpMethod.Get,
    CacheDurationMinutes = 10  // Cache for 10 minutes
};

Add Resilience (Retry + Circuit Breaker)

builder.Services.AddHttpClientUtility(options =>
{
    options.EnableResilience = true;
  options.ResilienceOptions.MaxRetryAttempts = 3;
    options.ResilienceOptions.RetryDelay = TimeSpan.FromSeconds(2);
});

All Features Enabled

builder.Services.AddHttpClientUtilityWithAllFeatures();

🔄 Upgrading from v1.x

If You DON'T Use Web Crawling

No code changes required! Simply upgrade:

dotnet add package WebSpark.HttpClientUtility --version 2.0.0

Your existing code continues to work exactly as before. All core HTTP features (authentication, caching, resilience, telemetry, etc.) are still in the base package with the same API.

If You DO Use Web Crawling

Three simple steps to migrate:

Step 1: Install the crawler package

dotnet add package WebSpark.HttpClientUtility.Crawler --version 2.0.0

Step 2: Add using directive

using WebSpark.HttpClientUtility.Crawler;

Step 3: Update service registration

// v1.x (old)
services.AddHttpClientUtility();

// v2.0 (new)
services.AddHttpClientUtility();
services.AddHttpClientCrawler();  // Add this line

That's it! Your crawler code (ISiteCrawler, SiteCrawler, SimpleSiteCrawler, etc.) works identically after these changes.

Need Help? See the detailed migration guide or open an issue.

📖 Documentation

• Getting Started Guide - Complete walkthrough
• Configuration Options - All settings explained
• Caching Guide - Response caching strategies
• Resilience Guide - Retry and circuit breaker patterns
• Web Crawling - Site crawler features
• Migration Guide - From raw HttpClient
• API Reference - Complete API documentation

🎓 Sample Projects

Explore working examples in the samples directory:

• BasicUsage - Simple GET/POST requests
• WithCaching - Response caching implementation
• WithResilience - Retry and circuit breaker patterns
• ConcurrentRequests - Parallel request processing
• WebCrawler - Site crawling example

🤝 Contributing

Contributions are welcome! See our Contributing Guide for details.

1. Fork the repository
2. Create a feature branch
3. Add tests for your changes
4. Ensure all tests pass
5. Submit a pull request

📊 Project Stats

• 252+ Unit Tests - 100% passing
• Supports .NET 8 LTS, .NET 9, & .NET 10 (Preview)
• MIT Licensed - Free for commercial use
• Active Maintenance - Regular updates

📦 Related Packages

Testing Package Features:

• FakeHttpResponseHandler - Mock HTTP responses without network calls
• Fluent API - Easy test setup with ForRequest().RespondWith()
• Sequential Responses - Test retry behavior with multiple responses
• Request Verification - Assert requests were made correctly
• Latency Simulation - Test timeout scenarios

dotnet add package WebSpark.HttpClientUtility.Testing

See the Testing documentation for examples.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

• GitHub Repository
• NuGet Package
• Changelog
• Issue Tracker
• Discussions

Questions or Issues? Open an issue or start a discussion!