Interflare.S3.LogParser 0.0.3

AWS S3 Access Log Parser

A parser for logs which follow the AWS S3 server access log format, fully equipped with the ability to retrieve the access logs from the bucket, and a streaming response API that allows for efficient pipeline processing with IAsyncEnumerable.

Installation

This repo publishes a single library package with all the included functionality, which you can add to your application as needed.

$ dotnet add package Interflare.S3.LogParser

Configuration

There is little to configure - the library does not run on its own, but it is best to use it via dependency injection to mock in unit testing. There is a helper extension to make this easier:

// Program.cs

var builder = WebApplication.CreateBuilder(args); // as usual

// Setup the log parser
builder.Services.AddAwsS3AccessLogParser();

Usage

This library is layered, meaning that you can make use of higher and higher abstractions depending on your use case.

Retrieve and parse the logs

A full end-to-end use case, the library is able to retrieve the log files from a storage bucket, and subsequently parse through them to give you strongly typed records, via the IS3AccessLogService (S3AccessLogService).

You supply an existing IAmazonS3 client, point it at the bucket, supply a prefix, and a range of timestamps to parse through. The library will list through the objects in the bucket matching the prefix and time range, and stream each object through a GZip decoder and finally the parser.

The results are returned via an IAsyncEnumerable, with the entire downloading/decoding/parsing system being streamed as the results are iterated.

public class MyClass(IAwsS3Client client, IS3AccessLogParserService parser)
{
    public async Task ReadMyLogs(DateTimeOffset from, DateTimeOffset to, CancellationToken cancellationToken)
    {
        IAsyncEnumerable<S3AccessLogEntry> accessLogs = parser.ParseAccessLogsAsync(
            client,
            bucketName: "my-log-bucket",
            prefix: "my-logs/another-bucket-",
            from,
            to,
            cancellationToken);
        
        await foreach (var log in accessLogs)
        {
            if (log.Operation != S3Operation.GetObject) continue;
            
            Uri? fullRequestUri = log.GetRequestUri(); // The full (relative) URI that was used - including query parameters
            Console.WriteLine($"{log.BucketName}{log.Key}: downloaded {log.SentBytes} bytes - {log.GetRequestUri()}");
        }
    }
}

Parse existing logs

If you already have the GZip decompressed access log files via some other means, you can iterate over each line to get a strongly typed record using the method in the IS3AccessLogParser (S3AccessLogParser), like so:

public class MyClass(IS3AccessLogParser parser) // <- Note, different interface
{
    public async Task ReadMyExistingLogs(string filePath)
    {
        var results = new List<S3AccessLogEntry>();
        
        foreach (var line in await File.ReadAllLinesAsync(filePath))
        {
            if (string.IsNullOrWhiteSpace(line)) continue;
            
            S3AccessLogEntry entry = parser.ParseLogLine(line);
            results.Add(entry);
        }
    }
}