BullseyeDev.Azure.Blob 1.0.10

Using SftpService in Azure Functions

Overview

The SftpService is designed to facilitate secure file transfers via SFTP within an Azure Function. It leverages Public Key Authentication to ensure robust and secure communication with the SFTP server. It provides methods to upload files and check if a file exists in a remote SFTP directory. This document details how to integrate SftpService into an Azure Function that reads data from Blob Storage and uploads it to an SFTP server.

Authentication Method

The SftpService exclusively uses Public Key Authentication, which is a secure method of authenticating with an SFTP server. This approach utilizes a private key file to establish an encrypted and secure connection.

How It Works

• The private key file path is provided as part of the SftpOptions configuration.
• The SftpService creates a PrivateKeyAuthenticationMethod using the private key to authenticate with the SFTP server.
• This ensures that passwords are not transmitted over the network, enhancing security.

Key Code Snippet:

new PrivateKeyAuthenticationMethod(sftpOptions.Username, keyFile)

Registering SftpService in Azure Functions

To use SftpService within an Azure Function, it must first be registered in the Program.cs file:

using BullseyeDev;
using Microsoft.Azure.Functions.Worker.Builder;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System;

var builder = FunctionsApplication.CreateBuilder(args);

builder.ConfigureFunctionsWebApplication();

// Determine environment (default to Development if not set)
var environment = Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT") ?? "Development";

// Load configuration from appropriate sources
builder.Configuration
    .AddJsonFile("local.settings.json", optional: environment == "Development", reloadOnChange: true) // Local dev only
    .AddEnvironmentVariables(); // Always loads environment variables

// Bind SFTP settings from configuration
builder.Services.Configure<SftpPrivateKeyOptions>(builder.Configuration.GetSection("SftpSettings"));
// Register SftpService as a singleton
builder.Services.AddSingleton<ISftpService, SftpService>();

builder.Build().Run();

This ensures SftpService is available for dependency injection and can be used across multiple Azure Functions.

Using SftpService in an Azure Function

Example: Uploading Files from Blob Storage to SFTP

The following Azure Function listens for new files in Blob Storage, reads the content, and uploads it to an SFTP server.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Azure.Storage.Blobs;

namespace BullseyeDev
{
    public class UploadCsvToSftpFunction
    {
        private readonly ISftpService _sftpService;
        private readonly SftpOptions _sftpOptions;
        private readonly string _archiveCsvContainer;
        private readonly string _processedFolder;
        private readonly string _failedFolder;

        public UploadCsvToSftpFunction(ISftpService sftpService, IOptions<SftpOptions> sftpOptions)
        {
            _sftpService = sftpService;
            _sftpOptions = sftpOptions.Value;
            _archiveCsvContainer = "bullseyedev-archive";
            _processedFolder = _sftpOptions.ProcessedFolder;
            _failedFolder = _sftpOptions.FailedFolder;
        }

        [Function("UploadCsvToSftp")]
        public async Task RunAsync(
            [BlobTrigger("bullseyedev-processed-csv/{name}", Connection = "AzureWebJobsStorage")] Stream blobStream,
            string name,
            ILogger log)
        {
            log.LogInformation($"Processing CSV file from Blob Storage: {name}");

            try
            {
                using var reader = new StreamReader(blobStream);
                string csvData = await reader.ReadToEndAsync();

                // Check if file already exists in processed or failed folder
                if (await _sftpService.FileExistsAsync($"{_processedFolder}/{name}") ||
                    await _sftpService.FileExistsAsync($"{_failedFolder}/{name}"))
                {
                    log.LogWarning($"File {name} already exists in processed or failed folder. Skipping.");
                    return;
                }

                // Upload file to SFTP 
                await _sftpService.UploadFileAsync(name, csvData);
                await MoveToArchiveAsync(name, csvData);

                log.LogInformation($"File {name} successfully uploaded and archived.");
            }
            catch (Exception ex)
            {
                log.LogError($"Error processing {name}: {ex.Message}");
            }
        }

        private async Task MoveToArchiveAsync(string fileName, string fileData)
        {
            var blobServiceClient = new BlobServiceClient(Environment.GetEnvironmentVariable("AzureWebJobsStorage"));
            var archiveBlobClient = blobServiceClient.GetBlobContainerClient(_archiveCsvContainer).GetBlobClient(fileName);
            using var stream = new MemoryStream(Encoding.UTF8.GetBytes(fileData));
            await archiveBlobClient.UploadAsync(stream, overwrite: true);
        }
    }
}

Configuration

To configure the SftpService and integrate it into your application, include the following settings in your appsettings.json or local.settings.json file:

"SftpSettings": {
    "Host": "sftp.server.com",
    "Username": "your-username",
    "PrivateKeyPath": "/path/to/private/key",
    "RemoteFolder": "/remote/folder",
    "ProcessedFolder": "bullseyedev-processed",
    "FailedFolder": "bullseyedev-failed",
    "RetryAttempts": 3,
    "BackoffFactor": 2.0
}

Key Fields in SftpOptions

• Host: The hostname or IP address of the SFTP server.
• Username: The username used for SFTP authentication.
• PrivateKeyPath: The path to the private key file for public key authentication.
• RemoteFolder: The directory on the SFTP server where files will be uploaded.
• ProcessedFolder: The folder for successfully processed files.
• FailedFolder: The folder for failed uploads.
• RetryAttempts: The number of retry attempts for handling transient errors.
• BackoffFactor: The exponential backoff multiplier for calculating delays between retries.

Final Notes

• SftpService is injected as a singleton, ensuring efficient and centralized SFTP handling.
• Azure Blob Storage acts as the trigger, automating file processing.
• Improved Resilience: Retry attempts and exponential backoff enhance robustness against transient errors.
• Customizable Configuration: Centralized settings make SftpService adaptable.
• Logging and Error Handling: Comprehensive logs aid debugging and monitoring.

This approach efficiently integrates SFTP file transfers within an Azure Function, leveraging Blob Storage for automation.

Additional Classes

• QueueService
• Instantiating SftpService Using Different Authentication Options