RemuxForge.Vulkan 0.2.0

RemuxForge.Vulkan

RemuxForge.Vulkan provides deterministic, GPU-accelerated SIFT feature extraction, reciprocal descriptor matching, homography RANSAC, and perceptual frame hashing for .NET applications.

The library keeps image normalization, Gaussian and Difference-of-Gaussians pyramids, keypoint detection and refinement, orientation assignment, descriptor generation, matching, and geometric verification on the Vulkan compute device.

Installation

dotnet add package RemuxForge.Vulkan --version 0.2.0

Supported image formats

VulkanImageFrame accepts packed 8-bit image buffers with an explicit row stride:

• VulkanPixelFormat.Gray8
• VulkanPixelFormat.Rgb24
• VulkanPixelFormat.Bgr24
• VulkanPixelFormat.Rgba32
• VulkanPixelFormat.Bgra32

Gray8 samples are normalized directly. Color inputs require an explicit VulkanRgbToGrayMatrix value:

• VulkanRgbToGrayMatrix.Bt601
• VulkanRgbToGrayMatrix.Bt709
• VulkanRgbToGrayMatrix.Bt2020

The selected matrix is applied to the stored RGB samples by the Vulkan normalization shader. RGBA and BGRA alpha samples are ignored by SIFT.

Quick start

using RemuxForge.Vulkan;

using VulkanVisionContext context = new VulkanVisionContext();
using VulkanSiftPipeline pipeline = context.CreateSiftPipeline();

VulkanImageFrame first = new VulkanImageFrame(
    identifier: 1,
    pixels: firstPixels,
    width: width,
    height: height,
    stride: width * 3,
    pixelFormat: VulkanPixelFormat.Rgb24,
    rgbToGrayMatrix: VulkanRgbToGrayMatrix.Bt709);

VulkanImageFrame second = new VulkanImageFrame(
    identifier: 2,
    pixels: secondPixels,
    width: width,
    height: height,
    stride: width * 3,
    pixelFormat: VulkanPixelFormat.Rgb24,
    rgbToGrayMatrix: VulkanRgbToGrayMatrix.Bt709);

VulkanSiftBatchRequest request = new VulkanSiftBatchRequest(
    new[] { first },
    new[] { second },
    new[] { new VulkanFramePair { FirstFrameIndex = 0, SecondFrameIndex = 0 } });

VulkanSiftBatchResult result = pipeline.Execute(request);
VulkanSiftPairResult pair = result.PairResults[0];

Each pair result reports its acceptance status, reject reason, keypoint and match counts, inlier ratio, geometric coverage, reprojection error, score, and row-major homography.

Batch execution

Execute extracts the supplied frames and evaluates the requested frame-index pairs. Prepare keeps compact SIFT features resident for repeated pair sets over the same bounded frame batch.

Frames with matching geometry, stride, pixel format, and RGB-to-gray matrix use the packed multi-frame extraction path. Mixed layouts are split into compatible extraction chunks while preserving caller identifiers and result ordering.

Progress reports expose uploaded and extracted frame counts, processed pairs, completed tiles, and resident bytes. Cancellation is accepted by every public execution entry point.

Perceptual hashing

VulkanHashPipeline computes the one-hundred-twenty-eight-bit gradient hash of grayscale analysis squares and measures how many frames of one track a candidate time offset explains against another.

using VulkanVisionContext context = new VulkanVisionContext();
using VulkanHashPipeline pipeline = context.CreateHashPipeline();

VulkanFrameHash[] hashes = pipeline.Extract(frames);

using VulkanHashPreparedBatch batch = pipeline.Prepare(
    new VulkanHashTrack(sourceHashes, sourceTimestampsMs),
    new VulkanHashTrack(languageHashes, languageTimestampsMs));

VulkanHashScanResult scan = batch.Execute(
    new[] { new VulkanHashScan(0, 1, 400, -30000.0, 20.0, 3001, 1, 14) }).Scans[0];

Extract accepts a tightly packed sequence of VulkanHashPipeline.FrameBytes squares, one byte per pixel in row order, and returns one hash per square. One workgroup hashes one square, so squares are worth handing over in blocks: VulkanHashPipeline.FramesPerSubmission is the block one submission covers. Prepare keeps both tracks resident so that repeated offset grids are measured without re-uploading them.

Each scan declares the measured source frames, the candidate offset grid, the number of neighbouring language frames explored around the located timestamp, and the Hamming distance that still counts as an explained frame. The result reports the explained frame count of every candidate, the best candidate, and its explained fraction.

Timestamps travel as their IEEE-754 values and the shader reproduces double-precision addition and ordering exactly, so a scan returns the same counts as an equivalent host implementation.

Memory and concurrency

VulkanVisionOptions controls device selection, the VRAM ceiling, validation layers, concurrent workloads, and pipeline-cache initialization.

The allocator derives its automatic pressure threshold from VK_EXT_memory_budget when available and otherwise uses the device-local heap. MaximumVramBytes can impose a lower ceiling. Extraction and matching are tiled, temporary workspaces are recycled, and resource exhaustion is reported with a typed exception.

The context owns the Vulkan runtime and pipeline cache. Pipelines and prepared batches implement IDisposable and should be released promptly.

Diagnostics

VulkanVisionDiagnostics reports:

• device capabilities and shader build metadata;
• upload, normalization, pyramid, descriptor, matching, RANSAC, readback, and host-wait timings;
• submit, dispatch, tile, frame, feature, pair, and rejection counters;
• current, peak, cached, wasted, and host-visible memory usage;
• validation-layer warnings and errors.

GetPipelineCacheData returns cache data tagged with device, driver, pipeline-cache UUID, and shader-manifest identity for reuse by a later context.

Build requirements

• .NET SDK 10
• Vulkan 1.2 loader
• Vulkan compute device with timeline semaphores
• glslc on PATH
• spirv-val on PATH
• MoltenVK with portability enumeration on macOS

Build the solution:

dotnet build RemuxForge.Vulkan.sln -c Release

Create the NuGet package:

dotnet pack RemuxForge.Vulkan/RemuxForge.Vulkan.csproj -c Release -o artifacts

The build compiles and validates each compute shader, generates the shader manifest, and embeds the SPIR-V modules in the assembly. The NuGet package includes the README, license, third-party notices, and ArrayFire derivation matrix.

Attribution

Algorithm derivation and third-party notices are documented in ARRAYFIRE-DERIVATION.md and THIRD-PARTY-NOTICES.md.