AlphaAppTools.UsefulBrowserTesting 10.7.0

AlphaAppTools.UsefulBrowserTesting

Tests that run a .NET WebAssembly app in a headless Chrome, Chromium or Edge, with no WebDriver. The test project declares a fixture, a collection and the app; the package publishes the app, serves it on a loopback port, opens it in the browser and hands the tests the report that the page of the app posts back. It is built for xunit.v3 (xunit.v3.extensibility.core 4.0.2, so the test project needs xunit.v3 4.0.2 or later), and its types are in the namespace UsefulBrowserTesting. It targets net10.0; the 10.6.x versions of the package are for xUnit v2 and target net8.0 as well.

dotnet add package AlphaAppTools.UsefulBrowserTesting

The XML documentation of BrowserAppFixture<TReport>, BrowserAppParameters<TReport> and BrowserFactAttribute holds the contract; this guide shows the files to copy.

What the machine needs

• A Chromium-based browser where the kit looks for it. Without one every [ BrowserFact ] test is reported as skipped and nothing is published. The places, in this order:
  • Windows: Chrome (Google\Chrome\Application\chrome.exe), Chromium (Chromium\Application\chrome.exe) and Edge (Microsoft\Edge\Application\msedge.exe), each under Program Files, Program Files (x86) and LocalAppData;
  • macOS: Chrome, Chromium and Edge as the applications Google Chrome, Chromium and Microsoft Edge in /Applications;
  • elsewhere: the PATH, by the names google-chrome, google-chrome-stable, chromium, chromium-browser, microsoft-edge and microsoft-edge-stable.
• The .NET SDK: the kit runs dotnet publish with the dotnet on the PATH. dotnet test in Microsoft.Testing.Platform mode, the mode of xunit.v3 test projects, sets no DOTNET_HOST_PATH environment variable; where one is set, as dotnet test in VSTest mode does, the kit takes the dotnet it names.
• The ASP.NET Core shared framework, which serves the app with Kestrel inside the test process: the package's FrameworkReference to Microsoft.AspNetCore.App flows to every project that references the package, so the test project takes it along.
• No WebAssembly workload. The wasm-tools workload, which Visual Studio installs, is ignored where it is installed: the app's csproj below sets UsingBrowserRuntimeWorkload to false.

The files a consumer writes

The names are examples: the test project is MyAppTest, the app MyAppTest_BrowserApp, its report MyReport and the fixture MyBrowserAppFixture. The app is a project of its own, found as <directory>/<ProjectName>/<ProjectName>.csproj in the folder of the test assembly or in any folder above it; the usual place is next to the test project's folder:

src/
    MyAppTest/                          the test project
        MyAppTest.csproj
        MyBrowserAppFixture.cs
        MyBrowserAppCollection.cs
        MyFeatureBrowserTest.cs
    MyAppTest_BrowserApp/               the app
        MyAppTest_BrowserApp.csproj
        MyReport.cs                     compiled by both projects
        Program.cs
        wwwroot/
            index.html
            main.js

The fixture

It derives from BrowserAppFixture<TReport> and passes all five parameters. Two minutes for the report and five for the publish are reasonable limits: a cold publish took 33 s on a developer machine, a warm one 3 to 5 s, so keep the publish limit well above the cold value.

// MyAppTest/MyBrowserAppFixture.cs
using FluentAssertions.Extensions;
using MyAppTest_BrowserApp;
using UsefulBrowserTesting;

namespace MyAppTest;

public sealed class MyBrowserAppFixture() : BrowserAppFixture<MyReport>( new()
{
    ProjectName = "MyAppTest_BrowserApp",
    TestAssembly = typeof( MyBrowserAppFixture ).Assembly,
    ReportTypeInfo = MyReportJsonContext.Default.MyReport,
    ReportTimeout = 2.Minutes(),
    PublishTimeout = 5.Minutes()
} );

The constructor checks the parameters and starts nothing; the run starts on the first GetReportAsync().

The collection

xUnit finds a collection definition only in the test assembly, so the package cannot declare it: every test project declares its own. The definition sets DisableParallelization = true.

// MyAppTest/MyBrowserAppCollection.cs
using Xunit;

namespace MyAppTest;

// The browser tests run alone, after all others: publishing the app and running a browser would take processor time from the tests that measure time.
[ CollectionDefinition( Name, DisableParallelization = true ) ]
public sealed class MyBrowserAppCollection : ICollectionFixture<MyBrowserAppFixture>
{
    public const string Name = "My browser app";
}

A test class

A test takes the fixture through its constructor, is marked [ BrowserFact ] and awaits GetReportAsync(). Every test of the collection gets the same report, so the app runs once however many tests read it. The log of the run goes to the test output from Dispose(). WriteToTestOutput() of UsefulTesting.Loggers finds the output of the running test through xunit.v3's ambient TestContext.Current, so the test class needs no ITestOutputHelper; WriteTo( testOutputHelper ) works as well.

// MyAppTest/MyFeatureBrowserTest.cs
using System;
using System.Threading.Tasks;
using FluentAssertions;
using UsefulBrowserTesting;
using UsefulTesting.Loggers;
using Xunit;

namespace MyAppTest;

[ Collection( MyBrowserAppCollection.Name ) ]
public class MyFeatureBrowserTest( MyBrowserAppFixture browserApp ) : IDisposable
{
    [ BrowserFact ]
    public async Task Feature_InBrowser_ReportsEveryValue()
    {
        // Act
        var report = await browserApp.GetReportAsync();

        // Assert
        report.IsBrowser.Should().BeTrue();
        report.Values.Should().BeEquivalentTo( [1, 2, 3], o => o.WithStrictOrdering() );
    }

    public void Dispose()
        => browserApp.Logger.WriteToTestOutput();
}

The test project's csproj

The test project is an xunit.v3 test project: it references the xunit.v3 package, sets <OutputType>Exe</OutputType> and runs under dotnet test in Microsoft.Testing.Platform mode. The release note of version 10.7.0 (ReleaseNotes/v10.7.0.md in the repository) describes the migration from xUnit v2.

The package reference, the project reference to the app and the report file that both projects compile. The project reference builds the app for each target framework of the test project; ReferenceOutputAssembly="false" and Private="false" keep the test project from referencing the app's assembly and from copying anything of it into the test output. With central package management the version of the package goes in Directory.Packages.props.

<ItemGroup>
    <PackageReference Include="AlphaAppTools.UsefulBrowserTesting" />
</ItemGroup>


<ItemGroup>
    <ProjectReference Include="..\MyAppTest_BrowserApp\MyAppTest_BrowserApp.csproj" ReferenceOutputAssembly="false" Private="false" />
</ItemGroup>


<ItemGroup>
    <Compile Include="..\MyAppTest_BrowserApp\MyReport.cs" Link="MyReport.cs" />
</ItemGroup>

The app's csproj

Copy this project file and reference the code under test. Its target framework is the test project's: the kit publishes the app for the target framework of the running tests, so the app of a test project with several target frameworks lists them all in TargetFrameworks.

• WasmEnableThreads false keeps the single-threaded runtime that Blazor WebAssembly runs on.
• PublishTrimmed false, CompressionEnabled false and InvariantGlobalization true keep the publish at seconds: the tests publish the app on every run.
• UsingBrowserRuntimeWorkload false builds and publishes the app as where the wasm-tools workload is not installed. Where it is, dotnet publish runs a nested publish of the app that builds its project references, which the kit's publish with --no-build refuses with NETSDK1085, and every build relinks the runtime with emcc. The property comes from the workload's own MSBuild targets; an app that needs the workload, for AOT or native libraries, cannot opt out.

<Project Sdk="Microsoft.NET.Sdk.WebAssembly">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <LangVersion>latestmajor</LangVersion>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  
  <PropertyGroup>
    <WasmEnableThreads>false</WasmEnableThreads>
  </PropertyGroup>

  
  <PropertyGroup>
    <UsingBrowserRuntimeWorkload>false</UsingBrowserRuntimeWorkload>
  </PropertyGroup>

  
  <PropertyGroup>
    <PublishTrimmed>false</PublishTrimmed>
    <CompressionEnabled>false</CompressionEnabled>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\MyApp\MyApp.csproj" />
  </ItemGroup>

</Project>

The page: wwwroot/index.html and wwwroot/main.js

Copy both files; only the title is the app's own. main.js starts the app and passes it the address of the server as its first argument. When Program.Main fails to start or exits with a code other than 0, it posts the failure to /error, so a broken app fails the test with a message instead of waiting for the ReportTimeout.

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <title>MyAppTest_BrowserApp</title>
    <script type="module" src="./main.js"></script>
</head>
<body></body>
</html>

// MyAppTest_BrowserApp/wwwroot/main.js
import { dotnet } from './_framework/dotnet.js';

// Program.Main posts its report to the address passed as its argument; a failure to start or to run it is posted here instead.
try {
    const exitCode = await dotnet.withApplicationArguments( window.location.origin + '/' ).run();

    if( exitCode !== 0 ) {
        await fetch( '/error', { method: 'POST', body: `Program.Main exited with code ${exitCode}` } );
    }
} catch( error ) {
    await fetch( '/error', { method: 'POST', body: String( error?.stack ?? error ) } );
}

The report and Program.Main

The report is a record with a JsonSerializerContext, in one file that the app and the test project both compile, so both sides read one contract. Program.Main runs the code under test and makes exactly one POST to results, relative to the address in args[ 0 ], with the report as JSON.

// MyAppTest_BrowserApp/MyReport.cs
using System.Collections.Generic;
using System.Text.Json.Serialization;

namespace MyAppTest_BrowserApp;

// The JSON the app posts to the test that serves it. The test project compiles this file as well, so both sides read one contract.
public record MyReport
{
    public required bool IsBrowser { get; init; }
    public required IReadOnlyList<int> Values { get; init; }
}

[ JsonSerializable( typeof( MyReport ) ) ]
public partial class MyReportJsonContext : JsonSerializerContext;

// MyAppTest_BrowserApp/Program.cs
using System;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace MyAppTest_BrowserApp;

public static class Program
{
    public static async Task Main( string[] args )
    {
        // Run the code under test here, and put what the tests assert on into the report.
        var report = new MyReport { IsBrowser = OperatingSystem.IsBrowser(), Values = [1, 2, 3] };

        using var client = new HttpClient { BaseAddress = new Uri( args[ 0 ] ) };
        using var response = await client.PostAsJsonAsync( "results", report, MyReportJsonContext.Default.MyReport );
        response.EnsureSuccessStatusCode();
    }
}

What one run does

The run happens once per fixture, on the first GetReportAsync() of a test that is not skipped; every test of the collection gets the same report.

1. Find the browser, where "What the machine needs" says. Without one the [ BrowserFact ] tests are skipped and nothing is published.
2. Find the app's project: <directory>/<ProjectName>/<ProjectName>.csproj, in the folder of TestAssembly or in any folder above it.
3. Publish the app: dotnet publish <project> --configuration <the test assembly's> --framework <the test assembly's> --no-build --output <folder>, into a temporary folder named <ProjectName>-*. It is --no-build because the project reference has built the app already, for each target framework of the test project. The publish is limited to PublishTimeout, measured from starting dotnet publish to its exit; past it the kit kills dotnet with its process tree and fails with the output so far.
4. Serve the wwwroot of the publish on http://127.0.0.1:<a free port>, with .wasm files as application/wasm. POST /results completes the report, POST /error fails it with what the page posted.
5. Open index.html in the browser, headless (--headless=new), with a profile of its own in a temporary folder named UsefulBrowserTesting-profile-*, the browser logging to its error output, and --no-sandbox on Linux, where the sandbox needs user namespaces, which a container running as root and the AppArmor profile of Ubuntu 24.04 deny.
6. Wait for one of the two POSTs, at most ReportTimeout from opening the browser.
7. Read the report through ReportTypeInfo.
8. Clean up: kill the browser's process tree while the process the kit started still runs, wait at most 10 seconds for its exit, delete the profile (10 tries, 200 ms apart) and delete the publish folder.

The Logger of the fixture records the run, for the test output:

Published MyAppTest_BrowserApp for net10.0 (Debug) in 3412 ms.
Opened http://127.0.0.1:55472/index.html in C:\Program Files\Google\Chrome\Application\chrome.exe.
Received the report after 6120 ms: {"IsBrowser":true,"Values":[1,2,3]}

When a run fails

GetReportAsync() returns a task that fails with these exceptions. The messages are exact, with {...} for what varies; a [ BrowserFact ] test is skipped without a browser, so the first row is for a test that asks for the report anyway.

A message that ends with a colon continues on the following lines: with the output of dotnet publish (the two publish rows) or with the last 200 lines of the error output of the browser (the no-report row). JsonException is the exception of System.Text.Json, which the kit does not catch.

A publish that fails with NETSDK1085: The 'NoBuild' property was set to true but the 'Build' target was invoked. comes from an app with project references that uses the wasm-tools workload: set UsingBrowserRuntimeWorkload to false in its csproj, as the app's csproj above does.

The constructor of the fixture checks the parameters: ArgumentNullException for a missing parameters record, test assembly, ReportTypeInfo or ProjectName, ArgumentException for a blank ProjectName, and ArgumentOutOfRangeException for a ReportTimeout or PublishTimeout that is not greater than zero.

A browser left behind

The clean-up ends the browser and deletes its profile. When it cannot, the log of the fixture says so, with one of these warnings:

The browser process {ProcessId} exited by itself (code {exitCode}) before the run ended: Chrome may have relaunched itself, for example to apply an update.
The browser profile {ProfileDirectory} could not be deleted: a browser may still be using it.

The first: on Windows a Chrome that finds a staged update when it starts swaps it in and relaunches itself as a new process. The process the kit started exits, so the kill misses the browser that keeps running with the profile of the kit, which is out of the kit's reach. The second: a browser still uses the profile folder.

A test host that is stopped, with Ctrl+C or by a hang dump, runs no clean-up at all, and leaves the browser, its profile and the publish folder behind.

End such a browser by its profile folder, UsefulBrowserTesting-profile-* in the temporary folder, then delete the folder. In PowerShell:

Get-CimInstance Win32_Process | Where-Object CommandLine -like '*UsefulBrowserTesting-profile-*' | ForEach-Object { Stop-Process -Id $_.ProcessId }

On Linux and macOS:

pkill -f UsefulBrowserTesting-profile-