Ossa.QualityDashboard.Accessibility 1.1.0-beta.9

Quality Dashboard

Pulls code-quality stats from SonarQube, security findings from OWASP ZAP reports and WCAG results from your sites' Playwright integration tests, and consolidates them into a one-page dashboard (A4 landscape) for steering committee / QA documents.

• Outputs: HTML (self-contained), PDF and Word (.docx), plus optional JSON of the underlying figures
• Multiple projects, one page: one row per project with a traffic light (RAG), key metrics, a WCAG score and grade, coverage, issue, ZAP and WCAG sparklines, and trend direction
• WCAG report per site: every page its tests visited with a score, grade and each failure, written as HTML straight after the test run, or as HTML, PDF or Word with quality-dashboard wcag (see WCAG). The dashboard itself carries only the WCAG summary, so it stays one page.
• Trends come from SonarQube's analysis history, plus a local SQLite database of past ZAP and WCAG results, over a configurable window (default 90 days)
• Installs as a .NET tool (quality-dashboard) from NuGet; designed for an Azure DevOps Windows agent

Install

Requires the .NET 8, 9 or 10 SDK (or later) on Windows x64 or Arm64. The package carries a build per framework and dotnet tool install picks the newest one your SDK supports; add --framework net8.0 to choose one explicitly.

dotnet tool install -g Ossa.QualityDashboard     # install
dotnet tool update -g Ossa.QualityDashboard      # upgrade
quality-dashboard --version

To pin it per repository instead, use a local tool manifest: dotnet new tool-manifest, then dotnet tool install Ossa.QualityDashboard and run it as dotnet quality-dashboard.

Quick start

# Preview the layout with built-in sample data (no SonarQube needed)
quality-dashboard --demo --out ./out

# Real run
$env:SONAR_TOKEN = '<your SonarQube user token>'
quality-dashboard --config dashboard.config.json --out ./out

Output: out/quality-dashboard.html, .pdf, .docx. From a clone of this repo, dotnet run --project src/QualityDashboard -f net10.0 -- <options> works the same without installing.

Command line

For a site's WCAG report (every page and its failures), use quality-dashboard wcag --report <wcag-report.json>: see WCAG report command.

Exit code is 0 on success and 1 for invalid arguments/config, when no project could be read, or when an output format failed. If one project's data can't be read, it still appears on the dashboard with the error shown, and in Azure DevOps a pipeline warning is raised.

Configuration

Start from the sample dashboard.config.json in the repo root (comments are allowed).

The SonarQube token needs Browse permission on each project. Branch analysis (branch) requires SonarQube Developer Edition or higher.

ZAP reports

Use the traditional JSON report:

• Docker scripts: zap-baseline.py -t https://app -J portal-api.json (or zap-full-scan.py / zap-api-scan.py)
• Automation Framework: a report job with template: traditional-json

Counts on the dashboard are distinct alert types per risk level (High / Medium / Low), matching the ZAP report summary. The "Key security findings" panel lists the top High/Medium alerts with the number of affected URLs.

ZAP history database

SonarQube keeps its own history, but a ZAP report is a one-off file. So each run stores the ZAP results in a SQLite database, and the dashboard draws a ZAP H+M trend (distinct High + Medium alert types per scan) from it.

• Location: %APPDATA%\QualityDashboard\history.db by default. Override it with historyDb in config or --history-db. The folder and schema are created on first run.
• Deduplication: each report is identified by a SHA-256 of its contents, so re-running the dashboard against the same ZAP report doesn't add a duplicate point.
• Scan date: taken from the report's @generated timestamp, or the time it was recorded if that's missing.
• Project key: history is stored against the project's id, falling back to sonarKey, then name. Set id if you expect to rename a project.
• Tables: zap_scan (one row per scan, with counts by risk), zap_alert (top alerts per scan) and accessibility_scan (one row per WCAG report: site score and failures by impact). You can query them directly with any SQLite tool. Databases from version 1.0 are upgraded in place.
• Resilience: if the database can't be opened, the dashboard is still produced without ZAP and WCAG trends, and a warning is raised.

A trend line needs at least two recorded scans in the window; until then the column shows "Building history".

Where to keep it in Azure DevOps. %APPDATA% belongs to whichever account runs the agent:

• Microsoft-hosted agents start from a clean VM every run, so %APPDATA% doesn't survive. The pipeline's historyAsArtifact option (on by default) downloads history.db from the previous run's quality-history artifact and publishes the updated copy at the end.
• Self-hosted agents keep %APPDATA% between runs (for an agent running as a service, that's the service account's profile, e.g. C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming). Turn historyAsArtifact off, or point historyDb at a fixed local folder such as D:\QualityDashboard\history.db.
• Avoid putting the database on a network share: SQLite file locking is unreliable over SMB.

WCAG

WCAG checks run inside each site's existing Playwright integration tests, so they see the pages exactly as the tests do: signed in, as each role, with dialogs open. The Ossa.QualityDashboard.Accessibility NuGet library runs axe-core on any Playwright IPage and writes one wcag-report.json per site. There are two outputs:

• The WCAG report (one per site): every page and its failures. The scanner writes it as wcag-report.html next to the JSON as soon as the tests finish. For PDF or Word, run quality-dashboard wcag on the JSON.
• The one-page dashboard reads the JSON like a ZAP report: it scores it, applies traffic-light and trend rules, keeps history, and shows a WCAG score, grade and trend per project. The page-by-page detail stays in the WCAG report.

Both are rendered by the same shared code, so the scores and layout always match.

The library works with xUnit, NUnit or MSTest (it only needs an IPage) and with any Playwright browser: Microsoft Edge (msedge channel, no browser download needed on Windows), Chrome, Chromium, Firefox or WebKit.

Scanning from integration tests

dotnet add package Ossa.QualityDashboard.Accessibility

NUnit with Microsoft.Playwright.NUnit: one auditor for the whole run, and one extra line in each test that reaches a page worth checking.

[SetUpFixture]
public class WcagSetup
{
    public static AccessibilityAuditor Auditor = null!;

    [OneTimeSetUp]
    public void Start() => Auditor = new(new AccessibilityOptions
    {
        SiteName = "Customer Portal",
        Standard = WcagStandard.Wcag22AA,   // per site: Wcag2A, Wcag2AA, Wcag21A, Wcag21AA, Wcag22AA (default), Wcag22AAA
    });

    [OneTimeTearDown]
    public Task Stop() => Auditor.WriteReportAsync();
}

public class OrderTests : PageTest
{
    [Test]
    public async Task Orders_list()
    {
        await Page.GotoAsync("/orders");                    // already signed in (see below)
        await WcagSetup.Auditor.ScanAsync(Page, "Orders list");
        // ... the test's own assertions
    }
}

To run NUnit tests in Edge, add a .runsettings (see tests/QualityDashboard.Accessibility.NUnitSmoke/edge.runsettings):

<RunSettings>
  <Playwright>
    <BrowserName>chromium</BrowserName>
    <LaunchOptions><Channel>msedge</Channel></LaunchOptions>
  </Playwright>
</RunSettings>

xUnit: hold the auditor in an assembly or collection fixture (IAsyncLifetime) and call WriteReportAsync() from DisposeAsync(). With plain Microsoft.Playwright, launch Edge with playwright.Chromium.LaunchAsync(new() { Channel = "msedge" }).

Signed-in pages. The auditor scans whatever page it's given, so reuse your tests' existing login. A common pattern: sign in once per role in the fixture, save await context.StorageStateAsync(new() { Path = "admin.json" }), then create each test's context with new BrowserNewContextOptions { StorageStatePath = "admin.json" }. Name scans per role or state when it matters, e.g. "Orders list (admin)" or "Checkout – address dialog".

Options (AccessibilityOptions):

Each ScanAsync call records one page result, keyed by name and browser. Scanning the same name in the same browser again replaces the earlier result. The query string, fragment and any credentials are stripped from URLs, so tokens never reach the report.

After the run, open TestResults/wcag-report.html for the results. To include the site on the dashboard, point its project at the JSON:

{ "name": "Customer Portal", "sonarKey": "portal-web", "accessibilityReport": "reports/customer-portal/wcag-report.json" }

WCAG report command

quality-dashboard wcag renders a WCAG report from the JSON in any format, e.g. for a QA document:

dotnet test
quality-dashboard wcag --report TestResults/wcag-report.json --out ./out --format html,pdf,docx

The report has the site score and counts, the pages worst first, the top failing rules, then a section per page listing each failure: impact, rule and WCAG success criteria, what to fix, the number of failing elements, and the first few elements (CSS selector and axe's explanation). Best-practice findings are shown but not scored. Exit code is 0 on success, 1 if the report can't be read or an output fails.

Scores and grades

The dashboard re-scores every report with these rules rather than trusting stored numbers, so reports from older library versions are scored the same way. Automated checks can't catch every WCAG issue, so treat the score as a floor and keep doing manual and assistive-technology testing.

Licences. The dashboard tool stays MIT-only: it just reads the JSON report. The scanner library depends on Deque.AxeCore.Playwright (MIT), which bundles axe-core (MPL-2.0, used unmodified). MPL-2.0 is file-level copyleft and puts no obligations on your own code.

How the traffic light and trend are worked out

Trend compares the first and last data point in the window for four signals:

• Coverage (SonarQube) counts as changed if it moved by ≥ 0.5 percentage points.
• Total issues (SonarQube bugs + vulnerabilities + code smells) count as changed if they moved by more than 2% (and at least 1).
• ZAP High + Medium alert types (history database) count as changed if they moved by 1 or more.
• WCAG site score (history database) counts as changed if it moved by 1 point or more.
• More good signals (coverage up, issues down, fewer ZAP alerts, higher WCAG score) than bad → Improving; more bad than good → Worsening; otherwise Stable. No signal with two or more points in the window → No history.

These rules are printed in the dashboard footer so readers know what each colour means.

Azure DevOps

pipelines/azure-pipelines.yml is a ready-to-use pipeline (weekly schedule + manual runs) on windows-latest:

1. Create a variable group quality-dashboard with a secret SonarToken.
2. Edit and commit dashboard.config.json in the repo root (the pipeline's default configPath), or point configPath at your own copy.
3. ZAP reports either live in the repo next to the config, or come from the pipeline that runs the scans: set the zapPipeline parameter, and that pipeline's zap-reports artifact is downloaded and used as --zap-root. In that case, set each zapReport to the file's path inside the artifact (e.g. portal-api.json).
4. Choose toolSource: source (default) builds the tool from this repo; nuget installs the published Ossa.QualityDashboard tool instead (optionally pinned with the template's toolVersion).
5. Run it. The dashboard appears as the quality-dashboard artifact on the run.

To add the dashboard to an existing pipeline (in any repo, once the tool is on NuGet), use the step template:

steps:
- template: pipelines/templates/quality-dashboard.yml
  parameters:
    configPath: dashboard.config.json
    sonarToken: $(SonarToken)
    toolSource: nuget
    zapRoot: $(Pipeline.Workspace)/zap   # optional

WCAG reports in a pipeline. In the site's test job, set QD_WCAG_REPORT to a folder you publish (e.g. $(Build.ArtifactStagingDirectory)/reports/customer-portal/wcag-report.json) and publish that folder as an artifact. In a later job of the same run, pass the artifact name to the template's reportsArtifact parameter: it's downloaded and used as the reports root, so accessibilityReport paths are relative to the artifact (e.g. customer-portal/wcag-report.json). The published folder already holds wcag-report.html; for PDF or Word, add a step running quality-dashboard wcag --report <json> --out <folder> in either job and publish its output.

Publishing the tool to NuGet

pipelines/publish-tool.yml builds, tests and packs the tool on every Git Flow branch, and publishes releases and prereleases to nuget.org.

Branching and versions. The repo follows Git Flow. Versions come from git history via the GitTools extension (gitversion-setup@4 / gitversion-execute@4) and GitVersion.yml; no tags are needed.

To release, branch release/1.2.0 from develop, stabilise it (each push publishes 1.2.0-beta.N), then merge it into main (publishes 1.2.0) and back into develop. Hotfixes work the same way from main with hotfix/1.2.1. The run's build number is set to the version. Don't commit to main directly: a direct commit keeps the previous version, so its push is skipped as a duplicate.

Install a prerelease with dotnet tool install -g Ossa.QualityDashboard --prerelease (or --version 1.2.0-beta.3).

• Setup:

  1. Install the GitTools extension in the Azure DevOps organisation.
  2. Create a nuget.org API key that can push Ossa.QualityDashboard*, and add it as the secret pipeline variable NuGetApiKey.
  3. Create a develop branch from main if you don't have one.

• GitHub Actions: if the repo is on GitHub, .github/workflows/publish-tool.yml does the same job: the same GitVersion calculation (GitTools gitversion/setup and execute actions), the same branches publishing, and the same artifacts. A build job on windows-latest tests and packs, and a separate publish job pushes to nuget.org through the nuget environment.

  The workflow uses nuget.org Trusted Publishing, so no long-lived API key is stored. GitHub issues a short-lived OIDC token, and the NuGet/login action exchanges it for a one-hour nuget.org key just before the push. To set it up:

  1. On nuget.org, go to your username → Trusted Publishing and add a policy with the GitHub repository owner and repository, Workflow File publish-tool.yml (file name only) and Environment nuget. Give it a scope that allows pushing new packages and new versions of Ossa.QualityDashboard*.
  2. In GitHub, add a secret NUGET_USER with the nuget.org username (profile name, not email) that owns the policy.
  3. Optionally, add required reviewers to the nuget environment so each publish needs approval.

  For a private repository, the policy starts out active for 7 days only, and becomes permanent after the first successful publish. Use either the Azure DevOps pipeline or this workflow for publishing, not both.

• Check the version locally: dotnet tool restore, then dotnet gitversion /showvariable FullSemVer. GitVersion is pinned in dotnet-tools.json to match the pipeline.

• Two packages, one version: Ossa.QualityDashboard (the tool) and Ossa.QualityDashboard.Accessibility (the WCAG scanner library for site test projects) are packed and published together. The tests drive the Microsoft Edge that's preinstalled on windows-latest, so no Playwright browser download is needed.

• Package layout: the Ossa.QualityDashboard tool package is multi-targeted to net8.0, net9.0 and net10.0. Each framework has its own tools/<tfm>/any folder and dependency group, and NuGet picks the nearest framework at install time. Native libraries (SkiaSharp, HarfBuzz, SQLite) use NuGet's runtimes/<rid>/native layout, trimmed to win-x64 and win-arm64, and the .NET host loads the right one at run time. Change TargetFrameworks and ToolPackRuntimes in the csproj to adjust either list.

Build a package locally with dotnet pack src/QualityDashboard -c Release (output in nupkg/), and install it with dotnet tool install -g Ossa.QualityDashboard --add-source ./nupkg.

Agent requirements: Windows, plus Microsoft Edge or Google Chrome for PDF output (both are on Microsoft-hosted windows-latest; Edge ships with Windows 10/11 and Windows Server 2022+). The template installs the .NET 10 SDK itself. ZAP's Docker images need Linux containers, so run the scan itself on a Linux job/pipeline and publish the JSON reports as an artifact.

Self-contained executable

dotnet publish src/QualityDashboard -c Release -f net10.0 -r win-x64 -o ./publish
./publish/QualityDashboard.exe --demo

This gives a single QualityDashboard.exe that needs no .NET install, which is handy for agents without internet access or the .NET SDK.

Development

dotnet test

The WCAG scanner tests launch Microsoft Edge through Playwright. To use another browser, set WCAG_TEST_BROWSER to chrome, chromium, firefox or webkit (the last three need pwsh tests/QualityDashboard.Accessibility.Tests/bin/Debug/net10.0/playwright.ps1 install <browser> first).

The page is designed to fit about 8 projects on one A4 page. With more, the table carries onto a second page (the header row repeats). WCAG detail by page is in the separate WCAG report, not on the dashboard.

Third-party packages: the tool uses DocumentFormat.OpenXml (MIT), Microsoft.Data.Sqlite (MIT) and ScottPlot (MIT). The scanner library uses Microsoft.Playwright (MIT) and Deque.AxeCore.Playwright (MIT), which bundles axe-core (MPL-2.0). No commercially licensed components.