Precept.Reporting.AzureDevOps 0.3.0

Precept.Reporting.AzureDevOps

Azure DevOps reporting for Precept: a test run per suite execution, with results filed against the Test Case work items the scenarios name — in their titles, or in a tag.

dotnet add package Precept.Reporting.AzureDevOps

public sealed class Startup : IPreceptStartup
{
    public void ConfigureServices(IServiceCollection services, PreceptSettings settings) =>
        services.AddPreceptAzureDevOpsReporting();
}

Scenario: [41207] A customer checks out with a stored card

{
  "Reporting": {
    "AzureDevOps": {
      "OrganizationUrl": "https://dev.azure.com/acme",
      "Project": "Web"
    }
  }
}

That scenario's result is filed against Test Case work item 41207, so the work item shows its own latest automated outcome and a test plan reports coverage from the run. A @tc:41207 tag does the same for a suite that prefers tags. A scenario covering two test cases names both — [41207][41208], or two tags — and is reported against each. Unmapped tests are still part of the run — set "OnlyMappedTests": true if the run should contain nothing else.

Add "TestPlanId": 412 — and optionally "TestSuiteId" — to file the run under a test plan. Precept then resolves the test point each scenario's work item occupies in that plan, which is what Azure DevOps requires of a planned result; a test that fills no point in the plan is left out of the run and counted in its comment. It needs no scope beyond Test Management, though Work Items (read) lets each planned result carry its test case's real revision. See running against a test plan.

The title is read with TestCaseTitlePattern, a regular expression with a group named id (^\s*(?:\[(?<id>\d+)\]\s*)+ by default), so 41207: A customer checks out or TC-41207 … is a pattern away. See linking scenarios to test cases.

This is the Test Results API rather than a .trx published by the PublishTestResults task; the difference is that link to the work item. Results are batched, so a thousand-scenario suite costs about ten requests, and a failing test's screenshots are uploaded as attachments of its result.

From a pipeline, pass the build's own token:

- script: dotnet test
  env:
    PRECEPT_REPORTING__AZUREDEVOPS__PERSONALACCESSTOKEN: $(System.AccessToken)
    PRECEPT_REPORTING__AZUREDEVOPS__BUILDID: $(Build.BuildId)

The job needs Allow scripts to access the OAuth token, and the build service identity needs test management rights on the project.

Test runs are created from a build agent only, so a test plan's latest outcome is the pipeline's and not whatever a developer last ran while writing the scenario. A local run files nothing and asks for nothing — the reporter is never started, so a missing token is not an error there. Add { "Reporting": { "CiOnly": false } } to precept.local.json to file from your own machine, and see CI and local runs for how a run is recognised as CI.

The test run's page is posted to the run's PreceptRunLinks as soon as Azure DevOps creates it, so a run that also reports to Teams gets an Open Azure DevOps button on its card — see linking a run's reports to each other.

Part of Precept, a .NET 10 test automation framework. Full documentation: reporting · configuration · artifacts