NovoNordisk.SpecTrack 1.2.10

NovoNordisk.SpecTrack

SpecTrack is a CLI tool designed to provide traceability between user requirement specifications as code and the executed test results. It facilitates the generation of various test reports (test report, validation reports, and custom reports), ensuring that your software development process meets and documents the predefined requirements.

Whether you adhere to the default configuration or customize the tool to fit your specific workflows, SpecTrack ensures a seamless integration into your CI/CD pipeline automation flow.

Installation

To install the latest stable version of SpecTrack as a NuGet package, run the following command:

dotnet tool install --global NovoNordisk.SpecTrack

To install a beta version (published from non-main branches), use the --prerelease flag:

dotnet tool install --global NovoNordisk.SpecTrack --prerelease

For more information about beta packages, see Preview Packages Documentation.

Prerequisites

SpecTrack, by design, does not include aggregators for various test frameworks such as xUnit, NUnit, Playwright, etc. Instead, it leverages the capabilities of the Allure Reports aggregator.

Pick your framework(s):

• https://allurereport.org/docs/xunit/
• https://allurereport.org/docs/reqnroll/
• https://allurereport.org/docs/playwright/
• ... or any other (nunit, pytest, cypress, ...) framework that Allure supports

SpecTrack does not use the "allure generate" command or any other Allure commands, except for accessing the actual test result files automatically generated by Allure. If you are curious why we use Allure Reports, then please read more in the Why Allure Reports?

Commands

Command: Extract requirements

spectrack extract requirements --name=mymodule --path='./requirements/**/**.md' --verbose

# Output: dist/mymodule-requirements.json

The requirements command is used for extracting requirements and mapping them to the SpecTrack Requirement domain model. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	true	-	Name prefix to the file that will be generated as an output.
--path	string	true	-	Path to the file location. Supports globbing.
--tags	array	false	["URS/URS.AC", "URS/DS/DS.AC", "URS/FS/FS.AC"]	Tags hierarchy for requirement definitions.
--pattern	regex	false	Auto-detected based on the extension of the --path defintion (.md or .adoc)	Used when extracting the requirement tags from the defined files in --path. Not recommended to update.
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

How to write Requirements as Code

SpecTrack supports both markdown and asciidoc when writing the requirements as code.

Example: ./requirements/create-account.md

`@URS:CreateAccount`
# URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

## Acceptance criteria

`@URS.AC:Min10CharPassword`
- URS.AC: Password should contain minimum 10 characters

Rules

• All root tags should be unique (in default SpecTrack configuration the root tag prefix is "URS")
• Tags within the same root (URS) are scoped, which means that they should be unique within the same URS/scope

INCORRECT tag structure:

- URS:CreateAccount
   - URS.AC:Min10CharPassword
- URS:CreateAccount // duplicate root-tag, will fail.
  - URS.AC:Example

CORRECT tag structure:

- URS:CreateAccount
   - URS.AC:Min10CharPassword
- URS:ResetPassword
  - - URS.AC:Min10CharPassword // URS.AC:Min10CharPassword is unique _within_ the same URS scope, which is approved.

Command: Extract test results

spectrack extract testresults --name=unittests --path='src/unittests/allure-results/*-result.json' --attachments-path='src/unittests/allure-results/*-attachment*' --verbose

# Output: dist/unittests-testresults.json
# Output: All the matched attachments will be stored in: dist/attachments/...

The testresults command is used for extracting test results and mapping them to the SpecTrack Test Result domain model. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	Yes	-	Name prefix to the file that will be generated as an output.
--path	string	Yes	-	Path to the executed test results location. Supports globbing.
--attachments-path	string	No	-	Path to executed test result attachments. Supports globbing.
--tags	array	false	["URS/URS.AC", "URS/DS/DS.AC", "URS/FS/FS.AC"]	Tags hierarchy for requirement definitions.
--exclude-tags	string	No	["draft"]	Define tags to exclude from the report.
--aggregator	string	No	"Allure"	Test result aggregator to parse executed test runs to SpecTrack domain models.
--check-testresults-exists	bool	No	true	Fail if no test results could be found based on the provided path.
--check-attachments-exists	bool	No	true	Fail if no test attachments could be found based on the provided path.
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

Microsoft.Testing.Platform TRX reports

Important: The MtpTrx aggregator only supports TRX reports generated by Microsoft.Testing.Platform. VSTest-generated TRX is not supported because VSTest does not expose MSTest TestProperty attributes in its report output.

The aggregator is test-framework independent as long as the framework publishes its metadata as MTP test properties. It has been verified with MSTest and xUnit v3.

spectrack extract testresults --name=dotnet-tests --path='src/backend/**/TestResults/*.trx' --aggregator=MtpTrx --verbose

Use the metadata key SpecTrack and values from the same configured tag hierarchy as requirement extraction. Class-level metadata is included in every test in that class.

MSTest

Use one TestProperty per requirement link:

[TestClass]
[TestProperty("SpecTrack", "URS:CreateAccount")]
public class UserAccountTests
{
    [TestMethod]
    [TestProperty("SpecTrack", "URS.AC:Min10CharPassword")]
    public void Password_MinimumLength_Validation()
    {
        // Test implementation
    }
}

xUnit v3

Use one Trait per requirement link:

Compatibility: xUnit 3.x defaults to MTP v1. Projects using MTP v2 packages must reference xunit.v3.mtp-v2 instead of xunit.v3; xUnit 4.0 and later default to MTP v2. Keep the xUnit MTP major version aligned with the other Microsoft.Testing.* packages to avoid runtime type-loading errors. See Choosing the Microsoft Testing Platform version.

[Trait("SpecTrack", "URS:CreateAccount")]
public class UserAccountTests
{
    [Fact]
    [Trait("SpecTrack", "URS.AC:Min10CharPassword")]
    public void Password_MinimumLength_Validation()
    {
        // Test implementation
    }
}

NUnit

Use one Property per requirement link:

[Property("SpecTrack", "URS:CreateAccount")]
public class UserAccountTests
{
    [Test]
    [Property("SpecTrack", "URS.AC:Min10CharPassword")]
    public void Password_MinimumLength_Validation()
    {
        // Test implementation
    }
}

A test must have a root requirement property (for example, URS:CreateAccount) when it has sub-requirement or acceptance-criteria properties. Add multiple SpecTrack properties or traits when a test covers multiple sub-requirements. Metadata with other keys is ignored.

Linking Allure reports to requirements

In this example, we are using xUnit and the xUnit aggregator from Allure.

There are different ways of creating the link from the test definition to the requirement.

Rules:

• There should always be a label named "req".
  • This should always refer to the "root" requirement tag (with default SpecTrack configuration, this would be the "URS:" tag.)
• Optional: Add a label named spec if you want to refer to any sub-requirements within the same URS tag.
  • You can add several spec-labels, if the same test case covers several sub-requirements (within the same URS tag).

using Allure.Xunit;
using Xunit;

[AllureLabel("req", "URS:CreateAccount")] // Tagging, the requirement
public class UserAccountTests
{
    [AllureLabel("spec", "URS.AC:Min10CharPassword")] // Tagging, the individual acceptance criteria
    // Optinal: Possibel to assign the same test definition to several sub-requirements within the same URS:CreateAccount
    // [AllureLabel("spec", "URS.AC:Max40CharPassword")]
    [Fact]
    public void Password_MinimumLength_Validation()
    {
        // Not important for SpecTrack...
    }
}

Command: Merge requirements & test results

# Output from 'extract requirements' command: dist/mymodule-requirements.json
# Output from 'extract testresults' command: dist/unittests-testresults.json

spectrack merge --name=requirements --requirements-path='dist/*-requirements.json' --testresults-path='dist/*-testresults.json' --verbose

# Output: dist/requirements-merged.json

The merge command is used to merge and validate the provided requirements with test result execution. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	Yes	-	Name prefix to the file that will be generated as an output.
--requirements-path	string	Yes	-	Path to SpecTrack extracted requirements. Supports globbing.
--testresults-path	string	Yes	-	Path to SpecTrack extracted test results. Supports globbing.
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

Command: Generating Reports

# Test Report
spectrack report --name test-report.html --dist reports --source 'dist/requirements-merged.json' --template '<TEST_REPORT_HTML>' --params-file params.spectrack --verbose

# Output: reports/test-report.html
# Output: reports/attachments/... (all attachments has been copied from the specified path)

# Validation Report:
spectrack report --name validation-report.adoc --params-file .spectrack --source 'dist/requirements-merged.json' --template '<VALIDATION_REPORT_ASCIIDOC>' --verbose

# Output:

The report command is used for generating reports (implementation report and test report) based on the merged requirements and test results. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	Yes	-	Name including the extension.<br>Example: --name my-test-report.html
--source, -s	string	Yes	-	File path to the source data used as input to the report template.
--template, -t	string	Yes	-	Path to the liquid template, or built-in SpecTrack template reference.
--attachments-path	string[]	No	-	Path to attachments directories. Supports globbing.
--inline-attachments	bool	No	false	Inline attachments in the test report. Combined attachment size limit of 200mb.
--params-file	string	No	-	Path location to the SpecTrack params file.
--params, -p	string	No	-	Additional params to be included in the report.<br>Expected syntax: --meta status=draft
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

Passing Parameters to SpecTrack Report CLI

When passing parameters to the SpecTrack report CLI, there are two methods available for interacting with the template engine (Read: How does the template engine works?).

1. Using a Parameter File: To start, you can utilize a file, typically named params.spectrack, which contains static variables. All dynamic parameters can then be passed as CLI parameters. Within the params.spectrack file, dynamic variables can be referenced. For instance:

productName=My Awesome Product
release=Release tag {tag}

2. Override Parameters: If you need to override specific parameters, you can achieve this by simply passing the --params tag=1.0.0 as a parameter to the spectrack report command. This approach allows for the straightforward adjustment of individual parameters as needed.

How does the template engine works?

The template engine used by SpecTrack operates as a liquid template parser behind the scenes. When utilizing the CLI, any data passed with --source (e.g., file.json) is sent to the specified template indicated by --template (e.g., mytemplate.liquid). This process allows for the dynamic generation of output based on the provided data and the template's structure.

SpecTrack uses https://github.com/sebastienros/fluid as the template engine based on the Liquid template language.

SpecTrack built-in templates

• Template: Test Report
  • --template <TEST_REPORT_HTML>
• Template: Validation Report
  • asciidoc:
    • --template <VALIDATION_REPORT_ASCIIDOC>
  • markdown:
    • --template <VALIDATION_REPORT_MARKDOWN>

Parameters:

Parameter	Required	Example	Description
productName	Yes	--params productName="SpecTrack"	Used to show the Product Name in the Test Report
pipeline.link	No	--params pipeline.link="https://dev.azure.com/orgname/project/_build/results?buildId=buildid"	Link to the pipeline from where the report has been generated.
pipeline.displayName	No	--params pipeline.displayName="Build ID #123"	Used to show the Product Name in the Test Report
git.commitId	No	--params git.commitId="commitId"	Used to show the Product Name in the Test Report
git.requirementLink	No	--params git.requirementLink="https://dev.azure.com/orgname/project/_git/gitrepo/?version=GC{git.commitId}\u0026path={spectrack:filePathRelativeToRoot}"	Creates a link for each requirement in the test report that links to the specific commitId and file location where your requirement is defined.<br>Note that {spectrack:filePathRelativeToRoot} is important and will be replaced runtime.
testTypeFilter	No	--params testTypeFilter='{"Acceptance Tests":["AcceptanceTests"],"Unit Tests":["UnitTests"]}'	Enables test type filtering in the Test Report with user-defined filter types. The value is a JSON object where keys are filter button labels and values are arrays of path substrings to match against the test fullName field (case-insensitive). Each filter can have multiple paths. Comma-separated strings are also supported as values (e.g. "path1,path2"). If not set, filter buttons are hidden.
sortPriority	No	--params sortPriority='["URS"]'	Controls the display order of requirement groups in the Test Report. The value is a JSON array of title prefixes, ordered by priority. Items whose title starts with the first prefix appear first, then the second prefix, and so on. Items not matching any prefix sort last. Within each group, items sort alphabetically. If not set, all items sort alphabetically (default behavior).

Test Type Filter Feature

The test type filter is an optional feature for the Test Report that allows users to filter and view tests by their type. When configured via the testTypeFilter parameter, the report displays toggle buttons above the test list for easy filtering.

Configuration Format:

The testTypeFilter parameter accepts a JSON object where:

• Keys are the display names for the filter buttons
• Values are arrays of path substrings to match against the test's fullName field (case-insensitive)

Each filter can specify multiple paths, allowing flexible matching across different test naming conventions.

{
  "Acceptance Tests": ["AcceptanceTests"],
  "Unit Tests": ["UnitTests"],
  "Integration Tests": ["MCRA.Backend.Test.IntegrationTests", "MyApp.IntegrationTests"],
  "Snapshot Tests": ["SnapshotTests"]
}

Values can also be comma-separated strings instead of arrays:

{
  "Integration Tests": "MCRA.Backend.Test.IntegrationTests,MyApp.IntegrationTests"
}

How to Use:

# Enable the filter feature with custom test types
spectrack report --name test-report.html --dist reports --source 'dist/requirements-merged.json' --template '<TEST_REPORT_HTML>' --params 'testTypeFilter={"Acceptance Tests":["AcceptanceTests"],"Unit Tests":["UnitTests"]}' --params-file params.spectrack --verbose

Or in a .spectrack params file:

testTypeFilter={"Acceptance Tests":["AcceptanceTests"],"Unit Tests":["UnitTests"],"Integration Tests":["MCRA.Backend.Test.IntegrationTests","MyApp.IntegrationTests"]}

Behavior:

• An "All Tests" button is always shown as the first filter option
• When a filter is applied, only tests whose fullName contains any of the configured path substrings are displayed
• Matching is case-insensitive
• Empty requirement nodes (with no matching tests) are automatically hidden to keep the UI clean
• Test summary statistics (pass/fail/broken counts) are recalculated to reflect the filtered results
• The filter is client-side only - no server processing required

Note: If the testTypeFilter parameter is not set, the filter buttons will not appear, and users will see the standard view showing all tests.

Sort Priority Feature

The sort priority is an optional feature for the Test Report that controls the display order of requirement groups. By default, all requirement nodes are sorted alphabetically by their title. When configured via the sortPriority parameter, items whose titles match specified prefixes are sorted to the top.

Configuration Format:

The sortPriority parameter accepts a JSON array of title prefix strings, ordered from highest to lowest priority:

["URS"]

Or with multiple priority groups:

["URS", "RC-NF"]

How to Use:

# URS items appear first, everything else sorted alphabetically after
spectrack report ... --params 'sortPriority=["URS"]'

# URS first, then RC-NF, then everything else
spectrack report ... --params 'sortPriority=["URS","RC-NF"]'

Or in a .spectrack params file:

sortPriority=["URS","RC-NF"]

Behavior:

• Items whose title starts with a prefix appearing earlier in the array are sorted first
• Items not matching any prefix are sorted last
• Within each priority group, items continue to sort alphabetically by title
• Sorting is applied recursively at every level of the requirement tree
• If not set, the default alphabetical sorting is used

Note: If the sortPriority parameter is not set, the default alphabetical sorting behavior is unchanged.

Test Report Organization

The Test Report displays requirements in the left panel organized in a hierarchical tree structure. Requirements are sorted alphabetically by their titles at each level by default. To customize the display order of requirement groups, see the Sort Priority Feature section.

(Only for Novo Nordisk employees: Find the SpecTrack team in our Developer Portal, to get the verified internal-only validation report template.)

Command: Validate tag coverage

spectrack validate tags --path='./requirements/**/**.md' --test-contexts './tests' './src/tests' --context . --verbose

# Output: Success or failure message indicating whether all requirement tags have apparent test coverage

The validate tags command is used for validating that all requirement tags have apparent test coverage in your test files. This command is designed to be run in CI/CD pipelines during Pull Request validation to provide early feedback about missing test coverage without needing to run all tests.

How it works:

The command extracts all tags from your requirements and then searches the specified test directories to check if each tag appears at least once. A tag that appears in any of the test context directories is considered to have apparent test coverage.

Important notes:

• This is a heuristic check that provides a "good enough" early validation. It checks if tags appear in your test directories.
• Parent references (e.g., @PARENT::URS:CreateAccount or @PARENT:URS:CreateAccount) are excluded from the count to avoid false positives.
• It does NOT check if a test is tagged with a valid combination of URS tag and child FS / AC tag.
• This does not replace running actual tests - it only validates that tags appear to be referenced in test files.
• By specifying test contexts, the command only searches those directories, making it much faster than scanning the entire repository.

Option	Type	Required	Default	Description
--path	string	Yes	-	Path to the requirement files. Supports globbing.
--test-contexts	string[]	Yes	-	Paths to directories containing test files. Multiple paths can be provided. Supports globbing.
--pattern	regex	No	Auto-detected based on file extension (.md or .adoc)	Used when extracting the requirement tags from the defined files in --path.
--tags	array	No	["URS/URS.AC", "URS/DS/DS.AC", "URS/FS/FS.AC"]	Tags hierarchy for requirement definitions.
--context	string	No	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--check-orphaned-tags	bool	No	true	Fail if a test context references a requirement tag with no matching declaration (e.g. a stale or misspelled tag left after a requirement was renamed or removed). Pass --check-orphaned-tags:false to disable. Note: since this compares against only the tags declared in --path, scoping --path to a subset of requirements while --test-contexts covers a broader test directory can produce false positives - widen --path or disable this check in that case.
--skip-coverage-check	bool	No	false	Skip the forward check that every requirement tag has apparent test coverage. Useful when adopting --check-orphaned-tags on a codebase with a pre-existing coverage backlog.
--verbose, -v	bool	No	false	Increase logging verbosity.

Example use in CI/CD:

# In your Pull Request pipeline, run this before running the full test suite
# Specify where your tests are located for faster execution
spectrack validate tags --path='./requirements/**/**.md' --test-contexts './tests' './src/IntegrationTests' --context .

# Exit code 0 = all tags have apparent coverage
# Exit code 1 = some tags are missing apparent coverage

Exit codes:

• 0: All requirement tags appear to have test coverage
• 1: One or more requirement tags do not appear to have test coverage

FAQ

Why Allure Reports?

Allure Reports provides a flexible and comprehensive way to aggregate test results from numerous testing frameworks. By using Allure Reports, SpecTrack ensures compatibility with a wide range of existing frameworks without the need to develop and maintain individual aggregators.

Using the Allure Reports aggregator ensures that SpecTrack can seamlessly integrate with your existing test frameworks to extract test results while focusing on providing high-quality, detailed compliance reports.

Test Result Aggregator Flexibility

SpecTrack is designed to be tool-agnostic. This means that while we currently use Allure Reports to extract test results, the underlying architecture of SpecTrack allows for flexibility in changing or adding aggregators as needed.

Markdown & Asciidoc Requirement example

Requirement definition in markdown:

`@URS:CreateAccount`
# URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

## Acceptance criteria

`@URS.AC:Min10CharPassword`
- URS.AC: Password should contain minimum 10 characters

Equivalent definition in asciidoc:

[[URS:CreateAccount]]
= URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

== Acceptance criteria

[[URS.AC:Min10CharPassword]]
- URS.AC: Password should contain minimum 10 characters

Supported --pattern

--pattern is dynamically set based on the --path extension:

Markdown (example: --path ./requirements/*.md)

^`@(?<draft>DRAFT::?)?(?<parent>PARENT::?)?(?<tag>(?<tagId>[^:].+):{1}(?<tagTitle>[^:].+))`\r?$(\n^(#|\*|-|\+|1\.)+\s(?<title>.+))?$

Asciidoc: (example: --path ./requirements/*.adoc)

^\[\[(?<draft>DRAFT::?)?(?<parent>PARENT::?)?(?<tag>(?<tagId>[^:].+):{1}(?<tagTitle>[^:].+))\]\]\r?$(\n^(=|\*|-|\.|1\.)+\s(?<title>.+))?$

Create your own custom "--pattern"

SpecTrack uses regex for pattern matching and can support various types of flat files. You can extend the "pattern" with your custom pattern.

Ensure that your custom regex pattern includes these specified groups:

• <parent>
  • The pattern to find parents (in case you split requirements into several files)
  • Example: PARENT::
• <tag>
  • The entire tag
  • Example: URS:MyExample
• <tagId>
  • The ID of the tag must match the tags defined in "requirements.tags"
  • Example: URS
• <tagTitle>
  • The title of the tag
  • Example: MyExample
• <title>
  • The title of the requirement