Ruledger 1.5.0

Ruledger

Ruledger implements Rule-Derived Test Design on top of Rulealize.

It walks a rule set and writes down, for every state it visits, everything that can be observed there: which inputs are legal, whether the state is terminal and with what Result, where each legal input leads, and what each of the rule set's projections says about it. That enumeration is the test design. It is mechanical, so it does not forget. It is also tautological, so it does not know what is correct: it states what the rule set decides, never what it should decide. Supplying that judgement is the developer's job, and it is the only part of the work Ruledger refuses to guess at.

Change the rule set, run it again, and the diff is the blast radius — which states gained or lost a legal input, which outcomes moved, where a projection says something else. Edits the developer made to the previous test design are carried over; whatever could not be carried is reported instead of silently reset. How far the walk goes is a number of states it may visit, and where it stopped for want of them is reported as that, never as nothing being there.

The walk is reproducible. The same rule set visits the same states in the same order every time: no randomness, no seed. Everything else — that number of states, and how hard the runtime is asked to look in each one — is a setting, and it is recorded alongside the test design.

Using it

From a shell, it is Ruledger.Cli: ruledger derive, ruledger diff and ruledger observe, and its tutorial is the half hour that shows what this is like — reversi changed one line at a time, then a shift roster, ending in a count of what did not happen.

From a program, it is this package, which is what the tool calls:

dotnet add package Ruledger

using Rulealize;
using Ruledger;

RuleRuntime runtime = new RuleRuntime().LoadPluginsFrom("plugin");

// Walk it, and write the test design down.
TestDesign design = TestDesign.Derive(runtime, File.ReadAllText("reversi.json"));
File.WriteAllText("reversi.test-design.json", design.ToJson());

// Apply that design to a later version of the rules, which is the diff.
TestDesignDiff diff = TestDesignDiff.Of(
    runtime, File.ReadAllText("reversi.test-design.json"), File.ReadAllText("reversi.json"));

foreach (StateChange change in diff.Changed)
{
    // change.Before and change.After are the state as each design has it; Lost, Gained,
    // Moved, Refusing, NotRefusing and ProjectionsMoved are what the rules decide
    // differently there.
}

A host that shows a diff its own way asks this for it, and gets what changed as the values the tool prints its lines from — never a second account of it worked out from two designs. diff.After is the design the new rules give, with the choices of the one applied carried across; what could not be carried is in its Edits.

A rule set whose initial state is only the shape of a case — who may sign and what may be requested both empty until somebody says — offers nothing from there, and is walked from a case instead:

TestDesign design = TestDesign.Derive(
    runtime, File.ReadAllText("signed-request.json"), components,
    File.ReadAllText("case.state.json"), settings: null, edits: null);

The state is read against the rule set's schema, a composite's components included, like any other. It travels in the design as design.Start, the document itself rather than where it was read from, so whatever replays the design begins where the walk began; and a diff walks the new version from it. Choices handed in beside a start are carried one by one, as they are across a change to the rules.

A rule set whose uses holds something many keeps each instance — a claim of a month, say — as a document of its own, in a store its requires names. The walk does not open that store. Evaluation is pure, so where the documents are kept changes no answer, and it gives the runtime a MemoryStateStore in its place; the store's plugin is still loaded, because the rule set is built against it. The documents a many begins its instances in are handed over by name:

TestDesign design = TestDesign.Derive(
    runtime, File.ReadAllText("claim-month.json"), components, starts,
    start: null, settings: null, edits: null);

A position is then every document, and a state carries them as Documents, each instance compared whole — ObservationScan.Collections names the collections that are; a move is asked of each document in turn and says which instance it is at. A step reads only the rule set's own document and the instance it is at — Rulealize refuses a guard that reads anything its transition does not write — so one already followed from the same documents is written down and not followed again, which is what lets a walk end where another instance can always be created. A collection whose keys are written as a schema is walked at every key the schema names, and otherwise only at the keys a person wrote in a choice.

doc/test-design.md is the form of the document both of them write.

The numbers

Every claim about Ruledger is a number, and every one of them comes out of a run:

dotnet test verify/Ruledger.Verify.csproj --logger "console;verbosity=detailed"

67 measurements, about four minutes. They walk the rule sets in verify/ruleset/, print what they found, and fail if any of it has moved. Nothing about this project is a figure somebody wrote down once: the numbers in these documents are the numbers that command prints, and the way to check one is to run it.

test/ holds the unit tests, which are a different question and take two seconds.

What v1 does

Every "out" above is a decision rather than a gap, and the reason is in the row. Two of them are worth saying twice. A line-oriented view of a test design is left out because the document already carries what one needs — measured, not asserted. A coverage number is left out and stays out, because a number standing in for quality is the thing this method was written against; how many places the walk stopped in front of is a count of places, and it is the only quantity reported.

Status

v1 is done and published. The library walks a rule set, works out by itself which state two positions are compared on, writes the test design down, applies one to a later version and carries the developer's edits across. The measurements all run. Ruledger and Ruledger.Cli are on nuget.org.

Since then it runs on the Rulealize that leaves a parameter open and lets an input validate its arguments — and name, with invalid, the refusal of a value the schema does not admit — and the test design it writes became ruledger/test-design/v2 for the one shape that added — a move waiting for a value. It is ruledger/test-design/v3 since a walk can begin in a state of a case, which the design carries as start, and ruledger/test-design/v4 since it writes down what a rule set's projections say: a host shows them, so a change to one is a change the rules made. It is ruledger/test-design/v5 since it walks a rule set that keeps instances as documents of their own, which needed a position of several documents, a move at an instance and a move waiting for a key. A v4, v3, v2 or v1 test design is still read; a v3 one said nothing about projections, so nothing about them is held against it.

Until Ruledger.Cli 1.3.0 the library shipped only inside the tool. It is a package of its own from 1.0.0, and the tool has a repository of its own, because a host that draws a diff on a screen has to be handed what changed rather than read it back out of lines meant for a person.

It is C#: it needs the Rulealize runtime, and reading the rule set statically is part of the same job, not a separate program in another language.

License

Apache-2.0. See LICENSE.