Modules

wingbox-evidence: technical evidence of a run, normalised and verified

A reviewer who was not there when a change was merged needs to know what was checked, on which commit, and what the tests reported. This library takes the facts a run produces and answers, the same way every time, whether a piece of evidence is well formed, complete and traceable to its source.

There is no package named wingbox-evidence to install yet. Today these capabilities are delivered through the pull request check; wingbox-evidence is the library they are going into.

What it is for

  • One record per run: the commit that was checked, the result with its reasons, the changed files with their digests, the tests and what the test reports hold about them.
  • A plain account of what a run did not analyse, so that a gap is seen as a gap.
  • The same format for every run, with a schema, so that another system can keep the records.

What it takes in and gives out

Takes in

  • From the code graph: what depends on a change, the boundary results with what was not checked, the languages that were left out.
  • From the pull request check: the base, the head and the commit that was checked, the findings, the tests that use what changed.
  • From the CI job: the test reports, read as data.

Gives out

  • Today: the evidence record of a run, one JSON document with a schema, with the SHA-256 of the report and of the plan written beside it.
  • In development: one package per run that verifies without the CI system, and the records of a period with their failures and gaps.

The rules it holds to

These hold in the pull request check today and are requirements of the library.

  • What was not checked is never a pass. A part that was not analysed, a language that was left out or a checkout that is not the commit named ends in UNKNOWN.
  • No language model, in the library or in its tests.
  • The code, the tests and the build of the repository are never run. Test reports are read as data.
  • The same inputs give the same result.
  • It states what was checked and what a record holds. Whether a control conforms is the reviewer's decision.

Status

What is true today. Available: in a released version. Merged, in the next release: on the main branch, not released. In development: an open issue on GitHub.

The evidence record of a run: one JSON document with a schemaAvailable
Pull request check 1.4.0 and later
Test evidence from JUnit XML reports: passed, failed, skipped or no record for each test that uses what changedAvailable
Pull request check 1.4.0 and later
The commit that was checked, and the SHA-256 of each changed file in that commitAvailable
Pull request check 1.4.0 and later
A result of PASS, FAIL, UNKNOWN or NOT_APPLICABLE with its reasonsAvailable
Pull request check 1.2.0 and later
Test evidence for each test case: a test is passed only when every case recorded for it passedAvailable
Pull request check 1.5.0 and later
No test found is a state of its own, never read as completeAvailable
Pull request check 1.5.0 and later
Where a test report comes from: a report that git tracks, that is older than the run or that names another commit is refusedAvailable
Pull request check 1.5.0 and later
A record is written also when the test step fails or the check does not runAvailable
Pull request check 1.5.0 and later
A change is judged with the policy files of the base commit, not with the ones it editsAvailable
Pull request check 1.5.0 and later
What the run covered, in the record: what was analysed and what was left outAvailable
Pull request check 1.5.0 and later
Separate verdicts, never folded into one: boundaries introduced by the change, boundaries there before it, the baseline, the tests, the coverage, and whether the record is eligible as evidenceAvailable
Pull request check 1.5.0 and later
An evidence package: one bundle per run, built and verified by a command, checkable without the CI systemIn development
Evidence over a period: every run with its failures, policy changes and gapsIn development
The evaluation model: separate result dimensions, reason codes, replay with an explicit timeIn development
Adapters for the dialects of test reports, with the original reports kept byte for byteIn development
The subject that was examined, and the provenance of the executionIn development
Sealing and archive: a manifest with a detached attestation, restore and replay of an old packageIn development
A profile for systems with a model in them: the versions of model, prompt, tools and data a result was obtained withIn development