Modules
wingbox-evidence: technical evidence of a run, normalised and verified
A reviewer who was not there when a change was merged needs to know what was checked, on which commit, and what the tests reported. This library takes the facts a run produces and answers, the same way every time, whether a piece of evidence is well formed, complete and traceable to its source.
There is no package named wingbox-evidence to install yet. Today these capabilities are delivered through the pull request check; wingbox-evidence is the library they are going into.
What it is for
- One record per run: the commit that was checked, the result with its reasons, the changed files with their digests, the tests and what the test reports hold about them.
- A plain account of what a run did not analyse, so that a gap is seen as a gap.
- The same format for every run, with a schema, so that another system can keep the records.
What it takes in and gives out
Takes in
- From the code graph: what depends on a change, the boundary results with what was not checked, the languages that were left out.
- From the pull request check: the base, the head and the commit that was checked, the findings, the tests that use what changed.
- From the CI job: the test reports, read as data.
Gives out
- Today: the evidence record of a run, one JSON document with a schema, with the SHA-256 of the report and of the plan written beside it.
- In development: one package per run that verifies without the CI system, and the records of a period with their failures and gaps.
The rules it holds to
These hold in the pull request check today and are requirements of the library.
- What was not checked is never a pass. A part that was not analysed, a language that was left out or a checkout that is not the commit named ends in UNKNOWN.
- No language model, in the library or in its tests.
- The code, the tests and the build of the repository are never run. Test reports are read as data.
- The same inputs give the same result.
- It states what was checked and what a record holds. Whether a control conforms is the reviewer's decision.
Status
What is true today. Available: in a released version. Merged, in the next release: on the main branch, not released. In development: an open issue on GitHub.
| Capability | State | Source |
|---|---|---|
| The evidence record of a run: one JSON document with a schema | Available Pull request check 1.4.0 and later | |
| Test evidence from JUnit XML reports: passed, failed, skipped or no record for each test that uses what changed | Available Pull request check 1.4.0 and later | |
| The commit that was checked, and the SHA-256 of each changed file in that commit | Available Pull request check 1.4.0 and later | |
| A result of PASS, FAIL, UNKNOWN or NOT_APPLICABLE with its reasons | Available Pull request check 1.2.0 and later | |
| Test evidence for each test case: a test is passed only when every case recorded for it passed | Available Pull request check 1.5.0 and later | |
| No test found is a state of its own, never read as complete | Available Pull request check 1.5.0 and later | |
| Where a test report comes from: a report that git tracks, that is older than the run or that names another commit is refused | Available Pull request check 1.5.0 and later | |
| A record is written also when the test step fails or the check does not run | Available Pull request check 1.5.0 and later | |
| A change is judged with the policy files of the base commit, not with the ones it edits | Available Pull request check 1.5.0 and later | |
| What the run covered, in the record: what was analysed and what was left out | Available Pull request check 1.5.0 and later | |
| Separate verdicts, never folded into one: boundaries introduced by the change, boundaries there before it, the baseline, the tests, the coverage, and whether the record is eligible as evidence | Available Pull request check 1.5.0 and later | |
| An evidence package: one bundle per run, built and verified by a command, checkable without the CI system | In development | |
| Evidence over a period: every run with its failures, policy changes and gaps | In development | |
| The evaluation model: separate result dimensions, reason codes, replay with an explicit time | In development | |
| Adapters for the dialects of test reports, with the original reports kept byte for byte | In development | |
| The subject that was examined, and the provenance of the execution | In development | |
| Sealing and archive: a manifest with a detached attestation, restore and replay of an old package | In development | |
| A profile for systems with a model in them: the versions of model, prompt, tools and data a result was obtained with | In development |
See more
- wingbox-acceptance: Runs acceptance criteria and writes the report from the records, never by hand.
- wingbox-controls: Formal rules and the mapping to ISO/IEC 27001, ISO/IEC 42001 and SOC 2, with the limits of what code evidence can show.
- What a run gives a team that prepares for an audit
- The pull request check
- The architecture task on GitHub