Evidence for an audit, from the code and the CI run
An audit asks for evidence of change control and secure development: what changed, what depends on it, what was checked and on which commit. Collecting that by hand from CI is slow and easy to get wrong. The Wingbox pull request check writes it down on every run, on your own runner, for a team that prepares for ISO/IEC 27001, ISO/IEC 42001 or SOC 2.
What a run records today
A run of the released pull request check, version 1.4.0 and later, leaves a report, a review plan and one JSON record. Together they hold:
The commit that was checked
The report names the commit whose files it read, and how that commit relates to the head of the pull request: the head itself, or its merge into the base.
What a change touches and what depends on it
The function or type that holds the changed lines, and the modules outside the pull request that use it, each with the file and the line.
Boundary findings
The layer, capability and pattern violations the pull request introduces: the ones the base branch does not have.
The tests that use what changed
Named by file. When the job hands over its JUnit XML reports, each test says what the reports hold about it. The check runs no test itself.
File digests
The SHA-256 of each changed file in the commit that was checked, of each test report, and of the report and the plan written beside the record.
One JSON record with a schema
wingbox-evidence.json holds the repository and the run, the base, the head, the commit that was checked, the result with its reasons, the changed files and the tests with their evidence. Its schema is in the repository.
Since 1.5.0 a run also records the evidence for each test case, the tests a change must pass, where each test report comes from, the policy files a change edits, what the run covered and separate verdicts, and a record is written also when the test step fails. The status of each.
What was not checked is never a pass
A run ends in PASS, FAIL, UNKNOWN or NOT_APPLICABLE. A run that could not check something says so in the result, with the reason, and in the record.
The result when something could not be checked or the tool ended abnormally. Never shown as a pass.
A part of the repository on which the boundary rules could not be checked is named, with the reason, and makes the result UNKNOWN.
Files of a language that were left out make the result UNKNOWN.
When the checkout is another commit, or holds a changed tracked file or an untracked source file, the result is UNKNOWN.
With fail-on: strict the job fails on FAIL and on UNKNOWN.
What is not claimed
ISO/IEC 27001
Support for evidence on secure development and change control, tied to the team's own statement of applicability and risk assessment. Not certification of a management system, and not the organisational requirements, which no code shows.
ISO/IEC 42001
Technical evidence for the life cycle and change of the code of systems with a model in them. Not governance, impact assessment or accountability, and not the behaviour of an agent at run time, none of which a code graph shows.
A profile that records the versions of model, prompt, tools and data is in development: wingbox #168.
SOC 2
Records over a period, with the failed runs included, for a reviewer to read. Not a positive report, and no conclusion on operating effectiveness from one passing snapshot.
Today each run writes its own record. The report over a period is in development: wingbox-action #25.
Technical evidence for a reviewer
A record states what was checked and what the run found. Whether a control conforms is the reviewer's decision.
The matrix of which evidence supports which control is in development: wingbox-action #26.
How it is built
Deterministic
Parsers build a graph of the code and the report is read from it. The same pull request gives the same report.
No language model
There is no model call, no API key and no prompt.
On your own runner
The graph is built and queried on your GitHub or GitLab runner. No code leaves it.
The repository is read, never run
The check reads files and asks git. Test reports are read as data.
The pull request check today, three libraries next
Today these capabilities are delivered through the pull request check. They are going into three libraries, each usable alone. None is published as a package yet; each page has a status table.
Normalises and verifies technical evidence: the record of a run, where its inputs come from, and what was and was not analysed.
The record of a run is available in the pull request check, 1.4.0 and later. The library is in development.
Runs acceptance criteria and writes the report from the records, never by hand.
The manifest of required tests is available in the pull request check, 1.5.0 and later. The library is in development.
Formal rules and the mapping to ISO/IEC 27001, ISO/IEC 42001 and SOC 2, with the limits of what code evidence can show.
In development.