Compliance

Evidence for an audit, from the code and the CI run

An audit asks for evidence of change control and secure development: what changed, what depends on it, what was checked and on which commit. Collecting that by hand from CI is slow and easy to get wrong. The Wingbox pull request check writes it down on every run, on your own runner, for a team that prepares for ISO/IEC 27001, ISO/IEC 42001 or SOC 2.

What a run records today

A run of the released pull request check, version 1.4.0 and later, leaves a report, a review plan and one JSON record. Together they hold:

The commit that was checked

The report names the commit whose files it read, and how that commit relates to the head of the pull request: the head itself, or its merge into the base.

What a change touches and what depends on it

The function or type that holds the changed lines, and the modules outside the pull request that use it, each with the file and the line.

Boundary findings

The layer, capability and pattern violations the pull request introduces: the ones the base branch does not have.

The tests that use what changed

Named by file. When the job hands over its JUnit XML reports, each test says what the reports hold about it. The check runs no test itself.

File digests

The SHA-256 of each changed file in the commit that was checked, of each test report, and of the report and the plan written beside the record.

One JSON record with a schema

wingbox-evidence.json holds the repository and the run, the base, the head, the commit that was checked, the result with its reasons, the changed files and the tests with their evidence. Its schema is in the repository.

The evidence record in the README

Since 1.5.0 a run also records the evidence for each test case, the tests a change must pass, where each test report comes from, the policy files a change edits, what the run covered and separate verdicts, and a record is written also when the test step fails. The status of each.

What was not checked is never a pass

A run ends in PASS, FAIL, UNKNOWN or NOT_APPLICABLE. A run that could not check something says so in the result, with the reason, and in the record.

UNKNOWN

The result when something could not be checked or the tool ended abnormally. Never shown as a pass.

Parts that were not checked

A part of the repository on which the boundary rules could not be checked is named, with the reason, and makes the result UNKNOWN.

Skipped languages

Files of a language that were left out make the result UNKNOWN.

A checkout that is not the commit named

When the checkout is another commit, or holds a changed tracked file or an untracked source file, the result is UNKNOWN.

With fail-on: strict the job fails on FAIL and on UNKNOWN.

What is not claimed

ISO/IEC 27001

Support for evidence on secure development and change control, tied to the team's own statement of applicability and risk assessment. Not certification of a management system, and not the organisational requirements, which no code shows.

ISO/IEC 42001

Technical evidence for the life cycle and change of the code of systems with a model in them. Not governance, impact assessment or accountability, and not the behaviour of an agent at run time, none of which a code graph shows.

A profile that records the versions of model, prompt, tools and data is in development: wingbox #168.

SOC 2

Records over a period, with the failed runs included, for a reviewer to read. Not a positive report, and no conclusion on operating effectiveness from one passing snapshot.

Today each run writes its own record. The report over a period is in development: wingbox-action #25.

Technical evidence for a reviewer

A record states what was checked and what the run found. Whether a control conforms is the reviewer's decision.

The matrix of which evidence supports which control is in development: wingbox-action #26.

How it is built

Deterministic

Parsers build a graph of the code and the report is read from it. The same pull request gives the same report.

No language model

There is no model call, no API key and no prompt.

On your own runner

The graph is built and queried on your GitHub or GitLab runner. No code leaves it.

The repository is read, never run

The check reads files and asks git. Test reports are read as data.

What we hold to, and how to check it