The full report with every table, exactly as its generator wrote it from the result files in the repository.
Wingbox benchmark: impact questions on public code
Wingbox 0.2.3 from PyPI, run of 2026-11-16. This file is written by make_report.py from the files in results/. Dockerfile.benchmark at the root of the repository runs the tests that need no model in a clean container and compares every result file with the committed one.
Summary
- 100 questions drawn at random with a fixed seed in 5 public repositories (TypeScript, Rust, Python):
wingbox impactnames the complete list of dependent files, with nothing outside it, in 100 of 100 (95 % interval 96 to 100 %). A required file is missing in 0 of 100. - A coding agent on 55 of these questions, with Wingbox against its built-in search alone: tokens -35 % (-47 % to -20 %), cost -51 % (-58 % to -43 %); complete answers 55 of 55 against 48 of 55.
- Every module dependency of the graph against the import statements of the code, in 7 corpora (3,795 dependencies in the largest): 0 wrong, 1 missing.
- Building the graph, the report and the map of fastapi (1,227 source files) from a fresh checkout takes 2.0 s on a hosted runner.
The question
For a file X: every file that depends on X directly or through one other file, where "A depends on B" means that A imports B. It is what a developer or an agent asks before changing X.
Reference lists come from resolvers that share no code with Wingbox, all in tools/: Python's ast (py_edges.py; on FastAPI its pairs equal the import graph of grimp, results/fastapi.grimp_check.json), an own TypeScript resolver (ts_resolve.py) and an own Rust resolver (rust_resolve.py).
Three kinds of file are accepted when an answer names them and are not required: in Python a package __init__.py executed on the way to the imported module; in TypeScript a file that imports the compiled output of X, which tsconfig.json maps back to X through outDir and rootDir; in Rust a file whose path passes through the module of X to reach a deeper module (crate::x::y). In each case a change of X can reach the file, and a reader of the import statements may or may not count it.
Sampling: per repository, 20 files drawn without replacement (seed 20261116) from the source files that at least one other file imports. Nothing else is filtered. The reference lists hold 1 to 503 files, median 9.
Test 1. The impact command, 100 questions
wingbox impact <file> 2 --all, no agent. "Complete and nothing extra": every required file is named and no named file is outside the accepted list.
| Repository | Language | Questions | Complete and nothing extra | With a missing file | Median answer, tokens |
|---|---|---|---|---|---|
| chrome-devtools-mcp | typescript | 20 | 20 | 0 | 306 |
| immich-server | typescript | 20 | 20 | 0 | 232 |
| agent-browser | rust | 20 | 20 | 0 | 159 |
| claudes-c-compiler | rust | 20 | 20 | 0 | 100 |
| fastapi | python | 20 | 20 | 0 | 86 |
| All | 100 | 100 (96 to 100 %) | 0 | 166 |
The interval is a 95 % Wilson interval. Mean recall 1.000, mean precision 1.000. In 90 of the 100 answers the list equals the required list file for file; in the other 10 Wingbox also names accepted files (19 in all). Tokens are characters of the output divided by four; the largest answer is 19,462 tokens for a list of 503 files.
Test 2. A coding agent with and without Wingbox, 55 questions
Claude Code, headless, model alias sonnet, no user settings, skills or connectors. Two setups: built-in search only (Read, Grep, Glob), and the same with the wingbox command added; the graph is built beforehand and the prompt names wingbox impact <file> 2 in one sentence. One trial per question and setup. The agent subset is drawn at random among the questions whose required list holds 3 to 120 files (up to 12 per repository). Tokens: everything the model read and wrote, cache included.
| Setup | Trials | Complete answers (95 % interval) | F1, mean (95 % interval) | Tokens, mean (95 % interval) | USD per answer | Turns |
|---|---|---|---|---|---|---|
| With Wingbox | 55 | 55 (93 to 100 %) | 1.000 (1.000 to 1.000) | 32,183 (28,229 to 36,523) | 0.031 | 3.0 |
| Built-in search only | 55 | 48 (76 to 94 %) | 0.979 (0.955 to 0.996) | 49,292 (42,863 to 56,703) | 0.063 | 5.0 |
- Tokens with Wingbox against search alone: -35 % (-47 % to -20 %); cost -51 % (-58 % to -43 %). Fewer tokens on 39 of the 55 questions, more on 16.
- Complete answer only with Wingbox on 7 questions, only without it on 0; exact sign test on these questions: p = 0.016.
| Repository | Questions | Tokens with Wingbox, mean | Tokens with search alone, mean | Change |
|---|---|---|---|---|
| chrome-devtools-mcp | 12 | 39,146 | 42,406 | -8 % |
| immich-server | 12 | 35,867 | 31,950 | +12 % |
| agent-browser | 12 | 35,158 | 78,529 | -55 % |
| claudes-c-compiler | 12 | 24,270 | 50,428 | -52 % |
| fastapi | 7 | 22,396 | 38,753 | -42 % |
Ratios are totals over the questions; their intervals are 95 % paired bootstrap intervals over the questions (10,000 resamples). The 110 trials cost 5.15 USD in total and ran with Wingbox 0.2.1 (the 100 answers of wingbox impact are identical in 0.2.1, 0.2.2 and 0.2.3); every trial with its scores is in results/agent.json.
Test 3. Are the edges in the code?
Every module dependency of the Wingbox graph against every import statement of the code. Wrong: an edge the reference neither requires nor accepts. Missing: a required edge the graph does not have.
| Corpus | Language | Dependencies in the code | Edges in the graph | Wrong | Missing |
|---|---|---|---|---|---|
| immich, server | TypeScript | 3,795 | 3,795 | 0 | 0 |
| chrome-devtools-mcp | TypeScript | 901 | 919 | 0 | 1 |
| agent-browser, Rust sources | Rust | 391 | 391 | 0 | 0 |
| claudes-c-compiler | Rust | 1,462 | 1,462 | 0 | 0 |
| FastAPI, whole tree | Python | 1,801 | 1,818 | 0 | 0 |
| FastAPI, the fastapi/ package | Python | 124 | 130 | 0 | 0 |
| nanoGPT, minGPT, micrograd | Python | 15 | 15 | 0 | 0 |
- chrome-devtools-mcp, missing:
src/third_party/index.ts->src/third_party/lighthouse-devtools-mcp-bundle.d.ts
The graph holds more edges than the code requires where it also records accepted ones (see "The question").
Test 4. Building the graph
Seconds from a fresh checkout to the graph, the report and the map: every run starts from a new export of the repository, with no earlier output and no cache. Median of 3 runs on a GitHub-hosted runner (Linux, AMD EPYC 9V45 96-Core Processor, 4 logical CPUs), workflow benchmark-build-time, script tools/time_builds.py, wingbox 0.2.3.
| Corpus | Source files | Build, s | Modules | Import edges |
|---|---|---|---|---|
| karpathy | 29 | 0.20 | 29 | 15 |
| chrome-devtools-mcp | 296 | 0.58 | 293 | 919 |
| agent-browser | 363 | 1.24 | 359 | 751 |
| claudes-c-compiler | 351 | 1.23 | 351 | 1,462 |
| immich-server | 707 | 1.34 | 707 | 3,795 |
| fastapi | 1,227 | 2.05 | 1,227 | 1,818 |
The size of the graph is read from the file each build wrote; the summary stops when two runs differ.
Scope
- One kind of question: which files import this file, directly or through one other file. Other questions are not measured here.
- Questions are drawn inside five repositories, so they are not independent in the strict sense; the per-repository rows show where a result comes from.
- Tests 1 and 3 are deterministic and need no model. Test 2 uses one agent and one model, one trial per question; a second run will differ in the details.
Versions and commits
| Item | Version or commit |
|---|---|
| Wingbox | 0.2.3 (PyPI wingbox) |
| grimp | 3.20 |
| nanoGPT | 3adf61e |
| minGPT | 37baab7 |
| micrograd | 7bc720e |
| fastapi | 2a1bd23 |
| chrome-devtools-mcp | 8ba2ac9 |
| agent-browser | 5338477 |
| claudes-c-compiler | 6f1b99a |
| immich | e5d6eb7 |
Reproduce
In a clean container (the repositories are fetched at the commits above; no model, no key):
git clone https://github.com/aileron-tech/wingbox && cd wingbox docker build -f Dockerfile.benchmark -t wingbox-benchmark . docker run --rm wingbox-benchmark # ends with REPRODUCED, or lists the result files that differ
results/docker_run.log is the output of that run for the committed results. Without a container:
benchmarks/impact/run.sh ~/wingbox-bench # tests 1 and 3; needs git and uv python3 -I benchmarks/impact/impact100.py ~/wingbox-bench agent # test 2: paid Claude Code sessions python3 -I benchmarks/impact/tools/time_builds.py ~/wingbox-bench # test 4 on this machine
| Path | Content |
|---|---|
run.sh, impact100.py, edge_test.py, verify.py, stats.py | the run, the sampling and the tests, the comparison with the committed results, the intervals |
tools/ | the reference resolvers and the build timing; none of them uses Wingbox code |
results/ | the questions, the reference lists (gold/), every row of the command test, every trial of the agent test, the edge test, the build times |