Measured on public code. Checked against the code.
100 questions drawn at random in five public repositories, an agent with and without Wingbox, every edge compared with the code. Intervals are 95 % confidence intervals. The scripts, the reference lists and every result file are public, and one container reproduces the run.
Scripts and results on GitHubRead the full reportquestions where wingbox impact names the complete list of dependent files (96 to 100 %).
tokens an agent reads per answer with Wingbox, against its own search alone (-47 % to -20 %). Cost: -51 %.
complete answers from the agent with Wingbox. With its own search alone: 48 of 55.
to build the graph of FastAPI, 1,227 files, from a fresh checkout: graph, report and map.
An agent with and without Wingbox
55 questions of the kind "which files depend on this file, directly or through one other file", in five public repositories (TypeScript, Rust, Python). Claude Code answers each one twice: with its built-in search only (Read, Grep, Glob), and with the wingbox command added. One trial per question and setup.
Data table
| Setup | Tokens per answer |
|---|---|
| With Wingbox | 32,183 |
| Built-in search only | 49,292 |
Data table
| Setup | USD per answer |
|---|---|
| With Wingbox | $0.031 |
| Built-in search only | $0.063 |
| Setup | Complete answers | F1, mean | Tokens, mean | USD | Turns |
|---|---|---|---|---|---|
| With Wingbox | 55 of 55 (93 to 100 %) | 1.000 (1.000 to 1.000) | 32,183 (28,229 to 36,523) | 0.031 | 3.0 |
| Built-in search only | 48 of 55 (76 to 94 %) | 0.979 (0.955 to 0.996) | 49,292 (42,863 to 56,703) | 0.063 | 5.0 |
- Tokens -35 % with Wingbox (-47 % to -20 %), cost -51 % (-58 % to -43 %). Fewer tokens on 39 of the 55 questions, more on the other 16.
- The complete list: with Wingbox on all 55 questions; with search alone on 48. On 7 questions only the agent with Wingbox was complete, on 0 only the agent without it (exact sign test, p = 0.016).
- With Wingbox
- Built-in search only
Data table
| Repository | With Wingbox | Built-in search only |
|---|---|---|
| chrome-devtools-mcp | 39,146 | 42,406 |
| immich-server | 35,867 | 31,950 |
| agent-browser | 35,158 | 78,529 |
| claudes-c-compiler | 24,270 | 50,428 |
| fastapi | 22,396 | 38,753 |
One question, with and without
Which files depend on src/backend/riscv/mod.rs (claudes-c-compiler), directly or through one other file? The reference list holds 32 files.
Impact of changing src/backend/riscv/mod.rs direct dependents 2 within 2 hops 32 Depth 1 src/backend/mod.rs imports src/backend/common.rs imports Depth 2 src/backend/arm/codegen/emit.rs imports src/backend/generation.rs imports src/backend/i686/codegen/alu.rs imports src/backend/i686/codegen/asm_emitter.rs imports src/backend/i686/codegen/atomics.rs imports src/backend/i686/codegen/calls.rs imports src/backend/i686/codegen/casts.rs imports src/backend/i686/codegen/comparison.rs imports src/backend/i686/codegen/emit.rs imports src/backend/i686/codegen/float_ops.rs imports src/backend/i686/codegen/globals.rs imports src/backend/i686/codegen/i128_ops.rs imports src/backend/i686/codegen/intrinsics.rs imports
2 turns, 16,482 tokens. All 32 files named, none missing.
13 turns, 147,818 tokens. 18 files named, 14 missing.
Before changing `src/backend/riscv/mod.rs`, I need a complete review list: every file of this repository, test files included, that depends on `src/backend/riscv/mod.rs` directly, or through exactly one other file. [...] Do not list `src/backend/riscv/mod.rs` itself. Answer with only a JSON array of repository-relative file paths.
This is the question where Wingbox saved the most. The chart above shows the mean and the repositories where it saved nothing.
Is the answer complete? 100 questions
No agent here. Per repository, 20 files are drawn at random (fixed seed) among the files that at least one other file imports; nothing else is filtered. wingbox impact is run on each, and the files it names are compared with a reference list built by resolvers that share no code with Wingbox: Python's own parser (equal to grimp on FastAPI), a TypeScript resolver and a Rust resolver. The lists hold 1 to 503 files, median 9.
| Repository | Language | Questions | Complete and nothing extra |
|---|---|---|---|
| chrome-devtools-mcp | TypeScript | 20 | 20 of 20 |
| immich-server | TypeScript | 20 | 20 of 20 |
| agent-browser | Rust | 20 | 20 of 20 |
| claudes-c-compiler | Rust | 20 | 20 of 20 |
| fastapi | Python | 20 | 20 of 20 |
| All | 100 | 100 of 100 (96 to 100 %) |
No required file is missing in any of the 100 answers. In 90 the answer equals the reference list file for file. In the other 10 Wingbox also names files the reference accepts without requiring them: scripts that import the compiled output of the file, which tsconfig maps back to the source, and a Python package file executed on the way to the import. The median answer is 166 tokens long.
Are the edges in the code?
Every file-to-file dependency of the Wingbox graph against every import statement of the code, read by a resolver that shares no code with Wingbox. Wrong: an edge the code does not have. Missing: a dependency of the code the graph does not have.
| Corpus | Language | Dependencies in the code | Wrong | Missing |
|---|---|---|---|---|
| immich, server | TypeScript | 3,795 | 0 | 0 |
| chrome-devtools-mcp | TypeScript | 901 | 0 | 1 |
| agent-browser, Rust sources | Rust | 391 | 0 | 0 |
| claudes-c-compiler | Rust | 1,462 | 0 | 0 |
| FastAPI, whole tree | Python | 1,801 | 0 | 0 |
| FastAPI, the fastapi/ package | Python | 124 | 0 | 0 |
| nanoGPT, minGPT, micrograd | Python | 15 | 0 | 0 |
The missing dependency in chrome-devtools-mcp: src/third_party/index.ts imports src/third_party/lighthouse-devtools-mcp-bundle.d.ts, the type declarations of a vendored bundle. Wingbox leaves vendored and minified bundles out of the graph.
How long the graph takes to build
Seconds from a fresh checkout to the graph, the report and the map, with no earlier output and no cache.
Data table
| Corpus | Build, seconds |
|---|---|
| fastapi (1,227 files) | 3.23 s |
| immich-server (707 files) | 1.84 s |
| agent-browser (363 files) | 2.06 s |
| claudes-c-compiler (351 files) | 2.23 s |
| chrome-devtools-mcp (296 files) | 0.79 s |
| karpathy (29 files) | 0.28 s |
The size of the graph each build wrote is recorded with the time (1,227 modules and 1,818 import edges for FastAPI), so a build that did less cannot pass as a fast one.
What this does not show
- One kind of question: which files import this file, directly or through one other file. It is the question asked before a change; other questions are not measured here.
- The questions are drawn inside five repositories, so they are not independent in the strict sense; the per-repository rows show where a result comes from.
- The agent test uses one agent and one model, one trial per question. A second run will differ in the details, which is what the intervals are for.
$ git clone https://github.com/aileron-tech/wingbox && cd wingbox$ docker build -f Dockerfile.benchmark -t wingbox-benchmark .$ docker run --rm wingbox-benchmarkThe container installs Wingbox 0.2.2 from PyPI, fetches the repositories at pinned commits, draws the same questions (seed 20261116), runs the tests that need no model and stops with an error when a result differs from the committed one: the log of that run ends with REPRODUCED, and a workflow repeats it on a hosted runner at every change of the benchmark. The agent test needs paid model sessions and is run apart (Wingbox 0.2.1, whose answers on the 100 questions are identical; model claude-sonnet-5-5); its trials are in the result files. Run of 2026-11-16.