Benchmark report

The full report with every table, exactly as its generator wrote it from the result files in the repository.

Wingbox benchmark: impact questions on public code

Wingbox 0.2.3 from PyPI, run of 2026-11-16. This file is written by make_report.py from the files in results/. Dockerfile.benchmark at the root of the repository runs the tests that need no model in a clean container and compares every result file with the committed one.

Summary

  • 100 questions drawn at random with a fixed seed in 5 public repositories (TypeScript, Rust, Python): wingbox impact names the complete list of dependent files, with nothing outside it, in 100 of 100 (95 % interval 96 to 100 %). A required file is missing in 0 of 100.
  • A coding agent on 55 of these questions, with Wingbox against its built-in search alone: tokens -35 % (-47 % to -20 %), cost -51 % (-58 % to -43 %); complete answers 55 of 55 against 48 of 55.
  • Every module dependency of the graph against the import statements of the code, in 7 corpora (3,795 dependencies in the largest): 0 wrong, 1 missing.
  • Building the graph, the report and the map of fastapi (1,227 source files) from a fresh checkout takes 2.0 s on a hosted runner.

The question

For a file X: every file that depends on X directly or through one other file, where "A depends on B" means that A imports B. It is what a developer or an agent asks before changing X.

Reference lists come from resolvers that share no code with Wingbox, all in tools/: Python's ast (py_edges.py; on FastAPI its pairs equal the import graph of grimp, results/fastapi.grimp_check.json), an own TypeScript resolver (ts_resolve.py) and an own Rust resolver (rust_resolve.py).

Three kinds of file are accepted when an answer names them and are not required: in Python a package __init__.py executed on the way to the imported module; in TypeScript a file that imports the compiled output of X, which tsconfig.json maps back to X through outDir and rootDir; in Rust a file whose path passes through the module of X to reach a deeper module (crate::x::y). In each case a change of X can reach the file, and a reader of the import statements may or may not count it.

Sampling: per repository, 20 files drawn without replacement (seed 20261116) from the source files that at least one other file imports. Nothing else is filtered. The reference lists hold 1 to 503 files, median 9.

Test 1. The impact command, 100 questions

wingbox impact <file> 2 --all, no agent. "Complete and nothing extra": every required file is named and no named file is outside the accepted list.

RepositoryLanguageQuestionsComplete and nothing extraWith a missing fileMedian answer, tokens
chrome-devtools-mcptypescript20200306
immich-servertypescript20200232
agent-browserrust20200159
claudes-c-compilerrust20200100
fastapipython2020086
All100100 (96 to 100 %)0166

The interval is a 95 % Wilson interval. Mean recall 1.000, mean precision 1.000. In 90 of the 100 answers the list equals the required list file for file; in the other 10 Wingbox also names accepted files (19 in all). Tokens are characters of the output divided by four; the largest answer is 19,462 tokens for a list of 503 files.

Test 2. A coding agent with and without Wingbox, 55 questions

Claude Code, headless, model alias sonnet, no user settings, skills or connectors. Two setups: built-in search only (Read, Grep, Glob), and the same with the wingbox command added; the graph is built beforehand and the prompt names wingbox impact <file> 2 in one sentence. One trial per question and setup. The agent subset is drawn at random among the questions whose required list holds 3 to 120 files (up to 12 per repository). Tokens: everything the model read and wrote, cache included.

SetupTrialsComplete answers (95 % interval)F1, mean (95 % interval)Tokens, mean (95 % interval)USD per answerTurns
With Wingbox5555 (93 to 100 %)1.000 (1.000 to 1.000)32,183 (28,229 to 36,523)0.0313.0
Built-in search only5548 (76 to 94 %)0.979 (0.955 to 0.996)49,292 (42,863 to 56,703)0.0635.0
  • Tokens with Wingbox against search alone: -35 % (-47 % to -20 %); cost -51 % (-58 % to -43 %). Fewer tokens on 39 of the 55 questions, more on 16.
  • Complete answer only with Wingbox on 7 questions, only without it on 0; exact sign test on these questions: p = 0.016.
RepositoryQuestionsTokens with Wingbox, meanTokens with search alone, meanChange
chrome-devtools-mcp1239,14642,406-8 %
immich-server1235,86731,950+12 %
agent-browser1235,15878,529-55 %
claudes-c-compiler1224,27050,428-52 %
fastapi722,39638,753-42 %

Ratios are totals over the questions; their intervals are 95 % paired bootstrap intervals over the questions (10,000 resamples). The 110 trials cost 5.15 USD in total and ran with Wingbox 0.2.1 (the 100 answers of wingbox impact are identical in 0.2.1, 0.2.2 and 0.2.3); every trial with its scores is in results/agent.json.

Test 3. Are the edges in the code?

Every module dependency of the Wingbox graph against every import statement of the code. Wrong: an edge the reference neither requires nor accepts. Missing: a required edge the graph does not have.

CorpusLanguageDependencies in the codeEdges in the graphWrongMissing
immich, serverTypeScript3,7953,79500
chrome-devtools-mcpTypeScript90191901
agent-browser, Rust sourcesRust39139100
claudes-c-compilerRust1,4621,46200
FastAPI, whole treePython1,8011,81800
FastAPI, the fastapi/ packagePython12413000
nanoGPT, minGPT, microgradPython151500
  • chrome-devtools-mcp, missing: src/third_party/index.ts -> src/third_party/lighthouse-devtools-mcp-bundle.d.ts

The graph holds more edges than the code requires where it also records accepted ones (see "The question").

Test 4. Building the graph

Seconds from a fresh checkout to the graph, the report and the map: every run starts from a new export of the repository, with no earlier output and no cache. Median of 3 runs on a GitHub-hosted runner (Linux, AMD EPYC 9V45 96-Core Processor, 4 logical CPUs), workflow benchmark-build-time, script tools/time_builds.py, wingbox 0.2.3.

CorpusSource filesBuild, sModulesImport edges
karpathy290.202915
chrome-devtools-mcp2960.58293919
agent-browser3631.24359751
claudes-c-compiler3511.233511,462
immich-server7071.347073,795
fastapi1,2272.051,2271,818

The size of the graph is read from the file each build wrote; the summary stops when two runs differ.

Scope

  • One kind of question: which files import this file, directly or through one other file. Other questions are not measured here.
  • Questions are drawn inside five repositories, so they are not independent in the strict sense; the per-repository rows show where a result comes from.
  • Tests 1 and 3 are deterministic and need no model. Test 2 uses one agent and one model, one trial per question; a second run will differ in the details.

Versions and commits

ItemVersion or commit
Wingbox0.2.3 (PyPI wingbox)
grimp3.20
nanoGPT3adf61e
minGPT37baab7
micrograd7bc720e
fastapi2a1bd23
chrome-devtools-mcp8ba2ac9
agent-browser5338477
claudes-c-compiler6f1b99a
immiche5d6eb7

Reproduce

In a clean container (the repositories are fetched at the commits above; no model, no key):

git clone https://github.com/aileron-tech/wingbox && cd wingbox
docker build -f Dockerfile.benchmark -t wingbox-benchmark .
docker run --rm wingbox-benchmark        # ends with REPRODUCED, or lists the result files that differ

results/docker_run.log is the output of that run for the committed results. Without a container:

benchmarks/impact/run.sh ~/wingbox-bench                                    # tests 1 and 3; needs git and uv
python3 -I benchmarks/impact/impact100.py ~/wingbox-bench agent             # test 2: paid Claude Code sessions
python3 -I benchmarks/impact/tools/time_builds.py ~/wingbox-bench           # test 4 on this machine
PathContent
run.sh, impact100.py, edge_test.py, verify.py, stats.pythe run, the sampling and the tests, the comparison with the committed results, the intervals
tools/the reference resolvers and the build timing; none of them uses Wingbox code
results/the questions, the reference lists (gold/), every row of the command test, every trial of the agent test, the edge test, the build times