Comparison report

The full report with every table, exactly as its generator wrote it from the result files in the repository.

Wingbox and Graphify on the same public tests

Run of 2026-11-16. Wingbox 0.2.0 from PyPI. Graphify 0.9.110 from PyPI (graphifyy), GRAPHIFY_BACKEND=none (no model, no key). Every number below is produced by the scripts in tools/ from the files in results/; this file is written by tools/make_report.py, and run.sh reproduces tests A, B and C from nothing.

Summary

  • Graphify's "71.5x fewer tokens per query" is the size of the whole corpus divided by the size of one graph answer. No answer is checked. On the code of its own example corpus Graphify publishes 8.8x; Graphify 0.9.110 gives 3.5x and 5.0x today, and the same formula gives Wingbox 78.1x (test A).
  • Asked which files depend on a file, wingbox impact names the exact list in 5 of 5 questions, graphify affected in 1 of 5 (test C).
  • A coding agent answers the same questions correctly in all three setups. It spends 25,598 tokens with Wingbox, 59,367 with its built-in search alone and 109,071 with Graphify: Wingbox -57 % against plain search, Graphify +84 % (test D).
  • Building the graph of fastapi (1,227 source files) takes Wingbox 2.2 s and Graphify 17.9 s on a hosted runner; over the five corpora Graphify's extract takes 3.5 to 19.2 times as long as the whole Wingbox build (test E).

What Graphify publishes in its repository

Read at Graphify-Labs/graphify, branch v8, commit 18cd78f (2026-11-15).

Published numberWhere it comes fromReproducible from the repository
71.5x fewer tokens per query; 8.8x on code onlygraphify benchmark (graphify/benchmark.py) on worked/karpathy-repos: nanoGPT, minGPT, micrograd, 5 papers, 4 imagesyes: formula, corpus list and five questions are published
12 of 12 inferred cross-file edges correctmanual audit under fastapi/ of the FastAPI repository (BENCHMARKS.md)yes: the repository is public
ERPNext 82.0 % against 70.8 %; LOCOMO; LongMemEvalBENCHMARKS.md names crosstool/run.py and memory/runner.pyno: neither folder, nor the questions, nor the gold facts are in the repository

The formula of graphify benchmark: corpus tokens = words x 100 / 75 (every file in the context); answer tokens = characters / 4; reduction = corpus tokens / mean answer tokens over the questions that matched something in the graph.

Test A. Graphify's token formula, Graphify's corpus

Corpus: the code of karpathy/nanoGPT, minGPT and micrograd: 29 Python files, 12,748 words, 16,997 tokens. Graphify's review states ~16,997 tokens for the same 29 files.

Tool and query setQueries answeredMean answer, tokensReduction
Graphify, published in worked/karpathy-repos/review.md (code only)not stated~1,9298.8x
Graphify 0.9.110, its five default questions, corpus counted by its own word counter1 of 54,9093.5x
Graphify 0.9.110, the five questions of its review2 of 53,4135.0x
Wingbox analyze, every module of the corpus29 of 2921878.1x
Wingbox impact <file> 2, every file of the corpus29 of 2966259.2x
Wingbox, the five questions of Graphify's review, one quick <keyword> per keyword2 of 542440.1x

The literal command graphify benchmark, run in the corpus folder after graphify extract . --code-only, estimates the corpus from the node count and prints 3.3x (results/graphify_benchmark_literal.txt).

Wingbox takes a module or a keyword, not a sentence, so its main row is every module of the corpus instead of five sentences. Largest single analyze answer: 617 tokens (27.5x). The whole Wingbox graph file is 1,834 tokens.

Graphify's published graph of the full corpus (worked/karpathy-repos/graph.json) holds 177 nodes and 246 links; its review states 285 and 340. Graphify 0.9.110 run on that file with the published 92,616 words and five questions gives 45.0x with 2 of 5 questions matched (published: 71.5x). Papers and images are outside Wingbox, so the full-corpus row has no Wingbox counterpart.

Same formula on FastAPI (1,223 Python files, 321,676 words, 428,901 tokens):

Tool and query setQueries answeredMean answer, tokensLargest answer, tokensReduction
Graphify 0.9.110, five default questions5 of 51,3691,634313.3x
Wingbox analyze, every module1,223 of 1,22362455,131687.5x
Wingbox impact <file> 2, every file1,223 of 1,22326127,6221645.1x
Wingbox, Graphify's five default questions by keyword5 of 51,078397.9x

Test B. Are the edges in the code?

The formula above does not check a single answer. This test does: every file-to-file dependency edge of each tool against the import statements of the code.

Reference: tools/py_edges.py reads every import with Python's own parser (ast) and resolves it to a file. Check of the reference itself: the import graph built by grimp 3.20 (the engine of import-linter) on FastAPI gives 1,801 file pairs; 1,801 of them are the reference's 1,801 pairs; 0 only in grimp, 0 only in the reference.

CorpusPairs in the codeToolEdgesWrongMissingPrecisionRecall
Karpathy repositories15Wingbox15001.0001.000
Karpathy repositories15Graphify, module-level imports15001.0001.000
FastAPI, whole tree1,801Wingbox1,818001.0001.000
FastAPI, whole tree1,801Graphify, module-level imports1,81222320.9880.982
FastAPI, fastapi/ package124Wingbox130001.0001.000
FastAPI, fastapi/ package124Graphify, module-level imports122071.0000.944

An edge to a package __init__.py that Python executes on the way to the named module counts as right and is not required. Graphify also links an import to the file that defines the imported name; with those links accepted its FastAPI figures are precision 0.994 and recall 0.991 (19 wrong, 16 missing).

Test C. The impact command without an agent

Question: every file that depends on file X directly or through one other file. Five files in four public repositories. wingbox impact <file> 2 and graphify affected "<file>" --depth 2 are run as they are and the files they name are compared with the reference list.

Reference lists (gold/), none taken from either tool: Q1 from an own TypeScript resolver (tools/ts_resolve.py); Q2 and Q3 from an own Rust resolver (tools/rust_resolve.py), every file of the two lists read against the source; Q4 and Q5 from tools/py_edges.py, equal to grimp. run.sh computes the five lists again and stops when one differs from the committed list.

QuestionReference filesWingbox: right / wrong / missingWingbox output, tokensGraphify: right / wrong / missingGraphify output, tokens
Q1 src/collectors/PageCollector.ts (chrome-devtools-mcp)3030 / 0 / 058530 / 0 / 01,324
Q2 cli/src/validation.rs (agent-browser)3030 / 0 / 044017 / 2 / 132,550
Q3 src/backend/elf/constants.rs (claudes-c-compiler)3131 / 0 / 049821 / 9 / 10625
Q4 fastapi/openapi/models.py (fastapi)2929 / 0 / 061429 / 366 / 032,188
Q5 fastapi/sse.py (fastapi)9999 / 0 / 01,99798 / 0 / 12,791

Exact list: Wingbox 5 of 5, Graphify 1 of 5. graphify affected works on functions and methods and follows calls and references as well as imports; restricted to its import relations (--relation imports_from --relation imports --relation re_exports --relation dynamic_import) it gives: Q1 28 / 0 / 2; Q2 5 / 2 / 25; Q3 21 / 9 / 10; Q4 28 / 3 / 1; Q5 98 / 0 / 1.

Test D. A coding agent answers the same five questions

Claude Code, headless, model alias sonnet (claude-sonnet-5-5), no user settings, skills or connectors. Three setups: built-in search only; search plus the wingbox command; search plus the graphify command. Graphs are built beforehand. The prompt names the tool's impact command in one sentence of the same shape for both tools (tools/agent_test.py). 3 repeats per question and setup, four trials at a time, setups interleaved. Score: F1 of the answer list against the reference list. Tokens: everything the model read and wrote, cache included. Seconds: wall clock of one trial while four run.

SetupTrialsF1 meanF1 lowestTokensTurnsSecondsUSD per trialRefused shell commands
Plain search (Read, Grep, Glob)150.9990.98459,3675.225.20.0751
Wingbox151.0001.00025,5982.511.20.0163
Graphify150.9970.975109,0717.834.90.09512
  • Wingbox against Graphify: tokens -77 %, time -68 %, cost -83 %.
  • Wingbox against plain search: tokens -57 %, time -56 %, cost -79 %.
  • Graphify against plain search: tokens +84 %, time +39 %, cost +27 %.
QuestionPlain: F1 / tokens / sWingbox: F1 / tokens / sGraphify: F1 / tokens / s
Q1 (chrome-devtools-mcp, 30 files)0.995 / 39,364 / 16.81.000 / 26,416 / 11.41.000 / 83,439 / 26.1
Q2 (agent-browser, 30 files)1.000 / 69,193 / 25.51.000 / 24,274 / 9.81.000 / 158,897 / 40.4
Q3 (claudes-c-compiler, 31 files)1.000 / 63,529 / 26.21.000 / 19,832 / 10.51.000 / 81,091 / 33.0
Q4 (fastapi, 29 files)1.000 / 57,892 / 26.41.000 / 19,181 / 8.11.000 / 65,159 / 25.8
Q5 (fastapi, 99 files)1.000 / 66,858 / 31.11.000 / 38,287 / 16.00.983 / 156,768 / 49.1

Every trial with its answer list is in results/agent_summary.json. The 45 trials cost 2.79 USD in total.

Test E. Building the graph

Seconds from a fresh checkout to a graph: every run starts from a new export of the repository, with no earlier output and no cache, and the two tools alternate. Median of 3 runs on a GitHub-hosted runner (Linux, AMD EPYC 9V45 96-Core Processor, 4 logical CPUs), workflow benchmark-build-time, script tools/time_builds.py. wingbox 0.2.0; graphify 0.9.110 with GRAPHIFY_BACKEND=none.

wingbox build . writes the graph, the report and the map. graphify extract . --code-only writes the graph; graphify cluster-only . --no-label --no-viz is the step this benchmark runs before graphify affected.

CorpusSource filesWingbox build, sGraphify extract, sGraphify cluster, sExtract / buildWingbox graphGraphify graph
karpathy290.230.800.303.5x29 modules, 15 import edges241 nodes, 363 links
chrome-devtools-mcp2960.5911.342.1819.2x293 modules, 918 import edges4,804 nodes, 11,240 links
agent-browser3631.2516.685.3113.3x359 modules, 751 import edges10,516 nodes, 27,103 links
claudes-c-compiler3511.277.565.246.0x351 modules, 1,462 import edges9,413 nodes, 28,384 links
fastapi1,2272.1717.856.378.2x1,227 modules, 1,818 import edges12,897 nodes, 26,044 links

The size of the graph is read from the file each build wrote, in every run; the summary stops when two runs of one tool differ. CPU seconds per step and every single run are in results/build_times.json.

Scope

  • Code only. Graphify also reads documents, papers and images with a model; that part is not tested here and Wingbox has no counterpart.
  • One kind of question in tests C and D: which files depend on this file. Five questions, three languages (TypeScript, Rust, Python), one agent, one model.
  • Tests A, B and C are deterministic and need no model. Test D depends on the model and varies from run to run.

Versions and commits

ItemVersion or commit
Wingbox0.2.0 (PyPI wingbox)
Graphify0.9.110 (PyPI graphifyy); repository read at commit 18cd78f
grimp3.20
nanoGPT3adf61e
minGPT37baab7
micrograd7bc720e
fastapi2a1bd23
chrome-devtools-mcp8ba2ac9
agent-browser5338477
claudes-c-compiler6f1b99a

Reproduce

git clone https://github.com/aileron-tech/wingbox && cd wingbox
benchmarks/graphify/run.sh ~/wingbox-bench    # tests A, B, C; needs git, curl and uv; no model, no key
python3 -I benchmarks/graphify/tools/agent_test.py ~/wingbox-bench run      # test D: paid Claude Code sessions
python3 -I benchmarks/graphify/tools/agent_test.py ~/wingbox-bench report
python3 -I benchmarks/graphify/tools/time_builds.py ~/wingbox-bench             # test E: build time on this machine

run.sh installs both tools from PyPI into its own environments, fetches the seven repositories at the commits above, builds both graphs, computes the reference lists and writes out/REPORT.md in the work directory. WINGBOX_SPEC=<path or requirement> runs the same tests on another Wingbox.

FolderContent
tools/every script of the run; the references use Python's ast, grimp and two own resolvers, never Wingbox or Graphify
gold/the five reference lists
results/the result files of the run this report is written from