Does graphify help coding agents on big codebases?

graphify turns a codebase into a graph of who calls what and who imports what. its promise is that an agent answers code questions faster, and with fewer tokens, when it has the graph.

in graphify issue #985, a user reported that claude code alone gave similar answers with fewer tokens. this benchmark tests that report.

the short answer: graph facts, offered without graphify's usual instructions, did not make strong agents clearly more accurate or cheaper. the test offered the facts in two ways: a hook prototype built for this test, and two graph tools. agents found the answer files with grep in the first few calls, then read the files to check them. the notes replaced none of that reading.

a revised hook now cuts tokens by about 13% against plain grep, at about the same accuracy. that result comes from a small run, so treat it as a direction.

for where graphify might help instead, a separate page lists six guesses with their evidence: graphify-benchmark-where-next.

the main test is small: 12 questions on 3 repositories, 360 trials. about 660 more trials, in earlier attempts and practice, led to it. the table in How big this is lists every run.

Words in this note

stating even the obvious just to be on the same page

word meaning
agent an AI model that answers by calling tools, such as claude code with sonnet 5.5, or codex with gpt-6.1-sol
call one tool use by the agent: one search, one file read, or one graph query
trial one agent answering one question once, from a fresh start
setup one way of equipping the agent, such as plain grep or the old hook; graphify-benchmark-method lists all five
plain grep the agent uses only its built-in search and file reading
hook a program that runs after each agent search and adds a note under the search results; this benchmark built it, and graphify has not released it
note the text the hook adds: where a searched name is defined, what calls it, and what imports it
graph tools two query tools built on graphify's code for this test, which the agent may call instead of searching; graphify's own server offers more tools
gold the correct answer list for a question, checked by hand against the source code
F1 a score from 0 to 1 for how closely the agent's answer list matches the gold list; how F1 is calculated
tokens all text the model reads and writes during a trial; this is what users pay for
spacing fixed an F1 score after removing the space in java parameter lists, (A, B) to (A,B); graphify-benchmark-new-hook explains why
old hook the first hook prototype, used in the main runs; it never shipped in graphify
new hook the revised hook described in graphify-benchmark-new-hook

How big this is

run size what it showed why the work moved on
1. questions about graphify's own code 11 questions, 1 small repository on caller questions, the old hook raised answer coverage from 0.77–0.83 to 1.00, with 9 to 15 trials per setup one small repository, and the questions were written for this test
2. swe-explore, a public bug-finding benchmark 12 bug reports, 6 repositories, 340 trials no accuracy gain; the old hook added about 7% cost; graph tools got 0 calls in 60 trials finding a bug's file is a different task from #985, and agents opened that file at about the second call
3. a 101-question set 14 repositories, 205 of 1,380 planned trials plain grep scored F1 0.96 to 1.00 too easy to show a difference, so the run was paused
4. a harder 27-question set 7 repositories, 72 trials sonnet 5.5 still scored near 1.0; haiku 4.5 ran out of calls in 15 of 24 tool trials too easy for sonnet, and its setup let a trial read files outside the repository
5. practice for the main test kubernetes, 6 questions, 47 trials the plain-grep score landed in range; the scoring rules got their final form practice never counts in the results
6. the main test 12 questions, 3 repositories, 2 agents, 360 trials the results in this note
7. follow-ups 21 trials the new hook (15 trials) and haiku 5.5 (6 trials) small; see graphify-benchmark-new-hook and graphify-benchmark-haiku

the main test is the only run with all four: large repositories, hand-checked gold, goals fixed before the runs, and a sandbox. it is still small. three repositories and one kind of question cannot support a claim about all codebases.

What happens

  1. a developer asks the agent a code question, for example: "which functions call exactly_one?"
  2. the agent searches the codebase with grep.
  3. with the hook installed, a note appears under the grep results.
  4. the agent opens files to confirm what grep and the note showed.
  5. the agent answers with a list of functions, files, or call chains.
  6. a script scores the list against gold, and counts the calls and tokens the agent used.

What the problem is

the main test ran 360 trials on three large open-source repositories: airflow (python), vs code (typescript) and spring (java). there were twelve questions, two agents, five setups each, and three repeats.

claude code, sonnet 5.5 codex, gpt-6.1-sol
F1 with plain grep 0.961 0.995
F1 with the old hook 0.982 0.997
tokens, old hook ÷ plain grep 1.17 (17% more) 0.95 (5% fewer)
trials where the agent used optional graph tools 1 of 36 4 of 36

the old hook did not meet the accuracy goal or the token goal on either agent. both goals were written down before the runs. with no graphify instructions, and a prompt that asked them to verify graph answers against the source, the agents almost never chose the graph tools. plain grep already scored above 0.95, so these questions left little room for any accuracy gain.

full tables: graphify-benchmark-results. how the test works: graphify-benchmark-method.

Why it happens

the agent reads files to check its answer, whatever the note says. so the note adds tokens and removes no calls.

an offline replay re-ran every search from the sonnet trials and rebuilt each note. three facts came out.

  1. grep showed gold files early: on average by call 0.3 to 2.7, depending on the question type.
  2. notes named 35% to 65% of the gold items on six questions, often in the first note.
  3. the agent made 22.9 calls per trial with the old hook, and 22.9 calls with plain grep.

the old hook also re-read a graph file of 227 MB to 576 MB on every search. that added about 36% to each trial's run time.

details: graphify-benchmark-why.

What to do

give the agent graph facts it can trust without opening the file. the new hook does this.

the new hook is not in a graphify release yet. it lives on a development branch, and these steps describe that branch.

  1. build a small index of the graph once, so each note takes about 0.04 seconds.
  2. under each caller in a note, print the line that makes the call and that line's code.
  3. leave out any line that grep already printed.
  4. install this hook by default, and drop the rule that tells agents to query graphify first.

Why this action

the new hook keeps accuracy and removes calls on questions the note can settle. a follow-up run tested sonnet 5.5 with the new hook on six of the twelve questions.

tokens per trial on the same six questions, sonnet 5.5 plain grep 291 thousand tokens, old hook 355 thousand, new hook 253 thousand. the new hook uses the fewest tokens. plain grep plain grep: 291k tokens per trial (6 questions, 3 repeats) 291k old hook old hook: 355k tokens per trial (6 questions, 3 repeats) 355k new hook new hook: 253k tokens per trial (6 questions, 1 or 2 repeats) 253k tokens per trial, sonnet 5.5, same six questions; bars start at zero
setup tokens per trial cost per trial F1 (spacing fixed)
plain grep 291k $0.29 0.976
old hook 355k $0.33 0.984
new hook 253k $0.28 0.979

on one question, the agent answered from the note in 1 to 3 calls. the old hook needed about 15 calls on that question.

the nearest other action is to tell agents to query the graph first. each query is an extra call, and the #985 report puts each query at about 2,000 tokens. the agent then still reads files to check. this benchmark did not test that action directly. with no graphify instructions, agents chose the optional graph tools in only 5 of 72 trials.

a more accurate graph also misses the result. the old notes already named many gold items, and the agent still checked them.

this run is small: 6 questions, 15 trials. read the limits in graphify-benchmark-new-hook before you quote the 13%.

What stays the same

Where to look

claim source
main results, both agents graphify-benchmark-results; benchmark/artifacts/harder-v2/runs/20261006-223148-f592be/report.json; benchmark/artifacts/harder-v2/codex-eval/phase-a-20261007-001535/FINAL_REPORT.md
goals written before the runs benchmark/design/harder-v2/SCORING.md
replay of every search benchmark/design/harder-v2/analysis/HOOK_DIAGNOSTIC.md
new hook code, as run benchmark/artifacts/implementations/3780e47041043f75c8f5db308e84a6311378bf547eb4b20486bb6904282a368f/graphify/grep_enrich.py
new hook run benchmark/design/hook-v2/DECISIONS.md; benchmark/design/hook-v2/analysis/compare.json
the user report graphify issue #985

The check

  1. the reader's case. sonnet 5.5, question E01 on airflow, new hook installed. the agent must answer with F1 1.0 in far fewer than 15 calls. it did: 3 calls and 1 call.
  2. the case that must still succeed. the same six questions must keep plain-grep accuracy. the new hook scored F1 0.979; plain grep scored 0.976, with the same spacing fix.
  3. the decoy. a wrong explanation says "the questions are too easy, so no tool can help."
    • that explanation passes the main runs: plain grep already scored above 0.95, and no setup showed a clear accuracy gain.
    • that explanation fails the new-hook run: on the same easy questions, the new hook cut tokens. an easy question still leaves calls to remove.

Done when

  1. the new hook runs on all 12 questions, with 3 repeats, next to fresh plain-grep trials.
  2. the token ratio, new hook ÷ plain grep, has an upper bound at 0.90 or below.
  3. the F1 difference has a lower bound at −0.02 or above.

Out of scope