a ratio under 1.0 means the hook used fewer tokens than plain grep. the goal was an upper range end at 0.90 or below.
comparison
token ratio
range
questions
old hook, sonnet 5.5
1.17
0.98 to 1.44
12, 3 repeats
old hook, gpt-6.1-sol
0.95
0.85 to 1.13
12, 3 repeats
new hook, sonnet 5.5
0.87
0.76 to 0.99
6, 1 or 2 repeats
the new hook row uses fewer questions and a different resampling. graphify-benchmark-new-hook explains the difference.
Claude Code with Sonnet 5.5 (180 trials)
setup
F1
tokens per trial
calls per trial
cost per trial
seconds per trial
plain grep
0.961
180k
20.0
$0.20
47
old hook
0.982
211k
20.1
$0.23
63
server grep
0.976
262k
27.0
$0.25
58
graph tools
0.962
274k
27.3
$0.25
60
no tools
0.009
4k
0
$0.02
6
comparison
F1 change (range)
token ratio (range)
accuracy goal
token goal
old hook vs plain grep
+0.021 (−0.011 to +0.070)
1.17 (0.98 to 1.44)
missed
missed
graph tools vs server grep
−0.015 (−0.099 to +0.017)
1.05 (0.82 to 1.30)
not tested
missed
the hook delivered a note in 33 of 36 old-hook trials. on the three E02 trials it ran and found nothing to add.
the agent called a graph tool in 1 of 36 graph-tools trials.
by repository, old hook vs plain grep: F1 change −0.002 (airflow), +0.027 (vs code), +0.036 (spring). token ratio 1.19, 1.26 and 1.05.
one trial scored F1 0 after a full answer. the agent wrote each chain with three steps where gold uses two. the score stands under the rules.
Codex with gpt-6.1-sol (180 trials)
setup
F1
tokens per trial
calls per trial
seconds per trial
plain grep
0.995
250k
8.2
102
old hook
0.997
237k
7.6
117
server grep
0.997
373k
51.1
148
graph tools
0.997
372k
53.2
116
no tools
0.000
7k
0
9
comparison
F1 change (range)
token ratio (range)
accuracy goal
token goal
old hook vs plain grep
+0.001 (−0.005 to +0.008)
0.95 (0.85 to 1.13)
missed
missed
graph tools vs server grep
0.000 (−0.008 to +0.005)
1.00 (0.86 to 1.11)
not tested
missed
the agent called a graph tool in 4 of 36 graph-tools trials, 5 calls in total.
codex runs several shell commands in one call, so its call counts are lower than claude's. compare calls within one agent only.
by repository, old hook vs plain grep: token ratio 0.88 (airflow), 0.92 (vs code), 1.06 (spring).
the included plan reports no dollar cost.
The no-tools check
no-tools trials scored F1 0.009 on sonnet and 0.000 on gpt-6.1-sol. one sonnet trial scored 0.32 from memory. so the models could not answer these questions without reading the code. that result does not prove the models never saw these repositories.
What these numbers do not show
the graph-tools rows say little about the graph itself. the agents almost never called the graph tools. the prompt asked them to verify graph answers against the source, and gave no graphify instructions; graphify-benchmark-method lists what the trials left out.
the two agents never mix in one average. each comparison stays inside one agent.
three repositories are too few for a general claim about all codebases.
