The new hook

16 October 2026

this page backs the "What to do" and "Why this action" sections of graphify-benchmark. the results here come from a small run. read the limits section before you quote a number.

What changed

the new hook keeps the same job: it adds a note under each search result. six things changed.

an index. the hook builds a small database of the graph once. each note then takes about 0.04 seconds instead of 1.3 to 4.5 seconds.
the calling line. under each caller, the note prints the line that makes the call and that line's code.
no repeats. the note leaves out any calling line that grep already printed.
paths. a search for a module path, such as airflow/utils/net, now gets a note about that module.
common names. a name defined in many places gets one line that lists where it is defined.
complete lists. each list states its full count, so the agent knows when it holds every caller.

the default install for claude code also changed. it now installs this hook. the rule that tells agents to query graphify first now needs the --strict flag.

Replay before the run

an offline replay re-ran the 366 logged searches from the main sonnet run through both hooks. this used no model.

	old hook	new hook
searches that got a note	204	253
gold items named across all notes	203 of 717	238 of 717
median wait per search (airflow, spring, vs code)	1.3, 2.2, 4.5 seconds	0.04 seconds each

The run

agent. claude code with sonnet 5.5, at medium effort, with the same limits as the main runs.
questions. six of the twelve: E01 and E04 (airflow), E06 and E08 (vs code), E10 and E12 (spring).
why these six. per repository, the run took the callers question and the other question that needed the most calls. this choice was fixed before the run.
trials. 12 new-hook trials, two per question. 6 plain-grep trials, one per question, to check whether the model had changed since the main run.
what ran. 15 of 18 trials. the run stopped at a weekly plan limit. the missing trials are the second new-hook repeat of E06 and E08, and the plain-grep E08 trial.
comparison. each new-hook result sits next to the main run's old-hook and plain-grep results on the same six questions.

Results

setup	F1	F1, spacing fixed	tokens per trial	calls per trial	cost per trial	seconds
new hook (this run)	0.861	0.979	253k	25.7	$0.28	70
old hook (main run)	0.984	0.984	355k	28.7	$0.33	88
plain grep (main run)	0.950	0.976	291k	28.1	$0.29	63
plain grep (this run)	0.915	0.915	459k	32.0	$0.41	88

comparison	token ratio (range)	cost ratio	F1 change, spacing fixed
new hook vs old hook	0.71 (0.61 to 0.83)	0.84	−0.005
new hook vs plain grep, main run	0.87 (0.76 to 0.99)	0.96	+0.002

the ranges resample questions within repositories, 10,000 times. with six questions, these ranges are rough.

Per question

question	new hook	old hook	plain grep
E01	1.00	1.00	0.98
E04	1.00	1.00	0.99
E06	0.95	0.99	0.95
E08	1.00	0.97	0.98
E10	0.69	0.94	0.80
E12	0.53	1.00	1.00

these are recorded scores. the new hook lost points on E10 and E12 to the spacing slip.

tokens per trial

question	new hook	old hook	plain grep
E01	28k	82k	66k
E04	53k	56k	51k
E06	401k	750k	557k
E08	499k	633k	491k
E10	215k	244k	234k
E12	322k	363k	345k

calls per trial

question	new hook	old hook	plain grep
E01	2	15	15
E04	4	5	5
E06	44	50	47
E08	42	43	41
E10	41	39	37
E12	20	21	23

the new-hook numbers for E06 and E08 come from one trial each. the old-hook and plain-grep numbers average three trials.

on E01, the two new-hook trials took 3 calls and 1 call. the agent answered from the note.

The spacing slip

two new-hook answers lost most of their score to one formatting slip. the agent wrote java parameter lists as (A, B), and gold uses (A,B).

E12, second repeat: F1 0.06 as recorded, 1.00 with the space removed.
E10, first repeat: F1 0.47 as recorded, 0.95 with the space removed.

the scoring rules count these items as wrong, and the recorded scores stand. the "spacing fixed" column re-scores every trial with the space removed. that column applies the same fix to the old runs.

the slip came from the model. graph labels carry no parameter lists, so the note did not show that format. the main run's plain-grep trials show the same slip: their spacing-fixed F1 is higher than their recorded F1.

Limits

size. six questions, one or two repeats, and 15 trials. this run shows a direction.
the token goal. the range for new hook vs plain grep ends at 0.99, above the 0.90 goal. the main-run goal is not met.
model drift. plain grep in this run used 1.52 times the tokens of plain grep in the main run, on five questions. this may be noise, since each question ran once. it may also mean the model behaves differently now. against this run's plain grep, the new hook ratio is 0.52. the main-run comparison is the one quoted, because it is the more cautious one.
reads. the new hook did not reduce file reads. the agent still opened about 14.6 files per trial. the savings came from fewer searches, and from questions that a note settled outright.
the install. these trials set up the hook directly. the new default install and the new CLAUDE.md text are untested.

Where to look

claim	source
the new hook, as run	`benchmark/artifacts/implementations/3780e47041043f75c8f5db308e84a6311378bf547eb4b20486bb6904282a368f/graphify/grep_enrich.py`; tests on the `grep-enrich-hook` branch, not public yet
the install change	the `grep-enrich-hook` branch, not public yet
the replay	`benchmark/design/hook-v2/analysis/replay_hook.py`; `benchmark/design/hook-v2/analysis/replay_hook.json`
the plan, written before the run	`benchmark/design/hook-v2/PROTOCOL.md`
the comparison	`benchmark/design/hook-v2/analysis/compare.py`; `benchmark/design/hook-v2/analysis/compare.json`
the trials	`benchmark/artifacts/hook-v2/runs/20261009-203025-82247a` (new hook), `benchmark/artifacts/hook-v2/runs/20261009-203025-2dceea` (plain grep)
the decision log	`benchmark/design/hook-v2/DECISIONS.md`

back to graphify-benchmark.