Why the notes saved no work

16 October 2026

this page backs the "Why it happens" section of graphify-benchmark. the data is the 36 old-hook trials and 36 plain-grep trials of sonnet 5.5.

How the check worked

the trial logs kept only the length of each note, not its text. but a note depends only on the search text and the graph. so a replay re-ran every logged search against the same graph and rebuilt each note.

all 366 rebuilt notes matched their logged lengths exactly. five more searches had failed, and the hook does not run on a failed search. the replay then followed each trial call by call.

this check used no model and printed no gold items.

What the replay showed

1. Grep finds the answer files early

with plain grep, 99% of gold files appeared in some search or read result. each gold file that a note named had already appeared in grep output, in the same call or earlier.

so finding the files took little effort. checking the files took the effort: about 11 file reads per trial.

2. The notes held answers, and the agent checked anyway

on five of these questions, the very first note already named gold items. the agent still made 8 to 42 more calls.

these are plain means over trials, so they differ from the repository-weighted means in graphify-benchmark-results.

3. The agent never followed new files from a note

the notes named 160 files that the agent had not seen yet. the agent read none of those files, searched none of them, and cited none of them.

4. The agent searched again for names a note had covered

104 of 371 searches looked only for names that an earlier note had already described.

5. Many searches got no note

that is 167 of 371 searches, or 45%.

6. The notes explain only part of the extra tokens

the notes cost about 5,600 tokens per trial, including re-reading them on later turns. old-hook trials used about 36,000 more tokens than plain-grep trials. most of the gap came from a few long vs code trials, which vary about threefold between repeats.

7. The hook slowed every search

the old hook re-read the whole graph file on every search. the median wait per search was 1.2 seconds (airflow), 2.1 seconds (spring) and 4.2 seconds (vs code). the longest wait was 18 seconds. trials took 68 seconds with the hook and 50 seconds without it.

What this means

the notes added information, and the agent did not use that information to skip work. a note saves calls only when the agent can trust the note without opening the file. the old note listed where each caller is defined. it did not show the line that makes the call. so the agent opened the file to check.

the new hook adds that line and its code to the note.

What this page does not show

• why the model decides to check. the logs show only what it did.
• whether a different model trusts notes more. this replay covers sonnet 5.5 only.

Where to look

back to graphify-benchmark.