How the graphify benchmark works

16 October 2026

this page explains how the test behind graphify-benchmark works. the words "agent", "trial", "setup", "gold", "F1" and "tokens" mean what the word list in graphify-benchmark says.

The repositories

the test uses three large, popular repositories in three languages. each one is pinned to a fixed commit.

a fourth repository, kubernetes (go), held only the practice questions. practice trials tune the setup and never count in the results.

The questions

there are twelve evaluation questions, four per repository. each question is a real review task with an exact answer list.

question E01 reads, in part: "Before changing airflow.utils.helpers.exactly_one, I need a complete caller review list."

the twelve gold lists hold 239 items in total. each gold list was built from the source code with two separate methods, such as a language server and a parser. a separate review then checked every item against the source. gold never came from the graphify graph, so the test does not grade graphify against itself.

The five setups

two pairs carry the result. old hook against plain grep shows what the hook adds. graph tools against server grep shows what the graph tools add through the same channel.

The agents and the limits

• agents. claude code with sonnet 5.5, and codex with gpt-6.1-sol. both ran at medium effort.
• limits. each trial allowed 120 calls and 1,200 seconds. a claude trial also stopped at $2.
• isolation. each trial ran inside a sandbox. the agent could read only that question's repository copy, never the gold lists.
• order. the trial order came from a fixed random seed. each setup ran three times per question.
• counts. 12 questions × 5 setups × 3 repeats gives 180 trials per agent, and 360 in total.

What the trials left out

each trial ran without the parts of graphify that a real install adds.

• no graphify instructions. claude trials ran with CLAUDE.md, memory and skills switched off. codex trials ran with project instructions switched off.
• no query-first hook. graphify claude install adds a hook that tells the agent to query graphify before each search. every trial ran without it.
• no command line. claude trials had no shell, so they could not run graphify query. codex trials searched through a read-only shell.
• two small graph tools. the graph-tools setups offered graph_query and graph_explain. graphify's own server offers more tools, with its own descriptions.
• a prompt that asked for checking. both server setups told the agent: graph tools "are optional when offered and their claims should be verified against source". the graph_explain description added "verify important details in source". this wording may have lowered graph use.
• the hook trials got no such wording. their prompt said only: use the Read, Grep and Glob tools. the agents checked files on their own.
• a code-only graph. the graphs came from code structure alone. graphify can also add a layer written by a language model; these graphs had none.
• one graphify version. the trials froze graphify at 0.9.73, from 2026-10-01. later releases changed extraction in many places.

The scoring

1. the agent returns a JSON list of items, such as path/to/file.py::function_name.
2. a script compares the list with gold, item by item, after trimming spaces around separators.
3. the script reports F1. F1 is 1.0 only when the list matches gold exactly.
4. the script averages repeats per question, then questions per repository.
5. each repository counts equally in the final average.

the script never asks a model to judge an answer. a misspelled item counts as wrong.

How F1 is calculated

each item is one string, such as path/to/file.py::function_name. the script compares the agent's items with the gold items as two sets.

1. the script cleans each item. it trims outer spaces and the spaces around ::, -> and |. it removes a leading ./ and the sandbox folder prefix.
2. the script drops duplicate items.
3. the script counts three numbers.
   • matches: items in both the answer and gold.
   • extras: items in the answer and missing from gold.
   • misses: items in gold and missing from the answer.
4. precision = matches ÷ answer items. this is the share of the answer that is correct.
5. recall = matches ÷ gold items. this is the share of gold that the answer found.
6. F1 = 2 × matches ÷ (answer items + gold items). this balances precision and recall.

after the cleaning in the first step, an item must match its gold item character for character. a wrong file, name, parameter list or chain order makes a different item.

an answer that is not valid JSON, or that abstains, scores 0. the full rules are in benchmark/design/harder-v2/SCORING.md.

The goals fixed before the runs

these rules were written in benchmark/design/harder-v2/SCORING.md before any evaluation trial.

1. calibration. plain grep must score F1 between 0.30 and 0.95. otherwise the questions are too easy or too hard to show a difference.
2. accuracy goal. the lower end of the F1 gain range must be above +0.02.
3. token goal. the upper end of the token ratio range must be at 0.90 or below. the F1 change must also stay at −0.02 or better.

each range comes from 10,000 resamples of repositories, questions and repeats. three repositories are few, so every result is exploratory.

the calibration rule failed for both agents. plain grep scored 0.961 on sonnet and 0.995 on gpt-6.1-sol. so the questions could show a cost difference, but they had little room to show an accuracy gain.

The earlier attempts

the main test came after four earlier attempts and a practice round, about 660 trials in total. the table "How big this is" in graphify-benchmark lists each attempt and the reason the work moved on. the pattern repeated each time.

1. on small or familiar repositories, plain grep scored near the maximum, so no setup could show a gain.
2. on bug-finding tasks, agents opened the right file at about the second call.
3. after the first small test, the old hook never showed a clear accuracy gain.

so the main test uses much larger repositories, questions that need many files, and a sandbox that keeps gold out of reach.

Cost

the claude runs cost $36.80 in total. the codex runs used an included chatgpt plan, so they have no dollar figure. all runs stayed inside the plan limits.

Where to look

back to graphify-benchmark.