GEB Benchmark

On the very same scenes, at the very same 4-way odds, identifying the structure beats carrying it across voices by 21–44 points.

Our adversarial split asks a model to pick a scene's motif from its three nearest confusables (4-way). A separate task asks it to match the same scene to the one story (of four) that shares its structure — same odds, same images. Every model does far better at the first: GPT-5.5 86% → 64%, GPT-5.2 80% → 36% (McNemar p<10^-8). The bottleneck is not fine discrimination; it is cross-voice transport.

Growing a model 7B → 14B lifts recognition by +27 points and moves mapping by exactly zero.

In the Qwen2.5 family, story → motif identification climbs 17% → 44% (p = 2×10^-9), while story → theorem matching stays pinned at 35% → 35% (14 items flip each way; p = 1.0). A bigger model reads structure better and transports it no better at all.

When a model is wrong, it is wrong toward the formally-nearest motif 2.2× more often than chance.

Across 1,065 identification errors, 44% land on a designed nearest-neighbor motif (chance 20%; permutation p < 10^-4). An independently derived graph-edit distance — read straight off the released skeleton programs, never used to build the benchmark — reproduces the same concentration. Three perceptual embeddings (DINOv2, CLIP, SigLIP) do not explain it: errors track the designed formal geometry more than any measured perceptual one.

Structure is easiest stated as a theorem, hardest carried across voices — a clean ordering once you correct for chance.

Because the tasks differ in difficulty floor (25-way vs 4-way), we score every cell as κ = (acc − chance)/(1 − chance). Pooled over the three frontier models, κ falls monotonically: theorem 0.85 > story 0.81 > skeleton 0.77 > scene 0.68 > cross-voice 0.31. Cross-voice is the lowest for every vision model in >99.9% of bootstrap resamples.

When two frontier models miss the same item, they pick the same wrong motif 75% of the time.

Two systems from different vendors, with different training data, converge on the same conceptual confusions on the same stimuli. The hard boundaries are properties of the structures, not quirks of one model — which is what a benchmark should measure.

The same stories in English and Chinese score identically (77.3% each), and when models err in both languages they err toward the same motif far above chance.

What the benchmark probes is structural, not lexical: the shape, not the words that carry it. This is why the dataset is English-primary with a Chinese mirror rather than tied to either language.