SpatialBench

Score your own model. Generate your own data.

SpatialBench is built for teams evaluating models that aren't public yet. Measure your model's spatial reasoning in a few lines of TypeScript, then generate as much exact-labeled training data as you need to improve it.

Evaluate

Find where your model breaks

Spatial reasoning is one of the few things frontier models still get wrong, and the scores spread wide. Point SpatialBench at your model and see exactly where it fails, broken down by skill, in a few lines. You pass a function that returns your model's answer; every item is scored against exact ground truth, no judge model. Works with any model you can call from code, including one that isn't public yet.

// @spatialbench/kit — publishing to npm soon; this is the API it ships with
// evaluate.ts
import spatialbench from "@spatialbench/kit";

// autocomplete on `spatialbench.` shows every benchmark + evaluate
const bench = await spatialbench.cubebench({ items: 200 });

const result = await spatialbench.evaluate(bench, async ({ image, question }) => {
  return myModel.ask(image, question);   // however you call your model
});

console.log(result.accuracy);   // 0.42
console.log(result.byType);     // { faceRead: 0.61, hiddenFace: 0.22, ... }
result.save("run.json");

The image arrives as { bytes, base64, mime }, so wiring any client is a one-liner. Scoring is exact string match against the engine, no judge model. If you'd rather not write the call, spatialbench.openaiCompatible({ baseURL, apiKey, model }) returns a ready-made function for any OpenAI-compatible server, including your own.

Train

Then generate the data to close the gap

The same engine that finds the gap can fill it. Generate as many training examples as you want, at any difficulty, each a rendered scene with a provably-correct answer, no human labeling. The answer is computed, not annotated, so the set is exact and can't be contaminated. Hold out whole skills, not just items, for a clean train/test split you own.

# FoldBench engine: fold paper, ask a spatial question, know the exact answer
from foldbench import generate

generate(
    n=10_000,          # as many as you like
    folds=3,             # difficulty dial
    seed=0,              # deterministic and reproducible
    out="train/",        # -> train/img_*.png + train/labels.jsonl
)
# labels.jsonl: {"image": "...", "question": "...", "answer": "r1c1, r1c2"}

Because the label is computed from the fold, not annotated, the set is exact and contamination-free by construction. Hold out task types, not just items, for a clean train/test split you control.