# Guide: QA and judging

> Free pixel checks, binary judging, and choosing among variants.

## Judging

- **Never ask a VLM for a 1-10 score.** A 1-10 rubric scored defective panels
  10/10 across the board `[measured]`; VLMs rank well and score badly
  (arXiv 2604.25235) `[community]`. Use binary yes/no checks and pairwise
  comparison.
- **Derive checks from the spec** you wrote (count, text string, colour,
  placement, cutout edge). Tag blockers. Include trap questions whose correct
  answer is "no" `[inference]`.
- **Tell the judge that any named detail it cannot see is false**, and run a
  no-reference control so the false-positive rate is known `[measured]`.
  Qwen3-VL-30B verified a "torn" object that was intact and a "lit" screen
  that was dark; it is lenient on fine detail. Percentages on tiny features
  are noise. Look at the image.
- **A judge comparing to a reference must see the reference** `[inference]`.
- **What `vvgen_judge_image` returns** `[measured]`: `results` per check (`true`, `false`, or `null` when the
  model gave no valid answer), `failed` (the checks answered false), `status` `pass` | `fail` | `inconclusive`,
  and `pass`, computed by vvgen as "every check true", never taken from the model. The reply is held to a
  strict schema that requires every check as a boolean; an incomplete reply is asked again once, then comes
  back `inconclusive` (`pass: false`, `invalid` says which items). Before this, Qwen3-VL answered 2 of 4
  checks and said `pass: true`. It still contradicts itself (a defect saying "shadows under the knight" next
  to `true` for "without shadows"): the judge is advisory, `checks` are the gate, your eyes are the last word.
- Grade with a different model than the one that generated; majority-vote
  flaky questions `[community]`. Validate on 30-100 human labels before
  trusting a threshold `[community]`.
- Learned preference scores (PickScore, HPS, ImageReward) rank candidates;
  they do not gate, and are photoreal-trained `[community]`.

## Deterministic gates first (cheap, before any model)

- Pixel dimensions against spec `[measured: one in eleven differed]`.
- OCR on art that must be text-free `[inference]`.
- pHash/dHash duplicate check against previous renders and against the
  references (a render closer to the plate than to the brief is a paste).
- Alpha checks on cutouts: halo/fringe pixels along the edge.
- Face/hand counts (photo-trained detectors miss stylised art; calibrate).

vincevangen ships these as `checks`: `alpha_coverage:<min>-<max>`, `edge_halo:<max>`,
`color_distance:<hex>:<max>` (CIELAB deltaE vs the dominant colour), `size:<w>x<h>`,
`components:<n>` (connected opaque regions, specks ignored; `1` = one subject and no
ground tile), `frames:<n>` (figures a sprite sheet shows, separated by background, counted along each row:
counts go to this check, never to the judge), `seam:<max>` (how visible the wrap is when tiled: wrap-edge
step / interior step, 1.0 = seamless, ~2 = a visible seam; the `tileable` op brings a model texture under 1.1), `palette:<n>` (distinct opaque colours; pixel art after
`pixelate` has at most its `colors`; pair with `edge_halo:0` for alpha 0/255). `vvgen check` / `vvgen_check_image` run them for free; on `gen` /
`asset` with n>1 they gate before the judge and `keep: best-passing` picks the
winner (score = mean check margin, averaged with the judge's defect count).
Rejected files stay on disk; the result lists why. Keep the person as the final
approver: the judge only catches gross defects.

## Retry ladder (max 3 rounds)

1. Lint the spec before spending.
2. One production render on `default` (a clear brief needs no drafts).
   Options wanted or an open brief: four drafts on `fast`, pick by eye, finish
   the pick with `from_draft` (an edit, dearer than one final).
3. Route by defect: local → `edit`; structural (count, pose, layout) →
   revise the prompt, re-render once.
4. Best of 2; then escalate to a human with the defect list.

Published feedback loops stop at 3; none claims more helps `[community]`.
Fixing the brief up front beat evaluator retries 7.7% to under 1%
`[community: T2I-Copilot]`. Framing hands larger or out of shot beats
repairing them `[community: HandRefiner 70/100]`. If more than ~20% of a
batch escalates, the template is broken; fix it, not the items.

## Provenance

Every output carries model, prompt, seed, refs, params, cost and the
caller; vincevangen writes this to its manifest and the sidecar JSON. Keep the
human-authored spec in version control: purely prompt-generated images may
not be copyrightable; the spec and its edit history are the authorship
evidence `[community: legal reasoning]`.