vincevangen
Sign in
Docs menu

QA and judging

Free pixel checks, binary judging, and choosing among variants.

This is the same playbook the agent skill carries. Names like fast and default are aliases; see Models for what they run on the hosted service; model names in the research notes below may include ones only the local tool offers. Prices are in credits.

Judging

  • Never ask a VLM for a 1-10 score. A 1-10 rubric scored defective panels 10/10 across the board [measured]; VLMs rank well and score badly (arXiv 2604.25235) [community]. Use binary yes/no checks and pairwise comparison.
  • Derive checks from the spec you wrote (count, text string, colour, placement, cutout edge). Tag blockers. Include trap questions whose correct answer is "no" [inference].
  • Tell the judge that any named detail it cannot see is false, and run a no-reference control so the false-positive rate is known [measured]. Qwen3-VL-30B verified a "torn" object that was intact and a "lit" screen that was dark; it is lenient on fine detail. Percentages on tiny features are noise. Look at the image.
  • A judge comparing to a reference must see the reference [inference].
  • What vvgen_judge_image returns [measured]: results per check (true, false, or null when the model gave no valid answer), failed (the checks answered false), status pass | fail | inconclusive, and pass, computed by vvgen as "every check true", never taken from the model. The reply is held to a strict schema that requires every check as a boolean; an incomplete reply is asked again once, then comes back inconclusive (pass: false, invalid says which items). Before this, Qwen3-VL answered 2 of 4 checks and said pass: true. It still contradicts itself (a defect saying "shadows under the knight" next to true for "without shadows"): the judge is advisory, checks are the gate, your eyes are the last word.
  • Grade with a different model than the one that generated; majority-vote flaky questions [community]. Validate on 30-100 human labels before trusting a threshold [community].
  • Learned preference scores (PickScore, HPS, ImageReward) rank candidates; they do not gate, and are photoreal-trained [community].

Deterministic gates first (cheap, before any model)

  • Pixel dimensions against spec [measured: one in eleven differed].
  • OCR on art that must be text-free [inference].
  • pHash/dHash duplicate check against previous renders and against the references (a render closer to the plate than to the brief is a paste).
  • Alpha checks on cutouts: halo/fringe pixels along the edge.
  • Face/hand counts (photo-trained detectors miss stylised art; calibrate).

vincevangen ships these as checks: alpha_coverage:<min>-<max>, edge_halo:<max>, color_distance:<hex>:<max> (CIELAB deltaE vs the dominant colour), size:<w>x<h>, components:<n> (connected opaque regions, specks ignored; 1 = one subject and no ground tile), frames:<n> (figures a sprite sheet shows, separated by background, counted along each row: counts go to this check, never to the judge), seam:<max> (how visible the wrap is when tiled: wrap-edge step / interior step, 1.0 = seamless, ~2 = a visible seam; the tileable op brings a model texture under 1.1), palette:<n> (distinct opaque colours; pixel art after pixelate has at most its colors; pair with edge_halo:0 for alpha 0/255). vvgen check / vvgen_check_image run them for free; on gen / asset with n>1 they gate before the judge and keep: best-passing picks the winner (score = mean check margin, averaged with the judge's defect count). Rejected files stay on disk; the result lists why. Keep the person as the final approver: the judge only catches gross defects.

Retry ladder (max 3 rounds)

  1. Lint the spec before spending.
  2. One production render on default (a clear brief needs no drafts). Options wanted or an open brief: four drafts on fast, pick by eye, finish the pick with from_draft (an edit, dearer than one final).
  3. Route by defect: local → edit; structural (count, pose, layout) → revise the prompt, re-render once.
  4. Best of 2; then escalate to a human with the defect list.

Published feedback loops stop at 3; none claims more helps [community]. Fixing the brief up front beat evaluator retries 7.7% to under 1% [community: T2I-Copilot]. Framing hands larger or out of shot beats repairing them [community: HandRefiner 70/100]. If more than ~20% of a batch escalates, the template is broken; fix it, not the items.

Provenance

Every output carries model, prompt, seed, refs, params, cost and the caller; vincevangen writes this to its manifest and the sidecar JSON. Keep the human-authored spec in version control: purely prompt-generated images may not be copyrightable; the spec and its edit history are the authorship evidence [community: legal reasoning].