Researchers developed a vision-language reward model using no human preference labels by training a linear head on a frozen Qwen3-VL backbone. The head scores original web captions against perturbed versions, with the optimal layer selected using minimal labeled data. This approach achieves performance comparable to or exceeding GPT-4o on three benchmarks while requiring only a fraction of the fine-tuning compute.
- Eliminates expensive human preference labeling by using caption perturbations as self-supervision signals.
- Layer selection is critical: one intermediate layer yields 5.8 points higher score than the output layer.
- Outperforms GPT-4o on VL-RewardBench with a 32B parameter model at roughly 0.1% of fine-tuning compute.
- A frozen backbone with a small linear head transfers effectively without re-tuning across benchmarks.