New Framework SoliReward Improves Video Generation Reward Modeling

Global: New Framework SoliReward Improves Video Generation Reward Modeling

A team of AI researchers has introduced a systematic framework called SoliReward to enhance post‑training alignment of video generation models with human preferences, according to a preprint posted on arXiv on December 12, 2025. The framework seeks to overcome labeling noise, underexplored model architectures, and reward‑hacking vulnerabilities that have limited existing reward‑model (RM) approaches.

Addressing Data Collection Challenges

SoliReward sources high‑quality, cost‑efficient data through single‑item binary annotations rather than the traditional in‑prompt pairwise labeling. After collection, preference pairs are constructed using a cross‑prompt pairing strategy, which the authors argue reduces annotation inconsistencies and improves the signal‑to‑noise ratio of the training set.

Innovative Model Architecture

The framework incorporates a Hierarchical Progressive Query Attention mechanism designed to enhance feature aggregation across video frames. This architecture, built on vision‑language models (VLMs), aims to capture both spatial and temporal cues more effectively than prior RM designs.

Refined Loss Function

To better accommodate win‑tie scenarios, the authors introduce a modified binary‑tournament (BT) loss that explicitly models ties. This adjustment regularizes the RM’s score distribution for positive samples, providing more nuanced preference signals and mitigating over‑focus on a small subset of top‑scoring outputs.

Empirical Validation

Benchmarks assessing physical plausibility, subject deformity, and semantic alignment demonstrate measurable gains in direct RM evaluation metrics. Moreover, downstream experiments show that video generation models fine‑tuned with SoliReward‑trained RMs produce outputs that align more closely with human judgments.

Broader Impact

By delivering a more reliable reward‑model pipeline, SoliReward could reduce the incidence of reward hacking and improve the overall safety and usability of AI‑generated video content, according to the study’s authors.

Next Steps

The research team plans to release code and benchmark suites publicly, inviting further scrutiny and extension by the broader AI community.

This report is based on information from arXiv, licensed under Academic Preprint / Open Access. Based on the abstract of the research paper. Full text available via ArXiv.