Intrinsic Confidence-Driven Optimization Boosts LLM Reasoning, Study Finds

Global: Intrinsic Confidence-Driven Optimization Boosts LLM Reasoning, Study Finds

Researchers at an unnamed institution released a revised version (v5) of their paper on arXiv in November 2025, introducing the Intrinsic Confidence-Driven Group Relative Preference Optimization (ICPO) method to improve reasoning in large language models (LLMs). The work aims to overcome persistent challenges in reinforcement learning with verifiable rewards (RLVR), such as coarse‑grained rewards, reward noise, and inefficient exploration, which have historically caused unstable training and entropy collapse.

Background on RLVR Limitations

RLVR frameworks rely on external reward signals to steer LLM outputs toward correct reasoning paths, yet many implementations suffer from imprecise feedback that fails to capture nuanced differences between candidate responses. This coarse granularity can amplify noise in the reward signal, leading models to overfit to suboptimal strategies and diminish overall performance.

The ICPO Approach

ICPO leverages the intrinsic probabilities that an LLM assigns to multiple generated responses as a proxy for self‑assessment. By comparing these probabilities under the same prompt, the method computes a preference advantage score for each candidate. This score is then combined with verifiable rewards, guiding the model toward higher‑quality answers while mitigating overconfidence and reward noise.

Preference Advantage Score Explained

The preference advantage score quantifies the relative likelihood of each response, effectively ranking them without external annotation. According to the authors, this internal metric “alleviates the issues of coarse‑grained rewards and reward noise,” and it “curbs overconfident errors” by elevating undervalued high‑quality outputs.

Experimental Validation

Comprehensive experiments were conducted across four general‑domain benchmarks and three mathematical benchmarks. The results indicate that ICPO consistently outperforms the previously established Group Relative Preference Optimization (GRPO) baseline, delivering steady improvements in reasoning accuracy across all test sets.

Implications and Future Directions

The study suggests that integrating intrinsic confidence measures with verifiable rewards can enhance LLM training stability and reasoning depth. The authors propose extending ICPO to other domains, such as code generation and multimodal reasoning, to further assess its generalizability.

This report is based on information from arXiv, licensed under Academic Preprint / Open Access. Based on the abstract of the research paper. Full text available via ArXiv.