As Large Language Models (LLMs) become deeply integrated into human life and increas- ingly influence decision-making, it’s crucial to evaluate whether and to what extent they exhibit subjective preferences, opinions, and beliefs. These tendencies may stem from biases within the models, which may shape their behavior, influence the advice and rec- ommendations they offer to users, and poten- tially reinforce certain viewpoints. This paper presents the Preference, Opinion, and Belief survey (POBs), a benchmark developed to as- sess LLMs’ subjective inclinations across so- cietal, cultural, ethical, and personal domains. We applied our benchmark to evaluate leading open- and closed-source LLMs, measuring de- sired properties such as reliability, neutrality, and consistency. In addition, we investigated the effect of increasing the test-time compute, through reasoning and self-reflection mecha- nisms, on those metrics. While effective in other tasks, our results show that these mech- anisms offer only limited gains in our domain. Furthermore, we reveal that newer model ver- sions are becoming less consistent and more biased toward specific viewpoints, highlighting a blind spot and a concerning trend.
A strong inverse correlation between consistency and neutrality across prompting setups.
Numerical shifts in TCI/NNI from direct to reasoning to reflection.
TCI/NNI across 16 controversial topics.
Topics are not independent. Heatmap shows how ideological positions cluster.
@misc{Kour2025pobs,
title={Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Models},
author={George Kour and Itay Nakash and Ateret Anaby Tavor and Michal Shmueli-Scheuer},
year={2025},
bibtex_show={true},
eprint={2505.19621},
selected={true},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2505.19621},
}