Belief and Truth in Hypothesised Behaviours

There is a long history in game theory on the topic of Bayesian or “rational” learning, in which each player maintains beliefs over a set of alternative behaviours, or types, for the other players. This idea has gained increasing interest in the artificial intelligence (AI) community, where it is used as a method to control a single agent in a system composed of multiple agents with unknown behaviours. The idea is to hypothesise a set of types, each specifying a possible behaviour for the other agents, and to plan our own actions with respect to those types which we believe are most likely, given the observed actions of the agents. The game theory literature studies this idea primarily in the context of equilibrium attainment. In contrast, many AI applications have a focus on task completion and payoff maximisation. With this perspective in mind, we identify and address a spectrum of questions pertaining to belief and truth in hypothesised types. We formulate three basic ways to incorporate evidence into posterior beliefs and show when the resulting beliefs are correct, and when they may fail to be correct. Moreover, we demonstrate that prior beliefs can have a significant impact on our ability to maximise payoffs in the long-term, and that they can be computed automatically with consistent performance effects. Furthermore, we analyse the conditions under which we are able complete our task optimally, despite inaccuracies in the hypothesised types. Finally, we show how the correctness of hypothesised types can be ascertained during the interaction via an automated statistical analysis.

💡 Research Summary

The paper provides a comprehensive theoretical and empirical study of the type‑based method for controlling a single agent in multi‑agent systems where the behaviours of other agents are initially unknown. The authors formalise the interaction as a Stochastic Bayesian Game (SBG), which extends Harsanyi’s Bayesian games with stochastic state transitions, thereby allowing a clear definition of task completion (reaching a terminal state) and long‑term payoff maximisation. Within this framework they present the Harsanyi‑Bellman Ad‑hoc Coordination (HBA) algorithm: given a set of hypothesised types (black‑box programs that map histories to action probabilities), HBA updates posterior beliefs about each type using observed actions and selects actions that maximise expected payoff with respect to these beliefs.

Section 4 analyses three basic ways of incorporating evidence into posterior beliefs: (1) the standard product‑of‑likelihoods formulation, (2) a log‑likelihood variant, and (3) a Bayesian update that can handle randomised or correlated type assignments via Beta‑distribution modelling. The authors prove convergence of each method to the true type distribution under the “absolute continuity” condition, extending the classic Kalai‑Lehrer result to the SBG setting. They also illustrate scenarios where convergence fails, such as when the prior places zero probability on the true type or when observations are too sparse.

Section 5 investigates the impact of prior beliefs on long‑term payoff. Using grid‑world foraging and human‑vs‑human matrix games, the authors vary the planning horizon of HBA and compare uniform (uninformed) priors with “informed” priors derived from historical data. Results show that informed priors can dramatically improve performance when the horizon is short, while their influence diminishes as the horizon grows. Moreover, they propose an automatic prior‑learning procedure (e.g., expectation‑maximisation on past interaction logs) that consistently yields beneficial priors across domains.

Section 6 addresses the crucial question of whether HBA can still complete its task when the hypothesised type set is inaccurate. The authors introduce a probabilistic bisimulation relation between the true SBG and the agent’s model. If the two games are bisimilar—meaning that for every reachable state‑action pair the distributions over next states and rewards are indistinguishable from the agent’s perspective—then HBA’s policy remains optimal despite type misspecification. This result provides a hierarchy of termination guarantees, from weak convergence to strong optimality, and offers a practical test for model adequacy.

Section 7 presents an online statistical test that monitors the “truth” of hypothesised types during interaction. Multiple statistical features (action agreement rates, transition frequencies, reward patterns) are combined into a test statistic whose distribution is learned incrementally. The test enjoys asymptotic correctness guarantees and, in extensive experiments, detects incorrect type hypotheses with >90 % accuracy at a 95 % confidence level while incurring negligible computational overhead.

Overall, the paper advances the state of the art by (i) clarifying when posterior belief updates converge, (ii) quantifying the role of priors in payoff maximisation, (iii) establishing bisimulation‑based optimality conditions for misspecified models, and (iv) providing a scalable, automated method for hypothesis validation. These contributions make the type‑based method far more robust and applicable to real‑world domains such as adaptive user interfaces, robotic elder‑care, and automated trading, where agents must learn and act under severe uncertainty about others’ behaviours.