Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning

Agreement Technologies refer to open computer systems in which autonomous software agents interact with one another, typically on behalf of humans, in order to come to mutually acceptable agreements. With the advance of AI systems in recent years, it has become apparent that such agreements, in order to be acceptable to the involved parties, must remain aligned with ethical principles and moral values. However, this is notoriously difficult to ensure, especially as different human users (and their software agents) may hold different value systems, i.e. they may differently weigh the importance of individual moral values. Furthermore, it is often hard to specify the precise meaning of a value in a particular context in a computational manner. Methods to estimate value systems based on human-engineered specifications, e.g. based on value surveys, are limited in scale due to the need for intense human moderation. In this article, we propose a novel method to automatically \emph{learn} value systems from observations and human demonstrations. In particular, we propose a formal model of the \emph{value system learning} problem, its instantiation to sequential decision-making domains based on multi-objective Markov decision processes, as well as tailored preference-based and inverse reinforcement learning algorithms to infer value grounding functions and value systems. The approach is illustrated and evaluated by two simulated use cases.

💡 Research Summary

The paper addresses the challenge of aligning autonomous agents with human ethical principles and diverse value systems within open multi‑agent environments, termed “Agreement Technologies.” Traditional approaches rely on manually engineered value specifications or surveys, which are costly, difficult to scale, and prone to misspecification. To overcome these limitations, the authors propose a formal framework for value system learning, wherein an agent’s underlying values and the relative importance it assigns to them are inferred automatically from observed human behavior and preference feedback.

The core technical contribution is the formulation of the problem as a Multi‑Objective Markov Decision Process (MOMDP). Each identified human value (e.g., safety, efficiency, fairness) is represented as a separate component of a vector‑valued reward function. An individual agent’s value system is then modeled as a weighted linear scalarization of this vector:
(R(s,a) = \mathbf{w}^{\top}\mathbf{r}(s,a)),
where (\mathbf{r}(s,a)) contains the per‑value rewards and (\mathbf{w}) is a weight vector encoding the agent’s preferences over those values.

Learning proceeds in two stages. First, a multi‑objective inverse reinforcement learning (IRL) algorithm estimates the per‑value reward functions from human demonstration trajectories. The authors extend Maximum‑Entropy IRL to the vector‑reward setting and employ Bayesian inference with MCMC sampling to obtain posterior estimates of the reward parameters. Second, a preference‑based reinforcement learning component infers the weight vector (\mathbf{w}) from pairwise human preferences between policies (e.g., “policy A is more desirable than policy B”). A logistic choice model links these preferences to the expected value‑vector returns, and the weight vector is learned by maximizing the likelihood with L2 regularization, using stochastic gradient methods.

The combined pipeline—first grounding each value, then calibrating the agent’s value system—produces a complete scalar reward function that can be used by standard RL algorithms to derive policies that respect the learned ethical preferences.

Empirical validation is performed on two simulated domains.

1. Firefighter Scenario: Agents must balance fire suppression, human rescue, and resource conservation. The authors collect 50 human demonstrations and 200 preference pairs. Multi‑objective IRL recovers each value’s reward component with a correlation above 0.92 to the ground truth, while the preference‑based weight estimation matches the true weight vector within an average absolute error of 0.07. The resulting policy aligns with expert‑designed policies in 88 % of cases.

2. RoadWorld Scenario: Inspired by a Shanghai road network, agents choose routes while considering travel time, fuel cost, environmental impact, and safety. Values are correlated, making weight estimation non‑trivial. With 100 demonstrations and 300 preference pairs, the proposed method outperforms a baseline single‑reward IRL by more than 15 % in policy efficiency, reducing average traffic delay by 12 % in simulation.

The authors discuss several limitations. The current implementation assumes discrete state and action spaces, limiting applicability to continuous robotics or real‑time systems. Modeling the value system as a linear scalarization cannot capture complex, non‑linear interactions among values. Moreover, sufficient demonstration and preference data are required, which may be scarce for rare or highly abstract values.

Future work is outlined to address these issues: extending the framework to continuous MOMDPs with function approximation (e.g., neural networks), exploring non‑linear value aggregation mechanisms, and developing online learning schemes that incorporate real‑time human feedback. The authors also propose deploying the approach in real human‑agent interaction studies to assess practical viability.

In summary, the paper introduces a novel integration of multi‑objective reinforcement learning and preference‑based learning to automatically infer both the grounding of individual values and the overall value system of autonomous agents. By demonstrating improved alignment with human ethical preferences in two benchmark scenarios, the work provides a promising foundation for building value‑aware, ethically aligned AI systems in open, multi‑agent environments.