Abstract
Multi-robot systems must simultaneously optimize competing objectives while maintaining coordinated behavior. Existing multi-agent reinforcement learning approaches often rely on fixed or centralized coordination, which limits adaptability and violates distributed constraints. This work introduces the Coordination-Informed Multi-Objective Reinforcement Learning (CIMORL) framework, integrating a distributed weight prediction mechanism, a privileged expert training strategy, and theoretical guarantees for Pareto-optimal solutions. We present the base CIMORL method alongside two sampling-based variants, CIMORL-TS (Tree Search) and CIMORL-MPPI (MPPI), which leverage privileged global information during training to enable fully decentralized deployment. Experimental validation in cooperative and adversarial scenarios demonstrates a $21.2\%$ hypervolume improvement and superior policy stability compared to state-of-the-art baselines. Real-world experiments with Crazyflie drones further validate the framework's robustness in resource allocation and multi-attacker multi-defend scenarios under partial observability.
Framework Approach
The core innovation of CIMORL lies in its theoretical foundations regarding decentralized weight synchronization and distributed opinion dynamics, enabling teams of robots to discover the optimal Pareto front seamlessly.
Lemma 1 (Global Weight Synchronization):
When the agents can achieve any combination of the $K$ objectives, Pareto-optimal solutions for the team $\mathcal{V}_s$ can be obtained through the Tchebycheff problem if and only if the agents' weight vectors $w$ are synchronized across the team.
In practical robotic scenarios, individual agents may only contribute to a specific subset of the objectives (e.g., spatial constraints). Hence, the strict global synchronization requirement is relaxed:
Lemma 2 (Coalition Synchronization):
Assume the team $N_s$ is partitioned into a set of coalitions $\mathcal{C}$, where each coalition strictly influences a disjoint subset of the $K$ objectives. Pareto-optimal solutions for the global team can be achieved if and only if agents synchronize their weight vectors locally within their respective coalitions. Different coalitions do not need to share the same weights.
Weight Opinion Dynamics & Clustering
To satisfy these local synchronization conditions, CIMORL leverages an opinion dynamics model. It relies solely on local partial observations $z$ and neighbor communication. Using a feature embedding $\Xi = g(z)$, the distributed weight evolution is governed by:
Here, $p_w$ acts as the concentration parameter of a Dirichlet distribution for sampling multi-objective weights. These dynamics form stable agreements, fulfilling the required clustering criteria:
Theorem 1:
The distributed system defined above clusters into different stable equilibria $\bar{x}^c$ dictated by the communication graph topology and the observed environment, enabling optimal multi-objective coverage through the coordinated dynamic formation of coalitions.
As a result of Theorem 1, agents naturally form coalitions to align their objective preferences stably over time. The trajectories below illustrate how agents with different initial random states successfully cluster into distinct subgroups, demonstrating the practical realization of our opinion dynamics.
(a) Concentration parameters trajectories for 20 agents clustering into subgroups.
(b) Final Dirichlet distributions.
Given the stability of these weight dynamics, we extend the formal guarantees to the entire closed-loop control system:
Theorem 2 (Closed-loop Contraction):
Under the joint control policy $\pi_{\theta}$ and weight dynamics presented above, the closed-loop multi-agent system is contracting if the small-gain condition: $$ \lambda_{\max}(F_{22}) \cdot a > L_{G} \cdot L_{w} $$ is satisfied, where $L_{G}$ represents the coupling strength between the weight and observation dynamics.
Experiments
We evaluated CIMORL and its sampling-augmented variants (CIMORL-TS and CIMORL-MPPI) across complex multi-agent domains. Before detailing the training performance, we outline the two primary environments used in our evaluations:
(a) Multi-resource assignment: Agents deliver resources avoiding collisions and maintaining connectivity.
(b) Attacker-defender: Teams compete to infiltrate opponent areas while defending their own territory.
Multi-Resource Assignment
In the Multi-Resource Assignment scenario, our method consistently reaches significantly higher hypervolume and expected utility metrics, reflecting superior Pareto front coverage, sparsity, and diversity compared to state-of-the-art baselines like MOMAPPO and MO-MIX.
Hypervolume indicator and expected utility results across multiple training runs (Resource Assignment).
Multi-Team Attacker-Defender
In the adversarial Attacker-Defender environment, the sampling-based variants similarly outperform baselines, demonstrating robust policy generation under multi-agent conflict constraints.
Hypervolume indicator and expected utility results across multiple training runs (Attacker-Defender).