Heterogeneous multiagent systems are characterized by diverse task distributions, which are prevalent in practical scenarios, such as distributed decision making and robotic collaboration. A significant challenge in these systems is the constraint of limited observations, where each agent has access only to partial information. Many studies facilitate information exchange by employing shared parameters among agents. However, this approach is generally more effective for homogeneous systems where agents have similar observation or action spaces. In heterogeneous systems, indiscriminate parameter sharing can significantly increase the exploration cost required for effective adaptation. To address this challenge, we propose a novel communication complementary graph model (CCGM) for enhancing collaboration in heterogeneous multiagent systems. Our approach builds upon the training framework of heterogeneous agent reinforcement learning (HARL) with trust region learning and nonparameter sharing. This model utilizes advantage function decomposition and sequential updates to promote policy convergence. Within this framework, we introduce a novel communication method inspired by signaling games, where agents acting as receivers, process messages from other agents alongside their own observations. CCGM aligns the messages with observations in a graph-based communication module, which establishes communication relationships and supplements observational information. Subsequently, agents generate self-interested information, which they then share with others as senders. We evaluate our algorithm across various environments, including multiagent particle environments (MPE) and multiagent MuJoCo (MAMuJoCo) robot experiments. The results demonstrate the effectiveness of CCGM in enhancing HARL-based algorithms.

中文翻译：

---

通过通信互补图增强异构多智能体系统中的协作


异构多智能体系统的特点是任务分布多样化，这在分布式决策和机器人协作等实际场景中普遍存在。这些系统中的一个重大挑战是有限观察的限制，其中每个代理只能访问部分信息。许多研究通过在代理之间采用共享参数来促进信息交换。然而，这种方法通常对于代理具有相似观察或动作空间的同质系统更有效。在异质系统中，不加区分的参数共享会显著增加有效适应所需的勘探成本。为了应对这一挑战，我们提出了一种新的通信互补图模型 （CCGM），用于增强异构多智能体系统中的协作。我们的方法建立在异构智能体强化学习 （HARL） 的训练框架之上，具有信任区域学习和非参数共享。该模型利用优势函数分解和顺序更新来促进策略收敛。在这个框架内，我们引入了一种受信令博弈启发的新型通信方法，其中代理充当接收者，处理来自其他代理的消息以及他们自己的观察结果。CCGM 将消息与基于图形的通信模块中的观测结果保持一致，该模块建立通信关系并补充观测信息。随后，代理生成自利信息，然后作为发件人与其他人共享这些信息。我们在各种环境中评估我们的算法，包括多智能体粒子环境 （MPE） 和多智能体 MuJoCo （MAMuJoCo） 机器人实验。 结果证明了 CCGM 在增强基于 HARL 的算法方面的有效性。