Deep reinforcement learning (DRL) provides a promising way for intelligent agents (e.g., autonomous vehicles) to learn to navigate complex scenarios. However, DRL with neural networks as function approximators is typically considered a black box with little explainability and often suffers from suboptimal performance, especially for autonomous navigation in highly interactive multiagent environments. To address these issues, we propose three auxiliary tasks with spatio-temporal relational reasoning and integrate them into the standard DRL framework, which improves the decision making performance and provides explainable intermediate indicators. We propose to explicitly infer the internal states (i.e., traits and intentions) of surrounding agents (e.g., human drivers) as well as to predict their future trajectories in the situations with and without the ego agent through counterfactual reasoning. These auxiliary tasks provide additional supervision signals to infer the behavior patterns of other interactive agents. Multiple variants of framework integration strategies are compared. We also employ a spatio-temporal graph neural network to encode relations between dynamic entities, which enhances both internal state inference and decision making of the ego agent. Moreover, we propose an interactivity estimation mechanism based on the difference between predicted trajectories in these two situations, which indicates the degree of influence of the ego agent on other agents. To validate the proposed method, we design an intersection driving simulator based on the Intelligent Intersection Driver Model that simulates vehicles and pedestrians. Our approach achieves robust and state-of-the-art performance in terms of standard evaluation metrics and provides explainable intermediate indicators (i.e., internal states, and interactivity scores) for decision making.

中文翻译：

---

具有内部状态推断和交互性估计的交互式自主导航


深度强化学习（DRL）为智能代理（例如自动驾驶车辆）学习驾驭复杂场景提供了一种有前景的方法。然而，以神经网络作为函数逼近器的 DRL 通常被认为是一个几乎没有可解释性的黑匣子，并且通常性能不佳，特别是对于高度交互的多智能体环境中的自主导航。为了解决这些问题，我们提出了三个具有时空关系推理的辅助任务，并将它们集成到标准的 DRL 框架中，从而提高了决策性能并提供了可解释的中间指标。我们建议明确推断周围主体（例如人类驾驶员）的内部状态（即特征和意图），并通过反事实推理来预测有或没有自我主体的情况下他们的未来轨迹。这些辅助任务提供额外的监督信号来推断其他交互代理的行为模式。比较了框架集成策略的多种变体。我们还采用时空图神经网络来编码动态实体之间的关系，这增强了自我代理的内部状态推断和决策。此外，我们提出了一种基于这两种情况下预测轨迹之间差异的交互性估计机制，该机制表明了自我代理对其他代理的影响程度。为了验证所提出的方法，我们设计了一个基于智能交叉口驾驶员模型的交叉口驾驶模拟器，用于模拟车辆和行人。 我们的方法在标准评估指标方面实现了稳健且最先进的性能，并为决策提供了可解释的中间指标（即内部状态和交互性分数）。