姚兴虎　 宋光鑫

摘 要：针对协作多智能体强化学习中的全局信用分配机制很难捕捉智能体之间的复杂协作关系及无法有效地处理非马尔可夫奖励信号的问题，提出了一种增强的协作多智能体强化学习中的全局信用分配机制。首先，设计了一种新的基于奖励高速路连接的全局信用分配结构，使得智能体在决策时能够考虑其所分得的局部奖励信号与团队的全局奖励信号;其次，通过融合多步奖励信号提出了一种能够适应非马尔可夫奖励的值函数估计方法。在星际争霸微操作实验平台上的多个复杂场景下的实验结果表明：所提方法不仅能够取得先进的性能，同时还能大大提高样本的利用率。

关键词：深度学习;强化学习;多智能体系统

中图分类号：TP181      文献标识码：A

Enhancing Global Credit Assignment Mechanism for Cooperative

Multi-Agent Reinforcement Learning

YAO Xing-hu1，SONG Guang-xin2

（1.College of Computer Science and Technology， Nanjing University

of Aeronautics and Astronautics， Nanjing， Jiangsu 211106， China;

2.College of Science， Nanjing University of Aeronautics and Astronautics， Nanjing， Jiangsu 211106， China）

Abstract：In order to solve the problem that the global credit assignment mechanism in cooperative multi-agent reinforcement learning is difficult to capture the complex cooperative relationship among agents and cannot effectively deal with non-Markov reward signals， an enhanced global credit assignment mechanism in cooperative multi-agent reinforcement learning is proposed. Firstly， a new global credit assignment structure based on reward highway connection is designed， which enables each agent to consider the local reward signal and the team's global reward signal when making decisions. Secondly， by integrating multi-step rewards， a new value function estimation method which can adapt to non-Markov rewards is proposed. The experimental results of several complex scenarios on the StarCraft multi-agent challenges show that the proposed method can not only achieve state-of-the-art performance， but also greatly improve the sample efficiency.

Key words：deep learning; reinforcement learning; multi-agent systems

現实世界中的很多问题都可以建模为多智能体强化学习问题。比如：多玩家电子游戏[1]，多无人车辆控制[2]，以及网络路由包的传输[3]。然而，完全合作的多智能体强化学习问题面临着两个主要的挑战：首先，智能体的联合动作空间随着智能体数量的增加，这使得直接利用单智能体算法进行联合建模会带来维数灾难;其次，当环境给出的奖励信号是针对所有智能体的全局奖励信号时，如何将这一全局奖励信号进行合理分配也是一个需要解决的问题。

对于上述问题，目前的主流方法是采用“中心训练-分散执行”的框架[4][5]。这类方法的核心是如何建立中心化值函数与非中心化值函数之间结构约束关系。……