Multi-Agent Covering Option Discovery Based on Kronecker Product of Factor Graphs
KAUST DepartmentComputer Science, King Abdullah University Of Science And Technology, Thuwal, Saudi Arabia
Computer Science Program
Permanent link to this recordhttp://hdl.handle.net/10754/680130
MetadataShow full item record
AbstractCovering option discovery has been developed to improve the exploration of reinforcement learning in single-agent scenarios, where only sparse reward signals are available. It aims to connect the most distant states identified through the Fiedler vector of the state transition graph. However, the approach cannot be directly extended to multi-agent scenarios, since the joint state space grows exponentially with the number of agents thus prohibiting efficient option computation. Existing research adopting options in multi-agent scenarios still relies on single-agent algorithms and fails to directly discover joint options that can improve the connectivity of the joint state space. In this paper, we propose a new algorithm to directly compute multi-agent options with collaborative exploratory behaviors while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as the Kronecker product of individual agents' state spaces, based on which we can directly estimate the Fiedler vector of the joint state space using the Laplacian spectrum of individual agents' transition graphs. This decomposition enables us to efficiently construct multi-agent joint options by encouraging agents to connect the sub-goal joint states which are corresponding to the minimum or maximum of the estimated joint Fiedler vector. Evaluation on multi-agent collaborative tasks shows that our algorithm can successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options, in terms of both faster exploration and higher cumulative rewards.
CitationChen, J., Chen, J., Lan, T., & Aggarwal, V. (2022). Multi-Agent Covering Option Discovery Based on Kronecker Product of Factor Graphs. IEEE Transactions on Artificial Intelligence, 1–13. https://doi.org/10.1109/tai.2022.3195818
SponsorsThis work was supported in part by Meta Platforms, Inc., Cisco Systems, Inc., and the U.S. National Science Foundation under grant 2114415.