KolmogorovSmirnov Test-Based Actively-Adaptive Thompson Sampling for Non-Stationary Bandits
KAUST DepartmentKing Abdullah University of Science and Technology, 127355 Thuwal, Makkah, Saudi Arabia,
Preprint Posting Date2021-05-30
Permanent link to this recordhttp://hdl.handle.net/10754/669415
MetadataShow full item record
AbstractWe consider the non-stationary multi-armed bandit (MAB) framework and propose a Kolmogorov-Smirnov (KS) test based Thompson Sampling (TS) algorithm named TS-KS, that actively detects change points and resets the TS parameters once a change is detected. In particular, for the two-armed bandit case, we derive bounds on the number of samples of the reward distribution to detect the change once it occurs. Consequently, we show that the proposed algorithm is asymptotically optimal with an arbitrarily high probability. Contrary to existing works in the literature that detect a change based on the estimation of mean rewards, our algorithm is able to detect a change when the underlying reward distribution changes even though the mean reward remains the same. Finally, to test the efficacy of the proposed algorithm, we employ it in a task-offloading scenario in wireless edge-computing. Our results show that the proposed TS-KS algorithm outperforms not only the static TS algorithm but also it performs better than other bandit algorithms designed for non-stationary environments. Moreover, the performance of TS-KS is at par with the state-of-the-art forecasting algorithms such as Facebook-PROPHET and ARIMA.
CitationGhatak, G., Mohanty, H., & Ur Rahman, A. (2021). KolmogorovSmirnov Test-Based Actively-Adaptive Thompson Sampling for Non-Stationary Bandits. IEEE Transactions on Artificial Intelligence, 1–1. doi:10.1109/tai.2021.3121653