Optimal Algorithms for Affinely Constrained, Distributed, Decentralized, Minimax, and High-Order Optimization Problems
Type
DissertationAuthors
Kovalev, Dmitry
Advisors
Richtarik, Peter
Committee members
Nesterov, YuriiNemirovski, Arkadi
Keyes, David E.
Wang, Di
Parsani, Matteo
Program
Computer ScienceDate
2022-09Permanent link to this record
http://hdl.handle.net/10754/682331Metadata
Show full item recordAbstract
Optimization problems are ubiquitous in all quantitative scientific disciplines, from computer science and engineering to operations research and economics. Developing algorithms for solving various optimization problems has been the focus of mathematical research for years. In the last decade, optimization research has become even more popular due to its applications in the rapidly developing field of machine learning. In this thesis, we discuss a few fundamental and well-studied optimization problem classes: decentralized distributed optimization (Chapters 2 to 4), distributed optimization under similarity (Chapter 5), affinely constrained optimization (Chapter 6), minimax optimization (Chapter 7), and high-order optimization (Chapter 8). For each problem class, we develop the first provably optimal algorithm: the complexity of such an algorithm cannot be improved for the problem class given. The proposed algorithms show state-of-the-art performance in practical applications, which makes them highly attractive for potential generalizations and extensions in the future.Citation
Kovalev, D. (2022). Optimal Algorithms for Affinely Constrained, Distributed, Decentralized, Minimax, and High-Order Optimization Problems [KAUST Research Repository]. https://doi.org/10.25781/KAUST-NFIDYScopus Count
Related items
Showing items related by title, author, creator and subject.
-
On Truly Block Eigensolvers via Riemannian Optimization(Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, 2018-04) [Conference Paper]We study theoretical properties of block solvers for the eigenvalue problem. Despite a recent surge of interest in such eigensolver analysis, truly block solvers have received relatively less attention, in contrast to the majority of studies concentrating on vector versions and non-truly block versions that rely on the deflation strategy. In fact, truly block solvers are more widely deployed in practice by virtue of its simplicity without compromise on accuracy. However, the corresponding theoretical analysis remains inadequate for first-order solvers, as only local and k-th gap-dependent rates of convergence have been established thus far. This paper is devoted to revealing significantly better or as-yet-unknown theoretical properties of such solvers. We present a novel convergence analysis in a unified framework for three types of first-order Riemannian solvers, i.e., deterministic, vanilla stochastic, and stochastic with variance reduction, that are to find top-k eigenvectors of a real symmetric matrix, in full generality. In particular, the issue of zero gaps between eigenvalues, to the best of our knowledge for the first time, is explicitly considered for these solvers, which brings new understandings, e.g., the dependence of convergence on gaps other than the k-th one. We thus propose the concept of generalized k-th gap. Three types of solvers are proved to converge to a globally optimal solution at a global, generalized k-th gap-dependent, and linear or sub-linear rate.
-
Optimal explicit strong stability preserving Runge–Kutta methods with high linear order and optimal nonlinear order(Mathematics of Computation, American Mathematical Society (AMS), 2015-04-10) [Article]High order spatial discretizations with monotonicity properties are often desirable for the solution of hyperbolic PDEs. These methods can advantageously be coupled with high order strong stability preserving time discretizations. The search for high order strong stability time-stepping methods with large allowable strong stability coefficient has been an active area of research over the last two decades. This research has shown that explicit SSP Runge-Kutta methods exist only up to fourth order. However, if we restrict ourselves to solving only linear autonomous problems, the order conditions simplify and this order barrier is lifted: explicit SSP Runge-Kutta methods of any linear order exist. These methods reduce to second order when applied to nonlinear problems. In the current work we aim to find explicit SSP Runge-Kutta methods with large allowable time-step, that feature high linear order and simultaneously have the optimal fourth order nonlinear order. These methods have strong stability coefficients that approach those of the linear methods as the number of stages and the linear order is increased. This work shows that when a high linear order method is desired, it may still be worthwhile to use methods with higher nonlinear order.
-
PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization(arXiv, 2021) [Conference Paper]In this paper, we propose a novel stochastic gradient estimator-ProbAbilistic Gradient Estimator (PAGE)-for nonconvex optimization. PAGE is easy to implement as it is designed via a small adjustment to vanilla SGD: in each iteration, PAGE uses the vanilla minibatch SGD update with probability p t or reuses the previous gradient with a small adjustment, at a much lower computational cost, with probability 1 - p(t). We give a simple formula for the optimal choice of p(t). Moreover, we prove the first tight lower bound Omega (n + root n/epsilon(2)), for non-convex finite-sum problems, which also leads to a tight lower bound Omega (b + root b/epsilon(2)) for non- convex online problems, where b := min{sigma(2)/epsilon(2), n} . Then, we show that PAGE obtains the optimal convergence results O(n + root n/epsilon(2)) (finite-sum) and O(b + root b/is an element of(2)) (online) matching our lower bounds for both nonconvex finite-sum and online problems. Besides, we also show that for nonconvex functions satisfying the Polyak-Lojasiewicz (PL) condition, PAGE can automatically switch to a faster linear convergence rate O(. log 1/epsilon). Finally, we conduct several deep learning experiments (e.g., LeNet, VGG, ResNet) on real datasets in PyTorch showing that PAGE not only converges much faster than SGD in training but also achieves the higher test accuracy, validating the optimal theoretical results and confirming the practical superiority of PAGE.
