Optimal Algorithms for Affinely Constrained, Distributed, Decentralized, Minimax, and High-Order Optimization Problems
Type
DissertationAuthors
Kovalev, Dmitry
Advisors
Richtarik, Peter
Committee members
Nesterov, YuriiNemirovski, Arkadi
Keyes, David E.
Wang, Di
Parsani, Matteo
Program
Computer ScienceDate
2022-09Permanent link to this record
http://hdl.handle.net/10754/682331Metadata
Show full item recordAbstract
Optimization problems are ubiquitous in all quantitative scientific disciplines, from computer science and engineering to operations research and economics. Developing algorithms for solving various optimization problems has been the focus of mathematical research for years. In the last decade, optimization research has become even more popular due to its applications in the rapidly developing field of machine learning. In this thesis, we discuss a few fundamental and well-studied optimization problem classes: decentralized distributed optimization (Chapters 2 to 4), distributed optimization under similarity (Chapter 5), affinely constrained optimization (Chapter 6), minimax optimization (Chapter 7), and high-order optimization (Chapter 8). For each problem class, we develop the first provably optimal algorithm: the complexity of such an algorithm cannot be improved for the problem class given. The proposed algorithms show state-of-the-art performance in practical applications, which makes them highly attractive for potential generalizations and extensions in the future.Citation
Kovalev, D. (2022). Optimal Algorithms for Affinely Constrained, Distributed, Decentralized, Minimax, and High-Order Optimization Problems [KAUST Research Repository]. https://doi.org/10.25781/KAUST-NFIDYScopus Count
Related items
Showing items related by title, author, creator and subject.
-
On Truly Block Eigensolvers via Riemannian Optimization(Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, 2018-04) [Conference Paper]We study theoretical properties of block solvers for the eigenvalue problem. Despite a recent surge of interest in such eigensolver analysis, truly block solvers have received relatively less attention, in contrast to the majority of studies concentrating on vector versions and non-truly block versions that rely on the deflation strategy. In fact, truly block solvers are more widely deployed in practice by virtue of its simplicity without compromise on accuracy. However, the corresponding theoretical analysis remains inadequate for first-order solvers, as only local and k-th gap-dependent rates of convergence have been established thus far. This paper is devoted to revealing significantly better or as-yet-unknown theoretical properties of such solvers. We present a novel convergence analysis in a unified framework for three types of first-order Riemannian solvers, i.e., deterministic, vanilla stochastic, and stochastic with variance reduction, that are to find top-k eigenvectors of a real symmetric matrix, in full generality. In particular, the issue of zero gaps between eigenvalues, to the best of our knowledge for the first time, is explicitly considered for these solvers, which brings new understandings, e.g., the dependence of convergence on gaps other than the k-th one. We thus propose the concept of generalized k-th gap. Three types of solvers are proved to converge to a globally optimal solution at a global, generalized k-th gap-dependent, and linear or sub-linear rate.
-
Optimal explicit strong stability preserving Runge–Kutta methods with high linear order and optimal nonlinear order(Mathematics of Computation, American Mathematical Society (AMS), 2015-04-10) [Article]High order spatial discretizations with monotonicity properties are often desirable for the solution of hyperbolic PDEs. These methods can advantageously be coupled with high order strong stability preserving time discretizations. The search for high order strong stability time-stepping methods with large allowable strong stability coefficient has been an active area of research over the last two decades. This research has shown that explicit SSP Runge-Kutta methods exist only up to fourth order. However, if we restrict ourselves to solving only linear autonomous problems, the order conditions simplify and this order barrier is lifted: explicit SSP Runge-Kutta methods of any linear order exist. These methods reduce to second order when applied to nonlinear problems. In the current work we aim to find explicit SSP Runge-Kutta methods with large allowable time-step, that feature high linear order and simultaneously have the optimal fourth order nonlinear order. These methods have strong stability coefficients that approach those of the linear methods as the number of stages and the linear order is increased. This work shows that when a high linear order method is desired, it may still be worthwhile to use methods with higher nonlinear order.
-
Distributed Optimization for Robot Networks: From Real-Time Convex Optimization to Game-Theoretic Self-Organization(Proceedings of the IEEE, Institute of Electrical and Electronics Engineers (IEEE), 2020-10-27) [Article]Recent advances in sensing, communication, and computing technologies have enabled the use of multirobot systems for practical applications such as surveillance, area mapping, and search and rescue. For such systems, a major challenge is to design decision rules that are real-time-implementable, require local information only, and guarantee some desired global performance. Distributed optimization provides a framework for designing such local decision-making rules for multirobot systems. In this article, we present a collection of selected results for distributed optimization for robot networks. We will focus on two special classes of problems: 1) real-time path planning for multirobot systems and 2) self-organization in multirobot systems using game-theoretic approaches. For multirobot path planning, we will present some recent approaches that are based on approximately solving distributed optimization problems over continuous and discrete domains of actions. The main idea underlying these approaches is that a variety of path planning problems can be formulated as convex optimization and submodular minimization problems over continuous and discrete action spaces, respectively. To generate local update rules that are efficiently implementable in real time, these approaches rely on approximate solutions to the global problems that can still guarantee some level of desired global performance. For game-theoretic self-organization, we will present a sampling of results for area coverage and real-time target assignment. In these results, the problems are formulated as games, and online updating rules are designed to enable teams of robots to achieve the collective objective in a distributed manner.
