## Search

Now showing items 1-10 of 13

JavaScript is disabled for your browser. Some features of this site may not work without it.

Author

Mishchenko, Konstantin (13)

Richtarik, Peter (11)Kovalev, Dmitry (3)Hanzely, Filip (2)Khaled, Ahmed (2)View MoreDepartment
Computer Science Program (13)

Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division (11)Computer Science (5)Applied Mathematics and Computational Science Program (3)Applied Mathematics and Computational Science (2)View MorePublisherarXiv (13)TypePreprint (13)Year (Issue Date)2019 (11)2018 (2)Item AvailabilityOpen Access (13)

Now showing items 1-10 of 13

- List view
- Grid view
- Sort Options:
- Relevance
- Title Asc
- Title Desc
- Issue Date Asc
- Issue Date Desc
- Submit Date Asc
- Submit Date Desc
- Results Per Page:
- 5
- 10
- 20
- 40
- 60
- 80
- 100

MISO is Making a Comeback With Better Proofs and Rates

Qian, Xun; Sailanbayev, Alibek; Mishchenko, Konstantin; Richtarik, Peter (arXiv, 2019-06-04) [Preprint]

MISO, also known as Finito, was one of the first stochastic variance reduced methods discovered, yet its popularity is fairly low. Its initial analysis was significantly limited by the so-called Big Data assumption. Although the assumption was lifted in subsequent work using negative momentum, this introduced a new parameter and required knowledge of strong convexity and smoothness constants, which is rarely possible in practice. We rehabilitate the method by introducing a new variant that needs only smoothness constant and does not have any extra parameters. Furthermore, when removing the strong convexity constant from the stepsize, we present a new analysis of the method, which no longer uses the assumption that every component is strongly convex. This allows us to also obtain so far unknown nonconvex convergence of MISO. To make the proposed method efficient in practice, we derive minibatching bounds with arbitrary uniform sampling that lead to linear speedup when the expected minibatch size is in a certain range. Our numerical experiments show that MISO is a serious competitor to SAGA and SVRG and sometimes outperforms them on real datasets.

Revisiting Stochastic Extragradient

Mishchenko, Konstantin; Kovalev, Dmitry; Shulgin, Egor; Richtarik, Peter; Malitsky, Yura (arXiv, 2019-05-27) [Preprint]

We consider a new extension of the extragradient method that is motivated by approximating implicit updates. Since in a recent work~\cite{chavdarova2019reducing} it was shown that the existing stochastic extragradient algorithm (called mirror-prox) of~\cite{juditsky2011solving} diverges on a simple bilinear problem, we prove guarantees for solving variational inequality that are more general than in~\cite{juditsky2011solving}. Furthermore, we illustrate numerically that the proposed variant converges faster than many other methods on the example of~\cite{chavdarova2019reducing}. We also discuss how extragradient can be applied to training Generative Adversarial Networks (GANs). Our experiments on GANs demonstrate that the introduced approach may make the training faster in terms of data passes, while its higher iteration complexity makes the advantage smaller. To further accelerate method's convergence on problems such as bilinear minimax, we combine the extragradient step with negative momentum~\cite{gidel2018negative} and discuss the optimal momentum value.

First Analysis of Local GD on Heterogeneous Data

Khaled, Ahmed; Mishchenko, Konstantin; Richtarik, Peter (arXiv, 2019-09-10) [Preprint]

We provide the first convergence analysis of local gradient descent for minimizing the average of smooth and convex but otherwise arbitrary functions. Problems of this form and local gradient descent as a solution method are of importance in federated learning, where each function is based on private data stored by a user on a mobile device, and the data of different users can be arbitrarily heterogeneous. We show that in a low accuracy regime, the method has the same communication complexity as gradient descent.

Better Communication Complexity for Local SGD

Khaled, Ahmed; Mishchenko, Konstantin; Richtarik, Peter (arXiv, 2019-09-10) [Preprint]

We revisit the local Stochastic Gradient Descent (local SGD) method and prove new convergence rates. We close the gap in the theory by showing that it works under unbounded gradients and extend its convergence to weakly convex functions. Furthermore, by changing the assumptions, we manage to get new bounds that explain in what regimes local SGD is faster that its non-local version. For instance, if the objective is strongly convex, we show that, up to constants, it is sufficient to synchronize $M$ times in total, where $M$ is the number of nodes. This improves upon the known requirement of Stich (2018) of $\sqrt{TM}$ synchronization times in total, where $T$ is the total number of iterations, which helps to explain the empirical success of local SGD.

SEGA: Variance Reduction via Gradient Sketching

Hanzely, Filip; Mishchenko, Konstantin; Richtarik, Peter (arXiv, 2018-09-09) [Preprint]

We propose a randomized first order optimization method--SEGA (SkEtchedGrAdient method)-- which progressively throughout its iterations builds avariance-reduced estimate of the gradient from random linear measurements(sketches) of the gradient obtained from an oracle. In each iteration, SEGAupdates the current estimate of the gradient through a sketch-and-projectoperation using the information provided by the latest sketch, and this issubsequently used to compute an unbiased estimate of the true gradient througha random relaxation procedure. This unbiased estimate is then used to perform agradient step. Unlike standard subspace descent methods, such as coordinatedescent, SEGA can be used for optimization problems with a non-separableproximal term. We provide a general convergence analysis and prove linearconvergence for strongly convex objectives. In the special case of coordinatesketches, SEGA can be enhanced with various techniques such as importancesampling, minibatching and acceleration, and its rate is up to a small constantfactor identical to the best-known rate of coordinate descent.

A Stochastic Penalty Model for Convex and Nonconvex Optimization with Big Constraints

Mishchenko, Konstantin; Richtarik, Peter (arXiv, 2018-10-31) [Preprint]

The last decade witnessed a rise in the importance of supervised learningapplications involving {\em big data} and {\em big models}. Big data refers tosituations where the amounts of training data available and needed causesdifficulties in the training phase of the pipeline. Big model refers tosituations where large dimensional and over-parameterized models are needed forthe application at hand. Both of these phenomena lead to a dramatic increase inresearch activity aimed at taming the issues via the design of newsophisticated optimization algorithms. In this paper we turn attention to the{\em big constraints} scenario and argue that elaborate machine learningsystems of the future will necessarily need to account for a large number ofreal-world constraints, which will need to be incorporated in the trainingprocess. This line of work is largely unexplored, and provides ampleopportunities for future work and applications. To handle the {\em bigconstraints} regime, we propose a {\em stochastic penalty} formulation which{\em reduces the problem to the well understood big data regime}. Ourformulation has many interesting properties which relate it to the originalproblem in various ways, with mathematical guarantees. We give a number ofresults specialized to nonconvex loss functions, smooth convex functions,strongly convex functions and convex constraints. We show through experimentsthat our approach can beat competing approaches by several orders of magnitudewhen a medium accuracy solution is required.

Distributed Learning with Compressed Gradient Differences

Mishchenko, Konstantin; Gorbunov, Eduard; Takáč, Martin; Richtarik, Peter (arXiv, 2019-01-26) [Preprint]

Training very large machine learning models requires a distributed computingapproach, with communication of the model updates often being the bottleneck.For this reason, several methods based on the compression (e.g., sparsificationand/or quantization) of the updates were recently proposed, including QSGD(Alistarh et al., 2017), TernGrad (Wen et al., 2017), SignSGD (Bernstein etal., 2018), and DQGD (Khirirat et al., 2018). However, none of these methodsare able to learn the gradients, which means that they necessarily suffer fromseveral issues, such as the inability to converge to the true optimum in thebatch mode, inability to work with a nonsmooth regularizer, and slowconvergence rates. In this work we propose a new distributed learningmethod---DIANA---which resolves these issues via compression of gradientdifferences. We perform a theoretical analysis in the strongly convex andnonconvex settings and show that our rates are vastly superior to existingrates. Our analysis of block-quantization and differences between $\ell_2$ and$\ell_\infty$ quantization closes the gaps in theory and practice. Finally, byapplying our analysis technique to TernGrad, we establish the first convergencerate for this method.

Stochastic Distributed Learning with Gradient Quantization and Variance Reduction

Horvath, Samuel; Kovalev, Dmitry; Mishchenko, Konstantin; Stich, Sebastian; Richtarik, Peter (arXiv, 2019-04-10) [Preprint]

We consider distributed optimization where the objective function is spreadamong different devices, each sending incremental model updates to a centralserver. To alleviate the communication bottleneck, recent work proposed variousschemes to compress (e.g.\ quantize or sparsify) the gradients, therebyintroducing additional variance $\omega \geq 1$ that might slow downconvergence. For strongly convex functions with condition number $\kappa$distributed among $n$ machines, we (i) give a scheme that converges in$\mathcal{O}((\kappa + \kappa \frac{\omega}{n} + \omega)$ $\log (1/\epsilon))$steps to a neighborhood of the optimal solution. For objective functions with afinite-sum structure, each worker having less than $m$ components, we (ii)present novel variance reduced schemes that converge in $\mathcal{O}((\kappa +\kappa \frac{\omega}{n} + \omega + m)\log(1/\epsilon))$ steps to arbitraryaccuracy $\epsilon > 0$. These are the first methods that achieve linearconvergence for arbitrary quantized updates. We also (iii) give analysis forthe weakly convex and non-convex cases and (iv) verify in experiments that ournovel variance reduced schemes are more efficient than the baselines.

A Stochastic Decoupling Method for Minimizing the Sum of Smooth and Non-Smooth Functions

Mishchenko, Konstantin; Richtarik, Peter (arXiv, 2019-05-27) [Preprint]

We consider the problem of minimizing the sum of three convex functions: i) a smooth function $f$ in the form of an expectation or a finite average, ii) a non-smooth function $g$ in the form of a finite average of proximable functions $g_j$, and iii) a proximable regularizer $R$. We design a variance reduced method which is able progressively learn the proximal operator of $g$ via the computation of the proximal operator of a single randomly selected function $g_j$ in each iteration only. Our method can provably and efficiently accommodate many strategies for the estimation of the gradient of $f$, including via standard and variance-reduced stochastic estimation, effectively decoupling the smooth part of the problem from the non-smooth part. We prove a number of iteration complexity results, including a general ${\cal O}(1/t)$ rate, ${\cal O}(1/t^2)$ rate in the case of strongly convex $f$, and several linear rates in special cases, including accelerated linear rate. For example, our method achieves a linear rate for the problem of minimizing a strongly convex function $f$ under linear constraints under no assumption on the constraints beyond consistency. When combined with SGD or SAGA estimators for the gradient of $f$, this leads to a very efficient method for empirical risk minimization with large linear constraints. Our method generalizes several existing algorithms, including forward-backward splitting, Douglas-Rachford splitting, proximal SGD, proximal SAGA, SDCA, randomized Kaczmarz and Point-SAGA. However, our method leads to many new specific methods in special cases; for instance, we obtain the first randomized variant of the Dykstra's method for projection onto the intersection of closed convex sets.

99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it

Mishchenko, Konstantin; Hanzely, Filip; Richtarik, Peter (arXiv, 2019-06-04) [Preprint]

It is well known that many optimization methods, including SGD, SAGA, andAccelerated SGD for over-parameterized models, do not scale linearly in theparallel setting. In this paper, we present a new version of block coordinatedescent that solves this issue for a number of methods. The core idea is tomake the sampling of coordinate blocks on each parallel unit independent of theothers. Surprisingly, we prove that the optimal number of blocks to be updatedby each of $n$ units in every iteration is equal to $m/n$, where $m$ is thetotal number of blocks. As an illustration, this means that when $n=100$parallel units are used, $99\%$ of work is a waste of time. We demonstrate thatwith $m/n$ blocks used by each unit the iteration complexity often remains thesame. Among other applications which we mention, this fact can be exploited inthe setting of distributed optimization to break the communication bottleneck.Our claims are justified by numerical experiments which demonstrate almost aperfect match with our theory on a number of datasets.

The export option will allow you to export the current search results of the entered query to a file. Different formats are available for download. To export the items, click on the button corresponding with the preferred download format.

By default, clicking on the export buttons will result in a download of the allowed maximum amount of items. For anonymous users the allowed maximum amount is 50 search results.

To select a subset of the search results, click "Selective Export" button and make a selection of the items you want to export. The amount of items that can be exported at once is similarly restricted as the full export.

After making a selection, click one of the export format buttons. The amount of items that will be exported is indicated in the bubble next to export format.