A Stochastic Penalty Model for Convex and Nonconvex Optimization with Big Constraints

Abstract

The last decade witnessed a rise in the importance of supervised learningapplications involving {\em big data} and {\em big models}. Big data refers tosituations where the amounts of training data available and needed causesdifficulties in the training phase of the pipeline. Big model refers tosituations where large dimensional and over-parameterized models are needed forthe application at hand. Both of these phenomena lead to a dramatic increase inresearch activity aimed at taming the issues via the design of newsophisticated optimization algorithms. In this paper we turn attention to the{\em big constraints} scenario and argue that elaborate machine learningsystems of the future will necessarily need to account for a large number ofreal-world constraints, which will need to be incorporated in the trainingprocess. This line of work is largely unexplored, and provides ampleopportunities for future work and applications. To handle the {\em bigconstraints} regime, we propose a {\em stochastic penalty} formulation which{\em reduces the problem to the well understood big data regime}. Ourformulation has many interesting properties which relate it to the originalproblem in various ways, with mathematical guarantees. We give a number ofresults specialized to nonconvex loss functions, smooth convex functions,strongly convex functions and convex constraints. We show through experimentsthat our approach can beat competing approaches by several orders of magnitudewhen a medium accuracy solution is required.