How Does Lipschitz Regularization Influence GAN Training?

Embargo End Date
2021-10-09

Type
Conference Paper

Authors
Qin, Yipeng
Mitra, Niloy
Wonka, Peter

KAUST Department
Visual Computing Center (VCC)
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Computer Science Program

KAUST Grant Number
CRG2018-3730

Online Publication Date
2020-10-10

Print Publication Date
2020

Date
2020-10-09

Abstract
Despite the success of Lipschitz regularization in stabilizing GAN training, the exact reason of its effectiveness remains poorly understood. The direct effect of K-Lipschitz regularization is to restrict the L2-norm of the neural network gradient to be smaller than a threshold K (e.g.,) such that. In this work, we uncover an even more important effect of Lipschitz regularization by examining its impact on the loss function: It degenerates GAN loss functions to almost linear ones by restricting their domain and interval of attainable gradient values. Our analysis shows that loss functions are only successful if they are degenerated to almost linear ones. We also show that loss functions perform poorly if they are not degenerated and that a wide range of functions can be used as loss function as long as they are sufficiently degenerated by regularization. Basically, Lipschitz regularization ensures that all loss functions effectively work in the same way. Empirically, we verify our proposition on the MNIST, CIFAR10 and CelebA datasets.

Citation
Qin, Y., Mitra, N., & Wonka, P. (2020). How Does Lipschitz Regularization Influence GAN Training? Lecture Notes in Computer Science, 310–326. doi:10.1007/978-3-030-58517-4_19

Acknowledgements
This work was supported in part by the KAUST Office of Sponsored Research (OSR) under Award No. OSR-CRG2018-3730.

Publisher
Springer Nature

Conference/Event Name
16th European Conference on Computer Vision, ECCV 2020

DOI
10.1007/978-3-030-58517-4_19

arXiv
1811.09567

Additional Links
http://link.springer.com/10.1007/978-3-030-58517-4_19http://orca.cf.ac.uk/133740/1/how_does_lipschitz_regularization_influence_gan_training.pdf

Permanent link to this record