End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging
Type
ArticleAuthors
Sitzmann, VincentDiamond, Steven
Peng, Yifan
Dun, Xiong
Boyd, Stephen
Heidrich, Wolfgang

Heide, Felix
Wetzstein, Gordon
KAUST Department
Visual Computing Center (VCC)Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Computer Science Program
Date
2018-07-31Online Publication Date
2018-07-31Print Publication Date
2018-07-30Permanent link to this record
http://hdl.handle.net/10754/630336
Metadata
Show full item recordAbstract
In typical cameras the optical system is designed first; once it is fixed, the parameters in the image processing algorithm are tuned to get good image reproduction. In contrast to this sequential design approach, we consider joint optimization of an optical system (for example, the physical shape of the lens) together with the parameters of the reconstruction algorithm.We build a fully-differentiable simulation model that maps the true source image to the reconstructed one. The model includes diffractive light propagation, depth and wavelength-dependent effects, noise and nonlinearities, and the image post-processing. We jointly optimize the optical parameters and the image processing algorithm parameters so as to minimize the deviation between the true and reconstructed image, over a large set of images. We implement our joint optimization method using autodifferentiation to efficiently compute parameter gradients in a stochastic optimization algorithm. We demonstrate the efficacy of this approach by applying it to achromatic extended depth of field and snapshot super-resolution imaging.Citation
Sitzmann V, Diamond S, Peng Y, Dun X, Boyd S, et al. (2018) End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics 37: 1–13. Available: http://dx.doi.org/10.1145/3197517.3201333.Sponsors
The authors would like to thank Xu Liu and Liang Xu from the State Key Lab of Modern Optical Instrumentation, Zhejiang University, and the KAUST Visual Computing Center for support in designing and prototyping of DOEs. This project was supported by an NSF CAREER award (IIS 1553333), an NSF Graduate Research Fellowship (DGE-114747), a Sloan Fellowship, a Terman Faculty Fellowship, a Stanford Graduate Fellowship, the Intel Compressive Sensing Alliance, and by the KAUST Office of Sponsored Research through the Visual Computing Center CCF grant.Journal
ACM Transactions on GraphicsAdditional Links
https://dl.acm.org/citation.cfm?doid=3197517.3201333ae974a485f413a2113503eed53cd6c53
10.1145/3197517.3201333