Now showing items 1-20 of 669

• #### Finding Nano-Ötzi: Semi-Supervised Volume Visualization for Cryo-Electron Tomography

(arXiv, 2021-04-04) [Preprint]
Cryo-Electron Tomography (cryo-ET) is a new 3D imaging technique with unprecedented potential for resolving submicron structural detail. Existing volume visualization methods, however, cannot cope with its very low signal-to-noise ratio. In order to design more powerful transfer functions, we propose to leverage soft segmentation as an explicit component of visualization for noisy volumes. Our technical realization is based on semi-supervised learning where we combine the advantages of two segmentation algorithms. A first weak segmentation algorithm provides good results for propagating sparse user provided labels to other voxels in the same volume. This weak segmentation algorithm is used to generate dense pseudo labels. A second powerful deep-learning based segmentation algorithm can learn from these pseudo labels to generalize the segmentation to other unseen volumes, a task that the weak segmentation algorithm fails at completely. The proposed volume visualization uses the deep-learning based segmentation as a component for segmentation-aware transfer function design. Appropriate ramp parameters can be suggested automatically through histogram analysis. Finally, our visualization uses gradient-free ambient occlusion shading to further suppress visual presence of noise, and to give structural detail desired prominence. The cryo-ET data studied throughout our technical experiments is based on the highest-quality tilted series of intact SARS-CoV-2 virions. Our technique shows the high impact in target sciences for visual data analysis of very noisy volumes that cannot be visualized with existing techniques.
• #### arthurlirui/refsepECCV2020: Code for Reflection Separation via Multi-bounce Polarization State Tracing

(Github, 2021-03-31) [Software]
Code for Reflection Separation via Multi-bounce Polarization State Tracing
• #### Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging

(arXiv, 2021-03-30) [Preprint]
We introduce Mask-ToF, a method to reduce flying pixels (FP) in time-of-flight (ToF) depth captures. FPs are pervasive artifacts which occur around depth edges, where light paths from both an object and its background are integrated over the aperture. This light mixes at a sensor pixel to produce erroneous depth estimates, which can adversely affect downstream 3D vision tasks. Mask-ToF starts at the source of these FPs, learning a microlens-level occlusion mask which effectively creates a custom-shaped sub-aperture for each sensor pixel. This modulates the selection of foreground and background light mixtures on a per-pixel basis and thereby encodes scene geometric information directly into the ToF measurements. We develop a differentiable ToF simulator to jointly train a convolutional neural network to decode this information and produce high-fidelity, low-FP depth reconstructions. We test the effectiveness of Mask-ToF on a simulated light field dataset and validate the method with an experimental prototype. To this end, we manufacture the learned amplitude mask and design an optical relay system to virtually place it on a high-resolution ToF sensor. We find that Mask-ToF generalizes well to real data without retraining, cutting FP counts in half.
• #### Labels4Free: Unsupervised Segmentation using StyleGAN

(arXiv, 2021-03-27) [Preprint]
We propose an unsupervised segmentation framework for StyleGAN generated objects. We build on two main observations. First, the features generated by StyleGAN hold valuable information that can be utilized towards training segmentation networks. Second, the foreground and background can often be treated to be largely independent and be composited in different ways. For our solution, we propose to augment the StyleGAN2 generator architecture with a segmentation branch and to split the generator into a foreground and background network. This enables us to generate soft segmentation masks for the foreground object in an unsupervised fashion. On multiple object classes, we report comparable results against state-of-the-art supervised segmentation networks, while against the best unsupervised segmentation approach we demonstrate a clear improvement, both in qualitative and quantitative metrics.

(arXiv, 2021-03-26) [Preprint]
Deep neural networks are vulnerable to small input perturbations known as adversarial attacks. Inspired by the fact that these adversaries are constructed by iteratively minimizing the confidence of a network for the true class label, we propose the anti-adversary layer, aimed at countering this effect. In particular, our layer generates an input perturbation in the opposite direction of the adversarial one, and feeds the classifier a perturbed version of the input. Our approach is training-free and theoretically supported. We verify the effectiveness of our approach by combining our layer with both nominally and robustly trained models, and conduct large scale experiments from black-box to adaptive attacks on CIFAR10, CIFAR100 and ImageNet. Our anti-adversary layer significantly enhances model robustness while coming at no cost on clean accuracy.
• #### Transfer Deep Learning for Reconfigurable Snapshot HDR Imaging Using Coded Masks

(Computer Graphics Forum, Wiley, 2021-03-11) [Article]
High dynamic range (HDR) image acquisition from a single image capture, also known as snapshot HDR imaging, is challenging because the bit depths of camera sensors are far from sufficient to cover the full dynamic range of the scene. Existing HDR techniques focus either on algorithmic reconstruction or hardware modification to extend the dynamic range. In this paper we propose a joint design for snapshot HDR imaging by devising a spatially varying modulation mask in the hardware and building a deep learning algorithm to reconstruct the HDR image. We leverage transfer learning to overcome the lack of sufficiently large HDR datasets available. We show how transferring from a different large-scale task (image classification on ImageNet) leads to considerable improvements in HDR reconstruction. We achieve a reconfigurable HDR camera design that does not require custom sensors, and instead can be reconfigured between HDR and conventional mode with very simple calibration steps. We demonstrate that the proposed hardware–software so lution offers a flexible yet robust way to modulate per-pixel exposures, and the network requires little knowledge of the hardware to faithfully reconstruct the HDR image. Comparison results show that our method outperforms the state of the art in terms of visual perception quality.
• #### ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computation

(arXiv, 2021-03-02) [Preprint]
We propose ZeroSARAH -- a novel variant of the variance-reduced method SARAH (Nguyen et al., 2017) -- for minimizing the average of a large number of nonconvex functions $\frac{1}{n}\sum_{i=1}^{n}f_i(x)$. To the best of our knowledge, in this nonconvex finite-sum regime, all existing variance-reduced methods, including SARAH, SVRG, SAGA and their variants, need to compute the full gradient over all $n$ data samples at the initial point $x^0$, and then periodically compute the full gradient once every few iterations (for SVRG, SARAH and their variants). Moreover, SVRG, SAGA and their variants typically achieve weaker convergence results than variants of SARAH: $n^{2/3}/\epsilon^2$ vs. $n^{1/2}/\epsilon^2$. ZeroSARAH is the first variance-reduced method which does not require any full gradient computations, not even for the initial point. Moreover, ZeroSARAH obtains new state-of-the-art convergence results, which can improve the previous best-known result (given by e.g., SPIDER, SpiderBoost, SARAH, SSRGD and PAGE) in certain regimes. Avoiding any full gradient computations (which is a time-consuming step) is important in many applications as the number of data samples $n$ usually is very large. Especially in the distributed setting, periodic computation of full gradient over all data samples needs to periodically synchronize all machines/devices, which may be impossible or very hard to achieve. Thus, we expect that ZeroSARAH will have a practical impact in distributed and federated learning where full device participation is impractical.
• #### Controlling wave fronts with tunable disordered non-Hermitian multilayers.

(Scientific reports, Springer Science and Business Media LLC, 2021-02-27) [Article]
Unique and flexible properties of non-Hermitian photonic systems attract ever-increasing attention via delivering a whole bunch of novel optical effects and allowing for efficient tuning light-matter interactions on nano- and microscales. Together with an increasing demand for the fast and spatially compact methods of light governing, this peculiar approach paves a broad avenue to novel optical applications. Here, unifying the approaches of disordered metamaterials and non-Hermitian photonics, we propose a conceptually new and simple architecture driven by disordered loss-gain multilayers and, therefore, providing a powerful tool to control both the passage time and the wave-front shape of incident light with different switching times. For the first time we show the possibility to switch on and off kink formation by changing the level of disorder in the case of adiabatically raising wave fronts. At the same time, we deliver flexible tuning of the output intensity by using the nonlinear effect of loss and gain saturation. Since the disorder strength in our system can be conveniently controlled with the power of the external pump, our approach can be considered as a basis for different active photonic devices.
• #### AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation

(arXiv, 2021-02-24) [Preprint]
In this paper, we propose a novel framework to translate a portrait photo-face into an anime appearance. Our aim is to synthesize anime-faces which are style-consistent with a given reference anime-face. However, unlike typical translation tasks, such anime-face translation is challenging due to complex variations of appearances among anime-faces. Existing methods often fail to transfer the styles of reference anime-faces, or introduce noticeable artifacts/distortions in the local shapes of their generated faces. We propose Ani- GAN, a novel GAN-based translator that synthesizes highquality anime-faces. Specifically, a new generator architecture is proposed to simultaneously transfer color/texture styles and transform local facial shapes into anime-like counterparts based on the style of a reference anime-face, while preserving the global structure of the source photoface. We propose a double-branch discriminator to learn both domain-specific distributions and domain-shared distributions, helping generate visually pleasing anime-faces and effectively mitigate artifacts. Extensive experiments qualitatively and quantitatively demonstrate the superiority of our method over state-of-the-art methods.
• #### An Optimal Algorithm for Strongly Convex Minimization under Affine Constraints

(arXiv, 2021-02-22) [Preprint]
Optimization problems under affine constraints appear in various areas of machine learning. We consider the task of minimizing a smooth strongly convex function $F(x)$ under the affine constraint $K x = b$, with an oracle providing evaluations of the gradient of $F$ and matrix-vector multiplications by $K$ and its transpose. We provide lower bounds on the number of gradient computations and matrix-vector multiplications to achieve a given accuracy. Then we propose an accelerated primal--dual algorithm achieving these lower bounds. Our algorithm is the first optimal algorithm for this class of problems.
• #### Shape-Tailored Deep Neural Networks

(arXiv, 2021-02-16) [Preprint]
We present Shape-Tailored Deep Neural Networks (ST-DNN). ST-DNN extend convolutional networks (CNN), which aggregate data from fixed shape (square) neighborhoods, to compute descriptors defined on arbitrarily shaped regions. This is natural for segmentation, where descriptors should describe regions (e.g., of objects) that have diverse shape. We formulate these descriptors through the Poisson partial differential equation (PDE), which can be used to generalize convolution to arbitrary regions. We stack multiple PDE layers to generalize a deep CNN to arbitrary regions, and apply it to segmentation. We show that ST-DNN are covariant to translations and rotations and robust to domain deformations, natural for segmentation, which existing CNN based methods lack. ST-DNN are 3-4 orders of magnitude smaller then CNNs used for segmentation. We show that they exceed segmentation performance compared to state-of-the-art CNN-based descriptors using 2-3 orders smaller training sets on the texture segmentation problem.
• #### Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization

(arXiv, 2021-02-14) [Preprint]
Large scale distributed optimization has become the default tool for the training of supervised machine learning models with a large number of parameters and training data. Recent advancements in the field provide several mechanisms for speeding up the training, including {\em compressed communication}, {\em variance reduction} and {\em acceleration}. However, none of these methods is capable of exploiting the inherently rich data-dependent smoothness structure of the local losses beyond standard smoothness constants. In this paper, we argue that when training supervised models, {\em smoothness matrices} -- information-rich generalizations of the ubiquitous smoothness constants -- can and should be exploited for further dramatic gains, both in theory and practice. In order to further alleviate the communication burden inherent in distributed optimization, we propose a novel communication sparsification strategy that can take full advantage of the smoothness matrices associated with local losses. To showcase the power of this tool, we describe how our sparsification technique can be adapted to three distributed optimization algorithms -- DCGD, DIANA and ADIANA -- yielding significant savings in terms of communication complexity. The new methods always outperform the baselines, often dramatically so.
• #### Distributed Second Order Methods with Fast Rates and Compressed Communication

(arXiv, 2021-02-14) [Preprint]
We develop several new communication-efficient second-order methods for distributed optimization. Our first method, NEWTON-STAR, is a variant of Newton's method from which it inherits its fast local quadratic rate. However, unlike Newton's method, NEWTON-STAR enjoys the same per iteration communication cost as gradient descent. While this method is impractical as it relies on the use of certain unknown parameters characterizing the Hessian of the objective function at the optimum, it serves as the starting point which enables us design practical variants thereof with strong theoretical guarantees. In particular, we design a stochastic sparsification strategy for learning the unknown parameters in an iterative fashion in a communication efficient manner. Applying this strategy to NEWTON-STAR leads to our next method, NEWTON-LEARN, for which we prove local linear and superlinear rates independent of the condition number. When applicable, this method can have dramatically superior convergence behavior when compared to state-of-the-art methods. Finally, we develop a globalization strategy using cubic regularization which leads to our next method, CUBIC-NEWTON-LEARN, for which we prove global sublinear and linear convergence rates, and a fast superlinear rate. Our results are supported with experimental results on real datasets, and show several orders of magnitude improvement on baseline and state-of-the-art methods in terms of communication complexity.
• #### Manhattan Room Layout Reconstruction from a Single 360 ∘ Image: A Comparative Study of State-of-the-Art Methods

(International Journal of Computer Vision, Springer Science and Business Media LLC, 2021-02-09) [Article]
Recent approaches for predicting layouts from 360∘ panoramas produce excellent results. These approaches build on a common framework consisting of three steps: a pre-processing step based on edge-based alignment, prediction of layout elements, and a post-processing step by fitting a 3D layout to the layout elements. Until now, it has been difficult to compare the methods due to multiple different design decisions, such as the encoding network (e.g., SegNet or ResNet), type of elements predicted (e.g., corners, wall/floor boundaries, or semantic segmentation), or method of fitting the 3D layout. To address this challenge, we summarize and describe the common framework, the variants, and the impact of the design decisions. For a complete evaluation, we also propose extended annotations for the Matterport3D dataset (Chang et al.: Matterport3d: learning from rgb-d data in indoor environments. arXiv:1709.06158, 2017), and introduce two depth-based evaluation metrics.
• #### A Visual Analytics Based Decision Making Environment for COVID-19 Modeling and Visualization

(IEEE, 2021-02-01) [Conference Paper]
Public health officials dealing with pandemics like COVID-19 have to evaluate and prepare response plans. This planning phase requires not only looking into the spatiotemporal dynamics and impact of the pandemic using simulation models, but they also need to plan and ensure the availability of resources under different spread scenarios. To this end, we have developed a visual analytics environment that enables public health officials to model, simulate, and explore the spread of COVID-19 by supplying county-level information such as population, demographics, and hospital beds. This environment facilitates users to explore spatiotemporal model simulation data relevant to COVID-19 through a geospatial map with linked statistical views, apply different decision measures at different points in time, and understand their potential impact. Users can drill-down to county-level details such as the number of sicknesses, deaths, needs for hospitalization, and variations in these statistics over time. We demonstrate the usefulness of this environment through a use case study and also provide feedback from domain experts. We also provide details about future extensions and potential applications of this work.
• #### Efficient exponential time integration for simulating nonlinear coupled oscillators

(Journal of Computational and Applied Mathematics, Elsevier BV, 2021-01-27) [Article]
In this paper, we propose an advanced time integration technique associated with explicit exponential Rosenbrock-based methods for the simulation of large stiff systems of nonlinear coupled oscillators. In particular, a novel reformulation of these systems is introduced and a general family of efficient exponential Rosenbrock schemes for simulating the reformulated system is derived. Moreover, we show the required regularity conditions and prove the convergence of these schemes for the system of coupled oscillators. We present an efficient implementation of this new approach and discuss several applications in scientific and visual computing. The accuracy and efficiency of our approach are demonstrated through a broad spectrum of numerical examples, including a nonlinear Fermi–Pasta–Ulam–Tsingou model, elastic and nonelastic deformations as well as collision scenarios focusing on relevant aspects such as stability and energy conservation, large numerical stiffness, high fidelity, and visual accuracy.
• #### Lost photon enhances superresolution

(arXiv, 2021-01-14) [Preprint]
Quantum imaging can beat classical resolution limits, imposed by diffraction of light. In particular, it is known that one can reduce the image blurring and increase the achievable resolution by illuminating an object by entangled light and measuring coincidences of photons. If an $n$-photon entangled state is used and the $n$th-order correlation function is measured, the point-spread function (PSF) effectively becomes $\sqrt n$ times narrower relatively to classical coherent imaging. Quite surprisingly, measuring $n$-photon correlations is not the best choice if an $n$-photon entangled state is available. We show that for measuring $(n-1)$-photon coincidences (thus, ignoring one of the available photons), PSF can be made even narrower. This observation paves a way for a strong conditional resolution enhancement by registering one of the photons outside the imaging area. We analyze the conditions necessary for the resolution increase and propose a practical scheme, suitable for observation and exploitation of the effect.
• #### MAAS: Multi-modal Assignation for Active Speaker Detection

(arXiv, 2021-01-11) [Preprint]
Active speaker detection requires a solid integration of multi-modal cues. While individual modalities can approximate a solution, accurate predictions can only be achieved by explicitly fusing the audio and visual features and modeling their temporal progression. Despite its inherent muti-modal nature, current methods still focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. In this paper we present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem, and provides a straightforward strategy where independent visual features from potential speakers in the scene are assigned to a previously detected speech event. Our experiments show that, an small graph data structure built from a single frame, allows to approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 88.8\%.
• #### The influence of the synthesis conditions on the magnetic behaviour of the densely packed arrays of Ni nanowires in porous anodic alumina membranes

(RSC Advances, Royal Society of Chemistry (RSC), 2021) [Article]
The densely packed arrays of Ni nanowires of 70 nm diameter and 6–12 μm length were obtained $\textit{via}$ electrodeposition into porous alumina membranes (PAAMs) of 55–75 μm thickness.
• #### Towards an End-to-End Analysis and Prediction System for Weather, Climate, and Marine Applications in the Red Sea

(Bulletin of the American Meteorological Society, American Meteorological Society, 2021-01) [Article]
AbstractThe Red Sea, home to the second-longest coral reef system in the world, is a vital resource for the Kingdom of Saudi Arabia. The Red Sea provides 90% of the Kingdom’s potable water by desalinization, supporting tourism, shipping, aquaculture, and fishing industries, which together contribute about 10%–20% of the country’s GDP. All these activities, and those elsewhere in the Red Sea region, critically depend on oceanic and atmospheric conditions. At a time of mega-development projects along the Red Sea coast, and global warming, authorities are working on optimizing the harnessing of environmental resources, including renewable energy and rainwater harvesting. All these require high-resolution weather and climate information. Toward this end, we have undertaken a multipronged research and development activity in which we are developing an integrated data-driven regional coupled modeling system. The telescopically nested components include 5-km- to 600-m-resolution atmospheric models to address weather and climate challenges, 4-km- to 50-m-resolution ocean models with regional and coastal configurations to simulate and predict the general and mesoscale circulation, 4-km- to 100-m-resolution ecosystem models to simulate the biogeochemistry, and 1-km- to 50-m-resolution wave models. In addition, a complementary probabilistic transport modeling system predicts dispersion of contaminant plumes, oil spill, and marine ecosystem connectivity. Advanced ensemble data assimilation capabilities have also been implemented for accurate forecasting. Resulting achievements include significant advancement in our understanding of the regional circulation and its connection to the global climate, development, and validation of long-term Red Sea regional atmospheric–oceanic–wave reanalyses and forecasting capacities. These products are being extensively used by academia, government, and industry in various weather and marine studies and operations, environmental policies, renewable energy applications, impact assessment, flood forecasting, and more.