Patch-Based Stochastic Attention for Image Editing

by   Nicolas Cherel, et al.

Attention mechanisms have become of crucial importance in deep learning in recent years. These non-local operations, which are similar to traditional patch-based methods in image processing, complement local convolutions. However, computing the full attention matrix is an expensive step with a heavy memory and computational load. These limitations curb network architectures and performances, in particular for the case of high resolution images. We propose an efficient attention layer based on the stochastic algorithm PatchMatch, which is used for determining approximate nearest neighbors. We refer to our proposed layer as a "Patch-based Stochastic Attention Layer" (PSAL). Furthermore, we propose different approaches, based on patch aggregation, to ensure the differentiability of PSAL, thus allowing end-to-end training of any network containing our layer. PSAL has a small memory footprint and can therefore scale to high resolution images. It maintains this footprint without sacrificing spatial precision and globality of the nearest neighbours, which means that it can be easily inserted in any level of a deep architecture, even in shallower levels. We demonstrate the usefulness of PSAL on several image editing tasks, such as image inpainting and image colorization.


page 5

page 7

page 8

page 12

page 14

page 15

page 16


High-Resolution Deep Image Matting

Image matting is a key technique for image and video editing and composi...

High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis

Recent advances in deep learning have shown exciting promise in filling ...

Augmenting Convolutional networks with attention-based aggregation

We show how to augment any convolutional network with an attention-based...

On Nearest Neighbors in Non Local Means Denoising

To denoise a reference patch, the Non-Local-Means denoising filter proce...

PASTA-GAN++: A Versatile Framework for High-Resolution Unpaired Virtual Try-on

Image-based virtual try-on is one of the most promising applications of ...

A Full-Image Full-Resolution End-to-End-Trainable CNN Framework for Image Forgery Detection

Due to limited computational and memory resources, current deep learning...

Iterative Patch Selection for High-Resolution Image Recognition

High-resolution images are prevalent in various applications, such as au...

Please sign up or login with your details

Forgot password? Click here to reset