A Unified Architecture of Semantic Segmentation and Hierarchical Generative Adversarial Networks for Expression Manipulation

by   Rumeysa Bodur, et al.

Editing facial expressions by only changing what we want is a long-standing research problem in Generative Adversarial Networks (GANs) for image manipulation. Most of the existing methods that rely only on a global generator usually suffer from changing unwanted attributes along with the target attributes. Recently, hierarchical networks that consist of both a global network dealing with the whole image and multiple local networks focusing on local parts are showing success. However, these methods extract local regions by bounding boxes centred around the sparse facial key points which are non-differentiable, inaccurate and unrealistic. Hence, the solution becomes sub-optimal, introduces unwanted artefacts degrading the overall quality of the synthetic images. Moreover, a recent study has shown strong correlation between facial attributes and local semantic regions. To exploit this relationship, we designed a unified architecture of semantic segmentation and hierarchical GANs. A unique advantage of our framework is that on forward pass the semantic segmentation network conditions the generative model, and on backward pass gradients from hierarchical GANs are propagated to the semantic segmentation network, which makes our framework an end-to-end differentiable architecture. This allows both architectures to benefit from each other. To demonstrate its advantages, we evaluate our method on two challenging facial expression translation benchmarks, AffectNet and RaFD, and a semantic segmentation benchmark, CelebAMask-HQ across two popular architectures, BiSeNet and UNet. Our extensive quantitative and qualitative evaluations on both face semantic segmentation and face expression manipulation tasks validate the effectiveness of our work over existing state-of-the-art methods.


page 1

page 4

page 8

page 11

page 12

page 13

page 14

page 15


Semantic Segmentation by Improved Generative Adversarial Networks

While most existing segmentation methods usually combined the powerful f...

FakeLocator: Robust Localization of GAN-Based Face Manipulations via Semantic Segmentation Networks with Bells and Whistles

Nowadays, full face synthesis and partial face manipulation by virtue of...

Adversarial Pixel-Level Generation of Semantic Images

Generative Adversarial Networks (GANs) have obtained extraordinary succe...

Pixel-based Facial Expression Synthesis

Facial expression synthesis has achieved remarkable advances with the ad...

Cross-View Image Synthesis using Conditional GANs

Learning to generate natural scenes has always been a challenging task i...

Enhanced 3DMM Attribute Control via Synthetic Dataset Creation Pipeline

While facial attribute manipulation of 2D images via Generative Adversar...

GANimation: Anatomically-aware Facial Animation from a Single Image

Recent advances in Generative Adversarial Networks (GANs) have shown imp...

Please sign up or login with your details

Forgot password? Click here to reset