Open Set Classification of GAN-based Image Manipulations via a ViT-based Hybrid Architecture

by   Jun Wang, et al.
Università di Siena

Classification of AI-manipulated content is receiving great attention, for distinguishing different types of manipulations. Most of the methods developed so far fail in the open-set scenario, that is when the algorithm used for the manipulation is not represented by the training set. In this paper, we focus on the classification of synthetic face generation and manipulation in open-set scenarios, and propose a method for classification with a rejection option. The proposed method combines the use of Vision Transformers (ViT) with a hybrid approach for simultaneous classification and localization. Feature map correlation is exploited by the ViT module, while a localization branch is employed as an attention mechanism to force the model to learn per-class discriminative features associated with the forgery when the manipulation is performed locally in the image. Rejection is performed by considering several strategies and analyzing the model output layers. The effectiveness of the proposed method is assessed for the task of classification of facial attribute editing and GAN attribution.


page 5

page 7

page 8


CAFE-GAN: Arbitrary Face Attribute Editing with Complementary Attention Feature

The goal of face attribute editing is altering a facial image according ...

MU-GAN: Facial Attribute Editing based on Multi-attention Mechanism

Facial attribute editing has mainly two objectives: 1) translating image...

Designing a 3D-Aware StyleNeRF Encoder for Face Editing

GAN inversion has been exploited in many face manipulation tasks, but 2D...

A Siamese-based Verification System for Open-set Architecture Attribution of Synthetic Images

Despite the wide variety of methods developed for synthetic image attrib...

TriPINet: Tripartite Progressive Integration Network for Image Manipulation Localization

Image manipulation localization aims at distinguishing forged regions fr...

FreeDrag: Point Tracking is Not What You Need for Interactive Point-based Image Editing

To serve the intricate and varied demands of image editing, precise and ...

MaLP: Manipulation Localization Using a Proactive Scheme

Advancements in the generation quality of various Generative Models (GMs...

Please sign up or login with your details

Forgot password? Click here to reset