Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval

by   Jianjun Lei, et al.
Beijing University of Posts and Telecommunications
University of Surrey
Tianjin University

Sketch-based image retrieval (SBIR) is a challenging task due to the large cross-domain gap between sketches and natural images. How to align abstract sketches and natural images into a common high-level semantic space remains a key problem in SBIR. In this paper, we propose a novel semi-heterogeneous three-way joint embedding network (Semi3-Net), which integrates three branches (a sketch branch, a natural image branch, and an edgemap branch) to learn more discriminative cross-domain feature representations for the SBIR task. The key insight lies with how we cultivate the mutual and subtle relationships amongst the sketches, natural images, and edgemaps. A semi-heterogeneous feature mapping is designed to extract bottom features from each domain, where the sketch and edgemap branches are shared while the natural image branch is heterogeneous to the other branches. In addition, a joint semantic embedding is introduced to embed the features from different domains into a common high-level semantic space, where all of the three branches are shared. To further capture informative features common to both natural images and the corresponding edgemaps, a co-attention model is introduced to conduct common channel-wise feature recalibration between different domains. A hybrid-loss mechanism is designed to align the three branches, where an alignment loss and a sketch-edgemap contrastive loss are presented to encourage the network to learn invariant cross-domain representations. Experimental results on two widely used category-level datasets (Sketchy and TU-Berlin Extension) demonstrate that the proposed method outperforms state-of-the-art methods.


page 1

page 3

page 4

page 8


Three-Stream Joint Network for Zero-Shot Sketch-Based Image Retrieval

The Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) is a challenging ta...

Fuse and Attend: Generalized Embedding Learning for Art and Sketches

While deep Embedding Learning approaches have witnessed widespread succe...

The Devil is in the Middle: Exploiting Mid-level Representations for Cross-Domain Instance Matching

Many vision problems require matching images of object instances across ...

Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval

Free-hand sketch-based image retrieval (SBIR) is a specific cross-view r...

Learning Two-Branch Neural Networks for Image-Text Matching Tasks

This paper investigates two-branch neural networks for image-text matchi...

MMFL-Net: Multi-scale and Multi-granularity Feature Learning for Cross-domain Fashion Retrieval

Instance-level image retrieval in fashion is a challenging issue owing t...

Cross-Paced Representation Learning with Partial Curricula for Sketch-based Image Retrieval

In this paper we address the problem of learning robust cross-domain rep...

Please sign up or login with your details

Forgot password? Click here to reset