Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generation

by   Xingchen Li, et al.

Recently, increasing efforts have been focused on Weakly Supervised Scene Graph Generation (WSSGG). The mainstream solution for WSSGG typically follows the same pipeline: they first align text entities in the weak image-level supervisions (e.g., unlocalized relation triplets or captions) with image regions, and then train SGG models in a fully-supervised manner with aligned instance-level "pseudo" labels. However, we argue that most existing WSSGG works only focus on object-consistency, which means the grounded regions should have the same object category label as text entities. While they neglect another basic requirement for an ideal alignment: interaction-consistency, which means the grounded region pairs should have the same interactions (i.e., visual relations) as text entity pairs. Hence, in this paper, we propose to enhance a simple grounding module with both object-aware and interaction-aware knowledge to acquire more reliable pseudo labels. To better leverage these two types of knowledge, we regard them as two teachers and fuse their generated targets to guide the training process of our grounding module. Specifically, we design two different strategies to adaptively assign weights to different teachers by assessing their reliability on each training sample. Extensive experiments have demonstrated that our method consistently improves WSSGG performance on various kinds of weak supervision.


page 1

page 2

page 8


Relation-aware Instance Refinement for Weakly Supervised Visual Grounding

Visual grounding, which aims to build a correspondence between visual ob...

Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding

Weakly supervised referring expression grounding (REG) aims at localizin...

Top-Down Viewing for Weakly Supervised Grounded Image Captioning

Weakly supervised grounded image captioning (WSGIC) aims to generate the...

Focusing On Targets For Improving Weakly Supervised Visual Grounding

Weakly supervised visual grounding aims to predict the region in an imag...

Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priors

Human-object interaction (HOI) detection aims to extract interacting hum...

Distributed Attention for Grounded Image Captioning

We study the problem of weakly supervised grounded image captioning. Tha...

ASOC: Adaptive Self-aware Object Co-localization

The primary goal of this paper is to localize objects in a group of sema...

Please sign up or login with your details

Forgot password? Click here to reset