Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration

by   Xuan Kan, et al.
Emory University

Relation prediction among entities in images is an important step in scene graph generation (SGG), which further impacts various visual understanding and reasoning tasks. Existing SGG frameworks, however, require heavy training yet are incapable of modeling unseen (i.e.,zero-shot) triplets. In this work, we stress that such incapability is due to the lack of commonsense reasoning,i.e., the ability to associate similar entities and infer similar relations based on general understanding of the world. To fill this gap, we propose CommOnsense-integrAted sCenegrapHrElation pRediction (COACHER), a framework to integrate commonsense knowledge for SGG, especially for zero-shot relation prediction. Specifically, we develop novel graph mining pipelines to model the neighborhoods and paths around entities in an external commonsense knowledge graph, and integrate them on top of state-of-the-art SGG frameworks. Extensive quantitative evaluations and qualitative case studies on both original and manipulated datasets from Visual Genome demonstrate the effectiveness of our proposed approach.


page 1

page 2

page 3

page 4


Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoning

Commonsense reasoning systems should be able to generalize to diverse re...

One-shot Scene Graph Generation

As a structured representation of the image content, the visual scene gr...

Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation

Scene graph generation (SGG) aims to capture a wide variety of interacti...

Zero-shot Visual Commonsense Immorality Prediction

Artificial intelligence is currently powering diverse real-world applica...

Exploiting Sentiment and Common Sense for Zero-shot Stance Detection

The stance detection task aims to classify the stance toward given docum...

Reasoning about Goals, Steps, and Temporal Ordering with WikiHow

We propose a suite of reasoning tasks on two types of relations between ...

COSMO: Conditional SEQ2SEQ-based Mixture Model for Zero-Shot Commonsense Question Answering

Commonsense reasoning refers to the ability of evaluating a social situa...

Please sign up or login with your details

Forgot password? Click here to reset