Syntax Tree Constrained Graph Network for Visual Question Answering

by   Xiangrui Su, et al.

Visual Question Answering (VQA) aims to automatically answer natural language questions related to given image content. Existing VQA methods integrate vision modeling and language understanding to explore the deep semantics of the question. However, these methods ignore the significant syntax information of the question, which plays a vital role in understanding the essential semantics of the question and guiding the visual feature refinement. To fill the gap, we suggested a novel Syntax Tree Constrained Graph Network (STCGN) for VQA based on entity message passing and syntax tree. This model is able to extract a syntax tree from questions and obtain more precise syntax information. Specifically, we parse questions and obtain the question syntax tree using the Stanford syntax parsing tool. From the word level and phrase level, syntactic phrase features and question features are extracted using a hierarchical tree convolutional network. We then design a message-passing mechanism for phrase-aware visual entities and capture entity features according to a given visual context. Extensive experiments on VQA2.0 datasets demonstrate the superiority of our proposed model.


page 2

page 11


Knowledge Detection by Relevant Question and Image Attributes in Visual Question Answering

Visual question answering (VQA) is a Multidisciplinary research problem ...

Exploiting Rich Syntax for Better Knowledge Base Question Answering

Recent studies on Knowledge Base Question Answering (KBQA) have shown gr...

Using the Web as an Implicit Training Set: Application to Noun Compound Syntax and Semantics

An important characteristic of English written text is the abundance of ...

Interpretable Visual Question Answering by Reasoning on Dependency Trees

Collaborative reasoning for understanding each image-question pair is ve...

A Focused Dynamic Attention Model for Visual Question Answering

Visual Question and Answering (VQA) problems are attracting increasing i...

Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering

This paper presents a novel method, termed Bridge to Answer, to infer co...

Knowledge-enriched, Type-constrained and Grammar-guided Question Generation over Knowledge Bases

Question generation over knowledge bases (KBQG) aims at generating natur...

Please sign up or login with your details

Forgot password? Click here to reset