Generalization to Novel Objects using Prior Relational Knowledge

by   Varun Kumar Vijay, et al.

To solve tasks in new environments involving objects unseen during training, agents must reason over prior information about those objects and their relations. We introduce the Prior Knowledge Graph network, an architecture for combining prior information, structured as a knowledge graph, with a symbolic parsing of the visual scene, and demonstrate that this approach is able to apply learned relations to novel objects whereas the baseline algorithms fail. Ablation experiments show that the agents ground the knowledge graph relations to semantically-relevant behaviors. In both a Sokoban game and the more complex Pacman environment, our network is also more sample efficient than the baselines, reaching the same performance in 5-10x fewer episodes. Once the agents are trained with our approach, we can manipulate agent behavior by modifying the knowledge graph in semantically meaningful ways. These results suggest that our network provides a framework for agents to reason over structured knowledge graphs while still leveraging gradient based learning approaches.


page 1

page 2

page 3

page 4


Open-World Visual Recognition Using Knowledge Graphs

In a real-world setting, visual recognition systems can be brought to ma...

Symbolic image detection using scene and knowledge graphs

Sometimes the meaning conveyed by images goes beyond the list of objects...

Building Knowledge Graphs About Political Agents in the Age of Misinformation

This paper presents the construction of a Knowledge Graph about relation...

Knowledge-driven Scene Priors for Semantic Audio-Visual Embodied Navigation

Generalisation to unseen contexts remains a challenge for embodied navig...

Learning to Deceive Knowledge Graph Augmented Models via Targeted Perturbation

Symbolic knowledge (e.g., entities, relations, and facts in a knowledge ...

Learning Structured Embeddings of Knowledge Graphs with Adversarial Learning Framework

Many large-scale knowledge graphs are now available and ready to provide...

CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities

This paper introduces a novel method for determining the best room to pl...

Please sign up or login with your details

Forgot password? Click here to reset