Plotting Behind the Scenes: Towards Learnable Game Engines

by   Willi Menapace, et al.

Game engines are powerful tools in computer graphics. Their power comes at the immense cost of their development. In this work, we present a framework to train game-engine-like neural models, solely from monocular annotated videos. The result-a Learnable Game Engine (LGE)-maintains states of the scene, objects and agents in it, and enables rendering the environment from a controllable viewpoint. Similarly to a game engine, it models the logic of the game and the underlying rules of physics, to make it possible for a user to play the game by specifying both high- and low-level action sequences. Most captivatingly, our LGE unlocks the director's mode, where the game is played by plotting behind the scenes, specifying high-level actions and goals for the agents in the form of language and desired states. This requires learning "game AI", encapsulated by our animation model, to navigate the scene using high-level constraints, play against an adversary, devise the strategy to win a point. The key to learning such game AI is the exploitation of a large and diverse text corpus, collected in this work, describing detailed actions in a game and used to train our animation model. To render the resulting state of the environment and its agents, we use a compositional NeRF representation used in our synthesis model. To foster future research, we present newly collected, annotated and calibrated large-scale Tennis and Minecraft datasets. Our method significantly outperforms existing neural video game simulators in terms of rendering quality. Besides, our LGEs unlock applications beyond capabilities of the current state of the art. Our framework, data, and models are available at


page 1

page 8

page 9

page 11

page 13

page 18

page 22

page 23


Visualising Game Engine Subsystem Coupling

Game engines support video game development by providing functionalities...

LuckyMera: a Modular AI Framework for Building Hybrid NetHack Agents

In the last few decades we have witnessed a significant development in A...

Game Engine Comparative Anatomy

Video game developers use game engines as a tool to manage complex aspec...

Multi-Environment Pretraining Enables Transfer to Action Limited Datasets

Using massive datasets to train large-scale models has emerged as a domi...

An Exploratory Approach for Game Engine Architecture Recovery

Game engines provide video game developers with a wide range of fundamen...

Neural Game Engine: Accurate learning of generalizable forward models from pixels

Access to a fast and easily copied forward model of a game is essential ...

Procedural Planetary Multi-resolution Terrain Generation for Games

Terrains are the main part of an electronic game. To reduce human effort...

Please sign up or login with your details

Forgot password? Click here to reset