Multimodal Dialogue Response Generation

by   Qingfeng Sun, et al.

Responsing with image has been recognized as an important capability for an intelligent conversational agent. Yet existing works only focus on exploring the multimodal dialogue models which depend on retrieval-based methods, but neglecting generation methods. To fill in the gaps, we first present a multimodal dialogue generation model, which takes the dialogue history as input, then generates a textual sequence or an image as response. Learning such a model often requires multimodal dialogues containing both texts and images which are difficult to obtain. Motivated by the challenge in practice, we consider multimodal dialogue generation under a natural assumption that only limited training examples are available. In such a low-resource setting, we devise a novel conversational agent, Divter, in order to isolate parameters that depend on multimodal dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of text-only dialogues and text-image pairs respectively, then the whole parameters can be well fitted using the limited training examples. Extensive experiments demonstrate our method achieves state-of-the-art results in both automatic and human evaluation, and can generate informative text and high-resolution image responses.


page 6

page 7

page 8


Low-Resource Knowledge-Grounded Dialogue Generation

Responding with knowledge has been recognized as an important capability...

A Knowledge-Grounded Multimodal Search-Based Conversational Agent

Multimodal search-based dialogue is a challenging new task: It extends v...

Improving Context Modelling in Multimodal Dialogue Generation

In this work, we investigate the task of textual response generation in ...

Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels

Generating an informative and attractive title for the product is a cruc...

ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation

Image-grounded dialogue systems benefit greatly from integrating visual ...

Multimodal Transformer with Pointer Network for the DSTC8 AVSD Challenge

Audio-Visual Scene-Aware Dialog (AVSD) is an extension from Video Questi...

CAB: Empathetic Dialogue Generation with Cognition, Affection and Behavior

Empathy is an important characteristic to be considered when building a ...

Please sign up or login with your details

Forgot password? Click here to reset