Look and Modify: Modification Networks for Image Captioning

by   Fawaz Sammani, et al.

Attention-based neural encoder-decoder frameworks have been widely used for image captioning. Many of these frameworks deploy their full focus on generating the caption from scratch by relying solely on the image features or the object detection regional features. In this paper, we introduce a novel framework that learns to modify existing captions from a given framework by modeling the residual information, where at each timestep the model learns what to keep, remove or add to the existing caption allowing the model to fully focus on "what to modify" rather than on "what to predict". We evaluate our method on the COCO dataset, trained on top of several image captioning frameworks and show that our model successfully modifies captions yielding better ones with better evaluation scores.


page 1

page 2

page 9


Reflective Decoding Network for Image Captioning

State-of-the-art image captioning methods mostly focus on improving visu...

Show, Edit and Tell: A Framework for Editing Image Captions

Most image captioning frameworks generate captions directly from images,...

Senti-Attend: Image Captioning using Sentiment and Attention

There has been much recent work on image captioning models that describe...

Going Beneath the Surface: Evaluating Image Captioning for Grammaticality, Truthfulness and Diversity

Image captioning as a multimodal task has drawn much interest in recent ...

Controllable Image Captioning

State-of-the-art image captioners can generate accurate sentences to des...

Interactive Attention AI to translate low light photos to captions for night scene understanding in women safety

There is amazing progress in Deep Learning based models for Image captio...

Image Captioning with Integrated Bottom-Up and Multi-level Residual Top-Down Attention for Game Scene Understanding

Image captioning has attracted considerable attention in recent years. H...

Please sign up or login with your details

Forgot password? Click here to reset