Referring Image Segmentation Using Text Supervision

by   Fang Liu, et al.

Existing Referring Image Segmentation (RIS) methods typically require expensive pixel-level or box-level annotations for supervision. In this paper, we observe that the referring texts used in RIS already provide sufficient information to localize the target object. Hence, we propose a novel weakly-supervised RIS framework to formulate the target localization problem as a classification process to differentiate between positive and negative text expressions. While the referring text expressions for an image are used as positive expressions, the referring text expressions from other images can be used as negative expressions for this image. Our framework has three main novelties. First, we propose a bilateral prompt method to facilitate the classification process, by harmonizing the domain discrepancy between visual and linguistic features. Second, we propose a calibration method to reduce noisy background information and improve the correctness of the response maps for target object localization. Third, we propose a positive response map selection strategy to generate high-quality pseudo-labels from the enhanced response maps, for training a segmentation network for RIS inference. For evaluation, we propose a new metric to measure localization accuracy. Experiments on four benchmarks show that our framework achieves promising performances to existing fully-supervised RIS methods while outperforming state-of-the-art weakly-supervised methods adapted from related areas. Code is available at


page 1

page 3

page 6

page 7

page 8


A Weakly Supervised Consistency-based Learning Method for COVID-19 Segmentation in CT Images

Acquiring count annotations generally requires less human effort than po...

Weakly-supervised segmentation of referring expressions

Visual grounding localizes regions (boxes or segments) in the image corr...

Inter-Image Communication for Weakly Supervised Localization

Weakly supervised localization aims at finding target object regions usi...

DeepErase: Weakly Supervised Ink Artifact Removal in Document Text Images

Paper-intensive industries like insurance, law, and government have long...

Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning

Referring Expression Segmentation (RES), which is aimed at localizing an...

Monte Carlo Linear Clustering with Single-Point Supervision is Enough for Infrared Small Target Detection

Single-frame infrared small target (SIRST) detection aims at separating ...

Shatter and Gather: Learning Referring Image Segmentation with Text Supervision

Referring image segmentation, the task of segmenting any arbitrary entit...

Please sign up or login with your details

Forgot password? Click here to reset