Unified Medical Image-Text-Label Contrastive Learning With Continuous Prompt

by   Yuhao Wang, et al.

Contrastive language-image Pre-training (CLIP) [13] can leverage large datasets of unlabeled Image-Text pairs, which have demonstrated impressive performance in various downstream tasks. Given that annotating medical data is time-consuming and laborious, Image-Text Pre-training has promising applications in exploiting large-scale medical image and radiology report datasets. However, medical Image-Text Pre-training faces several challenges, as follows: (1) Due to privacy concerns, the amount of available medical data is relatively small compared to natural data, leading to weaker generalization ability of the model. (2) Medical images are highly similar with only fine-grained differences in subtleties, resulting in a large number of false-negative sample pairs in comparison learning. (3) The hand-crafted Prompt usually differs from the natural medical image report, Subtle changes in wording can lead to significant differences in performance. In this paper, we propose a unified Image-Text-Label contrastive learning framework based on continuous prompts, with three main contributions. First, We unified the data of images, text, and labels, which greatly expanded the training data that the model could utilize. Second, we address the issue of data diversity and the impact of hand-crafted prompts on model performance by introducing continuous implicit prompts. Lastly, we propose a ImageText-Label contrastive Training to mitigate the problem of too many false-negative samples. We demonstrate through sufficient experiments that the Unified Medical Contrastive Learning (UMCL) framework exhibits excellent performance on several downstream tasks.


page 1

page 2

page 3

page 4


Multi-level Asymmetric Contrastive Learning for Medical Image Segmentation Pre-training

Contrastive learning, which is a powerful technique for learning image-l...

Positional Contrastive Learning for Volumetric Medical Image Segmentation

The success of deep learning heavily depends on the availability of larg...

Joint Learning of Localized Representations from Medical Images and Reports

Contrastive learning has proven effective for pre-training image models ...

The Role of Local Alignment and Uniformity in Image-Text Contrastive Learning on Medical Images

Image-text contrastive learning has proven effective for pretraining med...

GCL: Gradient-Guided Contrastive Learning for Medical Image Segmentation with Multi-Perspective Meta Labels

Since annotating medical images for segmentation tasks commonly incurs e...

Increasing Textual Context Size Boosts Medical Image-Text Matching

This short technical report demonstrates a simple technique that yields ...

Contrastive Learning of Medical Visual Representations from Paired Images and Text

Learning visual representations of medical images is core to medical ima...

Please sign up or login with your details

Forgot password? Click here to reset