Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

by   Hongliang He, et al.

Contrastive learning-based methods, such as unsup-SimCSE, have achieved state-of-the-art (SOTA) performances in learning unsupervised sentence embeddings. However, in previous studies, each embedding used for contrastive learning only derived from one sentence instance, and we call these embeddings instance-level embeddings. In other words, each embedding is regarded as a unique class of its own, whichmay hurt the generalization performance. In this study, we propose IS-CSE (instance smoothing contrastive sentence embedding) to smooth the boundaries of embeddings in the feature space. Specifically, we retrieve embeddings from a dynamic memory buffer according to the semantic similarity to get a positive embedding group. Then embeddings in the group are aggregated by a self-attention operation to produce a smoothed instance embedding for further analysis. We evaluate our method on standard semantic text similarity (STS) tasks and achieve an average of 78.30 and 79.42 RoBERTa-base, and RoBERTa-large respectively, a 2.05 improvement compared to unsup-SimCSE.


page 1

page 2

page 3

page 4


Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding

Contrastive learning has become a new paradigm for unsupervised sentence...

InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

Contrastive learning has been extensively studied in sentence embedding ...

Whitening-based Contrastive Learning of Sentence Embeddings

This paper presents a whitening-based contrastive learning method for se...

CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings

Traditional comparative learning sentence embedding directly uses the en...

Learning Positional Embeddings for Coordinate-MLPs

We propose a novel method to enhance the performance of coordinate-MLPs ...

An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning

Unsupervised sentence embeddings learning has been recently dominated by...

A Critique of the Smooth Inverse Frequency Sentence Embeddings

We critically review the smooth inverse frequency sentence embedding met...

Please sign up or login with your details

Forgot password? Click here to reset