Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings

by   Qian Chen, et al.
Alibaba Group

Prior studies diagnose the anisotropy problem in sentence representations from pre-trained language models, e.g., BERT, without fine-tuning. Our analysis reveals that the sentence embeddings from BERT suffer from a bias towards uninformative words, limiting the performance in semantic textual similarity (STS) tasks. To address this bias, we propose a simple and efficient unsupervised approach, Diagonal Attention Pooling (Ditto), which weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. Ditto can be easily applied to any pre-trained language model as a postprocessing operation. Compared to prior sentence embedding approaches, Ditto does not add parameters nor requires any learning. Empirical evaluations demonstrate that our proposed Ditto can alleviate the anisotropy problem and improve various pre-trained models on STS tasks.


page 1

page 2

page 3

page 4


On the Sentence Embeddings from Pre-trained Language Models

Pre-trained contextual representations like BERT have achieved great suc...

Towards preserving word order importance through Forced Invalidation

Large pre-trained language models such as BERT have been widely used as ...

Toward Better Storylines with Sentence-Level Language Models

We propose a sentence-level language model which selects the next senten...

A Simple and Plug-and-play Method for Unsupervised Sentence Representation Enhancement

Generating proper embedding of sentences through an unsupervised way is ...

Unsupervised Sentence-embeddings by Manifold Approximation and Projection

The concept of unsupervised universal sentence encoders has gained tract...

Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

The task of abductive natural language inference (αnli), to decide which...

Catching Attention with Automatic Pull Quote Selection

Pull quotes are an effective component of a captivating news article. Th...

Please sign up or login with your details

Forgot password? Click here to reset