Index-based, High-dimensional, Cosine Threshold Querying with Optimality Guarantees

by   Yuliang Li, et al.

Given a database of vectors, a cosine threshold query returns all vectors in the database having cosine similarity to a query vector above a given threshold θ. These queries arise naturally in many applications, such as document retrieval, image search, and mass spectrometry. The present paper considers the efficient evaluation of such queries, providing novel optimality guarantees and exhibiting good performance on real datasets. We take as a starting point Fagin's well-known Threshold Algorithm (TA), which can be used to answer cosine threshold queries as follows: an inverted index is first built from the database vectors during pre-processing; at query time, the algorithm traverses the index partially to gather a set of candidate vectors to be later verified for θ-similarity. However, directly applying TA in its raw form misses significant optimization opportunities. Indeed, we first show that one can take advantage of the fact that the vectors can be assumed to be normalized, to obtain an improved, tight stopping condition for index traversal and to efficiently compute it incrementally. Then we show that one can take advantage of data skewness to obtain better traversal strategies. In particular, we show a novel traversal strategy that exploits a common data skewness condition which holds in multiple domains including mass spectrometry, documents, and image databases. We show that under the skewness assumption, the new traversal strategy has a strong, near-optimal performance guarantee. The techniques developed in the paper are quite general since they can be applied to a large class of similarity functions beyond cosine.


page 1

page 2

page 3

page 4


qwLSH: Cache-conscious Indexing for Processing Similarity Search Query Workloads in High-Dimensional Spaces

Similarity search queries in high-dimensional spaces are an important ty...

Navigable Proximity Graph-Driven Native Hybrid Queries with Structured and Unstructured Constraints

As research interest surges, vector similarity search is applied in mult...

The Document Vectors Using Cosine Similarity Revisited

The current state-of-the-art test accuracy (97.42%) on the IMDB movie re...

COSIME: FeFET based Associative Memory for In-Memory Cosine Similarity Search

In a number of machine learning models, an input query is searched acros...

Efficient Approximate Search for Sets of Vectors

We consider a similarity measure between two sets A and B of vectors, th...

Finding Significant Subregions in Large Image Databases

Images have become an important data source in many scientific and comme...

Consistent and Flexible Selectivity Estimation for High-dimensional Data

Selectivity estimation aims at estimating the number of database objects...

Please sign up or login with your details

Forgot password? Click here to reset