Concrete Score Matching: Generalized Score Matching for Discrete Data

by   Chenlin Meng, et al.
Stanford University

Representing probability distributions by the gradient of their density functions has proven effective in modeling a wide range of continuous data modalities. However, this representation is not applicable in discrete domains where the gradient is undefined. To this end, we propose an analogous score function called the "Concrete score", a generalization of the (Stein) score for discrete settings. Given a predefined neighborhood structure, the Concrete score of any input is defined by the rate of change of the probabilities with respect to local directional changes of the input. This formulation allows us to recover the (Stein) score in continuous domains when measuring such changes by the Euclidean distance, while using the Manhattan distance leads to our novel score function in discrete domains. Finally, we introduce a new framework to learn such scores from samples called Concrete Score Matching (CSM), and propose an efficient training objective to scale our approach to high dimensions. Empirically, we demonstrate the efficacy of CSM on density estimation tasks on a mixture of synthetic, tabular, and high-dimensional image datasets, and demonstrate that it performs favorably relative to existing baselines for modeling discrete data.


page 8

page 9

page 20


Interpretation and Generalization of Score Matching

Score matching is a recently developed parameter learning method that is...

Interaction Models and Generalized Score Matching for Compositional Data

Applications such as the analysis of microbiome data have led to renewed...

Hyperspherical Dirac Mixture Reapproximation

We propose a novel scheme for efficient Dirac mixture modeling of distri...

Score-based Continuous-time Discrete Diffusion Models

Score-based modeling through stochastic differential equations (SDEs) ha...

Deep Energy Estimator Networks

Density estimation is a fundamental problem in statistical learning. Thi...

Sliced Score Matching: A Scalable Approach to Density and Score Estimation

Score matching is a popular method for estimating unnormalized statistic...

Classification via score-based generative modelling

In this work, we investigated the application of score-based gradient le...

Please sign up or login with your details

Forgot password? Click here to reset