Fast Relative Entropy Coding with A* coding

by   Gergely Flamich, et al.

Relative entropy coding (REC) algorithms encode a sample from a target distribution Q using a proposal distribution P, such that the expected codelength is π’ͺ(D_KL[Q || P]). REC can be seamlessly integrated with existing learned compression models since, unlike entropy coding, it does not assume discrete Q or P, and does not require quantisation. However, general REC algorithms require an intractable Ξ©(e^D_KL[Q || P]) runtime. We introduce AS* and AD* coding, two REC algorithms based on A* sampling. We prove that, for continuous distributions over ℝ, if the density ratio is unimodal, AS* has π’ͺ(D_∞[Q || P]) expected runtime, where D_∞[Q || P] is the RΓ©nyi ∞-divergence. We provide experimental evidence that AD* also has π’ͺ(D_∞[Q || P]) expected runtime. We prove that AS* and AD* achieve an expected codelength of π’ͺ(D_KL[Q || P]). Further, we introduce DAD*, an approximate algorithm based on AD* which retains its favourable runtime and has bias similar to that of alternative methods. Focusing on VAEs, we propose the IsoKL VAE (IKVAE), which can be used with DAD* to further improve compression efficiency. We evaluate A* coding with (IK)VAEs on MNIST, showing that it can losslessly compress images near the theoretically optimal limit.


page 1

page 2

page 3

page 4

βˆ™ 04/20/2023

Adaptive Greedy Rejection Sampling

We consider channel simulation protocols between two communicating parti...
βˆ™ 10/02/2020

Compressing Images by Encoding Their Latent Representations with Relative Entropy Coding

Variational Autoencoders (VAEs) have seen widespread use in learned imag...
βˆ™ 05/25/2022

Formalizing Preferences Over Runtime Distributions

When trying to solve a computational problem we are often faced with a c...
βˆ™ 05/24/2023

Greedy Poisson Rejection Sampling

One-shot channel simulation is a fundamental data compression problem co...
βˆ™ 07/11/2022

Accelerated Deep Lossless Image Coding with Unified Paralleleized GPU Coding Architecture

We propose Deep Lossless Image Coding (DLIC), a full resolution learned ...
βˆ™ 01/13/2020

Optimal Approximate Sampling from Discrete Probability Distributions

This paper addresses a fundamental problem in random variate generation:...
βˆ™ 03/29/2019

User Preference Aware Lossless Data Compression at the Edge

Data compression is an efficient technique to save data storage and tran...

Please sign up or login with your details

Forgot password? Click here to reset