Tackling Combinatorial Distribution Shift: A Matrix Completion Perspective

by   Max Simchowitz, et al.

Obtaining rigorous statistical guarantees for generalization under distribution shift remains an open and active research area. We study a setting we call combinatorial distribution shift, where (a) under the test- and training-distributions, the labels z are determined by pairs of features (x,y), (b) the training distribution has coverage of certain marginal distributions over x and y separately, but (c) the test distribution involves examples from a product distribution over (x,y) that is not covered by the training distribution. Focusing on the special case where the labels are given by bilinear embeddings into a Hilbert space H: ๐”ผ[z | x,y ]=โŸจ f_โ‹†(x),g_โ‹†(y)โŸฉ_H, we aim to extrapolate to a test distribution domain that is not covered in training, i.e., achieving bilinear combinatorial extrapolation. Our setting generalizes a special case of matrix completion from missing-not-at-random data, for which all existing results require the ground-truth matrices to be either exactly low-rank, or to exhibit very sharp spectral cutoffs. In this work, we develop a series of theoretical results that enable bilinear combinatorial extrapolation under gradual spectral decay as observed in typical high-dimensional data, including novel algorithms, generalization guarantees, and linear-algebraic results. A key tool is a novel perturbation bound for the rank-k singular value decomposition approximations between two matrices that depends on the relative spectral gap rather than the absolute spectral gap, a result that may be of broader independent interest.


page 1

page 2

page 3

page 4

โˆ™ 06/27/2012

A Combinatorial Algebraic Approach for the Identifiability of Low-Rank Matrix Completion

In this paper, we review the problem of matrix completion and expose its...
โˆ™ 02/10/2014

Universal Matrix Completion

The problem of low-rank matrix completion has recently generated a lot o...
โˆ™ 11/17/2012

The Algebraic Combinatorial Approach for Low-Rank Matrix Completion

We present a novel algebraic combinatorial view on low-rank matrix compl...
โˆ™ 06/09/2015

Symmetric Tensor Completion from Multilinear Entries and Learning Product Mixtures over the Hypercube

We give an algorithm for completing an order-m symmetric low-rank tensor...
โˆ™ 10/27/2022

From bilinear regression to inductive matrix completion: a quasi-Bayesian analysis

In this paper we study the problem of bilinear regression and we further...
โˆ™ 04/27/2023

Learning to Extrapolate: A Transductive Approach

Machine learning systems, especially with overparameterized deep neural ...
โˆ™ 08/03/2022

Equivariant Disentangled Transformation for Domain Generalization under Combination Shift

Machine learning systems may encounter unexpected problems when the data...

Please sign up or login with your details

Forgot password? Click here to reset