Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions

by   Yongqiang Cai, et al.
Beijing Normal University

In recent years, deep learning-based sequence modelings, such as language models, have received much attention and success, which pushes researchers to explore the possibility of transforming non-sequential problems into a sequential form. Following this thought, deep neural networks can be represented as composite functions of a sequence of mappings, linear or nonlinear, where each composition can be viewed as a word. However, the weights of linear mappings are undetermined and hence require an infinite number of words. In this article, we investigate the finite case and constructively prove the existence of a finite vocabulary V={ϕ_i: ℝ^d →ℝ^d | i=1,...,n} with n=O(d^2) for the universal approximation. That is, for any continuous mapping f: ℝ^d →ℝ^d, compact domain Ω and ε>0, there is a sequence of mappings ϕ_i_1, ..., ϕ_i_m∈ V, m ∈ℤ_+, such that the composition ϕ_i_m∘ ... ∘ϕ_i_1 approximates f on Ω with an error less than ε. Our results provide a linguistic perspective of composite mappings and suggest a cross-disciplinary study between linguistics and approximation theory.


page 1

page 2

page 3

page 4


Beyond the Central Limit Theorem: Universal and Non-universal Simulations of Random Variables by General Mappings

The Central Limit Theorem states that a standard Gaussian random variabl...

Multi-Level Composite Stochastic Optimization via Nested Variance Reduction

We consider multi-level composite optimization problems where each mappi...

Characterizations and constructions of n-to-1 mappings over finite fields

n-to-1 mappings have wide applications in many areas, especially in cryp...

Stochastic Feedforward Neural Networks: Universal Approximation

In this chapter we take a look at the universal approximation question f...

Neural tangent kernels, transportation mappings, and universal approximation

This paper establishes rates of universal approximation for the shallow ...

Solving for best linear approximates

Our goal is to finally settle a persistent problem in Diophantine Approx...

Please sign up or login with your details

Forgot password? Click here to reset