Extracting Weighted Automata for Approximate Minimization in Language Modelling

by   Clara Lacroce, et al.

In this paper we study the approximate minimization problem for language modelling. We assume we are given some language model as a black box. The objective is to obtain a weighted finite automaton (WFA) that fits within a given size constraint and which mimics the behaviour of the original model while minimizing some notion of distance between the black box and the extracted WFA. We provide an algorithm for the approximate minimization of black boxes trained for language modelling of sequential data over a one-letter alphabet. By reformulating the problem in terms of Hankel matrices, we leverage classical results on the approximation of Hankel operators, namely the celebrated Adamyan-Arov-Krein (AAK) theory. This allows us to use the spectral norm to measure the distance between the black box and the WFA. We provide theoretical guarantees to study the potentially infinite-rank Hankel matrix of the black box, without accessing the training data, and we prove that our method returns an asymptotically-optimal approximation.


page 1

page 2

page 3

page 4


Explaining Black Boxes on Sequential Data using Weighted Automata

Understanding how a learned black box works is of crucial interest for t...

Leveraging Extracted Model Adversaries for Improved Black Box Attacks

We present a method for adversarial input generation against black box m...

Learning event-driven switched linear systems

We propose an automata theoretic learning algorithm for the identificati...

Best-Effort Adversarial Approximation of Black-Box Malware Classifiers

An adversary who aims to steal a black-box model repeatedly queries the ...

Neural Modelling of Dynamic Systems with Time Delays Based on an Adjusted NEAT Algorithm

A problem related to the development of an algorithm designed to find an...

PROPS: Probabilistic personalization of black-box sequence models

We present PROPS, a lightweight transfer learning mechanism for sequenti...

Learning Interpretable Temporal Properties from Positive Examples Only

We consider the problem of explaining the temporal behavior of black-box...

Please sign up or login with your details

Forgot password? Click here to reset