SPORES: Sum-Product Optimization via Relational Equality Saturation for Large Scale Linear Algebra

by   Yisu Remy Wang, et al.

Machine learning algorithms are commonly specified in linear algebra (LA). LA expressions can be rewritten into more efficient forms, by taking advantage of input properties such as sparsity, as well as program properties such as common subexpressions and fusible operators. The complex interaction among these properties' impact on the execution cost poses a challenge to optimizing compilers. Existing compilers resort to intricate heuristics that complicate the codebase and add maintenance cost but fail to search through the large space of equivalent LA expressions to find the cheapest one. We introduce a general optimization technique for LA expressions, by converting the LA expressions into Relational Algebra (RA) expressions, optimizing the latter, then converting the result back to (optimized) LA expressions. One major advantage of this method is that it is complete, meaning that any equivalent LA expression can be found using the equivalence rules in RA. The challenge is the major size of the search space, and we address this by adopting and extending a technique used in compilers, called equality saturation. We integrate the optimizer into SystemML and validate it empirically across a spectrum of machine learning tasks; we show that we can derive all existing hand-coded optimizations in SystemML, and perform new optimizations that lead to speedups from 1.2X to 5X.


OLLIE: Derivation-based Tensor Program Optimizer

Boosting the runtime performance of deep neural networks (DNNs) is criti...

Equality Saturation for Tensor Graph Superoptimization

One of the major optimizations employed in deep learning frameworks is g...

Caviar: An E-graph Based TRS for Automatic Code Optimization

Term Rewriting Systems (TRSs) are used in compilers to simplify and prov...

Proving Equivalence Between Complex Expressions Using Graph-to-Sequence Neural Models

We target the problem of provably computing the equivalence between two ...

ReGiS: Regular Expression Simplification via Rewrite-Guided Synthesis

Expression simplification is an important task necessary in a variety of...

Functional Collection Programming with Semi-Ring Dictionaries

This paper introduces semi-ring dictionaries, a powerful class of compos...

Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations

Recent program synthesis techniques help users customize CAD models(e.g....

Please sign up or login with your details

Forgot password? Click here to reset