Efficiently Controlling Multiple Risks with Pareto Testing

by   Bracha Laufer-Goldshtein, et al.

Machine learning applications frequently come with multiple diverse objectives and constraints that can change over time. Accordingly, trained models can be tuned with sets of hyper-parameters that affect their predictive behavior (e.g., their run-time efficiency versus error rate). As the number of constraints and hyper-parameter dimensions grow, naively selected settings may lead to sub-optimal and/or unreliable results. We develop an efficient method for calibrating models such that their predictions provably satisfy multiple explicit and simultaneous statistical guarantees (e.g., upper-bounded error rates), while also optimizing any number of additional, unconstrained objectives (e.g., total run-time cost). Building on recent results in distribution-free, finite-sample risk control for general losses, we propose Pareto Testing: a two-stage process which combines multi-objective optimization with multiple hypothesis testing. The optimization stage constructs a set of promising combinations on the Pareto frontier. We then apply statistical testing to this frontier only to identify configurations that have (i) high utility with respect to our objectives, and (ii) guaranteed risk levels with respect to our constraints, with specifiable high probability. We demonstrate the effectiveness of our approach to reliably accelerate the execution of large-scale Transformer models in natural language processing (NLP) applications. In particular, we show how Pareto Testing can be used to dynamically configure multiple inter-dependent model attributes – including the number of layers computed before exiting, number of attention heads pruned, or number of text tokens considered – to simultaneously control and optimize various accuracy and cost metrics.


Constrained Multi-Objective Optimization for Automated Machine Learning

Automated machine learning has gained a lot of attention recently. Build...

All unconstrained strongly convex problems are weakly simplicial

A multi-objective optimization problem is C^r weakly simplicial if there...

Self-Evolutionary Optimization for Pareto Front Learning

Multi-task learning (MTL), which aims to improve performance by learning...

Learning the Pareto Front with Hypernetworks

Multi-objective optimization problems are prevalent in machine learning....

Pareto Navigation Gradient Descent: a First-Order Algorithm for Optimization in Pareto Set

Many modern machine learning applications, such as multi-task learning, ...

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

We introduce Learn then Test, a framework for calibrating machine learni...

Budgeted Classification with Rejection: An Evolutionary Method with Multiple Objectives

Classification systems are often deployed in resource-constrained settin...

Please sign up or login with your details

Forgot password? Click here to reset