Designing Optimal Behavioral Experiments Using Machine Learning

by   Simon Valentin, et al.

Computational models are powerful tools for understanding human cognition and behavior. They let us express our theories clearly and precisely, and offer predictions that can be subtle and often counter-intuitive. However, this same richness and ability to surprise means our scientific intuitions and traditional tools are ill-suited to designing experiments to test and compare these models. To avoid these pitfalls and realize the full potential of computational modeling, we require tools to design experiments that provide clear answers about what models explain human behavior and the auxiliary assumptions those models must make. Bayesian optimal experimental design (BOED) formalizes the search for optimal experimental designs by identifying experiments that are expected to yield informative data. In this work, we provide a tutorial on leveraging recent advances in BOED and machine learning to find optimal experiments for any kind of model that we can simulate data from, and show how by-products of this procedure allow for quick and straightforward evaluation of models and their parameters against real experimental data. As a case study, we consider theories of how people balance exploration and exploitation in multi-armed bandit decision-making tasks. We validate the presented approach using simulations and a real-world experiment. As compared to experimental designs commonly used in the literature, we show that our optimal designs more efficiently determine which of a set of models best account for individual human behavior, and more efficiently characterize behavior given a preferred model. We provide code to replicate all analyses as well as tutorial notebooks and pointers to adapt the methodology to other experimental settings.


page 11

page 12

page 13

page 14

page 15

page 17

page 36

page 39


Bayesian Optimal Experimental Design for Simulator Models of Cognition

Bayesian optimal experimental design (BOED) is a methodology to identify...

Machine learning based co-creative design framework

We propose a flexible, co-creative framework bringing together multiple ...

Robust Experimental Designs for Model Calibration

A computer model can be used for predicting an output only after specify...

Using Machine Learning to Guide Cognitive Modeling: A Case Study in Moral Reasoning

Large-scale behavioral datasets enable researchers to use complex machin...

Computing the Performance of A New Adaptive Sampling Algorithm Based on The Gittins Index in Experiments with Exponential Rewards

Designing experiments often requires balancing between learning about th...

Constrained Submodular Optimization for Vaccine Design

Advances in machine learning have enabled the prediction of immune syste...

The Finite-Horizon Two-Armed Bandit Problem with Binary Responses: A Multidisciplinary Survey of the History, State of the Art, and Myths

In this paper we consider the two-armed bandit problem, which often natu...

Please sign up or login with your details

Forgot password? Click here to reset