Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing

by   Tyler Cody, et al.

This paper demonstrates the systematic use of combinatorial coverage for selecting and characterizing test and training sets for machine learning models. The presented work adapts combinatorial interaction testing, which has been successfully leveraged in identifying faults in software testing, to characterize data used in machine learning. The MNIST hand-written digits data is used to demonstrate that combinatorial coverage can be used to select test sets that stress machine learning model performance, to select training sets that lead to robust model performance, and to select data for fine-tuning models to new domains. Thus, the results posit combinatorial coverage as a holistic approach to training and testing for machine learning. In contrast to prior work which has focused on the use of coverage in regard to the internal of neural networks, this paper considers coverage over simple features derived from inputs and outputs. Thus, this paper addresses the case where the supplier of test and training sets for machine learning models does not have intellectual property rights to the models themselves. Finally, the paper addresses prior criticism of combinatorial coverage and provides a rebuttal which advocates the use of coverage metrics in machine learning applications.


page 1

page 3


Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error

Machine learning models are increasingly used in practice. However, many...

MLCheck- Property-Driven Testing of Machine Learning Models

In recent years, we observe an increasing amount of software with machin...

Training on test data: Removing near duplicates in Fashion-MNIST

MNIST and Fashion MNIST are extremely popular for testing in the machine...

Identifying Model Weakness with Adversarial Examiner

Machine learning models are usually evaluated according to the average c...

Test Selection for Deep Learning Systems

Testing of deep learning models is challenging due to the excessive numb...

Quality of Data in Machine Learning

A common assumption exists according to which machine learning models im...

Combinatorial Sequence Testing Using Behavioral Programming and Generalized Coverage Criteria

This paper tackles three main issues regarding test design: (1) it propo...

Please sign up or login with your details

Forgot password? Click here to reset