Tutorials on Testing Neural Networks

by   Nicolas Berthier, et al.

Deep learning achieves remarkable performance on pattern recognition, but can be vulnerable to defects of some important properties such as robustness and security. This tutorial is based on a stream of research conducted since the summer of 2018 at a few UK universities, including the University of Liverpool, University of Oxford, Queen's University Belfast, University of Lancaster, University of Loughborough, and University of Exeter. The research aims to adapt software engineering methods, in particular software testing methods, to work with machine learning models. Software testing techniques have been successful in identifying software bugs, and helping software developers in validating the software they design and implement. It is for this reason that a few software testing techniques – such as the MC/DC coverage metric – have been mandated in industrial standards for safety critical systems, including the ISO26262 for automotive systems and the RTCA DO-178B/C for avionics systems. However, these techniques cannot be directly applied to machine learning models, because the latter are drastically different from traditional software, and their design follows a completely different development life-cycle. As the outcome of this thread of research, the team has developed a series of methods that adapt the software testing techniques to work with a few classes of machine learning models. The latter notably include convolutional neural networks, recurrent neural networks, and random forest. The tools developed from this research are now collected, and publicly released, in a GitHub repository: <https://github.com/TrustAI/DeepConcolic>, with the BSD 3-Clause licence. This tutorial is to go through the major functionalities of the tools with a few running examples, to exhibit how the developed techniques work, what the results are, and how to interpret them.


page 10

page 20

page 21

page 22

page 25

page 26

page 38


Open Problems in Engineering Machine Learning Systems and the Quality Model

Fatal accidents are a major issue hindering the wide acceptance of safet...

Repository for Reusing Artifacts of Artificial Neural Networks

Artificial Neural Networks (ANNs) replaced conventional software systems...

Towards an interdisciplinary, socio-technical analysis of software ecosystem health

This extended abstract presents the research goals and preliminary resea...

Software Testing for Machine Learning

Machine learning has become prevalent across a wide variety of applicati...

Patching as Translation: the Data and the Metaphor

Machine Learning models from other fields, like Computational Linguistic...

TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing

Machine learning models are notoriously difficult to interpret and debug...

Recurrent Neural Networks for Fuzz Testing Web Browsers

Generation-based fuzzing is a software testing approach which is able to...

Please sign up or login with your details

Forgot password? Click here to reset