Analysis of Failures and Risks in Deep Learning Model Converters: A Case Study in the ONNX Ecosystem

by   Purvish Jajal, et al.

Software engineers develop, fine-tune, and deploy deep learning (DL) models. They use and re-use models in a variety of development frameworks and deploy them on a range of runtime environments. In this diverse ecosystem, engineers use DL model converters to move models from frameworks to runtime environments. However, errors in converters can compromise model quality and disrupt deployment. The failure frequency and failure modes of DL model converters are unknown. In this paper, we conduct the first failure analysis on DL model converters. Specifically, we characterize failures in model converters associated with ONNX (Open Neural Network eXchange). We analyze past failures in the ONNX converters in two major DL frameworks, PyTorch and TensorFlow. The symptoms, causes, and locations of failures (for N=200 issues), and trends over time are also reported. We also evaluate present-day failures by converting 8,797 models, both real-world and synthetically generated instances. The consistent result from both parts of the study is that DL model converters commonly fail by producing models that exhibit incorrect behavior: 33 of converted models fell into this category. Our results motivate future research on making DL software simpler to maintain, extend, and validate.


An Empirical Study of Challenges in Converting Deep Learning Models

There is an increase in deploying Deep Learning (DL)-based software syst...

An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms

Deep Learning (DL) has recently achieved tremendous success. A variety o...

Self-Claimed Assumptions in Deep Learning Frameworks: An Exploratory Study

Deep learning (DL) frameworks have been extensively designed, implemente...

Ivy: Templated Deep Learning for Inter-Framework Portability

We introduce Ivy, a templated Deep Learning (DL) framework which abstrac...

Systematic Evaluation of Deep Learning Models for Failure Prediction

With the increasing complexity and scope of software systems, their depe...

Comparing the costs of abstraction for DL frameworks

High level abstractions for implementing, training, and testing Deep Lea...

DeepStability: A Study of Unstable Numerical Methods and Their Solutions in Deep Learning

Deep learning (DL) has become an integral part of solutions to various i...

Please sign up or login with your details

Forgot password? Click here to reset