On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHub

by   Guoming Long, et al.

The tremendous success of Deep Learning (DL) has significantly boosted the number of open-sourced DL frameworks hosted on GitHub. Among others, performance and accuracy bugs are critical factors that affect the reputation of these DL frameworks, therefore understanding the practice of discovering and investigating them for DL is important. In this paper, we conduct an exploratory study on the nature of reporting performance and accuracy bugs bugs for DL frameworks, aiming to improve our knowledge on this topic. Our study covers 10 most popular open-sourced DL frameworks on GitHub (e.g., TensorFlow, Keras, and PyTorch), based on which we sample 664 representative performance and accuracy bugs bug reports out of a total population of 22,522. Through systematic analysis of these samples, our key findings are: (1) low speed is the primary reason that a performance bug related report is submitted but we see no consistent pattern for accuracy related ones; (2) most of the reports are about issues encountered in the training stage; (3) only a small proportion of the reports provide insufficient information to investigate; (4) the majority of the performance and accuracy bugs bug reports (from 69 are not related to the actual bug or regarded as unclassified; (5) around 50 of the performance and accuracy bug reports, which indeed reveal bugs, are not resolved by direct patches. Deriving from the above, we discuss a set of actionable implications to the researchers, maintainers, and report submitters on this subject. To promote open science, the labeled dataset has been made publicly available at https://tinyurl.com/4x3tap9w.


page 3

page 7


Multifaceted Hierarchical Report Identification for Non-Functional Bugs in Deep Learning Frameworks

Non-functional bugs (e.g., performance- or accuracy-related bugs) in Dee...

Toward Understanding Deep Learning Framework Bugs

DL frameworks are the basis of constructing all DL programs and models, ...

Self-Claimed Assumptions in Deep Learning Frameworks: An Exploratory Study

Deep learning (DL) frameworks have been extensively designed, implemente...

Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow

Deep Learning (DL) frameworks are now widely used, simplifying the creat...

Too Few Bug Reports? Exploring Data Augmentation for Improved Changeset-based Bug Localization

Modern Deep Learning (DL) architectures based on transformers (e.g., BER...

Challenges in Migrating Imperative Deep Learning Programs to Graph Execution: An Empirical Study

Efficiency is essential to support responsiveness w.r.t. ever-growing da...

Characterizing Performance Bugs in Deep Learning Systems

Deep learning (DL) has been increasingly applied to a variety of domains...

Please sign up or login with your details

Forgot password? Click here to reset