Dead or Alive: Continuous Data Profiling for Interactive Data Science

by   Will Epperson, et al.

Profiling data by plotting distributions and analyzing summary statistics is a critical step throughout data analysis. Currently, this process is manual and tedious since analysts must write extra code to examine their data after every transformation. This inefficiency may lead to data scientists profiling their data infrequently, rather than after each transformation, making it easy for them to miss important errors or insights. We propose continuous data profiling as a process that allows analysts to immediately see interactive visual summaries of their data throughout their data analysis to facilitate fast and thorough analysis. Our system, AutoProfiler, presents three ways to support continuous data profiling: it automatically displays data distributions and summary statistics to facilitate data comprehension; it is live, so visualizations are always accessible and update automatically as the data updates; it supports follow up analysis and documentation by authoring code for the user in the notebook. In a user study with 16 participants, we evaluate two versions of our system that integrate different levels of automation: both automatically show data profiles and facilitate code authoring, however, one version updates reactively and the other updates only on demand. We find that both tools facilitate insight discovery with 91 originating from the tools rather than manual profiling code written by users. Participants found live updates intuitive and felt it helped them verify their transformations while those with on-demand profiles liked the ability to look at past visualizations. We also present a longitudinal case study on how AutoProfiler helped domain scientists find serendipitous insights about their data through automatic, live data profiles. Our results have implications for the design of future tools that offer automated data analysis support.


EDAssistant: Supporting Exploratory Data Analysis in Computational Notebooks with In-Situ Code Search and Recommendation

Using computational notebooks (e.g., Jupyter Notebook), data scientists ...

SeeChart: Enabling Accessible Visualizations Through Interactive Natural Language Interface For People with Visual Impairments

Web-based data visualizations have become very popular for exploring dat...

The Quest for Omnioculars: Embedded Visualization for Augmenting Basketball Game Viewing Experiences

Sports game data is becoming increasingly complex, often consisting of m...

Notable: On-the-fly Assistant for Data Storytelling in Computational Notebooks

Computational notebooks are widely used for data analysis. Their interle...

code::proof: Prepare for most weather conditions

Computational tools for data analysis are being released daily on reposi...

One DSL to Rule Them All: IDE-Assisted Code Generation for Agile Data Analysis

Data analysis is at the core of scientific studies, a prominent task tha...

Themisto: Towards Automated Documentation Generation in Computational Notebooks

Computational notebooks allow data scientists to express their ideas thr...

Please sign up or login with your details

Forgot password? Click here to reset