Dataset Management Platform for Machine Learning

by   Ze Mao, et al.

The quality of the data in a dataset can have a substantial impact on the performance of a machine learning model that is trained and/or evaluated using the dataset. Effective dataset management, including tasks such as data cleanup, versioning, access control, dataset transformation, automation, integrity and security, etc., can help improve the efficiency and speed of the machine learning process. Currently, engineers spend a substantial amount of manual effort and time to manage dataset versions or to prepare datasets for machine learning tasks. This disclosure describes a platform to manage and use datasets effectively. The techniques integrate dataset management and dataset transformation mechanisms. A storage engine is described that acts as a source of truth for all data and handles versioning, access control etc. The dataset transformation mechanism is a key part to generate a dataset (snapshot) to serve different purposes. The described techniques can support different workflows, pipelines, or data orchestration needs, e.g., for training and/or evaluation of machine learning models.


page 1

page 2

page 3

page 4


Replicate or Relocate? Non-Uniform Access in Parameter Servers

Parameter servers (PSs) facilitate the implementation of distributed tra...

The CSIRO Crown-of-Thorn Starfish Detection Dataset

Crown-of-Thorn Starfish (COTS) outbreaks are a major cause of coral loss...

Jenkins Pipelines: A Novel Approach to Machine Learning Operations (MLOps)

Machine Learning is a widely popular field that is being used in an incr...

A Data Quality-Driven View of MLOps

Developing machine learning models can be seen as a process similar to t...

MSC: A Dataset for Macro-Management in StarCraft II

Macro-management is an important problem in StarCraft, which has been st...

Study on the Data Storage Technology of Mini-Airborne Radar Based on Machine Learning

The data rate of airborne radar is much higher than the wireless data tr...

The Trip to The Enterprise Gourmet Data Product Marketplace through a Self-service Data Platform

Data Analytics provides core business reporting needs in many software c...

Please sign up or login with your details

Forgot password? Click here to reset