Data management and execution systems for the Rubin Observatory Science Pipelines

by   Nate B. Lust, et al.

We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algorithms can be written against. This abstraction layer manages the complexities of the large data volumes expected and allows algorithms to be written independently, yet be tied together automatically into a coherent processing pipeline. This system consists of tools which execute these pipelines by transforming them into execution graphs which contain concrete data stored in the Butler. The pipeline infrastructure is designed to be scalable in nature, allowing execution on environments ranging from a laptop all the way up to multi-facility data centers. This presentation will focus on the data management aspects as well as an overview on the creation of pipelines and the corresponding execution graphs.


page 1

page 2

page 3

page 4


Adding Workflow Management Flexibility to LSST Pipelines Execution

Data processing pipelines need to be executed at scales ranging from sma...

Comparing graph data science libraries for querying and analysing datasets: towards data science queries on graphs

This paper presents an experimental study to compare analysis tools with...

The Vera C. Rubin Observatory Data Butler and Pipeline Execution System

The Rubin Observatory's Data Butler is designed to allow data file locat...

JITA4DS: Disaggregated execution of Data Science Pipelines between the Edge and the Data Centre

This paper targets the execution of data science (DS) pipelines supporte...

An Alternative to Cells for Selective Execution of Data Science Pipelines

Data Scientists often use notebooks to develop Data Science (DS) pipelin...

Putting Data Science Pipelines on the Edge

This paper proposes a composable "Just in Time Architecture" for Data Sc...

A milestone for FaaS pipelines; object storage vs VM-driven data exchange

Serverless functions provide high levels of parallelism, short startup t...

Please sign up or login with your details

Forgot password? Click here to reset