funcX: Federated Function as a Service for Science

by   Zhuozhao Li, et al.

funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. funcX's cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints–enabling users to route functions to endpoints based on specific needs. funcX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. funcX implements various container management strategies to execute functions with high performance and efficiency on diverse funcX endpoints. funcX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for funcX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than 130 000 concurrent workers. We show that funcX's container warming-aware routing algorithm can reduce the completion time for 3000 functions by up to 61 in-memory data store can speed up data transfers by up to 3x compared to a shared file system.


page 1

page 2

page 3

page 4


funcX: A Federated Function Serving Fabric for Science

Exploding data volumes and velocities, new computational methods and pla...

Serverless Supercomputing: High Performance Function as a Service for Science

Growing data volumes and velocities are driving exciting new methods acr...

Empirical Investigation of Factors influencing Function as a Service Performance in Different Cloud/Edge System Setups

Experimental data can aid in gaining insights about a system operation, ...

A Multi-Protocol, Secure, and Dynamic Data Storage Integration Frameworkfor Multi-tenanted Science Gateway Middleware

Science gateways are user-centric, end-to-end cyberinfrastructure for ma...

TxForest: A DSL for Concurrent Filestores

Many systems use ad hoc collections of files and directories to store pe...

Proactive Serverless Function Resource Management

This paper introduces a new primitive to serverless language runtimes ca...

BUDAMAF: Data Management in Cloud Federations

Data management has always been a multi-domain problem even in the simpl...

Please sign up or login with your details

Forgot password? Click here to reset