Performance Evaluation of Big Data Processing Strategies for Neuroimaging

by   Valérie Hayot-Sasson, et al.

Neuroimaging datasets are rapidly growing in size as a result of advancements in image acquisition methods, open-science and data sharing. However, it remains that current neuroimaging data processing engines do not employ strategies adopted by Big Data processing engines to improve processing time. Here, we evaluate three Big Data processing strategies (in-memory computing, data-locality and lazy-evaluation) on typical neuroimaging use cases, represented by the BigBrain dataset. We contrast the various processing strategies by using Apache Spark and Nipype as our representative Big Data and neuroimaging processing engines on Dell EMC's Top-500 cluster. Big Data thresholds were modeled by comparing the data/compute ratio of the application to the filesystem/concurrent processes ratio. This model acknowledges the fact that page caching provided by the Linux kernel is critical to the performance of Big Data applications. Results show that in-memory computing alone speeds-up executions by a factor of up to 1.6, whereas when combined with data locality, this factor reaches 5.3. Lazy evaluation strategies were found to increase the likelihood of cache hits, further improving processing time. Such important speed-up values are likely to be observed on typical image processing operations performed on images of size larger than 75GB. A ballpark speculation from our model showed that in-memory computing alone will not speed-up current functional MRI analyses unless coupled with data locality and processing around 280 subjects concurrently. In addition, we observe that emulating in-memory computing using in-memory file systems (tmpfs) does not reach the performance of an in-memory engine, presumably due to swapping to disk and the lack of data cleanup. We conclude that Big Data processing strategies are worth developing for neuroimaging applications.


A Survey on Spark Ecosystem for Big Data Processing

With the explosive increase of big data in industry and academic fields,...

A performance comparison of Dask and Apache Spark for data-intensive neuroimaging pipelines

In the past few years, neuroimaging has entered the Big Data era due to ...

Modeling the Linux page cache for accurate simulation of data-intensive applications

The emergence of Big Data in recent years has resulted in a growing need...

Faasm: Lightweight Isolation for Efficient Stateful Serverless Computing

Serverless computing is an excellent fit for big data processing because...

Intermediate Data Caching Optimization for Multi-Stage and Parallel Big Data Frameworks

In the era of big data and cloud computing, large amounts of data are ge...

Performance Evaluation of an Algorithm-based Asynchronous Checkpoint-Restart Fault Tolerant Application Using Mixed MPI/GPI-2

One of the hardest challenges of the current Big Data landscape is the l...

TRANSMUT-SPARK: Transformation Mutation for Apache Spark

We propose TRANSMUT-Spark, a tool that automates the mutation testing pr...

Please sign up or login with your details

Forgot password? Click here to reset