An enhanced parallelisation model for performance prediction of apache spark on a multinode hadoop cluster

Nasim Ahmed*, Andre L.C. Barczak, Mohammad A. Rashid, Teo Susnjak

*Corresponding author for this work

Research output: Contribution to journalArticleResearchpeer-review

2 Citations (Scopus)
1 Downloads (Pure)

Abstract

Big data frameworks play a vital role in storing, processing, and analysing large datasets. Apache Spark has been established as one of the most popular big data engines for its efficiency and reliability. However, one of the significant problems of the Spark system is performance prediction. Spark has more than 150 configurable parameters, and configuration of so many parameters is challenging task when determining the suitable parameters for the system. In this paper, we proposed two distinct parallelisation models for performance prediction. Our insight is that each node in a Hadoop cluster can communicate with identical nodes, and a certain function of the non-parallelisable runtime can be estimated accordingly. Both models use simple equations that allows us to predict the runtime when the size of the job and the number of executables are known. The proposed models were evaluated based on five HiBench workloads, Kmeans, PageRank, Graph (NWeight), SVM, and WordCount. The workload’s empirical data were fitted with one of the two models meeting the accuracy requirements. Finally, the experimental findings show that the model can be a handy and helpful tool for scheduling and planning system deployment.

Original languageEnglish
Article number65
JournalBig Data and Cognitive Computing
Volume5
Issue number4
DOIs
Publication statusPublished - Dec 2021
Externally publishedYes

Fingerprint

Dive into the research topics of 'An enhanced parallelisation model for performance prediction of apache spark on a multinode hadoop cluster'. Together they form a unique fingerprint.

Cite this