SparkEdgeEmu: An Emulation Framework for Edge-Enabled Apache Spark Deployments

Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Edge Computing emerges as a stable and efficient solution for IoT data processing and analytics. With big data distributed engines to be deployed on edge infrastructures, users seek solutions to evaluate the performance of their analytics queries. In this paper, we introduce SparkEdgeEmu, an interactive framework designed for researchers and practitioners who need to inspect the performance of Spark analytic jobs without the edge topology setup burden. SparkEdgeEmu provides: (i) parameterizable template-based use cases for edge infrastructures, (ii) real-time emulated environments serving ready-to-use Spark clusters, (iii) a unified and interactive programming interface for the framework’s execution and query submission, and (vi) utilization metrics from the underlying emulated topology as well as performance and quantitative metrics from the deployed queries. We evaluate the usability of our framework in a smart city use case and extract useful performance hints for the Apache Spark code execution.

Original languageEnglish
Title of host publicationEuro-Par 2023
Subtitle of host publicationParallel Processing - 29th International Conference on Parallel and Distributed Computing, Proceedings
EditorsJosé Cano, Marios D. Dikaiakos, George A. Papadopoulos, Miquel Pericàs, Rizos Sakellariou
PublisherSpringer Science and Business Media Deutschland GmbH
Pages154-168
Number of pages15
ISBN (Print)9783031396977
DOIs
Publication statusPublished - 2023
Event29th International European Conference on Parallel and Distributed Computing, Euro-Par 2023 - Limassol, Cyprus
Duration: 28 Aug 20231 Sept 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14100 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference29th International European Conference on Parallel and Distributed Computing, Euro-Par 2023
Country/TerritoryCyprus
CityLimassol
Period28/08/231/09/23

Keywords

  • Big Data
  • Edge Computing
  • Internet of Things

Fingerprint

Dive into the research topics of 'SparkEdgeEmu: An Emulation Framework for Edge-Enabled Apache Spark Deployments'. Together they form a unique fingerprint.

Cite this