Cloud-Agnostic and Lightweight Big Data Processing Platform in Multiple Clouds Using Docker Swarm and Terraform

Research output: Chapter in Book/Published conference outputConference publication

Abstract

Big data necessitates the processing of data at scale and efficiency, which is beyond the capabilities of traditional IT infrastructure. Cloud-based big data processing platforms are the most efficient and established infrastructure to accomplish the big data processing requirements. Now organisations are recognising the requirement of a multi-cloud big data processing platform, as it has several benefits such as lower risk, resilience, flexibility, cost-effectiveness, and performance efficiency. However, the orchestration process of a multi-cloud platform is a very complex process due to the interoperability issue between clouds. Therefore, this paper proposes a distinctive application of Docker Swarm in building a cloud-agnostic and lightweight big data processing platform with the assistance of Infrastructure as Code (IaC) tool Terraform. It presents the architectural design and simulated implementation of the proposed big data processing platform in multiple clouds using Docker Swarm. Then, it illustrates the process of making this Docker-based big data platform, a cloud-agnostic and lightweight platform by using Terraform. Subsequently, it demonstrates the reason for using Terraform by comparing it with popular IaC tools. Finally, it performs a comparative evaluation of the proposed containerization based platform with the traditional virtualization based platform to expound the relative efficacy of the Docker Swarm-based big data processing platform.
Original languageEnglish
Title of host publicationUK Workshop on Computational Intelligence (UKCI 2021)
EditorsT. Jansen, R. Jensen, N. Mac Parthaláin, Lin C.M.
PublisherSpringer
Pages519-531
ISBN (Electronic)978-3-030-87094-2
ISBN (Print)978-3-030-87093-5
DOIs
Publication statusPublished - 17 Nov 2021
EventUK Workshop on Computational Intelligence (UKCI 2021) - Aberystwyth, United Kingdom
Duration: 8 Sept 202110 Sept 2021

Publication series

NameAdvances in Intelligent Systems and Computing
PublisherSpringer
Volume1409
ISSN (Print)2194-5357

Conference

ConferenceUK Workshop on Computational Intelligence (UKCI 2021)
Abbreviated titleUKCI 2021
Country/TerritoryUnited Kingdom
CityAberystwyth
Period8/09/2110/09/21

Keywords

  • Big Data Processing Platform
  • Docker Swarm
  • Infrastructure as Code
  • IaC
  • TerraForm
  • Container
  • Virtual Machine
  • Containerization
  • Virtualization

Fingerprint

Dive into the research topics of 'Cloud-Agnostic and Lightweight Big Data Processing Platform in Multiple Clouds Using Docker Swarm and Terraform'. Together they form a unique fingerprint.

Cite this