Docker container-based big data processing system in multiple clouds for everyone

Research output: Chapter in Book/Report/Conference proceedingConference publication

Abstract

Big data processing is progressively becoming essential for everyone to extract the meaningful information from their large volume of data irrespective of types of users and their application areas. Big data processing is a broad term and includes several operations such as the storage, cleaning, organization, modelling, analysis and presentation of data at a scale and efficiency. For ordinary users, the significant challenges are the requirement of the powerful data processing system and its provisioning, installation of complex big data analytics and difficulty in their usage. Docker is a container-based virtualization technology and it has recently introduced Docker Swarm for the development of various types of multi-cloud distributed systems, which can be helpful in solving all above problems for ordinary users. However, Docker is predominantly used in the software development industry, and less focus is given to the data processing aspect of this container-based technology. Therefore, this paper proposes the Docker container-based big data processing system in multiple clouds for everyone, which explores another potential dimension of Docker for big data analysis. This Docker container-based system is an inexpensive and user-friendly framework for everyone who has the knowledge of basic IT skills. Additionally, it can be easily developed on a single machine, multiple machines or multiple clouds. This paper demonstrates the architectural design and simulated development of the proposed Docker container-based big data processing system in multiple clouds. Subsequently, it illustrates the automated provisioning of big data clusters using two popular big data analytics, Hadoop and Pachyderm (without Hadoop) including the Web-based GUI interface Hue for easy data processing in Hadoop.
Original languageEnglish
Title of host publication2017 IEEE International Systems Engineering Symposium (ISSE)
PublisherIEEE
Pages1-7
Number of pages7
ISBN (Electronic)978-1538634035
ISBN (Print)978-1538634042
DOIs
Publication statusPublished - 30 Oct 2017
Event2017 IEEE International Systems Engineering Symposium (ISSE) - Vienna, Austria
Duration: 11 Oct 201713 Oct 2017

Conference

Conference2017 IEEE International Systems Engineering Symposium (ISSE)
CountryAustria
CityVienna
Period11/10/1713/10/17

Bibliographical note

© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Fingerprint Dive into the research topics of 'Docker container-based big data processing system in multiple clouds for everyone'. Together they form a unique fingerprint.

Cite this