A genetic algorithm-based job scheduling model for big data analytics

Big data analytics (BDA) applications are a new category of software applications that process large amounts of data using scalable parallel processing infrastructure to obtain hidden value. Hadoop is the most mature open-source big data analytics framework, which implements the MapReduce programmin...

Descripción completa

Detalles Bibliográficos
Autores principales: Lu, Qinghua, Li, Shanshan, Zhang, Weishan, Zhang, Lei
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer International Publishing 2016
Materias:
Research
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4923081/
https://www.ncbi.nlm.nih.gov/pubmed/27429611
http://dx.doi.org/10.1186/s13638-016-0651-z
_version_ 1782439679265603584
author Lu, Qinghua
Li, Shanshan
Zhang, Weishan
Zhang, Lei
author_facet Lu, Qinghua
Li, Shanshan
Zhang, Weishan
Zhang, Lei
author_sort Lu, Qinghua
collection PubMed
description Big data analytics (BDA) applications are a new category of software applications that process large amounts of data using scalable parallel processing infrastructure to obtain hidden value. Hadoop is the most mature open-source big data analytics framework, which implements the MapReduce programming model to process big data with MapReduce jobs. Big data analytics jobs are often continuous and not mutually separated. The existing work mainly focuses on executing jobs in sequence, which are often inefficient and consume high energy. In this paper, we propose a genetic algorithm-based job scheduling model for big data analytics applications to improve the efficiency of big data analytics. To implement the job scheduling model, we leverage an estimation module to predict the performance of clusters when executing analytics jobs. We have evaluated the proposed job scheduling model in terms of feasibility and accuracy.
format Online
Article
Text
id pubmed-4923081
institution National Center for Biotechnology Information
language English
publishDate 2016
publisher Springer International Publishing
record_format MEDLINE/PubMed
spelling pubmed-49230812016-07-13 A genetic algorithm-based job scheduling model for big data analytics Lu, Qinghua Li, Shanshan Zhang, Weishan Zhang, Lei EURASIP J Wirel Commun Netw Research Big data analytics (BDA) applications are a new category of software applications that process large amounts of data using scalable parallel processing infrastructure to obtain hidden value. Hadoop is the most mature open-source big data analytics framework, which implements the MapReduce programming model to process big data with MapReduce jobs. Big data analytics jobs are often continuous and not mutually separated. The existing work mainly focuses on executing jobs in sequence, which are often inefficient and consume high energy. In this paper, we propose a genetic algorithm-based job scheduling model for big data analytics applications to improve the efficiency of big data analytics. To implement the job scheduling model, we leverage an estimation module to predict the performance of clusters when executing analytics jobs. We have evaluated the proposed job scheduling model in terms of feasibility and accuracy. Springer International Publishing 2016-06-27 2016 /pmc/articles/PMC4923081/ /pubmed/27429611 http://dx.doi.org/10.1186/s13638-016-0651-z Text en © Lu et al. 2016 Open Access This article is distributed under the termsof the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
spellingShingle Research
Lu, Qinghua
Li, Shanshan
Zhang, Weishan
Zhang, Lei
A genetic algorithm-based job scheduling model for big data analytics
title A genetic algorithm-based job scheduling model for big data analytics
title_full A genetic algorithm-based job scheduling model for big data analytics
title_fullStr A genetic algorithm-based job scheduling model for big data analytics
title_full_unstemmed A genetic algorithm-based job scheduling model for big data analytics
title_short A genetic algorithm-based job scheduling model for big data analytics
title_sort genetic algorithm-based job scheduling model for big data analytics
topic Research
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4923081/
https://www.ncbi.nlm.nih.gov/pubmed/27429611
http://dx.doi.org/10.1186/s13638-016-0651-z
work_keys_str_mv AT luqinghua ageneticalgorithmbasedjobschedulingmodelforbigdataanalytics
AT lishanshan ageneticalgorithmbasedjobschedulingmodelforbigdataanalytics
AT zhangweishan ageneticalgorithmbasedjobschedulingmodelforbigdataanalytics
AT zhanglei ageneticalgorithmbasedjobschedulingmodelforbigdataanalytics
AT luqinghua geneticalgorithmbasedjobschedulingmodelforbigdataanalytics
AT lishanshan geneticalgorithmbasedjobschedulingmodelforbigdataanalytics
AT zhangweishan geneticalgorithmbasedjobschedulingmodelforbigdataanalytics
AT zhanglei geneticalgorithmbasedjobschedulingmodelforbigdataanalytics

Ejemplares similares