AN EFFICIENT TRAFFIC-AWARE SEPARATION AND AGGRIGATION USING MAPREDUCE FOR BIG DATA APPLICATIONS

Abstract

The MapReduce programming model simplifies the processing large datasets. Mapreduce is typically used to do distributed computing on cluster of computers, exploiting parallel map tasks and reduce tasks. Although many efforts have been made to improve the performance of MapReduce jobs, they ignore the network traffic generated in the shuffle phase, which plays a critical role in performance enhancement. Traditionally, a hash function is used to separate intermediate data among reduce tasks however, it is not traffic-efficient. In this paper, we study to reduce network traffic cost for a MapReduce job by designing a novel intermediate data separation scheme. A decomposition-based distributed algorithm is proposed to deal
with the large-scale optimization problem for big data application and an online algorithm is also designed to adjust data separation and aggregation in a dynamic manner. Finally, extensive simulation results demonstrate that our proposals can significantly reduce network
traffic cost.