Towards Automatic Parameter Tuning of Stream Processing Systems

M. Bilal, M. Canini
Proceedings of the 2017 Symposium on Cloud Computing, pp.189-20, (2017)

Towards Automatic Parameter Tuning of Stream Processing Systems

Keywords

Streaming applications, Streaming systems

Abstract

‚ÄčOptimizing the performance of big-data streaming applications has become a daunting and time-consuming task: parameters may be tuned from a space of hundreds or even thousands of possible configurations. In this paper, we present a framework for automating parameter tuning for stream-processing systems. Our framework supports standard black-box optimization algorithms as well as a novel gray-box optimization algorithm. We demonstrate the multiple benefits of automated parameter tuning in optimizing three benchmark applications in Apache Storm. Our results show that a hill-climbing algorithm that uses a new heuristic sampling approach based on Latin Hypercube provides the best results. Our gray-box algorithm provides comparable results while being two to five times faster.

Code

DOI: 10.1145/3127479.3127492

‚Äč

Sources

Website PDF

See all publications 2017