Analyzing Efficient Stream Processing on Modern Hardware

Publication

Steffen Zeuch, Bonaventura Del Monte, Jeyhun Karimov, Clemens Lutz, Manuel Renz, Jonas Traub, Sebastian Breß, Tilmann Rabl, Volker Markl

In: Proceedings of the VLDB Endowment (PVLDB) 12 5 Pages 516-530 VLDB Endowment 2019.

Abstract

Modern Stream Processing Engines (SPEs) process large data volumes under tight latency constraints. Many SPEs execute processing pipelines using message passing on sharednothing architectures and apply a partition-based scale-out strategy to handle high-velocity input streams. Furthermore, many state-of-the-art SPEs rely on a Java Virtual Machine to achieve platform independence and speed up system development by abstracting from the underlying hardware. In this paper, we show that taking the underlying hardware into account is essential to exploit modern hardware efficiently. To this end, we conduct an extensive experimental analysis of current SPEs and SPE design alternatives optimized for modern hardware. Our analysis highlights potential bottlenecks and reveals that state-of-the-art SPEs are not capable of fully exploiting current and emerging hardware trends, such as multi-core processors and high-speed networks. Based on our analysis, we describe a set of design changes to the common architecture of SPEs to scale-up on modern hardware. We show that the single-node throughput can be increased by up to two orders of magnitude compared to state-of-the-art SPEs by applying specialized code generation, fusing operators, batch-style parallelization strategies, and optimized windowing. This speedup allows for deploying typical streaming applications on a single or a few nodes instead of large clusters.

Projekte

E2Data - European Extreme Performing Big Data Stacks,
BBDCII - Berliner Big Data Center

zeuch-analyzing-efficient-stream-processing-on-modern-hardware.pdf (pdf, 485 KB )

Analyzing Efficient Stream Processing on Modern Hardware

Abstract

Projekte

Follow us on:

Editor

Newsletter

Locations