Overview
Stream analytics on real-time events has an insatiable demand for high throughput and low latency. StreamBox is a novel stream processing engine optimized for multicore and hybrid memory.
StreamBox on a multicore machine
-
Exploits the parallelism and memory hierarchies in modern multicore hardware
-
Supports out-of-order data processing
-
Scales to a large number (56) of cores
-
Achieves throughput on a par with distributed engines on medium-size clusters
-
Delivers latencies of the tens of milliseconds, which are 20× shorter than popular distributed streaming engines
StreamBox on high bandwidth hybrid memory
-
Exploits the emerging hybrid memories combining commodity DDR4 DRAM and 3D-stacked DRAM
-
Replaces Hash with Sort using high task parallelism, wide SIMD (AVX-512), and sequential memory access to harness abundant bandwidth of 3D-stacked DRAM
-
Dynamically extracts Key Pointer Aarry (KPA) from full records on DRAM to 3D-stacked DRAM to minimize the use of the precious capacity
-
Dynamically manages KPA placement based on both 3D-stacked DRAM’s high bandwidth and limited capacity, and normal DRAM’s high capacity and limited bandwidth
-
Achieves the best reported records per second for streaming with ingress on a single machine (Intel Knights Landing)
Related Publication
-
[USENIX ATC’17] "StreamBox: Modern Stream Processing on a Multicore Machine," Hongyu Miao, Heejin Park, Myeongjae Jeon, Gennady Pekhimenko, Kathryn S. McKinley, and Felix Xiaozhu Lin, in Proc. of USENIX Annual Technical Conference, July 2017.
-
[ASPLOS’19] "StreamBox-HBM: Stream Analytics on High Bandwidth Hybrid Memory," Hongyu Miao, Myeongjae Jeon, Gennady Pekhimenko, Kathryn S. McKinley, and Felix Xiaozhu Lin, in Proc. of the 24th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, April 2019.
-
[USENIX ATC’19] "StreamBox-TZ: Secure Stream Analytics at the Edge with TrustZone," Heejin Park, Shuang Zhai, Long Lu, and Felix Xiaozhu Lin, in Proc. of USENIX Annual Technical Conference, July 2019.
Source Code on a Multicore Machine
Where
wget ftp://ftp.ecn.purdue.edu/xzl/software/streambox/streambox-last.tar.gz
Build & Run
-
Dependencies
/*Ubuntu 16.04*/
$sudo apt-get install
g++ \
libtbb-dev \
automake \
autoconf \
autoconf-archive \
libtool \
libboost-all-dev \
libevent-dev \
libdouble-conversion-dev \
libgoogle-glog-dev \
libgflags-dev \
liblz4-dev \
liblzma-dev \
libsnappy-dev \
make \
zlib1g-dev \
binutils-dev \
libjemalloc-dev \
libssl-dev
-
Set compilation parameters
Edit /path/to/streambox/CMakeLists.txt
-
Build an application
$cd /path/to/streambox/ $/usr/bin/cmake -DCMAKE_BUILD_TYPE=Release -G "CodeBlocks - Unix Makefiles" \ /path/to/streambox /*Build Wordcount*/ $make test-wc.bin
-
Run an application
$cd /path/to/streambox $./test-wc.bin
Source Code on Intel Knights Landing with Hybrid Memory
Where
wget ftp://ftp.ecn.purdue.edu/xzl/software/streambox/streambox-knl.tar.gz
Build & Run
Please check out the HOW_TO_RUN_BENCHMARKS in source code for detailed instructions on how to compile and run all benchmarks.
Source Code on HiKey Board with TrustZone
Where
wget ftp://ftp.ecn.purdue.edu/xzl/software/streambox/streambox-tz.tar.gz
Build & Run
Please check out the README for detailed instructions on how to compile both normal and secure world and run all benchmarks.