AuthorsZ. Vrba, P. Halvorsen, C. Griwodz, P. Beskow, H. Espeland and D. Johansen
TitleThe Nornir Run-Time System for Parallel Programs Using Kahn Process Networks on Multi-Core Machines - a Flexible Alternative to MapReduce
AfilliationMedia, Communication Systems
StatusPublished
Publication TypeJournal Article
Year of Publication2010
JournalThe Journal of Supercomputing
NumberOnline first
Pagination1-27
Abstract

Even though shared-memory concurrency is a paradigm frequently used for developing parallel applications on small- and middle-sized machines, experience has shown that it is hard to use. This is largely caused by synchro- nization primitives which are low-level, inherently nondeterministic, and, consequently, non-intuitive to use. In this paper, we present the Nornir run-time system. Nornir is comparable to well-known frameworks such as MapReduce and Dryad that are recognized for their efficiency and simplicity. Unlike these frameworks, Nornir also supports process structures containing branches and cycles. Nornir is based on the formalism of Kahn process networks, which is a shared-nothing, message-passing model of concurrency. We deem this model a simple and deterministic alternative to shared-memory concurrency. Experiments with real and synthetic benchmarks on up to 8 CPUs show that per- formance in most cases scales almost linearly with the number of CPUs, when not limited by data dependencies. We also show that the modeling flexibility allows Nornir to outperform its MapReduce counter-parts using well-known benchmarks.

Notes

In print: January 2013, Volume 63, Issue 1, pp 191-217

DOI10.1007/s11227-010-0503-2
Citation KeySimula.mpg.30