Dryad: distributed data-parallel programs from sequential building blocks

Citation: Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly (2007/03/21) Dryad: distributed data-parallel programs from sequential building blocks. EuroSys (RSS)
DOI (original publisher): 10.1145/1272996.1273005
Semantic Scholar (metadata): 10.1145/1272996.1273005
Sci-Hub (fulltext): 10.1145/1272996.1273005
Internet Archive Scholar (search for fulltext): Dryad: distributed data-parallel programs from sequential building blocks
Download: https://doi.org/10.1145/1272996.1273005
Tagged: Computer Science (RSS) parallel systems (RSS)

Summary

Placeholder

Introduction

• Single-core frequency slowing down, big clusters exist, need to write parallel programs.
• Automatic parallelization is too hard.
• GPU shaders, MapReduce, and parallel databases fix the communication topology, and allow the programmer to supply the nodes.
• Contributions: built a high-performance distributed processing engine (Dryad), evaluation of that engine, and programming interface (low- and high-levels).

Dryad System

• Channels can be implemented with TCP, files, or shared memory, and the choice is transparent to user-code.
• SQL can be mapped to Dryad.

Dryad Graphs

•