3 ms·
1000+ machines to sort 1TB should work out to each machine reading, sorting and writing rows around 32MB/s on machines with 4 SATA disks per node. While it is
by bayareaguy 17y ago
1000+ machines to sort 1TB should work out to each machine reading, sorting and writing rows around 32MB/s on machines with 4 SATA disks per node.
While it is kind of neat that Hadoop makes it easy to run big jobs on clusters almost nobody can afford, it doesn't look like it's very efficient.
For comparison, check out http://www.ordinal.com http://www.ordinal.com
- oconnor0 17y agoIs their minute to sort 34 GB on 1 machine? I couldn't tell.