3 ms·
A minute to sort 1TB on a system with 11TB of ram?
by bayareaguy 17y ago
A minute to sort 1TB on a system with 11TB of ram?
- wmf 17y agoIIRC the rules require the data to be read from disk, sorted, and written back to disk, so that's at least 32GB/s of disk I/O by my calculations. Also, the cluster has fairly weak bisection bandwidth.
- bayareaguy 17y ago1000+ machines to sort 1TB should work out to each machine reading, sorting and writing rows around 32MB/s on machines with 4 SATA disks per node. While it is kind of neat that Hadoop makes it easy to run big jobs on clusters almost nobody can afford, it doesn't look like it's very efficient. For comparison, check out http://www.ordinal.com http://www.ordinal.com
- oconnor0 17y agoIs their minute to sort 34 GB on 1 machine? I couldn't tell.