16 ms·
If my math is right: (68sec * 1024) / 60 / 60 / 4 --- (68 sec for 1TB) (1024 to make 1TB) /60/60 (hours) and /4 as there are 4x as many computers.... So ideal
by jwilliams 18y ago
If my math is right:
(68sec * 1024) / 60 / 60 / 4
--- (68 sec for 1TB) (1024 to make 1TB) /60/60 (hours) and /4 as there are 4x as many computers....
So ideally taking the 1TB sort to 1PB on 4x the hardware would be 4.83hrs.
Google's 1PB sort was in 6 hours - So that's fairly linear. Impressive.
- jmtulloss 18y agoThe article implied they were using hard disk sizes. IE. 1 Petabyte is 1000 Terabytes. Not that it makes that much of a difference.
- lutorm 18y agoIsn't the mapreduce sort NlogN in the number of entries? That would make 1e13 log 1e13/(1e10 log 1e10) 68s/4 = 6.1h. So they were actually faster than the possible scaling? Doesn't sound right...
- russell 18y agoSounds dead on to me. The typical sort is O(N logN). O can be slightly different between the 1T runs and 1P runs, e.g. slightly difference latency or pattern of disk failures. Every sort is sensitive to the data distribution. There is probably an interesting distribution of results across the runs.
- bd 18y agoThey mention that they did some extra optimizations ("stragglers") for petabyte test (that were not there yet for the terabyte test).
- jwilliams 18y agoThis is a really good point. I was assuming that the sort was actually a partial sort - returning the top "x" (admittedly - even then isn't linear). However, I was also assuming (n << x), which would almost certainly be the case with a large data set. It's a fairly common approach, but not sure why I automatically assumed it. If the sort is complete, then yeah, that changes things.