5 ms·
Yes, but the key difference here is that you can download the Hadoop sources and try it out Yahoo's way yourself.
by tc 17y ago
Yes, but the key difference here is that you can download the Hadoop sources and try it out Yahoo's way yourself.
- neilc 17y agoSure, you also just need 3600 machines.
- diego 17y agoYou don't have to sort a terabyte in 62 seconds, you could do it on 10 machines and it would probably take you less than a day. If you are a company that needs to sort a petabyte once in a while and your data resides on the Amazon cloud, you could get 3600 high-cpu EC2 instances and do it for about $12k. A petabyte is on the order of 100k bytes for every person on the planet. If you have that much data that you need to sort in less than a day, you can afford it.
- neilc 17y agoI'd actually be curious to see what would happen if you tried to sort a petabyte with 3800 EC2 nodes. I wonder how much worse your performance would be? EC2 instance-local storage is pretty slow by default (the "first-write" problem), but if you used multiple EBS volumes on each node you might be able to get pretty good I/O performance.
- sanswork 17y agoJust a small correction. Using their mapreduce large instances it would cost you less that $3500.