8 ms·
0) Get your data into Amazon's cloud, and rent a server on EC2 with 68GB of memory. Spot instances are a buck an hour. 1) Do sort -u --buffer-size=60GB FILE
by lsb 16y ago
0) Get your data into Amazon's cloud, and rent a server on EC2 with 68GB of memory. Spot instances are a buck an hour.
1) Do sort -u --buffer-size=60GB FILE. You'll sort it all in memory, and that'll be a great speedup.
It's easier to scale up than scale out if you have the money, and your dataset fits in memory, so don't bother with Hadoop for something as simple as that. What do you want to do after you get the uniques?
- trin_ 16y agobut wouldnt that also include uploading 35gb of data and finally downloading xxGB (result size)? i think when you add this time you can just take your multicore desktop and let it run over night.