3 ms·
Previously on HN, a similar approach demonstrating that command-line tools can be hundreds of times faster than the fancy distributed approach (for certain clas
by jmcphers 8y ago
Previously on HN, a similar approach demonstrating that command-line tools can be hundreds of times faster than the fancy distributed approach (for certain classes of problems):
"Command-line tools can be faster than your Hadoop cluster"
https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462
- girst 8y agoIt's not just about "raw" speed, but also about how much mental time you spend thinking about your setup. even if you didn't know about zgrep already, that's at most minutes of thinking ahead vs. potential hours or days setting up cloud providers, big toolchains, and whatever else just to get started. since we're throwing links around, the (tangentially related) Knuth vs. McIlroy saga comes to mind: https://news.ycombinator.com/item?id=15264997 https://news.ycombinator.com/item?id=15264997
- obvious_throwaw 8y ago> even if you didn't know about zgrep already Interestingly, the dumps come with a brief README file telling you... to use the zgrep and example arguments :)
- saberience 8y agoHadoop is intended to be used if you have exceeded terabyte scale of data and reached petabytes. That is when simply moving the data around has become incredibly difficult due to network IO limitations. Let me know when you have a solution for using command line tools for sorting through our 4PB of data in our Hadoop cluster! Use the right tools for the right jobs :)
- patrickg_zill 8y agoYou don't have any indexes?