5 ms·

You can actually get a lot just from grep (well, zgrep, which searches compressed files) zgrep -A '@\[^.]\+\.ch' /path/to/collections/**/*.gz will spit ou

by girst 8y ago

You can actually get a lot just from grep (well, zgrep, which searches compressed files) zgrep -A '@\[^.]\+\.ch' /path/to/collections/**/*.gz will spit out (almost) all swiss email addresses. sprinkle some grep -v, wc and sort|uniq here and there and bob's your uncle :-) (I'm in no way trying to dismiss the efforts--my method won't get you as much detail, but is enough to find you and your friends who asked to be searched in the dumps)

jmcphers 8y ago
Previously on HN, a similar approach demonstrating that command-line tools can be hundreds of times faster than the fancy distributed approach (for certain classes of problems): "Command-line tools can be faster than your Hadoop cluster" https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462
- girst 8y ago
  It's not just about "raw" speed, but also about how much mental time you spend thinking about your setup. even if you didn't know about zgrep already, that's at most minutes of thinking ahead vs. potential hours or days setting up cloud providers, big toolchains, and whatever else just to get started. since we're throwing links around, the (tangentially related) Knuth vs. McIlroy saga comes to mind: https://news.ycombinator.com/item?id=15264997 https://news.ycombinator.com/item?id=15264997
  - obvious_throwaw 8y ago
    > even if you didn't know about zgrep already Interestingly, the dumps come with a brief README file telling you... to use the zgrep and example arguments :)
- saberience 8y ago
  Hadoop is intended to be used if you have exceeded terabyte scale of data and reached petabytes. That is when simply moving the data around has become incredibly difficult due to network IO limitations. Let me know when you have a solution for using command line tools for sorting through our 4PB of data in our Hadoop cluster! Use the right tools for the right jobs :)
  - patrickg_zill 8y ago
    You don't have any indexes?
nwallin 8y ago
Yes, I have difficulty imagining that the bottleneck here is anything other than reading the file from a spinning-rust disk. It would probably be faster to iterate over all the files line by line (eg, sequentially) in a single thread than spinning up 40 threads and having them skip around all over the files, thrashing the disk. I had this come up recently myself. An operation was performing slowly, so I split it up to use multiple cores in parallel. Got slower. Looked at performance monitor and disk utilization was pegged at 100%, while CPU utilization was ~80% of one core. Went back to the old code, and disk utilization was still pegged at 100%, but it had a marginally higher transfer rate because it was reading one file at a time rather than several all at once. I should have examined the performance characteristics more carefully in the beginning, but I didn't, because I assumed I'm smarter than I am. Good lesson. This is almost certainly the wrong tool for the job, unless the job is to learn how to use Spark.

by girst 8y ago

jmcphers 8y ago

Previously on HN, a similar approach demonstrating that command-line tools can be hundreds of times faster than the fancy distributed approach (for certain classes of problems): "Command-line tools can be faster than your Hadoop cluster" https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462

girst 8y ago
It's not just about "raw" speed, but also about how much mental time you spend thinking about your setup. even if you didn't know about zgrep already, that's at most minutes of thinking ahead vs. potential hours or days setting up cloud providers, big toolchains, and whatever else just to get started. since we're throwing links around, the (tangentially related) Knuth vs. McIlroy saga comes to mind: https://news.ycombinator.com/item?id=15264997 https://news.ycombinator.com/item?id=15264997
- obvious_throwaw 8y ago
  > even if you didn't know about zgrep already Interestingly, the dumps come with a brief README file telling you... to use the zgrep and example arguments :)
saberience 8y ago
Hadoop is intended to be used if you have exceeded terabyte scale of data and reached petabytes. That is when simply moving the data around has become incredibly difficult due to network IO limitations. Let me know when you have a solution for using command line tools for sorting through our 4PB of data in our Hadoop cluster! Use the right tools for the right jobs :)
- patrickg_zill 8y ago
  You don't have any indexes?

nwallin 8y ago

Yes, I have difficulty imagining that the bottleneck here is anything other than reading the file from a spinning-rust disk. It would probably be faster to iterate over all the files line by line (eg, sequentially) in a single thread than spinning up 40 threads and having them skip around all over the files, thrashing the disk. I had this come up recently myself. An operation was performing slowly, so I split it up to use multiple cores in parallel. Got slower. Looked at performance monitor and disk utilization was pegged at 100%, while CPU utilization was ~80% of one core. Went back to the old code, and disk utilization was still pegged at 100%, but it had a marginally higher transfer rate because it was reading one file at a time rather than several all at once. I should have examined the performance characteristics more carefully in the beginning, but I didn't, because I assumed I'm smarter than I am. Good lesson. This is almost certainly the wrong tool for the job, unless the job is to learn how to use Spark.