9 ms·

https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...

taneq 8y ago
That's the one, thanks!
- adamdrake 8y ago
  Author here. If I can answer any questions, feel free to ask away!
  - itronitron 8y ago
    I'm interested to know if you have any pointers on learning resources for awk.
    - pedro84 8y ago
      I learned from The AWK Programming Language: https://news.ycombinator.com/item?id=13451454 https://news.ycombinator.com/item?id=13451454
    - adamdrake 8y ago
      I use it from time to time, but I certainly wouldn't consider myself an expert. When I need to do something beyond my current knowledge I search around a bit until I find the awk syntax I need to get the job done. Sorry I can't be more helpful!
    - mmt 8y ago
      I'll second the sibling comment about the original AWK book (The AWK Programming Language). I'd also suggest just using it more. If you find yourself wanting to use grep, cut or sed (especially if you need more than one!) for a one-liner, try using awk instead, if only for the practice. Once you're accustomed to some of the idioms, simplicity, and built-in looping, it will feel like a more natural, casual text-processing tool that you'd reach for automatically. Lastly, search for "cookbooks" or libraries or other collections of useful scripts to see how others have used the language, in more advanced ways. Personally, I don't necessarily find it as something to aspire to (i.e. use of the language for complex, multi-line programs) as much as a way to better understand some of the power of the quirkier features that may, otherwise, be less obvious. One of my favorite non-obvious behaviors is bypassing the input/loop functionality by having the only pattern be BEGIN. This results in an output-only program that makes for a convenient way of playing around with things like printf or just language syntax features (or differences between awk/mawk/gawk), without having to redirect input from /dev/null or worrying about hitting control-D twice and exiting the shell (for those of us who abhor ignoreeof): awk 'BEGIN{print "Hello, world!"}'
  - taneq 8y ago
    No real questions, just that I love reading about this sort of optimization - Abrash's Black Book is a favourite that still gets pulled out every now and then. Thanks for the fun post! It always astounds me these days when someone manages to release slow software for a desktop computer despite modern systems being orders of magnitude faster than the first desktop computers while often being no more responsive. Edit: Also I'm amused by how much of a nerve this seems to have hit. I guess some people are defensive about their high performance computing approaches... :)
    - mmt 8y ago
      > I guess some people are defensive about their high performance computing approaches... Fortunately, when articles about real HPC clusters (aka Supercomputers) make it to the front page, they need no defense, only an occasional explanation that what makes them (extra) special are the high-bandwidth, low-latency interconnects. (Those interconnects make a very strong effort at Fallacies of Distributed Computing numbers 2 and 3[1] and tend to neatly take care of the remaining 6). To be fair, though, I don't think the claim for the more common distributed systems is that they're "high performance" so much as that they're scalable (and have other benefits of being multi-node like resilience). [1] The bandwidth may well be indistinguishable from infinite if it exceeds local (e.g. CPU-memory, CPU-CPU, CPU-GPU) bandwidths. I don't think that's yet true for something like multi-socket Intel systems, but it might be possible with enough interface cards. I didn't look at the specs on those POWER9 chips.
potatoyogurt 8y ago
in other words, "I used a chainsaw to cut an apple and it SUCKED at it." If you're processing an amount of data that comfortably fits in memory on a single machine, then obviously Hadoop is going to perform poorly in comparison. The costs of scheduling a job onto N mappers/reducers, transferring code to each node, waiting for the slowest mapper/reducer to finish, transferring data from mappers -> reducers, replicating output on HDFS, etc. are well-understood. It's true that many people try to use Hadoop when they'd be better served with simpler solutions, but that does not justify the amount of shade that the author throws at it.
- itronitron 8y ago
  Hadoop has a very narrow best fit use case but it has been oversold as the best solution for big data.
  - potatoyogurt 8y ago
    Best fit is something you can argue a lot about. There are a lot of data processing tools out there now, many of which have come out after Hadoop. But if the comparison is against some process running on a single machine, then the use case is not narrow. It includes basically anything where you're processing more than 1TB of data in non-trivial ways (i.e. not just a map operation) and are okay with batch processing.
    - PeterisP 8y ago
      The issue with that is that for most organizations their key business data (and all its recorded history) fits in RAM of a sufficiently beefy workstation. They want to call it Big Data to stroke their egos, and properly acquiring, cleaning and integrating that data can take a LOT of effort so that data can be quite expensive and worthy of any glorious label they can think of; but my experience is that processing more than 1TB of meaningful data actually is a narrow use case, which matters in two specific categories: the (relatively few) very large multinational companies, and processing of raw video/audio/image data; and the majority of people working on data analysis end up with business needs that can be satisfied by relatively simple methods on relatively small datasets.
malisper 8y ago
Hmm... I don't think the comparison says anything about Hadoop vs Bash. The programs they are running are completely different. In that article, the author runs a regular expression that looks lines that start with "Result" in the pgn files. Then they do a tiny bit of processing with awk. In the original article[0] the author completely parses the pgn files. Then they extract the result from the parsed file. I wouldn't be surprised if the difference came completely from that. I would love to see someone run benchmark the exact same program and see how much of a difference there really is between Hadoop and Bash. [0] https://web.archive.org/web/20140119221101/http://tomhayden3.com/2013/12/27/chess-mr-job/ https://web.archive.org/web/20140119221101/http://tomhayden3...
- mmt 8y ago
  It's probably easy enough to simulate the "bash" implementation done in a distributed way. Doing that python library full parsing locally, 10,000 lines (or however much memory one can configure up to) at a time might also be easy enough. Come to think of it, just benchmarking a single-threaded run of enough lines of each against each other should give you an idea, especially with even basic profiling from time(1) to estimate how much was CPU and how much was I/O. That said, I don't think that such a discrepancy, if it exists, takes away from what I took to be the author's overall point, which is that old, simple tools, are underused, despite being much faster and good enough, i.e. worse is better [1]. [1] This an example of sacrificing correctness for simplicity in implementation, something detailed in the "rise of worse is better" essay, not merely an excuse for sloppiness in engineering.

taneq 8y ago

That's the one, thanks!

adamdrake 8y ago
Author here. If I can answer any questions, feel free to ask away!
- itronitron 8y ago
  I'm interested to know if you have any pointers on learning resources for awk.
  - pedro84 8y ago
    I learned from The AWK Programming Language: https://news.ycombinator.com/item?id=13451454 https://news.ycombinator.com/item?id=13451454
  - adamdrake 8y ago
    I use it from time to time, but I certainly wouldn't consider myself an expert. When I need to do something beyond my current knowledge I search around a bit until I find the awk syntax I need to get the job done. Sorry I can't be more helpful!
  - mmt 8y ago
    I'll second the sibling comment about the original AWK book (The AWK Programming Language). I'd also suggest just using it more. If you find yourself wanting to use grep, cut or sed (especially if you need more than one!) for a one-liner, try using awk instead, if only for the practice. Once you're accustomed to some of the idioms, simplicity, and built-in looping, it will feel like a more natural, casual text-processing tool that you'd reach for automatically. Lastly, search for "cookbooks" or libraries or other collections of useful scripts to see how others have used the language, in more advanced ways. Personally, I don't necessarily find it as something to aspire to (i.e. use of the language for complex, multi-line programs) as much as a way to better understand some of the power of the quirkier features that may, otherwise, be less obvious. One of my favorite non-obvious behaviors is bypassing the input/loop functionality by having the only pattern be BEGIN. This results in an output-only program that makes for a convenient way of playing around with things like printf or just language syntax features (or differences between awk/mawk/gawk), without having to redirect input from /dev/null or worrying about hitting control-D twice and exiting the shell (for those of us who abhor ignoreeof): awk 'BEGIN{print "Hello, world!"}'
- taneq 8y ago
  No real questions, just that I love reading about this sort of optimization - Abrash's Black Book is a favourite that still gets pulled out every now and then. Thanks for the fun post! It always astounds me these days when someone manages to release slow software for a desktop computer despite modern systems being orders of magnitude faster than the first desktop computers while often being no more responsive. Edit: Also I'm amused by how much of a nerve this seems to have hit. I guess some people are defensive about their high performance computing approaches... :)
  - mmt 8y ago
    > I guess some people are defensive about their high performance computing approaches... Fortunately, when articles about real HPC clusters (aka Supercomputers) make it to the front page, they need no defense, only an occasional explanation that what makes them (extra) special are the high-bandwidth, low-latency interconnects. (Those interconnects make a very strong effort at Fallacies of Distributed Computing numbers 2 and 3[1] and tend to neatly take care of the remaining 6). To be fair, though, I don't think the claim for the more common distributed systems is that they're "high performance" so much as that they're scalable (and have other benefits of being multi-node like resilience). [1] The bandwidth may well be indistinguishable from infinite if it exceeds local (e.g. CPU-memory, CPU-CPU, CPU-GPU) bandwidths. I don't think that's yet true for something like multi-socket Intel systems, but it might be possible with enough interface cards. I didn't look at the specs on those POWER9 chips.

potatoyogurt 8y ago

in other words, "I used a chainsaw to cut an apple and it SUCKED at it." If you're processing an amount of data that comfortably fits in memory on a single machine, then obviously Hadoop is going to perform poorly in comparison. The costs of scheduling a job onto N mappers/reducers, transferring code to each node, waiting for the slowest mapper/reducer to finish, transferring data from mappers -> reducers, replicating output on HDFS, etc. are well-understood. It's true that many people try to use Hadoop when they'd be better served with simpler solutions, but that does not justify the amount of shade that the author throws at it.

itronitron 8y ago
Hadoop has a very narrow best fit use case but it has been oversold as the best solution for big data.
- potatoyogurt 8y ago
  Best fit is something you can argue a lot about. There are a lot of data processing tools out there now, many of which have come out after Hadoop. But if the comparison is against some process running on a single machine, then the use case is not narrow. It includes basically anything where you're processing more than 1TB of data in non-trivial ways (i.e. not just a map operation) and are okay with batch processing.
  - PeterisP 8y ago
    The issue with that is that for most organizations their key business data (and all its recorded history) fits in RAM of a sufficiently beefy workstation. They want to call it Big Data to stroke their egos, and properly acquiring, cleaning and integrating that data can take a LOT of effort so that data can be quite expensive and worthy of any glorious label they can think of; but my experience is that processing more than 1TB of meaningful data actually is a narrow use case, which matters in two specific categories: the (relatively few) very large multinational companies, and processing of raw video/audio/image data; and the majority of people working on data analysis end up with business needs that can be satisfied by relatively simple methods on relatively small datasets.

malisper 8y ago

Hmm... I don't think the comparison says anything about Hadoop vs Bash. The programs they are running are completely different. In that article, the author runs a regular expression that looks lines that start with "Result" in the pgn files. Then they do a tiny bit of processing with awk. In the original article[0] the author completely parses the pgn files. Then they extract the result from the parsed file. I wouldn't be surprised if the difference came completely from that. I would love to see someone run benchmark the exact same program and see how much of a difference there really is between Hadoop and Bash. [0] https://web.archive.org/web/20140119221101/http://tomhayden3.com/2013/12/27/chess-mr-job/ https://web.archive.org/web/20140119221101/http://tomhayden3...

mmt 8y ago
It's probably easy enough to simulate the "bash" implementation done in a distributed way. Doing that python library full parsing locally, 10,000 lines (or however much memory one can configure up to) at a time might also be easy enough. Come to think of it, just benchmarking a single-threaded run of enough lines of each against each other should give you an idea, especially with even basic profiling from time(1) to estimate how much was CPU and how much was I/O. That said, I don't think that such a discrepancy, if it exists, takes away from what I took to be the author's overall point, which is that old, simple tools, are underused, despite being much faster and good enough, i.e. worse is better [1]. [1] This an example of sacrificing correctness for simplicity in implementation, something detailed in the "rise of worse is better" essay, not merely an excuse for sloppiness in engineering.