3 ms·
Hmm... I don't think the comparison says anything about Hadoop vs Bash. The programs they are running are completely different. In that article, the author run
by malisper 8y ago
Hmm... I don't think the comparison says anything about Hadoop vs Bash. The programs they are running are completely different.
In that article, the author runs a regular expression that looks lines that start with "Result" in the pgn files. Then they do a tiny bit of processing with awk.
In the original article[0] the author completely parses the pgn files. Then they extract the result from the parsed file.
I wouldn't be surprised if the difference came completely from that.
I would love to see someone run benchmark the exact same program and see how much of a difference there really is between Hadoop and Bash.
[0] https://web.archive.org/web/20140119221101/http://tomhayden3.com/2013/12/27/chess-mr-job/ https://web.archive.org/web/20140119221101/http://tomhayden3...
- mmt 8y agoIt's probably easy enough to simulate the "bash" implementation done in a distributed way. Doing that python library full parsing locally, 10,000 lines (or however much memory one can configure up to) at a time might also be easy enough. Come to think of it, just benchmarking a single-threaded run of enough lines of each against each other should give you an idea, especially with even basic profiling from time(1) to estimate how much was CPU and how much was I/O. That said, I don't think that such a discrepancy, if it exists, takes away from what I took to be the author's overall point, which is that old, simple tools, are underused, despite being much faster and good enough, i.e. worse is better [1]. [1] This an example of sacrificing correctness for simplicity in implementation, something detailed in the "rise of worse is better" essay, not merely an excuse for sloppiness in engineering.