2 ms·

There was a post on HN back in 2016 pointing out that some tasks in Hadoop could be executed more quickly by just using standard Unix tools and pipes. It annoys

by dhosek 5y ago

There was a post on HN back in 2016 pointing out that some tasks in Hadoop could be executed more quickly by just using standard Unix tools and pipes. It annoys me to no end that very few languages provide out of the box mechanisms to be able to emulate that sort of simple parallelism. Although I did find a Google proposal for the C++ standard library that, at the very least, made it clear how to roll this functionality myself should I so choose.

lmm 5y ago
Everything's quicker if it doesn't have to work. Hadoop doesn't just run the code, it gives you job management, log collection, error reporting...
- dhosek 5y ago
  Yes, but it was more than that in the difference. It's that for many (maybe most) tasks, a simple pipeline-style parallelization makes much more sense than going through all the pain of map-reduce. If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard.
  - lmm 5y ago
    > If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard. You can do that with an actor model by giving each actor thread affinity, but it's rarely what you want since it's normally a lot more efficient to move the code to the data than move the data to the code. In a factory the machines are heavier than the products they're working on so it makes sense to move the machines to the materials; for most code tasks it's much more like harvesting a field or something where it's much more efficient to move the code to the data.
wging 5y ago
"Scalability! But at what COST?" is another classic in the category (though it's not precisely about on-box parallelism if I recall; the authors' acronym "COST" is the "Configuration that Outperforms a Single Thread"). https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

by dhosek 5y ago

lmm 5y ago

Everything's quicker if it doesn't have to work. Hadoop doesn't just run the code, it gives you job management, log collection, error reporting...

dhosek 5y ago
Yes, but it was more than that in the difference. It's that for many (maybe most) tasks, a simple pipeline-style parallelization makes much more sense than going through all the pain of map-reduce. If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard.
- lmm 5y ago
  > If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard. You can do that with an actor model by giving each actor thread affinity, but it's rarely what you want since it's normally a lot more efficient to move the code to the data than move the data to the code. In a factory the machines are heavier than the products they're working on so it makes sense to move the machines to the materials; for most code tasks it's much more like harvesting a field or something where it's much more efficient to move the code to the data.

wging 5y ago

"Scalability! But at what COST?" is another classic in the category (though it's not precisely about on-box parallelism if I recall; the authors' acronym "COST" is the "Configuration that Outperforms a Single Thread"). https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...