4 ms·

This is so true - so many bad Perl, Python and R jobs “requiring” supercomputer resources that a decent programmer would allow to run at better speed on a singl

This is so true - so many bad Perl, Python and R jobs “requiring” supercomputer resources that a decent programmer would allow to run at better speed on a single modern server!

dhosek 5y ago
There was a post on HN back in 2016 pointing out that some tasks in Hadoop could be executed more quickly by just using standard Unix tools and pipes. It annoys me to no end that very few languages provide out of the box mechanisms to be able to emulate that sort of simple parallelism. Although I did find a Google proposal for the C++ standard library that, at the very least, made it clear how to roll this functionality myself should I so choose.
- lmm 5y ago
  Everything's quicker if it doesn't have to work. Hadoop doesn't just run the code, it gives you job management, log collection, error reporting...
  - dhosek 5y ago
    Yes, but it was more than that in the difference. It's that for many (maybe most) tasks, a simple pipeline-style parallelization makes much more sense than going through all the pain of map-reduce. If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard.
    - lmm 5y ago
      > If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard. You can do that with an actor model by giving each actor thread affinity, but it's rarely what you want since it's normally a lot more efficient to move the code to the data than move the data to the code. In a factory the machines are heavier than the products they're working on so it makes sense to move the machines to the materials; for most code tasks it's much more like harvesting a field or something where it's much more efficient to move the code to the data.
- wging 5y ago
  "Scalability! But at what COST?" is another classic in the category (though it's not precisely about on-box parallelism if I recall; the authors' acronym "COST" is the "Configuration that Outperforms a Single Thread"). https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
azalemeth 5y ago
Part of the reason behind this is that it's really, really hard to hire good programmers in academia -- salaries are difficult to put into grants and equipment is easier. It may well be the case that an übernerd could get your algo a significant runtime improvement, but sometimes you're massively unlikely to actually be able to get an übernerd. On the other hand, saying "I and doing big science, give me time on this cluster" is often more of a positive thing to write.
- ta988 5y ago
  And at the same time we give indecent salaries to PIs, Deans and basketball coaches. We waste large amount of money on inefficient processes and we take 5 years to develop projects that would have taken 6 months with a good dev and in the end would have cost less in maintenance and improvement. I needed to hire devs for a project, it was impossible to do.
- pinewurst 5y ago
  What I've seen is that low-salaried ops people are viewed as a substitute for capital expenditure in this environment. It strongly discourages purchasing software and even non-white box hardware. "Just have IT cobble together something open source", is the message from administration and PIs. The result is ramshackle, inefficient systems running on low bidder hardware from often dubious "HPC" resellers.

This is so true - so many bad Perl, Python and R jobs “requiring” supercomputer resources that a decent programmer would allow to run at better speed on a single modern server!

dhosek 5y ago

There was a post on HN back in 2016 pointing out that some tasks in Hadoop could be executed more quickly by just using standard Unix tools and pipes. It annoys me to no end that very few languages provide out of the box mechanisms to be able to emulate that sort of simple parallelism. Although I did find a Google proposal for the C++ standard library that, at the very least, made it clear how to roll this functionality myself should I so choose.

lmm 5y ago
Everything's quicker if it doesn't have to work. Hadoop doesn't just run the code, it gives you job management, log collection, error reporting...
- dhosek 5y ago
  Yes, but it was more than that in the difference. It's that for many (maybe most) tasks, a simple pipeline-style parallelization makes much more sense than going through all the pain of map-reduce. If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard.
  - lmm 5y ago
    > If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard. You can do that with an actor model by giving each actor thread affinity, but it's rarely what you want since it's normally a lot more efficient to move the code to the data than move the data to the code. In a factory the machines are heavier than the products they're working on so it makes sense to move the machines to the materials; for most code tasks it's much more like harvesting a field or something where it's much more efficient to move the code to the data.
wging 5y ago
"Scalability! But at what COST?" is another classic in the category (though it's not precisely about on-box parallelism if I recall; the authors' acronym "COST" is the "Configuration that Outperforms a Single Thread"). https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

azalemeth 5y ago

Part of the reason behind this is that it's really, really hard to hire good programmers in academia -- salaries are difficult to put into grants and equipment is easier. It may well be the case that an übernerd could get your algo a significant runtime improvement, but sometimes you're massively unlikely to actually be able to get an übernerd. On the other hand, saying "I and doing big science, give me time on this cluster" is often more of a positive thing to write.

ta988 5y ago
And at the same time we give indecent salaries to PIs, Deans and basketball coaches. We waste large amount of money on inefficient processes and we take 5 years to develop projects that would have taken 6 months with a good dev and in the end would have cost less in maintenance and improvement. I needed to hire devs for a project, it was impossible to do.
pinewurst 5y ago
What I've seen is that low-salaried ops people are viewed as a substitute for capital expenditure in this environment. It strongly discourages purchasing software and even non-white box hardware. "Just have IT cobble together something open source", is the message from administration and PIs. The result is ramshackle, inefficient systems running on low bidder hardware from often dubious "HPC" resellers.