7 ms·

Meanwhile, HPC software will continue to be written with C+MPI, or Fortran+MPI. Source: was involved in development of multiple HPC frameworks, attended multip

by floren 5y ago

Meanwhile, HPC software will continue to be written with C+MPI, or Fortran+MPI. Source: was involved in development of multiple HPC frameworks, attended multiple HPC conferences, despite wailing and gnashing of teeth all actual software continues to be written the same way it's always been.

brrrrrm 5y ago
It'd be quite a boring world if observations of the past could reliably predict the future.
- deleted 5y ago
  [deleted]
- AnimalMuppet 5y ago
  I'm not an expert. But with current CPUs, doesn't HPC critically depend on memory alignment? (For two reasons: Avoiding cache misses, and enabling SIMD.) Sure, algorithms matter, probably more than memory alignment. But when you've got the algorithm perfect and you still need more speed, you're probably going to need to be able to tune memory alignment. C lets you control memory alignment, in a way that very few other languages do. Until the competitors in HPC catch up to that, C is going to continue to outrun those other languages. (Now, for doing the tweaking to get the most out of the algorithm, do I want to do that in C? No. I want to do it in something that makes me care about fewer details while I experiment.)
  - brrrrrm 5y ago
    You're correct that low level details very much matter in the HPC space. The types of optimizations described in this paper are exactly that! To elucidate, check out Halide's cool visualizations/documentation (which this paper compares itself to) https://halide-lang.org/tutorials/tutorial_lesson_05_scheduling_1.html https://halide-lang.org/tutorials/tutorial_lesson_05_schedul...
  - mhh__ 5y ago
    Nothing C does is even remotely special anymore. If anything it's total crap by modern standards because it makes performant abstractions harder. C code uses a lot of linked lists because linked lists are easy. C isn't the Lingua Franca of fast software anymore. The reason why C programs can sometimes be faster, today, however is not an intrinsic property of the language but rather its inability to allow incompetent programmers to hide their bad data structures. C also encourages programming styles that are harder to optimize so some loop optimizations are no longer possible.
    - jstimpfle 5y ago
      > C code uses a lot of linked lists because linked lists are easy. Strong disagree. I don't see any reason why a competent programmer would hand-code a linked list, which is more of a hassle to make than a simple array (e.g. using buffer pointer + capacity and/or length field), unless the linked list makes actual sense for performance.
      - mhh__ 5y ago
        I'm not saying you're duty bound to do it but rather giving an example of something which in my experience is encouraged by C since it's the path of least friction. Try writing a type that can automatically switch layout from AOS to SOA in C, you just can't do it.
        uluyol 5y ago
        Out of curiosity, what are some languages that can automatically switch layout between struct-of-arrays and array-of-structs?
        mhh__ 5y ago
        The only language I know for sure to do it for you (as in you don't have to write the type) was Jai a while back (I'm told Blow removed that feature). The only language I've actually done it in, is D. It's probably doable in many other nu-C languages these days, but D at very least can make it basically seamless as long as you do some try-and-break-shit testing to make sure nothing is relying on saving pointers when they shouldn't. This obviously constrains the definition of automatic ;) I don't have my implementation to hand because it grew out of patch that failed due to aforementioned pointer-saving in code that I'm not paid enough to refactor (), but here's one someone else made https://github.com/nordlow/phobos-next/blob/master/src/nxt/soa_petar_kirov.d https://github.com/nordlow/phobos-next/blob/master/src/nxt/s... there's another one in that repository too. I've never used those particular implementations but they're both by people I know so hopefully they're not too bad. A more subtle thing, which I haven't used in anger, but would like to try at some point is to use programmer annotations (probably in the form of user defined attributes) to try and group things so things are stored such that temporal locality <=> spacial locality, but I've never bothered to actually do it. There are some arrays of structs in an old bit of the D compiler that are roughly the size of a cacheline, and aren't accessed particularly uniformly. I profiled this and found that something like 75% of all LLC misses (hitting DRAM) were due to 2 particularly miserable lines... inside an O(n^2) algorithm.
  - pjmlp 5y ago
    Most compiled languages allow to control alignment even before C was an idea.
  - KolenCh 5y ago
    Memory alignment is important and languages let you control that and/or make them align automatically given sensible assumptions does have an edge here. Just a random example, I wrote a two lines Python function using Numba jit, because I know in my mind that some Python behavior is going to make it less performant and it is a high performance kernel so we need that fast (and also lower memory footprint.) Compiling using Numba jit is no brainer because I just add one more line there with like 10 seconds effort. But the code base I am merging to has a policy against Numba (reasonably as we are targeting HPC platform where Numba has some performance problem related to oversubscribing if not setting up carefully.) So I end up rewrote that in C++ and wrap it with pybind11. And the result is that it is faster by around 30%. Since the algorithm is entirely trivial, the only explanation I have is exactly memory alignment, where I can control that in C++, but in Numba jit there's no way (both to guarantee the array allocated is aligned, or tell the compiler to assume that.) (The 30% number is also reasonable in textbook examples.) P.S. But it has a cost: > Showing 14 changed files with 230 additions and 36 deletions. > from https://github.com/hpc4cmb/toast/commit/a38d1d6dbcc97001a1ad1c315bb08cf1eecc74c7 https://github.com/hpc4cmb/toast/commit/a38d1d6dbcc97001a1ad... where the 36 deletions are mostly just documentations. So it's an ~200 lines effort comparing to a 4 line effort...
- taeric 5y ago
  In this space, it is more they you don't want to lose all of the progress made in these older libraries. Coming up with a new ecosystem is an immediate race to parity with the older. And a lot of smart people were involved with that. (Such that you aren't competing with a single idea or implementation, but an ecosystem of them.) Hats off for making a good shot. But don't be surprised to see reluctance to move.
  - brrrrrm 5y ago
    To clarify, I’m not in any way involved with the paper described in the article. Just interested in the space :)
  - sarch 5y ago
    There's very promising work on upgrading older libraries/legacy code with a technique called verified lifting. The technique has been used successfully at Adobe to automatically lift image processing code written in C++ to use Halide. The technique also guarantees semantic equivalence so users can trust the lifted code. Paper: https://dl.acm.org/doi/pdf/10.1145/3355089.3356549 https://dl.acm.org/doi/pdf/10.1145/3355089.3356549
    - derefr 5y ago
      Before reading the paper: is "verified lifting" kind of like "deterministic decompilation" — where you ensure, with every modification to the generated HLL source, that it continues to compile back into the original LL object code? (See e.g. the Mario 64 decompilation — https://github.com/n64decomp/sm64 https://github.com/n64decomp/sm64 — , whose reverse-engineering process at all times kept a source tree that could build the original byte-identical ROMs of the games, despite being increasingly [manually] rewritten in an HLL.)
- Avshalom 5y ago
  That's pretty literally the framework of science.
agumonkey 5y ago
No mention of chapel ? just asking, it's been in development for years and I just saw its name on the benchmarkgame so some people still have interest in it.
- dapids 5y ago
  I too was surprised.
Bostonian 5y ago
Do you see use of coarrays, introduced in Fortran 2008?
- adgjlsfhk1 5y ago
  It's a kind of good idea, but has a few fatal flaws. The biggest is that they don't work well when you want a combination of GPU and CPU parallelism in the same codebase. The other is that most current fortran compilers don't compile them well (see https://fortran-lang.discourse.group/t/coarrays-not-ready-for-prime-time https://fortran-lang.discourse.group/t/coarrays-not-ready-fo...).
nine_k 5y ago
I don't see why such a specialized language may not have a good C or Fortran interface. Also, MPI even allows units written in different languages to talk via messages, AFAIK. I even suppose that the orchestration layer can be in basically any convenient language, even in Python, while the underlying compute modules may be written in highly-optimized compiled languages; see Thensorflow and PyTorch for hihgly successful examples.
eslaught 5y ago
I'm helping to organize a workshop about alternatives to MPI in HPC. You can see a sample of past programs at [1]. But you're right: today, MPI is dominant. I suspect this will change, if only because HPC and data center computing is converging, and the (much larger) data center side of things is very much not based on MPI (e.g., see TensorFlow). Personally, I find it reassuring that many of the dominant solutions from the data center side have more in common with the upstart HPC solutions, than they do with the HPC status quo. I'd like to think that at some point we'll be able to bring around the rest of the HPC community too. [1]: https://sourceryinstitute.github.io/PAW/ https://sourceryinstitute.github.io/PAW/
- pjmlp 5y ago
  I still remember when PVM vs MPI articles were common.
fouc 5y ago
The staying power of Fortran is amazing.
- Mikeb85 5y ago
  Well it does keep evolving. It's also pretty close to perfect for writing fast numeric code without too much trouble.
KolenCh 5y ago
Python and Julia are a (may be a distant) 3rd and 4th. Using Python on HPC system, depending on who you ask, is kind of cheating as the main work is always C/C++ (or Fortran). So it is still really just C/C++ and Fortran. Julia is really promising as it is the first new language that can efficiently run on HPC and can use other form of message passing. Also we should not forget UPC, a special kind of C that has its own parallel programming model (not MPI.) It has been used on HPC system as well but perhaps only from a selected few. (This probably reveal where I come from.)
- pjmlp 5y ago
  Speaking from how we were already doing at CERN, 20 years ago, Python was plainly more confortable to use than UNIX shell for program orchestration, that was all, and the CMT build system used Python as well. Real work was always C++ and Fortran.
  - KolenCh 5y ago
    Oh, CERN? I think they have made PyROOT which is horrible? (And even a jupyter kernel for interactive use.) In terms of scripting, a professor once showed us a workflow script from his old days written entirely in bash and is very very long and very complex. He basically said if your script gets that complicated it should be written in Python nowadays. But Python’s HPC presence is deeper than scripting in that sense. Machine learning frameworks definitely is a contributing factor, but even more traditional HPC workflows are benefited when stuffs have Python API, especially with mpi4py. I once used something called mpi bash that I need to compile it myself on an HPC platform. It is much less well known and used, perhaps because of what the professor said—if it needs MPI, probably even for scripting it should be scripted in Python instead.
pjmlp 5y ago
C++ and MPI in what concerns high energy physics.
ksaj 5y ago
I actually use MPI on a Raspberry Pi cluster, using the clusterHAT with 4 Raspberry Pi Zeros controlled by a Pi 3. The whole unit is about the size of a Big Mac. Obviously it's not very performant, but the lessons learned on how to program with MPI in mind is where the value lies.
hulitu 5y ago
Because it works. Nobody wants to risk a big budget to create something new from scratch with the risk that it will not work.

by floren 5y ago

brrrrrm 5y ago

It'd be quite a boring world if observations of the past could reliably predict the future.

deleted 5y ago
[deleted]
AnimalMuppet 5y ago
I'm not an expert. But with current CPUs, doesn't HPC critically depend on memory alignment? (For two reasons: Avoiding cache misses, and enabling SIMD.) Sure, algorithms matter, probably more than memory alignment. But when you've got the algorithm perfect and you still need more speed, you're probably going to need to be able to tune memory alignment. C lets you control memory alignment, in a way that very few other languages do. Until the competitors in HPC catch up to that, C is going to continue to outrun those other languages. (Now, for doing the tweaking to get the most out of the algorithm, do I want to do that in C? No. I want to do it in something that makes me care about fewer details while I experiment.)
- brrrrrm 5y ago
  You're correct that low level details very much matter in the HPC space. The types of optimizations described in this paper are exactly that! To elucidate, check out Halide's cool visualizations/documentation (which this paper compares itself to) https://halide-lang.org/tutorials/tutorial_lesson_05_scheduling_1.html https://halide-lang.org/tutorials/tutorial_lesson_05_schedul...
- mhh__ 5y ago
  Nothing C does is even remotely special anymore. If anything it's total crap by modern standards because it makes performant abstractions harder. C code uses a lot of linked lists because linked lists are easy. C isn't the Lingua Franca of fast software anymore. The reason why C programs can sometimes be faster, today, however is not an intrinsic property of the language but rather its inability to allow incompetent programmers to hide their bad data structures. C also encourages programming styles that are harder to optimize so some loop optimizations are no longer possible.
  - jstimpfle 5y ago
    > C code uses a lot of linked lists because linked lists are easy. Strong disagree. I don't see any reason why a competent programmer would hand-code a linked list, which is more of a hassle to make than a simple array (e.g. using buffer pointer + capacity and/or length field), unless the linked list makes actual sense for performance.
    - mhh__ 5y ago
      I'm not saying you're duty bound to do it but rather giving an example of something which in my experience is encouraged by C since it's the path of least friction. Try writing a type that can automatically switch layout from AOS to SOA in C, you just can't do it.
      - uluyol 5y ago
        Out of curiosity, what are some languages that can automatically switch layout between struct-of-arrays and array-of-structs?
        mhh__ 5y ago
        The only language I know for sure to do it for you (as in you don't have to write the type) was Jai a while back (I'm told Blow removed that feature). The only language I've actually done it in, is D. It's probably doable in many other nu-C languages these days, but D at very least can make it basically seamless as long as you do some try-and-break-shit testing to make sure nothing is relying on saving pointers when they shouldn't. This obviously constrains the definition of automatic ;) I don't have my implementation to hand because it grew out of patch that failed due to aforementioned pointer-saving in code that I'm not paid enough to refactor (), but here's one someone else made https://github.com/nordlow/phobos-next/blob/master/src/nxt/soa_petar_kirov.d https://github.com/nordlow/phobos-next/blob/master/src/nxt/s... there's another one in that repository too. I've never used those particular implementations but they're both by people I know so hopefully they're not too bad. A more subtle thing, which I haven't used in anger, but would like to try at some point is to use programmer annotations (probably in the form of user defined attributes) to try and group things so things are stored such that temporal locality <=> spacial locality, but I've never bothered to actually do it. There are some arrays of structs in an old bit of the D compiler that are roughly the size of a cacheline, and aren't accessed particularly uniformly. I profiled this and found that something like 75% of all LLC misses (hitting DRAM) were due to 2 particularly miserable lines... inside an O(n^2) algorithm.
- pjmlp 5y ago
  Most compiled languages allow to control alignment even before C was an idea.
- KolenCh 5y ago
  Memory alignment is important and languages let you control that and/or make them align automatically given sensible assumptions does have an edge here. Just a random example, I wrote a two lines Python function using Numba jit, because I know in my mind that some Python behavior is going to make it less performant and it is a high performance kernel so we need that fast (and also lower memory footprint.) Compiling using Numba jit is no brainer because I just add one more line there with like 10 seconds effort. But the code base I am merging to has a policy against Numba (reasonably as we are targeting HPC platform where Numba has some performance problem related to oversubscribing if not setting up carefully.) So I end up rewrote that in C++ and wrap it with pybind11. And the result is that it is faster by around 30%. Since the algorithm is entirely trivial, the only explanation I have is exactly memory alignment, where I can control that in C++, but in Numba jit there's no way (both to guarantee the array allocated is aligned, or tell the compiler to assume that.) (The 30% number is also reasonable in textbook examples.) P.S. But it has a cost: > Showing 14 changed files with 230 additions and 36 deletions. > from https://github.com/hpc4cmb/toast/commit/a38d1d6dbcc97001a1ad1c315bb08cf1eecc74c7 https://github.com/hpc4cmb/toast/commit/a38d1d6dbcc97001a1ad... where the 36 deletions are mostly just documentations. So it's an ~200 lines effort comparing to a 4 line effort...
taeric 5y ago
In this space, it is more they you don't want to lose all of the progress made in these older libraries. Coming up with a new ecosystem is an immediate race to parity with the older. And a lot of smart people were involved with that. (Such that you aren't competing with a single idea or implementation, but an ecosystem of them.) Hats off for making a good shot. But don't be surprised to see reluctance to move.
- brrrrrm 5y ago
  To clarify, I’m not in any way involved with the paper described in the article. Just interested in the space :)
- sarch 5y ago
  There's very promising work on upgrading older libraries/legacy code with a technique called verified lifting. The technique has been used successfully at Adobe to automatically lift image processing code written in C++ to use Halide. The technique also guarantees semantic equivalence so users can trust the lifted code. Paper: https://dl.acm.org/doi/pdf/10.1145/3355089.3356549 https://dl.acm.org/doi/pdf/10.1145/3355089.3356549
  - derefr 5y ago
    Before reading the paper: is "verified lifting" kind of like "deterministic decompilation" — where you ensure, with every modification to the generated HLL source, that it continues to compile back into the original LL object code? (See e.g. the Mario 64 decompilation — https://github.com/n64decomp/sm64 https://github.com/n64decomp/sm64 — , whose reverse-engineering process at all times kept a source tree that could build the original byte-identical ROMs of the games, despite being increasingly [manually] rewritten in an HLL.)
Avshalom 5y ago
That's pretty literally the framework of science.

agumonkey 5y ago

No mention of chapel ? just asking, it's been in development for years and I just saw its name on the benchmarkgame so some people still have interest in it.

dapids 5y ago
I too was surprised.

Bostonian 5y ago

Do you see use of coarrays, introduced in Fortran 2008?

adgjlsfhk1 5y ago
It's a kind of good idea, but has a few fatal flaws. The biggest is that they don't work well when you want a combination of GPU and CPU parallelism in the same codebase. The other is that most current fortran compilers don't compile them well (see https://fortran-lang.discourse.group/t/coarrays-not-ready-for-prime-time https://fortran-lang.discourse.group/t/coarrays-not-ready-fo...).

nine_k 5y ago

I don't see why such a specialized language may not have a good C or Fortran interface. Also, MPI even allows units written in different languages to talk via messages, AFAIK. I even suppose that the orchestration layer can be in basically any convenient language, even in Python, while the underlying compute modules may be written in highly-optimized compiled languages; see Thensorflow and PyTorch for hihgly successful examples.

eslaught 5y ago

I'm helping to organize a workshop about alternatives to MPI in HPC. You can see a sample of past programs at [1]. But you're right: today, MPI is dominant. I suspect this will change, if only because HPC and data center computing is converging, and the (much larger) data center side of things is very much not based on MPI (e.g., see TensorFlow). Personally, I find it reassuring that many of the dominant solutions from the data center side have more in common with the upstart HPC solutions, than they do with the HPC status quo. I'd like to think that at some point we'll be able to bring around the rest of the HPC community too. [1]: https://sourceryinstitute.github.io/PAW/ https://sourceryinstitute.github.io/PAW/

pjmlp 5y ago
I still remember when PVM vs MPI articles were common.

fouc 5y ago

The staying power of Fortran is amazing.

Mikeb85 5y ago
Well it does keep evolving. It's also pretty close to perfect for writing fast numeric code without too much trouble.

KolenCh 5y ago

Python and Julia are a (may be a distant) 3rd and 4th. Using Python on HPC system, depending on who you ask, is kind of cheating as the main work is always C/C++ (or Fortran). So it is still really just C/C++ and Fortran. Julia is really promising as it is the first new language that can efficiently run on HPC and can use other form of message passing. Also we should not forget UPC, a special kind of C that has its own parallel programming model (not MPI.) It has been used on HPC system as well but perhaps only from a selected few. (This probably reveal where I come from.)

pjmlp 5y ago
Speaking from how we were already doing at CERN, 20 years ago, Python was plainly more confortable to use than UNIX shell for program orchestration, that was all, and the CMT build system used Python as well. Real work was always C++ and Fortran.
- KolenCh 5y ago
  Oh, CERN? I think they have made PyROOT which is horrible? (And even a jupyter kernel for interactive use.) In terms of scripting, a professor once showed us a workflow script from his old days written entirely in bash and is very very long and very complex. He basically said if your script gets that complicated it should be written in Python nowadays. But Python’s HPC presence is deeper than scripting in that sense. Machine learning frameworks definitely is a contributing factor, but even more traditional HPC workflows are benefited when stuffs have Python API, especially with mpi4py. I once used something called mpi bash that I need to compile it myself on an HPC platform. It is much less well known and used, perhaps because of what the professor said—if it needs MPI, probably even for scripting it should be scripted in Python instead.

pjmlp 5y ago

C++ and MPI in what concerns high energy physics.

ksaj 5y ago

I actually use MPI on a Raspberry Pi cluster, using the clusterHAT with 4 Raspberry Pi Zeros controlled by a Pi 3. The whole unit is about the size of a Big Mac. Obviously it's not very performant, but the lessons learned on how to program with MPI in mind is where the value lies.

hulitu 5y ago

Because it works. Nobody wants to risk a big budget to create something new from scratch with the risk that it will not work.