6 ms·

Reflections on 30 years of HPC programming

146 points by matt_d 6mo ago

riffraff 6mo ago
Perhaps one issue lacking discussion in the article is how easy it is to find devs? I've never worked in HPC but it seems it should be relatively simple to find a C/C++ dev that can pick up OpenMP, or one that already knows it, compared to hiring people who know Chapel. The "scaling down" factor (how easy or interesting it is to use tool X for small use) seems a disadvantage of HPC-only languages, which creates a barrier to entry and a reduction in available workforce.
- kinow 6mo ago
  I think hpc devs need an extra set of skills that are not so common. Such as parallel file systems, batch schedulers, NUMA, infiniband, and probably some domain-specific knowledge for the apps they will develop. This knowledge is also probably a bit niche, like climate modelling, earthquake simulation, lidar data processing, and so it goes. And even knowing OpenMP or MPI may not suffice if the site uses older versions or heterogeneous approaches with CUDA, FPGA, etc. Knowing the language and the shared/distributed mem libs help, but if your project needs a new senior dev than it may be a bit hard to find (although popularity of company/HPC, salary, and location also play a role).
  - physicsguy 6mo ago
    You tend to only learn these things as they become a problem too. That's super super domain specific and it doesn't always translate between areas of research. So for e.g. when I did HPC simulation codes in magnetics, there was little point focusing on some of these areas because our codes were dominated by the long-range interaction cost which limited compute scaling. All of our effort was tuning those algorithms to the absolute max. We tried heterogenous CPU + GPU but had very mixed results, and at that time (2010s) the GPU memory wasn't large enough for the problems we cared about either. I then moved to CFD in industry. The concerns there were totally different since everything is grid local. Partitioning over multi-GPU is simple since only the boundaries need to be exchanged on each iteration. The problems there were much more on the memory bandwidth and parallel file system performance side. Basically, you have to learn to solve whatever challenges get thrown up by the specific domain problem. > And even knowing OpenMP or MPI may not suffice if the site uses older versions To be fair, you always have the option of compiling yourself, but most people I met in academia didn't have the background to do this. Spack and EasyBuild make this much much easier.
  - bluedino 6mo ago
    Most developers have no clue about any of that stuff. It's all abstracted out.
- KaiserPro 6mo ago
  I worked in HPC adjacent fields for a while (up until 40gig ethernet was cheap enough to roll out to all the edge nodes) There are a couple of big things that are difficult to get your head around: 1) when and where to dispatch and split jobs (ie whats the setup cost of spinning up n binaries on n machines vs threading on y machines) 2) data exchange primitives, Shared file systems have quirks, and a they differ from system to system. But most of the time its better/easier/faster to dump shit to a file system than some fancy database/object store. Until its not. Distributed queues are great, unless you're using them wrong. Most of the time you need to use them wrong. (the share memory RPC is a whole another beast that fortunatly I've never had to work with directly. ) 3) dealing with odd failures. As the number of parallel jobs increase the chance of getting a failure reaches 1. You need to bake in failure modes at the very start. 4) loading/saving data is often a bottle neck, lots of efficiecny comes from being clever in what you load, and _where_ you load it. (ie you have data affinity, which might be location based, or topology based, and you don't often have control over where your stuff is placed.)
jandrewrogers 6mo ago
I can easily explain this, having worked in this space. The new languages don’t actually solve any urgent problems. How people imagine scalable parallelism works and how it actually works doesn’t have a lot of overlap. The code is often boringly single-threaded because that is optimal for performance. The single biggest resource limit in most HPC code is memory bandwidth. If you are not addressing this then you are not addressing a real problem for most applications. For better or worse, C++ is really good at optimizing for memory bandwidth. Most of the suggested alternative languages are not. It is that simple. The new languages address irrelevant problems. It is really difficult to design a language that is more friendly to memory bandwidth than C++. And that is the resource you desperately need to optimize for in most cases.
- bruce343434 6mo ago
  What does it mean to be friendly to memory bandwidth, and why does C++ excel at it, over, say, Fortran or C or Rust?
  - lugu 6mo ago
    Parent talks about new languages, as per the article Fortran or C doing fine. I speculate the benefit of C++ over Rust how it let programmers instruct the compiler of warranty that goes beyong the initial semantic of the language. See __restrict, __builtin_prefetch and __builtin_assume_aligned. The programming language is a space for conversations between compiler builders and hardware designers.
    - _flux 6mo ago
      I believe __restrict, and __builtin_prefetch/__builtin_assume are compiler extensions, not part of the C++ language as is, and different compilers implement (or don't) these differently. The rust compiler actually has similar things, but they're not available in stable builds. I suppose there are some issues if principle why not to include them in stable. E.g: https://doc.rust-lang.org/std/intrinsics/fn.prefetch_read_data.html https://doc.rust-lang.org/std/intrinsics/fn.prefetch_read_da... Maybe some time in the future good acceptable abstractions will be conceived for them.. Perhaps using just using nightly builds for HPC is not that far out, though.
swiftcoder 6mo ago
It's interesting that none of the actor-based languages ever made it into this space. Feels like something with the design philosophy of Erlang would be pretty suitable to exploit millions of cores and a variety of interconnects...
- jacquesm 6mo ago
  That will never happen. The overhead is massive.
  - Joel_Mckay 6mo ago
    People did try to create an OTP in HDL at one point. And Erlang has already run many telecom infrastructures for decades. Surprising given how fragile the multi-host implementation has proven. Erlang/Elixir are neat languages, and right next to Julia for fun. =3
    - jacquesm 6mo ago
      Yes, it's absolutely amazing. But I'm intimately familiar with how the Erlang VM works and it would seem to me to be a very bad match for HPC on the number crunching side, though it likely would do quite well on the orchestration side but that would require the people writing the rest of the code to change their way of working completely. And given how much of that is still F77 I highly doubt they would be willing to make that investment without the promise of some massive gain.
      - Joel_Mckay 6mo ago
        For simple AMQP, it has performed rather well for our use-cases over the years. Haven't personally deployed this version yet. ymmv =3 https://www.rabbitmq.com/docs/quorum-queues https://www.rabbitmq.com/docs/quorum-queues
        jacquesm 6mo ago
        Of course it does, that's an ideal usecase. But the topic is a different kettle of fish. And I'm saying that as a complete fan of Erlang, it is one of the few pieces of software 'done right' that I'm aware of. Unfortunately that comes at the price of being unsuitable for highly optimized number crunching unless you want to break out the C-compiler and write an extension. Python is similar, but there the extensions have been grafted on so well lots of people forget that they are not part of the language itself. In the Erlang world you'd have a lot of leaks and conversions to make something like that work and it would likely never be as transparent as python, which in many ways is both the new BASIC and the new FORTRAN.
kevinten10 6mo ago
[flagged]
jpecar 6mo ago
All these fancy HPC languages are all nice and dandy, but the hard reality I see on our cluster is that most of the work is done in Python, R and even Perl and awk. MPI barely reached us and people still prefer huge single machines to proper distributed computing. Yeah, bioinformatics is from another planet.
- jpecar 6mo ago
  To add on this, what I see gaining traction are "workflow managers", tools that let people specify flow of data through various tools. These can figure out how to parallelize things on their own so users are not burdened with this task. So from what I see actual programming language doesn't matter as much as how the work is organized. Anything helping people simplify this task is of immediate benefit to the science.
  - jkh1 6mo ago
    Most of the time in bio-related fields, we need high-throughput computing not high-performance computing.
  - JohnWabwire 6mo ago
    [dead]
- jltsiren 6mo ago
  Bioinformatics is an outlier within HPC. It's less about numerical computing and more about processing string data with weird algorithms and data structures that are rarely used anywhere else. Distributed computing never really took off in bioinformatics, because most tasks are conveniently small. For example, a human genome is small enough that you can run most tasks involving a single genome on an average cost-effective server in a reasonable time. And that was already true 10–15 years ago. And if you have a lot of data, it usually means that you have many independent tasks. Which is nice from the perspective of a tool developer. You don't have to deal with the bureaucracy of distributed computing, as it's the user's responsibility. C++ is popular for developing bioinformatics tools. Some core tools are written in C, but actual C developers are rare. And Rust has become popular with new projects — to the extent that I haven't really seen C++20 or newer in the field.
DamonHD 6mo ago
I used to edit an HPC trade rag in the early 90s, so this was an interesting read!
crabbone 6mo ago
As someone who worked for a while and still works in HPC, my impression from this field as compared to eg. programming in finance sector or programming for storage sector is that... HPC is so backwards and far behind, it's really amazing how it's portrayed as some sort of a champion of the field. That's not to say that new things don't happen there, it's just that I find a lot of old stuff that was shown to be bad decades ago still being in vogue in HPC. Probably because it's a relatively small field with a lot of people there being academics and not a lot of migration to/from other fields. You've probably never heard of `module` (either Tcl or Lmod). This is a staple of HPC world. What this thing does is it sources or (tries to) remove some shell variables and functions into the shell used either interactively or by a batch job. This is a beyond atrocious idea to handle your working environment. The information leaks, becomes stale, you often end up loading the wrong thing into your environment. It's simply amazing how bad this thing is. And yet, it's just everywhere in HPC. Another example: running anything in HPC, basically, means running Slurm batch jobs. There are alternatives, but those are even worse (eg. OpenPBS). When you dig into the configuration of these tools, you realize they've been written for pre-systemd Linux and are held together by a shoestring of shell scripting. They seldom if at all do the right thing when it comes to logging or general integration with the environment they run in. They can be simultaneously on the bleeding edge (eg. cgroup integration or accelerator driver integration) and be completely backwards when it comes to having a sensible service definition for systemd (eg. try to manage their service dependencies on their own instead of relying on systemd to do that for them). In other words, imagine a steam-punk world, but now it's in software. That's sort of how HPC feels like after a decade or so in more popular programming fields. Also, a lot of code written for HPC is written the way it is not because the writer chose the language or the environment. The typical setup is: university IT created a cluster with whatever tools they managed to put there eons ago, and you, the code writer, have to deal with... using CentOS6 by authenticating to university's AD... in your browser... through JupyterLab interface. And there's nothing you can do about it because the IT isn't there, is incompetent to the bone and as long as you can get your work done somehow, you'd prefer that over fighting to perfect your toolchain. Bottom line, unless a language somehow becomes indispensable in this world, no matter its advantages, it's not going to be used because of the huge inertia and general unwillingness to do beyond the minimum.
chinabot 6mo ago
There has been a very big adoption of ENGLISH as a programming language in the last year or so, and, painful as it sounds, AI is already generating machine code without compilers, so let's see where we are in 2030.
RhysU 6mo ago
> we have failed to broadly adopt any new compiled programming languages for HPC The article neglects that all of C, C++, and Fortran have evolved over the last 30 years. Also, you'll find significant advances in the HPC library ecosystem over the trailing years. Consider, for example, Trilinos (https://trilinos.github.io/index.html https://trilinos.github.io/index.html) or Dakota (https://dakota.sandia.gov/about-dakota/ https://dakota.sandia.gov/about-dakota/) both of which push a ton of domain-agnostic capabilities into a C++ library instead of bolting them into a bespoke language. Communities of users tend to coalesce around shared libraries not creating new languages.
- pjmlp 6mo ago
  The authors are aware, as the Chapel compiler makes use of LLVM.
  - deleted 6mo ago
    [deleted]
  - RhysU 6mo ago
    The author's framing "we have failed" suggests otherwise. This section, https://chapel-lang.org/blog/posts/30years/#ok-then-why https://chapel-lang.org/blog/posts/30years/#ok-then-why, does not mention libraries at all.
    - pjmlp 6mo ago
      Not really, you should actually read that section a few times as well. > A fact of life in HPC is that the community has many large, long-lived codes written in languages like Fortran, C, and C++ that remain important. Such codes keep those languages at the forefront of peoples’ minds and sometimes lead to the belief that we can’t adopt new languages. > In large part because of the previous point, our programming notations tend to take a bottom-up approach. “What does this new hardware do, and how can we expose it to the programmer from C/C++?” The result is the mash-up of notations that we have today, like C++, MPI, OpenMP, and CUDA. While they allow us to program our systems, and are sufficient for doing so, they also leave a lot to be desired as compared to providing higher-level approaches that abstract away the specifics of the target hardware. Nothing there suggests the languages don't improve, especially anyone that follows ISO knows where many of improvements to Fortran, C and C++ are coming from. For example, C++26 is probably going to get BLAS into the standard library, senders/receivers is being sponsored by CUDA money. Another thing you missed from the author background, is that Chapel is sponsored by HPE and Intel, and one of the main targets are HPE Cray EX/XC systems, they know pretty well what is happening.
chatmasta 6mo ago
HPC is heavily skewed toward academia, and it doesn’t have a lot of overlap with compiler nerds. I think this explains it.
Xcelerate 6mo ago
I wonder how much of the programming language problem is due to churn of the user base. Looking over many comments in this thread, I see “Oh, back when I did HPC...” I used Titan for my own work back in 2012. But after my PhD, I never touched HPC again. So the people writing the code use what’s there but don’t stay long enough to help incentivize new or better languages. Now on the hardware side (e.g., design of interconnects), that more commonly seems to be a full career. The other issue is that to really get the value out of these machines, you sort of have to tailor your code to the machine itself to some degree. The DOE likes to fund projects that really show off the unique capabilities of supercomputers, and if your project could in principle be done on the cloud or a university cluster, it’s likely to be rejected at the proposal stage. So it’s sort of “all or nothing” in the sense that many codebases for HPC are one-off or even have machine-specific adaptations (e.g., see LAMMPS). No new general purpose language would really make this easier.
- kinow 6mo ago
  Churn of the user base could be playing a role in this, but I think it may not be too significant. In Europe there are multiple universities with HPC masters, which provide new users/devs to HPC. I worked with HPCs in New Zealand, and now I am doing the same in Spain. We hire multiple people from other HPC centers in Germany/UK/Italy, and equally lose people to those sites. I think the field is actually increasing with AI, digital twins, more industry projects (CFD, oil, models for fisheries, simulations for health diseases, etc.).
nnevatie 6mo ago
Usually a new language is facing the ecosystem mass issue - the previously used language, e.g., C++ has already the critical mass with available libraries and frameworks. Getting to the same level of ecosystem maturity with a new language will take a long time, as seen with Rust.
ivell 6mo ago
I think Mojo has a good chance to become suitable for HPC.
- rirze 6mo ago
  I don't understand why people are so excited for Mojo. I don't get the impression it will replace anything but computational scripting that was done previously in Python. HPC is a different beast as far as I'm aware.
  - convolvatron 6mo ago
    I don't think that's accurate. Mojo is explicitly compiling tensor graphs that run on accelerators. it's not like PyTorch where python is providing the chassis but not the engine. I don't think its going to be a good general HPC language just because its targeting a specific set of AI workloads, but they have shown some examples of synthesizing code which is comparable to hand-written kernels. but its not out of the question from first principles
  - pjmlp 6mo ago
    Additionally, as Intel, AMD and NVidia embrace Python JITs for their GPGPU, across all three major desktop OSes, the Mojo value becomes devalued. Why rewrite in Mojo, if Intel, AMD and NVidia SDKs give first class support for Python, including writing kernels in a language subset.
shevy-java 6mo ago
> Could the reason be that language design is dead, as was asserted by an anonymous reviewer on one of our team’s papers ~30 years ago? It may not be dead, but it seems much harder for languages to gain adoption. I think there are several reasons; I also suspect AI contributes a bit to this. People usually specialize in one or two language, so the more languages exist, the less variety we may see with regards to people ACTUALLY using the language. If I would be, say, 15 years old, I may pick up python and just stick with it rather than experiment and try out many languages. Or perhaps not even write software at all, if AI auto-writes most of it anyway.
pklausler 6mo ago
Honestly, if a language can't succeed in HPC alongside (or against) Fortran with its glacial rate of buggy evolution and poor track record of portability, and C++ with its never-ending attempts at parallelism, then it's not what HPC needs. (What HPC does need, IMNSHO, is to disband or disregard WG5/J3, get people who know what they're doing to fix the features they've botched or neglected for thirty years, and then have new procurements include RFCs that demand the fixed portable Fortran from system integrators rather than the ISO "standard".)
hpcgroup 6mo ago
[dead]
hpcdude 6mo ago
HPCdude here, and this is a mostly correct article, but here are what it misses: 1) It mentions in passing the hardware abstraction not being as universal as it seemed. This is more and more true, once we started doing fpgas, then asics, and as ARM and other platforms starting making headway, it fractured things a bit. GPUs too: I'm still a bit upset about CUDA winning over OpenCL, but Vulkan compute gives me hope. I haven't messed with SYCL but it might be a future possibility too. 2) The real crux is the practical, production interfaces that HPC end users get. Normally I'm not exposing an entire cluster (or sub cluster) to a researcher. I give them predefined tools which handle the computation across nodes for them (SLURM is old but still huge in the HPC space for a reason! When I searched the article for "slurm" I got 0 hits!) When it comes to science, reproducibility is the name of the game, and having more idempotent and repeatable code structures are what help us gain real insights that others can verify or take and run with into new problem/solutions. Ad-hoc HPC programming doesn't do that well, like existing languages slurm and other orchestration layers handle. Sidenote: One of the biggest advances recently is in RDMA improvements (remote direct memory access), because the RAM needs of these datasets are growing to crazy numbers, and often you have nodes being underutilized that are happy to help. I've only done RoCE myself though and not much with Infiniband, (sorry Yale, thats why I flubbed on the interview) but honestly, I still really like RoCE for cluster side and LACP for front facing ingress/egress. The point is existing tooling can be massaged and often we don't need new languages. I did some work with Mellanox/Weka prior to them being bought by Nvidia on optimizing the kernel shims for NFSv4 for example. Old tech made fast again.
- ablob 6mo ago
  afaik, you always have to break open abstractions for more performance. If you ignore cache-levels in your program you're gonna have a bad time - and depending on the system the layout (and with it how you should use it) is different. The same is true for how machines are interconnected. Depending on the wiring you have different throughput-values when sharing data between nodes. The whole area screams "not universal" to me.
  - suuuuuuuu 6mo ago
    I think people working on these abstractions would claim that an appropriate abstraction does involve the exact features you mention. Cache hierarchies and vectorization are common to CPUs and GPUs - in some sense, the numbers parametrizing them are all that differs. With a good abstraction, this machine-tuning can be automated.
guywithahat 6mo ago
I like the idea of chapel, but I'm not sure I agree with a lot of their design choices. Some of the parallelization features seem like they just copied OpenMP without meaningfully improving on it. They also kept exceptions, which are generally on their way out, especially in compiled languages (Go, Rust, Zig, and while they exist in modern C++ they are introducing more ways to not use them). I think a new HPC language is possible but I'm not sure this is the one
- bradcray 6mo ago
  I disagree with the characterization that Chapel's parallelization features copied OpenMP without improving upon it: * Chapel's support for task parallelism predates OpenMP's (~2004 vs. ~2007, where Wikipedia cites Chapel's tasks as being inspiration for OpenMP's, along with Cilk and X10). Chapel's tasks are also arguably more general-purpose (akin to threads) in terms of their ability to synchronize, support data-driven producer/consumer patterns, etc. * Chapel's forall loops are similar to OpenMP's loop-based parallelization pragmas, though OpenMP wasn't a source of inspiration in their design. Where OpenMP pragmas select from a menu of parallelization strategies baked into the specification and implementation, Chapel's forall loops invoke user-defined parallel iterators that permit abstracting a particular parallel pattern (say, multidimensional tiled iteration or tree traversal) into a named subroutine. These iterators can optionally be made methods of data structures and/or placed within libraries, and can be re-used across a program or multiple programs. One such library, DynamicIters, was community-contributed and specifically inspired by OpenMP's dynamic and guided scheduling strategies. * Chapel supports parallel zippered iteration, in which two or more data structures and/or parallel iterators can be traversed in a coordinated manner. * Chapel's parallelism can span multiple compute nodes via its shared namespace, which obviates the need for explicit communication; whereas OpenMP is limited to a single compute node or process unless mixed with MPI, SHMEM, or the like (and even then, OpenMP doesn't gain a cross-node view of parallel computation). * In Chapel, parallelism can be expressed implicitly, for example, by passing an array argument to a subroutine or operator that is expecting a scalar (e.g., `var B = sin(A);` or `var C = A + B;`).
saltcured 6mo ago
I was a student intern in a parallel computing research group around that first reference point of 1995. My career went other ways, working more on distributed systems instead of programming language theory or implementation. But, when I encountered OpenCL and CUDA about ten years ago, I was struck by just how much these were delivering the SPMD parallel programming model in finished products. Around 1995, these were often C dialects with some wonky compiler that each research group just barely kept together. By 2015, they were just bundled up inside a graphics driver or similarly commoditized runtime environment. Also, the GPU of 2015 was delivering the throughput we dreamed of in supercomputers back then. A teraFLOP went from a strategic theme to something you could deploy to your desktop.
rramadass 6mo ago
Relevant: The Art of High Performance Computing (a comprehensive series of textbooks) - https://theartofhpc.com/ https://theartofhpc.com/ Previous discussion - https://news.ycombinator.com/item?id=38815334 https://news.ycombinator.com/item?id=38815334

146 points by matt_d 6mo ago

Perhaps one issue lacking discussion in the article is how easy it is to find devs? I've never worked in HPC but it seems it should be relatively simple to find a C/C++ dev that can pick up OpenMP, or one that already knows it, compared to hiring people who know Chapel. The "scaling down" factor (how easy or interesting it is to use tool X for small use) seems a disadvantage of HPC-only languages, which creates a barrier to entry and a reduction in available workforce.

kinow 6mo ago
I think hpc devs need an extra set of skills that are not so common. Such as parallel file systems, batch schedulers, NUMA, infiniband, and probably some domain-specific knowledge for the apps they will develop. This knowledge is also probably a bit niche, like climate modelling, earthquake simulation, lidar data processing, and so it goes. And even knowing OpenMP or MPI may not suffice if the site uses older versions or heterogeneous approaches with CUDA, FPGA, etc. Knowing the language and the shared/distributed mem libs help, but if your project needs a new senior dev than it may be a bit hard to find (although popularity of company/HPC, salary, and location also play a role).
- physicsguy 6mo ago
  You tend to only learn these things as they become a problem too. That's super super domain specific and it doesn't always translate between areas of research. So for e.g. when I did HPC simulation codes in magnetics, there was little point focusing on some of these areas because our codes were dominated by the long-range interaction cost which limited compute scaling. All of our effort was tuning those algorithms to the absolute max. We tried heterogenous CPU + GPU but had very mixed results, and at that time (2010s) the GPU memory wasn't large enough for the problems we cared about either. I then moved to CFD in industry. The concerns there were totally different since everything is grid local. Partitioning over multi-GPU is simple since only the boundaries need to be exchanged on each iteration. The problems there were much more on the memory bandwidth and parallel file system performance side. Basically, you have to learn to solve whatever challenges get thrown up by the specific domain problem. > And even knowing OpenMP or MPI may not suffice if the site uses older versions To be fair, you always have the option of compiling yourself, but most people I met in academia didn't have the background to do this. Spack and EasyBuild make this much much easier.
- bluedino 6mo ago
  Most developers have no clue about any of that stuff. It's all abstracted out.
KaiserPro 6mo ago
I worked in HPC adjacent fields for a while (up until 40gig ethernet was cheap enough to roll out to all the edge nodes) There are a couple of big things that are difficult to get your head around: 1) when and where to dispatch and split jobs (ie whats the setup cost of spinning up n binaries on n machines vs threading on y machines) 2) data exchange primitives, Shared file systems have quirks, and a they differ from system to system. But most of the time its better/easier/faster to dump shit to a file system than some fancy database/object store. Until its not. Distributed queues are great, unless you're using them wrong. Most of the time you need to use them wrong. (the share memory RPC is a whole another beast that fortunatly I've never had to work with directly. ) 3) dealing with odd failures. As the number of parallel jobs increase the chance of getting a failure reaches 1. You need to bake in failure modes at the very start. 4) loading/saving data is often a bottle neck, lots of efficiecny comes from being clever in what you load, and _where_ you load it. (ie you have data affinity, which might be location based, or topology based, and you don't often have control over where your stuff is placed.)

jandrewrogers 6mo ago

I can easily explain this, having worked in this space. The new languages don’t actually solve any urgent problems. How people imagine scalable parallelism works and how it actually works doesn’t have a lot of overlap. The code is often boringly single-threaded because that is optimal for performance. The single biggest resource limit in most HPC code is memory bandwidth. If you are not addressing this then you are not addressing a real problem for most applications. For better or worse, C++ is really good at optimizing for memory bandwidth. Most of the suggested alternative languages are not. It is that simple. The new languages address irrelevant problems. It is really difficult to design a language that is more friendly to memory bandwidth than C++. And that is the resource you desperately need to optimize for in most cases.

bruce343434 6mo ago
What does it mean to be friendly to memory bandwidth, and why does C++ excel at it, over, say, Fortran or C or Rust?
- lugu 6mo ago
  Parent talks about new languages, as per the article Fortran or C doing fine. I speculate the benefit of C++ over Rust how it let programmers instruct the compiler of warranty that goes beyong the initial semantic of the language. See __restrict, __builtin_prefetch and __builtin_assume_aligned. The programming language is a space for conversations between compiler builders and hardware designers.
  - _flux 6mo ago
    I believe __restrict, and __builtin_prefetch/__builtin_assume are compiler extensions, not part of the C++ language as is, and different compilers implement (or don't) these differently. The rust compiler actually has similar things, but they're not available in stable builds. I suppose there are some issues if principle why not to include them in stable. E.g: https://doc.rust-lang.org/std/intrinsics/fn.prefetch_read_data.html https://doc.rust-lang.org/std/intrinsics/fn.prefetch_read_da... Maybe some time in the future good acceptable abstractions will be conceived for them.. Perhaps using just using nightly builds for HPC is not that far out, though.

swiftcoder 6mo ago

It's interesting that none of the actor-based languages ever made it into this space. Feels like something with the design philosophy of Erlang would be pretty suitable to exploit millions of cores and a variety of interconnects...

jacquesm 6mo ago
That will never happen. The overhead is massive.
- Joel_Mckay 6mo ago
  People did try to create an OTP in HDL at one point. And Erlang has already run many telecom infrastructures for decades. Surprising given how fragile the multi-host implementation has proven. Erlang/Elixir are neat languages, and right next to Julia for fun. =3
  - jacquesm 6mo ago
    Yes, it's absolutely amazing. But I'm intimately familiar with how the Erlang VM works and it would seem to me to be a very bad match for HPC on the number crunching side, though it likely would do quite well on the orchestration side but that would require the people writing the rest of the code to change their way of working completely. And given how much of that is still F77 I highly doubt they would be willing to make that investment without the promise of some massive gain.
    - Joel_Mckay 6mo ago
      For simple AMQP, it has performed rather well for our use-cases over the years. Haven't personally deployed this version yet. ymmv =3 https://www.rabbitmq.com/docs/quorum-queues https://www.rabbitmq.com/docs/quorum-queues
      - jacquesm 6mo ago
        Of course it does, that's an ideal usecase. But the topic is a different kettle of fish. And I'm saying that as a complete fan of Erlang, it is one of the few pieces of software 'done right' that I'm aware of. Unfortunately that comes at the price of being unsuitable for highly optimized number crunching unless you want to break out the C-compiler and write an extension. Python is similar, but there the extensions have been grafted on so well lots of people forget that they are not part of the language itself. In the Erlang world you'd have a lot of leaks and conversions to make something like that work and it would likely never be as transparent as python, which in many ways is both the new BASIC and the new FORTRAN.

kevinten10 6mo ago

[flagged]

jpecar 6mo ago

All these fancy HPC languages are all nice and dandy, but the hard reality I see on our cluster is that most of the work is done in Python, R and even Perl and awk. MPI barely reached us and people still prefer huge single machines to proper distributed computing. Yeah, bioinformatics is from another planet.

jpecar 6mo ago
To add on this, what I see gaining traction are "workflow managers", tools that let people specify flow of data through various tools. These can figure out how to parallelize things on their own so users are not burdened with this task. So from what I see actual programming language doesn't matter as much as how the work is organized. Anything helping people simplify this task is of immediate benefit to the science.
- jkh1 6mo ago
  Most of the time in bio-related fields, we need high-throughput computing not high-performance computing.
- JohnWabwire 6mo ago
  [dead]
jltsiren 6mo ago
Bioinformatics is an outlier within HPC. It's less about numerical computing and more about processing string data with weird algorithms and data structures that are rarely used anywhere else. Distributed computing never really took off in bioinformatics, because most tasks are conveniently small. For example, a human genome is small enough that you can run most tasks involving a single genome on an average cost-effective server in a reasonable time. And that was already true 10–15 years ago. And if you have a lot of data, it usually means that you have many independent tasks. Which is nice from the perspective of a tool developer. You don't have to deal with the bureaucracy of distributed computing, as it's the user's responsibility. C++ is popular for developing bioinformatics tools. Some core tools are written in C, but actual C developers are rare. And Rust has become popular with new projects — to the extent that I haven't really seen C++20 or newer in the field.

DamonHD 6mo ago

I used to edit an HPC trade rag in the early 90s, so this was an interesting read!

crabbone 6mo ago

As someone who worked for a while and still works in HPC, my impression from this field as compared to eg. programming in finance sector or programming for storage sector is that... HPC is so backwards and far behind, it's really amazing how it's portrayed as some sort of a champion of the field. That's not to say that new things don't happen there, it's just that I find a lot of old stuff that was shown to be bad decades ago still being in vogue in HPC. Probably because it's a relatively small field with a lot of people there being academics and not a lot of migration to/from other fields. You've probably never heard of `module` (either Tcl or Lmod). This is a staple of HPC world. What this thing does is it sources or (tries to) remove some shell variables and functions into the shell used either interactively or by a batch job. This is a beyond atrocious idea to handle your working environment. The information leaks, becomes stale, you often end up loading the wrong thing into your environment. It's simply amazing how bad this thing is. And yet, it's just everywhere in HPC. Another example: running anything in HPC, basically, means running Slurm batch jobs. There are alternatives, but those are even worse (eg. OpenPBS). When you dig into the configuration of these tools, you realize they've been written for pre-systemd Linux and are held together by a shoestring of shell scripting. They seldom if at all do the right thing when it comes to logging or general integration with the environment they run in. They can be simultaneously on the bleeding edge (eg. cgroup integration or accelerator driver integration) and be completely backwards when it comes to having a sensible service definition for systemd (eg. try to manage their service dependencies on their own instead of relying on systemd to do that for them). In other words, imagine a steam-punk world, but now it's in software. That's sort of how HPC feels like after a decade or so in more popular programming fields. Also, a lot of code written for HPC is written the way it is not because the writer chose the language or the environment. The typical setup is: university IT created a cluster with whatever tools they managed to put there eons ago, and you, the code writer, have to deal with... using CentOS6 by authenticating to university's AD... in your browser... through JupyterLab interface. And there's nothing you can do about it because the IT isn't there, is incompetent to the bone and as long as you can get your work done somehow, you'd prefer that over fighting to perfect your toolchain. Bottom line, unless a language somehow becomes indispensable in this world, no matter its advantages, it's not going to be used because of the huge inertia and general unwillingness to do beyond the minimum.

chinabot 6mo ago

There has been a very big adoption of ENGLISH as a programming language in the last year or so, and, painful as it sounds, AI is already generating machine code without compilers, so let's see where we are in 2030.

RhysU 6mo ago

> we have failed to broadly adopt any new compiled programming languages for HPC The article neglects that all of C, C++, and Fortran have evolved over the last 30 years. Also, you'll find significant advances in the HPC library ecosystem over the trailing years. Consider, for example, Trilinos (https://trilinos.github.io/index.html https://trilinos.github.io/index.html) or Dakota (https://dakota.sandia.gov/about-dakota/ https://dakota.sandia.gov/about-dakota/) both of which push a ton of domain-agnostic capabilities into a C++ library instead of bolting them into a bespoke language. Communities of users tend to coalesce around shared libraries not creating new languages.

pjmlp 6mo ago
The authors are aware, as the Chapel compiler makes use of LLVM.
- deleted 6mo ago
  [deleted]
- RhysU 6mo ago
  The author's framing "we have failed" suggests otherwise. This section, https://chapel-lang.org/blog/posts/30years/#ok-then-why https://chapel-lang.org/blog/posts/30years/#ok-then-why, does not mention libraries at all.
  - pjmlp 6mo ago
    Not really, you should actually read that section a few times as well. > A fact of life in HPC is that the community has many large, long-lived codes written in languages like Fortran, C, and C++ that remain important. Such codes keep those languages at the forefront of peoples’ minds and sometimes lead to the belief that we can’t adopt new languages. > In large part because of the previous point, our programming notations tend to take a bottom-up approach. “What does this new hardware do, and how can we expose it to the programmer from C/C++?” The result is the mash-up of notations that we have today, like C++, MPI, OpenMP, and CUDA. While they allow us to program our systems, and are sufficient for doing so, they also leave a lot to be desired as compared to providing higher-level approaches that abstract away the specifics of the target hardware. Nothing there suggests the languages don't improve, especially anyone that follows ISO knows where many of improvements to Fortran, C and C++ are coming from. For example, C++26 is probably going to get BLAS into the standard library, senders/receivers is being sponsored by CUDA money. Another thing you missed from the author background, is that Chapel is sponsored by HPE and Intel, and one of the main targets are HPE Cray EX/XC systems, they know pretty well what is happening.

chatmasta 6mo ago

HPC is heavily skewed toward academia, and it doesn’t have a lot of overlap with compiler nerds. I think this explains it.

Xcelerate 6mo ago

I wonder how much of the programming language problem is due to churn of the user base. Looking over many comments in this thread, I see “Oh, back when I did HPC...” I used Titan for my own work back in 2012. But after my PhD, I never touched HPC again. So the people writing the code use what’s there but don’t stay long enough to help incentivize new or better languages. Now on the hardware side (e.g., design of interconnects), that more commonly seems to be a full career. The other issue is that to really get the value out of these machines, you sort of have to tailor your code to the machine itself to some degree. The DOE likes to fund projects that really show off the unique capabilities of supercomputers, and if your project could in principle be done on the cloud or a university cluster, it’s likely to be rejected at the proposal stage. So it’s sort of “all or nothing” in the sense that many codebases for HPC are one-off or even have machine-specific adaptations (e.g., see LAMMPS). No new general purpose language would really make this easier.

kinow 6mo ago
Churn of the user base could be playing a role in this, but I think it may not be too significant. In Europe there are multiple universities with HPC masters, which provide new users/devs to HPC. I worked with HPCs in New Zealand, and now I am doing the same in Spain. We hire multiple people from other HPC centers in Germany/UK/Italy, and equally lose people to those sites. I think the field is actually increasing with AI, digital twins, more industry projects (CFD, oil, models for fisheries, simulations for health diseases, etc.).

nnevatie 6mo ago

Usually a new language is facing the ecosystem mass issue - the previously used language, e.g., C++ has already the critical mass with available libraries and frameworks. Getting to the same level of ecosystem maturity with a new language will take a long time, as seen with Rust.

ivell 6mo ago

I think Mojo has a good chance to become suitable for HPC.

rirze 6mo ago
I don't understand why people are so excited for Mojo. I don't get the impression it will replace anything but computational scripting that was done previously in Python. HPC is a different beast as far as I'm aware.
- convolvatron 6mo ago
  I don't think that's accurate. Mojo is explicitly compiling tensor graphs that run on accelerators. it's not like PyTorch where python is providing the chassis but not the engine. I don't think its going to be a good general HPC language just because its targeting a specific set of AI workloads, but they have shown some examples of synthesizing code which is comparable to hand-written kernels. but its not out of the question from first principles
- pjmlp 6mo ago
  Additionally, as Intel, AMD and NVidia embrace Python JITs for their GPGPU, across all three major desktop OSes, the Mojo value becomes devalued. Why rewrite in Mojo, if Intel, AMD and NVidia SDKs give first class support for Python, including writing kernels in a language subset.

shevy-java 6mo ago

> Could the reason be that language design is dead, as was asserted by an anonymous reviewer on one of our team’s papers ~30 years ago? It may not be dead, but it seems much harder for languages to gain adoption. I think there are several reasons; I also suspect AI contributes a bit to this. People usually specialize in one or two language, so the more languages exist, the less variety we may see with regards to people ACTUALLY using the language. If I would be, say, 15 years old, I may pick up python and just stick with it rather than experiment and try out many languages. Or perhaps not even write software at all, if AI auto-writes most of it anyway.

pklausler 6mo ago

Honestly, if a language can't succeed in HPC alongside (or against) Fortran with its glacial rate of buggy evolution and poor track record of portability, and C++ with its never-ending attempts at parallelism, then it's not what HPC needs. (What HPC does need, IMNSHO, is to disband or disregard WG5/J3, get people who know what they're doing to fix the features they've botched or neglected for thirty years, and then have new procurements include RFCs that demand the fixed portable Fortran from system integrators rather than the ISO "standard".)

hpcgroup 6mo ago

[dead]

hpcdude 6mo ago

HPCdude here, and this is a mostly correct article, but here are what it misses: 1) It mentions in passing the hardware abstraction not being as universal as it seemed. This is more and more true, once we started doing fpgas, then asics, and as ARM and other platforms starting making headway, it fractured things a bit. GPUs too: I'm still a bit upset about CUDA winning over OpenCL, but Vulkan compute gives me hope. I haven't messed with SYCL but it might be a future possibility too. 2) The real crux is the practical, production interfaces that HPC end users get. Normally I'm not exposing an entire cluster (or sub cluster) to a researcher. I give them predefined tools which handle the computation across nodes for them (SLURM is old but still huge in the HPC space for a reason! When I searched the article for "slurm" I got 0 hits!) When it comes to science, reproducibility is the name of the game, and having more idempotent and repeatable code structures are what help us gain real insights that others can verify or take and run with into new problem/solutions. Ad-hoc HPC programming doesn't do that well, like existing languages slurm and other orchestration layers handle. Sidenote: One of the biggest advances recently is in RDMA improvements (remote direct memory access), because the RAM needs of these datasets are growing to crazy numbers, and often you have nodes being underutilized that are happy to help. I've only done RoCE myself though and not much with Infiniband, (sorry Yale, thats why I flubbed on the interview) but honestly, I still really like RoCE for cluster side and LACP for front facing ingress/egress. The point is existing tooling can be massaged and often we don't need new languages. I did some work with Mellanox/Weka prior to them being bought by Nvidia on optimizing the kernel shims for NFSv4 for example. Old tech made fast again.

ablob 6mo ago
afaik, you always have to break open abstractions for more performance. If you ignore cache-levels in your program you're gonna have a bad time - and depending on the system the layout (and with it how you should use it) is different. The same is true for how machines are interconnected. Depending on the wiring you have different throughput-values when sharing data between nodes. The whole area screams "not universal" to me.
- suuuuuuuu 6mo ago
  I think people working on these abstractions would claim that an appropriate abstraction does involve the exact features you mention. Cache hierarchies and vectorization are common to CPUs and GPUs - in some sense, the numbers parametrizing them are all that differs. With a good abstraction, this machine-tuning can be automated.

guywithahat 6mo ago

I like the idea of chapel, but I'm not sure I agree with a lot of their design choices. Some of the parallelization features seem like they just copied OpenMP without meaningfully improving on it. They also kept exceptions, which are generally on their way out, especially in compiled languages (Go, Rust, Zig, and while they exist in modern C++ they are introducing more ways to not use them). I think a new HPC language is possible but I'm not sure this is the one

bradcray 6mo ago
I disagree with the characterization that Chapel's parallelization features copied OpenMP without improving upon it: * Chapel's support for task parallelism predates OpenMP's (~2004 vs. ~2007, where Wikipedia cites Chapel's tasks as being inspiration for OpenMP's, along with Cilk and X10). Chapel's tasks are also arguably more general-purpose (akin to threads) in terms of their ability to synchronize, support data-driven producer/consumer patterns, etc. * Chapel's forall loops are similar to OpenMP's loop-based parallelization pragmas, though OpenMP wasn't a source of inspiration in their design. Where OpenMP pragmas select from a menu of parallelization strategies baked into the specification and implementation, Chapel's forall loops invoke user-defined parallel iterators that permit abstracting a particular parallel pattern (say, multidimensional tiled iteration or tree traversal) into a named subroutine. These iterators can optionally be made methods of data structures and/or placed within libraries, and can be re-used across a program or multiple programs. One such library, DynamicIters, was community-contributed and specifically inspired by OpenMP's dynamic and guided scheduling strategies. * Chapel supports parallel zippered iteration, in which two or more data structures and/or parallel iterators can be traversed in a coordinated manner. * Chapel's parallelism can span multiple compute nodes via its shared namespace, which obviates the need for explicit communication; whereas OpenMP is limited to a single compute node or process unless mixed with MPI, SHMEM, or the like (and even then, OpenMP doesn't gain a cross-node view of parallel computation). * In Chapel, parallelism can be expressed implicitly, for example, by passing an array argument to a subroutine or operator that is expecting a scalar (e.g., `var B = sin(A);` or `var C = A + B;`).

saltcured 6mo ago

I was a student intern in a parallel computing research group around that first reference point of 1995. My career went other ways, working more on distributed systems instead of programming language theory or implementation. But, when I encountered OpenCL and CUDA about ten years ago, I was struck by just how much these were delivering the SPMD parallel programming model in finished products. Around 1995, these were often C dialects with some wonky compiler that each research group just barely kept together. By 2015, they were just bundled up inside a graphics driver or similarly commoditized runtime environment. Also, the GPU of 2015 was delivering the throughput we dreamed of in supercomputers back then. A teraFLOP went from a strategic theme to something you could deploy to your desktop.

rramadass 6mo ago

Relevant: The Art of High Performance Computing (a comprehensive series of textbooks) - https://theartofhpc.com/ https://theartofhpc.com/ Previous discussion - https://news.ycombinator.com/item?id=38815334 https://news.ycombinator.com/item?id=38815334