4 ms·

So which language has handled ILP, OOO, delay slots, cache hints, etc. particularly well? Anything that's ergonomic and actually used (i.e. not awkward and acad

So which language has handled ILP, OOO, delay slots, cache hints, etc. particularly well? Anything that's ergonomic and actually used (i.e. not awkward and academic)?

gumby 5y ago
But that's my point: the dominance of a C monoculture has relegated alternatives to being "awkward and academic". When you're in a C straightjacket it's hard to take advantages of other architectures like transputer, connection machine, or cell. Ever run bsd unix on a Cray? It was dog slow because the CPU assumed very deep pipelining and always had to continually resynchronize because of the frequent branches in the C code. GPGPU is the sole recent exception, and even then it's still at the margins. And we have CUDA which is an attempt to, yes, get back to the C model. These modern CPUs benchmark pretty well, but in practice little code can really take advantage. For example the M1 is a multicore screamer, yet most apps are still restricted to a single one.
- my123 5y ago
  > And we have CUDA which is an attempt to, yes, get back to the C model CUDA allows different programming models, it's the point of having the PTX virtual machine as the exposed programming model. CUDA C++ is popular, but is far from the only option people have. (and it can go very deep, https://www.ibm.com/docs/en/sdk-java-technology/8?topic=egpulwo-how-jit-compiler-uses-gpu-linux-windows-only https://www.ibm.com/docs/en/sdk-java-technology/8?topic=egpu... for example)
  - paulmd 5y ago
    It's far less popular than the C/C++ bindings (at least at an end-user level), but NVIDIA actually has official support for FORTRAN on CUDA as well. iirc you can mix and match at the linker level, and there's a fair amount of library code that uses FORTRAN for performance reasons. FORTRAN has tighter restrictions around what you can do in terms of aliasing/etc, which allows compilers to be more aggressive about optimizations without as much undefined-behavior "this looks correct and will run fine until we add another optimization and it doesn't" shenanigans. So in practice it can be faster than C, and it's actually quite popular among the HPC community and other performance-sensitive segments. It's not like C has to be this way, with all the "undefined behavior" nonsense, it just is an artifact of a programming language written in the early 70s. C makes a lot of sense when a compiler looks a lot more like an assembler with a bit of optimization sprinkled in, and isn't performing super deep introspection and deciding that this entire function can be optimized away because of some arcane "undefined behavior" rule.
    - my123 5y ago
      > at least at an end-user level I'd say that the most popular option at the end-user level today is a whole level of abstraction above: using Python w/ PyTorch (or less frequently, TensorFlow). Combined with CuPy and cuNumeric, which is available at https://developer.nvidia.com/cunumeric https://developer.nvidia.com/cunumeric and handles scaling to multiple machines with Python-written code, through the Legate runtime (https://nv-legate.github.io/legate.core/README.html https://nv-legate.github.io/legate.core/README.html). This covers a huge amount of what the CUDA userbase uses.
- deleted 5y ago
  [deleted]
titzer 5y ago
These are all microarchitectural things that should not be surfaced in language, IMHO. This is why compilers exist, to raise the level of abstraction. For example, other than sidechannels (i.e. Spectre), OOO is not observable, nor is ILP or cache hints. Delay slots are observable at the ISA level but typically compilers do the work of dealing with them. Thank god delay slots didn't make it into programming languages; they were a wart of ISA design surfaced because of a lack of decent branch prediction. Can you imagine if you had a bizarre feature where the next line of code after a branch still executed even if the branch was taken? That'd be bonkers.
- SuchAnonMuchWow 5y ago
  OOO is definitely observable when dealing with multithreading. That's why you have things like memory order constraints on atomic operations in C, and why you need to be extremely careful with how you place things like memory fences when doing anything fine-grained between several threads.
  - titzer 5y ago
    Fair point, but I'd qualify that OOO is only observable for racy programs. In languages with shared memory concurrency, but safety, races are either statically disallowed (Rust), or their results are constrained by a (weak) memory model. Java, e.g. only makes rough guarantees that you don't see out-of-thin air values and that you can't violate the type system. Java also is a little stricter around volatile memory access (they are sequentially consistent, iirc). Also worth noting, some reorderings that you see are not due to hardware, but due to compiler optimizations. Compilers have to obey the memory model, too. Generally, most programmers shouldn't need to be reasoning about OOO or memory order and be using fences. Rather, they should use libraries that are written by experts that use extra-lingual mechanisms (like compare-swap or fences) to guarantee a higher contract.
    - zozbot234 5y ago
      Rust has atomic types that can observe races. Such "racy" code is useful because some concurrent code doesn't need the strictest guarantees (sequential consistency). Though, compared to C, Rust does not use the "Consume" memory ordering because so far it has turned out to be ill-defined.
Annatar 5y ago
Fortran. This essay sums it up rather well: https://news.ycombinator.com/item?id=31123165 https://news.ycombinator.com/item?id=31123165

So which language has handled ILP, OOO, delay slots, cache hints, etc. particularly well? Anything that's ergonomic and actually used (i.e. not awkward and academic)?

gumby 5y ago

But that's my point: the dominance of a C monoculture has relegated alternatives to being "awkward and academic". When you're in a C straightjacket it's hard to take advantages of other architectures like transputer, connection machine, or cell. Ever run bsd unix on a Cray? It was dog slow because the CPU assumed very deep pipelining and always had to continually resynchronize because of the frequent branches in the C code. GPGPU is the sole recent exception, and even then it's still at the margins. And we have CUDA which is an attempt to, yes, get back to the C model. These modern CPUs benchmark pretty well, but in practice little code can really take advantage. For example the M1 is a multicore screamer, yet most apps are still restricted to a single one.

my123 5y ago
> And we have CUDA which is an attempt to, yes, get back to the C model CUDA allows different programming models, it's the point of having the PTX virtual machine as the exposed programming model. CUDA C++ is popular, but is far from the only option people have. (and it can go very deep, https://www.ibm.com/docs/en/sdk-java-technology/8?topic=egpulwo-how-jit-compiler-uses-gpu-linux-windows-only https://www.ibm.com/docs/en/sdk-java-technology/8?topic=egpu... for example)
- paulmd 5y ago
  It's far less popular than the C/C++ bindings (at least at an end-user level), but NVIDIA actually has official support for FORTRAN on CUDA as well. iirc you can mix and match at the linker level, and there's a fair amount of library code that uses FORTRAN for performance reasons. FORTRAN has tighter restrictions around what you can do in terms of aliasing/etc, which allows compilers to be more aggressive about optimizations without as much undefined-behavior "this looks correct and will run fine until we add another optimization and it doesn't" shenanigans. So in practice it can be faster than C, and it's actually quite popular among the HPC community and other performance-sensitive segments. It's not like C has to be this way, with all the "undefined behavior" nonsense, it just is an artifact of a programming language written in the early 70s. C makes a lot of sense when a compiler looks a lot more like an assembler with a bit of optimization sprinkled in, and isn't performing super deep introspection and deciding that this entire function can be optimized away because of some arcane "undefined behavior" rule.
  - my123 5y ago
    > at least at an end-user level I'd say that the most popular option at the end-user level today is a whole level of abstraction above: using Python w/ PyTorch (or less frequently, TensorFlow). Combined with CuPy and cuNumeric, which is available at https://developer.nvidia.com/cunumeric https://developer.nvidia.com/cunumeric and handles scaling to multiple machines with Python-written code, through the Legate runtime (https://nv-legate.github.io/legate.core/README.html https://nv-legate.github.io/legate.core/README.html). This covers a huge amount of what the CUDA userbase uses.
deleted 5y ago
[deleted]

titzer 5y ago

These are all microarchitectural things that should not be surfaced in language, IMHO. This is why compilers exist, to raise the level of abstraction. For example, other than sidechannels (i.e. Spectre), OOO is not observable, nor is ILP or cache hints. Delay slots are observable at the ISA level but typically compilers do the work of dealing with them. Thank god delay slots didn't make it into programming languages; they were a wart of ISA design surfaced because of a lack of decent branch prediction. Can you imagine if you had a bizarre feature where the next line of code after a branch still executed even if the branch was taken? That'd be bonkers.

SuchAnonMuchWow 5y ago
OOO is definitely observable when dealing with multithreading. That's why you have things like memory order constraints on atomic operations in C, and why you need to be extremely careful with how you place things like memory fences when doing anything fine-grained between several threads.
- titzer 5y ago
  Fair point, but I'd qualify that OOO is only observable for racy programs. In languages with shared memory concurrency, but safety, races are either statically disallowed (Rust), or their results are constrained by a (weak) memory model. Java, e.g. only makes rough guarantees that you don't see out-of-thin air values and that you can't violate the type system. Java also is a little stricter around volatile memory access (they are sequentially consistent, iirc). Also worth noting, some reorderings that you see are not due to hardware, but due to compiler optimizations. Compilers have to obey the memory model, too. Generally, most programmers shouldn't need to be reasoning about OOO or memory order and be using fences. Rather, they should use libraries that are written by experts that use extra-lingual mechanisms (like compare-swap or fences) to guarantee a higher contract.
  - zozbot234 5y ago
    Rust has atomic types that can observe races. Such "racy" code is useful because some concurrent code doesn't need the strictest guarantees (sequential consistency). Though, compared to C, Rust does not use the "Consume" memory ordering because so far it has turned out to be ill-defined.

Annatar 5y ago

Fortran. This essay sums it up rather well: https://news.ycombinator.com/item?id=31123165 https://news.ycombinator.com/item?id=31123165