4 ms·

I can't help thinking that we need a new scientific programming language that is properly vectorised and directly targets GPUs, as opposed to all these hacks th

I can't help thinking that we need a new scientific programming language that is properly vectorised and directly targets GPUs, as opposed to all these hacks that unroll loops. Now that a large proportion of data science and scientfic computing is GPU centric, aren't there new languages designed with these parallel architectures foremost in mind? Doesn't functional programming map very well to this? It almost seems to me that just as we're about to get c-speed in Python via JIT tech, so it'll almost immediately be left behind because the GPU is where it's at.

fulafel 10y ago
There are some attempts, like http://futhark-lang.org/ http://futhark-lang.org/ One problem is that you have to codegen to meager intermediate languages that are fragmented & buggy (OpenCL or GLSL), or go proprietary with just one set of bugs and features (CUDA) but with much limited user base. Also GPU software stacks are very crashy, have roots in the culture of testing all shaders for driver bugs before shipping in apps ("your game/app is buggy and poorly qa'd if user pc crashes") and only recently has been slightly exposed to dynamic workloads by WebGL (which is only a OpenGL ES 2 subset, impls invest major engineering effort to work around driver bugs, and still had/has big crash problems).
- vegabook 10y ago
  Do you think that Vulkan, which also provides a compute API, and appears to be gaining rapid momentum, might solve some of these issues?
  - fulafel 10y ago
    Vulkan's SPIR-V might turn out to be a good common target eventually. There's potential but it's all in the execution, is it going to be good enough on majority of platforms? Then there's Apple who are anti-Vulkan so there goes one major platform. It's also possible that everyone ends up implementing CUDA (AMD, Google, OTOY, PGI have impls out or underway) and it'll get supported as a codegen target.
  - Athas 10y ago
    OpenCL is fine, but not well supported (because NVIDIA). Vulkan is also fine, so again, the only question is support. I don't know whether Vulkan can supplant OpenCL for computation, or whether it's even supposed to. None of these solve the problem that the underlying drivers are still terrible (when I test Futhark code I routinely have to restart our GPU servers, as the NVIDIA kernel module crashes). Hopefully, the smaller surface area of OpenCL and Vulkan compared to OpenGL will eventually result in better drivers. I am also very hopeful about AMDs switch to an open source driver model.
Athas 10y ago
One problem is that functional programming style is often focused on recursion - not just in its functions, but also its data structures. Trees, linked lists and graphs are GPU poison, while arrays map very well to the hardware. However, arrays don't have the pleasant recursive structure that allows you to encode nifty properties (and do pattern matching) the way that is popular in functional programming. However, array programming is basically functional programming, and does map very well to parallel execution in general. While most array programming is nowadays of the fairly simple form you see in libraries like Numpy, older languages like APL show how far you can go. It's a somewhat alien style of programming to most modern programmers, though. My rule of thumb: parallel programming is hard. Functional programming makes it trivial to make parallel programs that are correct, but they still might not be fast.
- infinite8s 10y ago
  If those recursive structures are static then it's fairly easy to map them into an array form.
- vegabook 10y ago
  Is Rust the right way forward then? Functional enough without obsessing about linked-memory structures / recursion etc?
  - Athas 10y ago
    I don't think so. Rust has excellent support for concurrency, but does not have any interesting parallel constructs. Which means that while it can probably be fine for coarse-grained multicore parallelism, it does not provide the fine-grain parallelism you need on something like a GPU.
    - steveklabnik 10y ago
      Not built in, but there are libraries, like rayon or crossbeam.

fulafel 10y ago

There are some attempts, like http://futhark-lang.org/ http://futhark-lang.org/ One problem is that you have to codegen to meager intermediate languages that are fragmented & buggy (OpenCL or GLSL), or go proprietary with just one set of bugs and features (CUDA) but with much limited user base. Also GPU software stacks are very crashy, have roots in the culture of testing all shaders for driver bugs before shipping in apps ("your game/app is buggy and poorly qa'd if user pc crashes") and only recently has been slightly exposed to dynamic workloads by WebGL (which is only a OpenGL ES 2 subset, impls invest major engineering effort to work around driver bugs, and still had/has big crash problems).

vegabook 10y ago
Do you think that Vulkan, which also provides a compute API, and appears to be gaining rapid momentum, might solve some of these issues?
- fulafel 10y ago
  Vulkan's SPIR-V might turn out to be a good common target eventually. There's potential but it's all in the execution, is it going to be good enough on majority of platforms? Then there's Apple who are anti-Vulkan so there goes one major platform. It's also possible that everyone ends up implementing CUDA (AMD, Google, OTOY, PGI have impls out or underway) and it'll get supported as a codegen target.
- Athas 10y ago
  OpenCL is fine, but not well supported (because NVIDIA). Vulkan is also fine, so again, the only question is support. I don't know whether Vulkan can supplant OpenCL for computation, or whether it's even supposed to. None of these solve the problem that the underlying drivers are still terrible (when I test Futhark code I routinely have to restart our GPU servers, as the NVIDIA kernel module crashes). Hopefully, the smaller surface area of OpenCL and Vulkan compared to OpenGL will eventually result in better drivers. I am also very hopeful about AMDs switch to an open source driver model.

Athas 10y ago

One problem is that functional programming style is often focused on recursion - not just in its functions, but also its data structures. Trees, linked lists and graphs are GPU poison, while arrays map very well to the hardware. However, arrays don't have the pleasant recursive structure that allows you to encode nifty properties (and do pattern matching) the way that is popular in functional programming. However, array programming is basically functional programming, and does map very well to parallel execution in general. While most array programming is nowadays of the fairly simple form you see in libraries like Numpy, older languages like APL show how far you can go. It's a somewhat alien style of programming to most modern programmers, though. My rule of thumb: parallel programming is hard. Functional programming makes it trivial to make parallel programs that are correct, but they still might not be fast.

infinite8s 10y ago
If those recursive structures are static then it's fairly easy to map them into an array form.
vegabook 10y ago
Is Rust the right way forward then? Functional enough without obsessing about linked-memory structures / recursion etc?
- Athas 10y ago
  I don't think so. Rust has excellent support for concurrency, but does not have any interesting parallel constructs. Which means that while it can probably be fine for coarse-grained multicore parallelism, it does not provide the fine-grain parallelism you need on something like a GPU.
  - steveklabnik 10y ago
    Not built in, but there are libraries, like rayon or crossbeam.