4 ms·

I started with Lush a long time ago before being obliged to switch to MATLAB and then C. Recently I have been trying to learn Futhark which is a ML like langua

I started with Lush a long time ago before being obliged to switch to MATLAB and then C. Recently I have been trying to learn Futhark which is a ML like language generating parallel code (E.g. OpenCL or CUDA) from pure functional code with nested array structures. I find this exciting because when using the GPU with a thin layer like OpenCL directly, you have to write kernels from scratch without much abstraction possible. Higher level APIs like PyTorch or Jax give you easy ways to flexibly structure the computation but you just have to hope for kernel fusion. In Futhark this is in principle explicit and well handled. ISPC is another example of a C like language where SIMD style parallelism is unambiguous while staying relatively high level.

p1esk 6y ago
If your primary interest is in parallel code that runs on a GPU, and not autograd features of Pytorch/Jax, take a look at CuPy: https://cupy.dev/ https://cupy.dev/
- marmaduke 6y ago
  Right, but does CuPy fuse kernels? I can’t find anything in the docs about it.
  - p1esk 6y ago
    Not sure, but you can write your own cuda kernels inline if you want to.
- dTal 6y ago
  That's extremely cool. Julia has something very similar, in the form of CuArrays [0]. I do wish we had some better cross-platform solutions though, instead of cementing nVidia's hegemony. There's no technical reason why GPGPU should be any more difficult on AMD hardware. [0] https://github.com/JuliaGPU/CuArrays.jl https://github.com/JuliaGPU/CuArrays.jl
  - eigenspace 6y ago
    > I do wish we had some better cross-platform solutions though, instead of cementing nVidia's hegemony. There's no technical reason why GPGPU should be any more difficult on AMD hardware. Then look no further than GPUCompiler.jl. Tim factored out all the platform agnostic parts of CUDA.jl (successor of CuArrays.jl and CudaNative.jl) into GPUCompiler.jl and that is now being used by the AMDGPU.jl package. It works today!
    - dTal 6y ago
      Thanks! I had hit some of those pages before, but their ontology isn't immediately obvious, so thanks for the clarification. In particular I've definitely looked at AMDGPU.jl recently, but as the page says: "not all features (and especially performance) are up to par with CUDA.jl" - hence my impression of AMD being a second-class citizen. Still, it's great that we have anything at all, and I'm extremely grateful for the hard work of those in the Julia community. It's a remarkable achievement.
      - eigenspace 6y ago
        > In particular I've definitely looked at AMDGPU.jl recently, but as the page says: "not all features (and especially performance) are up to par with CUDA.jl" - hence my impression of AMD being a second-class citizen. That just takes developer time to fix. Currently it’s basically just a one-man show.
  - marmaduke 6y ago
    OpenCL was pretty good in this a few years ago. I had some kernels which ran across AMD and Nvidia GPUs as well as many core Xeon systems without changing anything. To squeeze the highest performance out, you still had to tune for the device, but it was a great start. Now, all vendors seem to have dropped support for OpenCL, though it will live on nicely in PortableCL, which implements it atop LLVM, allowing it to target anything LLVM does, which finally goes further to achieve the open-ness the original standard aspired to, than the vendors themselves at the time.
scottlocklin 6y ago
Yeah, I've been looking for some higher level way of taking advantage of GPUs. Pascal Jasmin wrote a doodad for J talking to Arrayfire, but I never got it to work properly. https://github.com/Pascal-J/Jfire https://github.com/Pascal-J/Jfire Lush: it just felt comfy for a data science workflow. No DB adapters, or even real csv readers, but you could do whatever you needed in the interpreted language and go pretty fast with the compiled bits. Still feels like the future.

p1esk 6y ago

If your primary interest is in parallel code that runs on a GPU, and not autograd features of Pytorch/Jax, take a look at CuPy: https://cupy.dev/ https://cupy.dev/

marmaduke 6y ago
Right, but does CuPy fuse kernels? I can’t find anything in the docs about it.
- p1esk 6y ago
  Not sure, but you can write your own cuda kernels inline if you want to.
dTal 6y ago
That's extremely cool. Julia has something very similar, in the form of CuArrays [0]. I do wish we had some better cross-platform solutions though, instead of cementing nVidia's hegemony. There's no technical reason why GPGPU should be any more difficult on AMD hardware. [0] https://github.com/JuliaGPU/CuArrays.jl https://github.com/JuliaGPU/CuArrays.jl
- eigenspace 6y ago
  > I do wish we had some better cross-platform solutions though, instead of cementing nVidia's hegemony. There's no technical reason why GPGPU should be any more difficult on AMD hardware. Then look no further than GPUCompiler.jl. Tim factored out all the platform agnostic parts of CUDA.jl (successor of CuArrays.jl and CudaNative.jl) into GPUCompiler.jl and that is now being used by the AMDGPU.jl package. It works today!
  - dTal 6y ago
    Thanks! I had hit some of those pages before, but their ontology isn't immediately obvious, so thanks for the clarification. In particular I've definitely looked at AMDGPU.jl recently, but as the page says: "not all features (and especially performance) are up to par with CUDA.jl" - hence my impression of AMD being a second-class citizen. Still, it's great that we have anything at all, and I'm extremely grateful for the hard work of those in the Julia community. It's a remarkable achievement.
    - eigenspace 6y ago
      > In particular I've definitely looked at AMDGPU.jl recently, but as the page says: "not all features (and especially performance) are up to par with CUDA.jl" - hence my impression of AMD being a second-class citizen. That just takes developer time to fix. Currently it’s basically just a one-man show.
- marmaduke 6y ago
  OpenCL was pretty good in this a few years ago. I had some kernels which ran across AMD and Nvidia GPUs as well as many core Xeon systems without changing anything. To squeeze the highest performance out, you still had to tune for the device, but it was a great start. Now, all vendors seem to have dropped support for OpenCL, though it will live on nicely in PortableCL, which implements it atop LLVM, allowing it to target anything LLVM does, which finally goes further to achieve the open-ness the original standard aspired to, than the vendors themselves at the time.

scottlocklin 6y ago

Yeah, I've been looking for some higher level way of taking advantage of GPUs. Pascal Jasmin wrote a doodad for J talking to Arrayfire, but I never got it to work properly. https://github.com/Pascal-J/Jfire https://github.com/Pascal-J/Jfire Lush: it just felt comfy for a data science workflow. No DB adapters, or even real csv readers, but you could do whatever you needed in the interpreted language and go pretty fast with the compiled bits. Still feels like the future.