4 ms·

The follow-up paper should include the deep learning and linear algebra compilers: - Halide - Tiramisu - Ngraph - Weld - PyTorch JIT & PyTorch Glow and Ten

The follow-up paper should include the deep learning and linear algebra compilers: - Halide - Tiramisu - Ngraph - Weld - PyTorch JIT & PyTorch Glow and Tensor Comprehensions - TVM - Futhark - A toy MLIR language build on the linalg dialect. I'm sure Google would love to help them. to name the main ones. Everyone is moving towards a compiler approach. I'm actively researching the domain and I think Halide is the most promising: by separating the algorithm from the schedule you can leave the algorithm side to the researchers and optimization is done by HPC devs or automated methods without having to rewrite in C/Fortran. Much more attempts in the past: - https://github.com/mratsim/Arraymancer/issues/347#issuecomment-459351890 https://github.com/mratsim/Arraymancer/issues/347#issuecomme... - My own domain specific language and compiler: https://github.com/numforge/laser/tree/master/laser/lux_compiler/core https://github.com/numforge/laser/tree/master/laser/lux_comp... there are a low_level_design_considerations.md and challenges.md that should be interesting as well. Unfortunately I didn't go beyond the proof of concept as I need to revamp the Nim multithreading runtime to improve composition of parallel linear algebra kernels. One thing for sure is that OpenMP is today something that needs to be workaround due to it's lack of composability and GCC implementation is a contention point due to using a single queue protected by a lock. While traditional linear algebra algorithms can deal with static distribution and no load balancing, some algorithms would hugely benefits from a more flexible task system, for example Beam Search.

Loq 7y ago
moving towards a compiler approach. What do you mean by "compiler approach" in this context?
- mratsim 7y ago
  Instead of having to reimplement a series of linear algebra transformation as coarse-grained transformation on ndarrays/tensors, you create a domain-specific language that still give you the mathematical feel. This produce a computation graph that is optimized and compiled across the whole processing pipelines i.e. similar to Dask and Tensorflow delayed computations but you allow manual tuning or autotuning on block size (to best use L1, L2 or GPU cache), on memoization versus recomputing intermediate stages, on swizzling/re-ordering/re-packing inputs to improve cache access for micro-kernels and/or allow their SIMD vectorization. Furthermore a compiler approach will lend itself to autodifferentiation like what Julia, Swift for Tensorflow and Halide are doing.
Jhsto 7y ago
How viable do you think SPIR-V is for computation? I am thinking of researching (that is, creating my own) DSLs targeting SPIR-V.
- mratsim 7y ago
  Good question. I can't find a single SPIR-V AXPY (a*X+y) example which is the Hello World of linear algebra in Vulkan/SPIR-V so I'm not optimistic on ease of use. You can probably do texture hacks like what people did in 2006 when they realized that GPU were fast for linear algebra and OpenCL/Cuda did not exist (https://igoro.com/archive/how-gpu-came-to-be-used-for-general-computation/ https://igoro.com/archive/how-gpu-came-to-be-used-for-genera...) The best would probably be to follow this Halide issue and the corresponding branches: https://github.com/halide/Halide/issues/2296 https://github.com/halide/Halide/issues/2296. Note that Halide is a compiler so it automates lots of the boilerplate that is probably needed to Vulkan for compute compared to Cuda.
pretty_dumm_guy 7y ago
Hi, Does one need to understand compiler technology fully to get into this field? I have been trying to get a grip over MLIR and GLOW source code but have failed miserably so far. Could you please give me some pointers which would help me in this regard ? Thanks

Loq 7y ago

moving towards a compiler approach. What do you mean by "compiler approach" in this context?

mratsim 7y ago
Instead of having to reimplement a series of linear algebra transformation as coarse-grained transformation on ndarrays/tensors, you create a domain-specific language that still give you the mathematical feel. This produce a computation graph that is optimized and compiled across the whole processing pipelines i.e. similar to Dask and Tensorflow delayed computations but you allow manual tuning or autotuning on block size (to best use L1, L2 or GPU cache), on memoization versus recomputing intermediate stages, on swizzling/re-ordering/re-packing inputs to improve cache access for micro-kernels and/or allow their SIMD vectorization. Furthermore a compiler approach will lend itself to autodifferentiation like what Julia, Swift for Tensorflow and Halide are doing.

Jhsto 7y ago

How viable do you think SPIR-V is for computation? I am thinking of researching (that is, creating my own) DSLs targeting SPIR-V.

mratsim 7y ago
Good question. I can't find a single SPIR-V AXPY (a*X+y) example which is the Hello World of linear algebra in Vulkan/SPIR-V so I'm not optimistic on ease of use. You can probably do texture hacks like what people did in 2006 when they realized that GPU were fast for linear algebra and OpenCL/Cuda did not exist (https://igoro.com/archive/how-gpu-came-to-be-used-for-general-computation/ https://igoro.com/archive/how-gpu-came-to-be-used-for-genera...) The best would probably be to follow this Halide issue and the corresponding branches: https://github.com/halide/Halide/issues/2296 https://github.com/halide/Halide/issues/2296. Note that Halide is a compiler so it automates lots of the boilerplate that is probably needed to Vulkan for compute compared to Cuda.

pretty_dumm_guy 7y ago

Hi, Does one need to understand compiler technology fully to get into this field? I have been trying to get a grip over MLIR and GLOW source code but have failed miserably so far. Could you please give me some pointers which would help me in this regard ? Thanks