9 ms·

Very good move indeed. This is what a smart JIT should be able to do so that the programmer does not have to worry about various hardware-specific optimizations

Very good move indeed. This is what a smart JIT should be able to do so that the programmer does not have to worry about various hardware-specific optimizations. Now since they are able to loop unroll it by analysing AST and converting into SIMD, it must also be possible to forward instructions to GPU/GPGPU in the future with little more effort. Similar efforts on JVM in the past - https://astojanov.github.io/blog/2017/12/20/scala-simd.html https://astojanov.github.io/blog/2017/12/20/scala-simd.html LLVM - https://llvm.org/docs/Vectorizers.html https://llvm.org/docs/Vectorizers.html GCC - https://www.gnu.org/software/gcc/projects/tree-ssa/vectorization.html https://www.gnu.org/software/gcc/projects/tree-ssa/vectoriza...

chrisseaton 7y ago
Note that of course the JVM's top-tier JIT compilers (C2, Graal EE) have already done this autovectorisation for many years - this is just the first time code has been contributed for the open source version of Graal, because previously it was an Enterprise-only feature in that compiler. For example open source code in C2 https://github.com/openjdk/jdk/blob/497cb3f4f4ae7f86c173139643adf8ec8401921e/src/hotspot/share/opto/superword.cpp https://github.com/openjdk/jdk/blob/497cb3f4f4ae7f86c1731396.... > Now since they are able to loop unroll it by analysing AST There's no AST analysis going on here - it operates at the graph level. > it must also be possible to forward instructions to GPU/GPGPU in the future with little more effort Hah. People have been trying general GPU offload for a decade and haven't really got anywhere yet. The blocking issue is finding work that is coarse-grained enough to make the context-switch worth it.
- gmueckl 7y ago
  Efficient CPU implementations and efficient GPU implementations that solve the same problem are typically completely different algorithms for anything that is more complex than a big loop. Programming 64 4-wide or 8-wide extremely powerful cores is very different from programming 400 limited cores, each 16 wide.
- apta 7y ago
  > this is just the first time code has been contributed for the open source version of Graal, because previously it was an Enterprise-only feature in that compiler. Is the Enterprise-only implementation different from what has been contributed to the open source version?
  - repolfx 7y ago
    Yes. This is Twitter's own implementation of the idea. It may not get accepted. The maintainer of Graal asked Twitter some tough questions about whether they'd really be maintaining it over the long run and how much it really helped when they first proposed the patch and didn't seem to get any answers. This seems like a continuation of the problem that has made Java a money hole for first Sun and then Oracle. Any attempt to stabilise its losses by making a better Java with proprietary extensions causes companies like Twitter or Red Hat to duplicate those features and submit competing implementations themselves, rather than simply pay the original team. Those duplicates then have to either both be maintained or one has to be rejected. Given this sort of behaviour it's not clear that there's any sustainable way to fund core Java development, other than indefinite cross-subsidisation by other revenue lines.
    - apta 7y ago
      > Given this sort of behaviour it's not clear that there's any sustainable way to fund core Java development, other than indefinite cross-subsidisation by other revenue lines. Wouldn't it move to a model similar to that of C++? Perhaps have a committee that agree to what features get committed and maintained? There are obviously many large companies with deep pockets who heavily rely on the JVM for their core business.
pjmlp 7y ago
NVidia is already doing contributions for GPGPU programming with Graal. https://github.com/NVIDIA/grcuda https://github.com/NVIDIA/grcuda This is why OpenCL got so little love, Khronos just kept focusing on C until it was too late.
- stochastic_monk 7y ago
  NVIDIA's emphasis on modern C++ was very smart as well. Compare writing CUDA to OpenCL. I have hopes for ROC.
  - pjmlp 7y ago
    Yes, pretty much that, although supporting Fortran, introducing PTX early on, and offering GPGPU debuggers were very clever moves as well.
- kllrnohj 7y ago
  > This is why OpenCL got so little love, Khronos just kept focusing on C until it was too late. I think the major problem with OpenCL was that the primary author pushing it, Apple, had no presence in either the HPC nor high-end gaming industries (and still doesn't). Keep in mind OpenCL wasn't a Khronos creation, it was Apple's. Khronos adopted it in version 1.1, but the original 1.0 release was Apple-only as part of the Snow Leopard release. As such, nobody that can drive GPU hardware sales was pushing it. Therefore OpenCL support from AMD & Nvidia was slow and bad, because it didn't help their bottom lines. It really wasn't because it was C instead of C++. It was because the drivers were bad and it was made "generic" too early on. There was no good programming guide for it, and it's super critical for GPGPU to know things like warp width and other programming information that OpenCL just doesn't have, because it's generic-ish. This generic-ish constraint is still a problem today, and is a major reason CUDA outperforms OpenCL so badly. By contrast Nvidia drove CUDA hard, because it sold GPUs to the HPC crowd. And HPC crowds adopted it because it actually let them go faster with real programming guides on how to achieve performance from the GPU, along with actual documentation on what you should & shouldn't expect from actual hardware.
  - 7y ago
OskarS 7y ago
Does the CLR for .NET do auto-vectorization? I've always found it very snappy as VMs go. What about Mono?
- truth_seeker 7y ago
  I am not sure about recent enhancement if any but until last year it wasn't possible unless you use Vectors. https://www.codeproject.com/Articles/1223361/Benchmarking-NET-Core-SIMD-performance-vs-Intel-IS https://www.codeproject.com/Articles/1223361/Benchmarking-NE...
- chrisseaton 7y ago
  .NET has a much simpler kind of JIT than the JVM - it doesn't do any dynamic optimisations or auto-vectorisation.
  - pjmlp 7y ago
    RyuJIT does auto-vectorization and .NET Core supports tiered compilation now. On .NET we don't need to wait for the Vector API, it is already here.
    - chrisseaton 7y ago
      > On .NET we don't need to wait for the Vector API, it is already here. I think the point is people don't want an API - they want it to happen automatically for conventional code.
      - pjmlp 7y ago
        Which even for C++ compilers isn't up to what intrisics are capable of.
  - merb 7y ago
    I'm not sure if that is still true for .net core 3 Edit: Looks like it still is not done: https://github.com/dotnet/coreclr/issues/20486 https://github.com/dotnet/coreclr/issues/20486
tom_mellior 7y ago
> Similar efforts on JVM in the past - https://astojanov.github.io/blog/2017/12/20/scala-simd.html https://astojanov.github.io/blog/2017/12/20/scala-simd.html If I understand this correctly, this doesn't do any autovectorization. It introduces a clever way to allow programmers to use vector intrinsics without the JVM having to understand them. But the programmer does have to use those intrinsics manually, there is no automatic vectorization.
- truth_seeker 7y ago
  > It introduces a clever way to allow programmers to use vector intrinsics without the JVM having to understand them. But the programmer does have to use those intrinsics manually, there is no automatic vectorization. Yep but that CLEVER WAY/LAYER can be very central to build higher-level API. So for a programmer point of view, who deals directly with higher-level (scala API as shown in example) dont have to bother about low-level intrinsics.
  - truth_seeker 7y ago
    Also, have a look at the paper on LMS: Explicit SIMD instructions into JVM using LMS - https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/166022/Toskov_Ivaylo.pdf https://www.research-collection.ethz.ch/bitstream/handle/20....
lidHanteyk 7y ago
PyPy/RPython too: https://pypyvecopt.blogspot.com/2015/08/the-end-of-summer-pypy-simd.html https://pypyvecopt.blogspot.com/2015/08/the-end-of-summer-py... https://morepypy.blogspot.com/2015/10/automatic-simd-vectorization-support-in.html https://morepypy.blogspot.com/2015/10/automatic-simd-vectori...
kllrnohj 7y ago
> it must also be possible to forward instructions to GPU/GPGPU in the future with little more effort. Highly unlikely. On most (all?) high-end systems the GPU or other compute accelerator has its own memory, and is connected to the CPU via a teeny tiny straw (PCI-E). The cost of shipping the data over and shipping the results back is astronomical. It's worth paying if you can ideally leave the data there or if the computation is so intense that the execution is just so much faster to pay for the overhead and then some. But a "smart JIT" is unlikely to figure that out. If we were talking a homogeneous shared memory system, like most mobile devices, then maybe. But there's still non-trivial costs & setup associated with that.
imtringued 7y ago
That doesn't actually address the real problem. The constraints of GPU hardware and SIMD instructions limit the type of problems that they can solve. Not every algorithm can be executed on a GPU or via SIMD with good performance. If you want the autovectorizer to optimize your code then it has to obey these restrictions but the problem with autovectorizers is that they do not produce errors if they fail to vectorize. If you make even a tiny mistake in your code then your application will perform slower than expected. Without building a benchmark suite and a scalar version as a reference implementation it's going to be very difficult to keep track of whether vectorization happens as intended or not.

chrisseaton 7y ago

Note that of course the JVM's top-tier JIT compilers (C2, Graal EE) have already done this autovectorisation for many years - this is just the first time code has been contributed for the open source version of Graal, because previously it was an Enterprise-only feature in that compiler. For example open source code in C2 https://github.com/openjdk/jdk/blob/497cb3f4f4ae7f86c173139643adf8ec8401921e/src/hotspot/share/opto/superword.cpp https://github.com/openjdk/jdk/blob/497cb3f4f4ae7f86c1731396.... > Now since they are able to loop unroll it by analysing AST There's no AST analysis going on here - it operates at the graph level. > it must also be possible to forward instructions to GPU/GPGPU in the future with little more effort Hah. People have been trying general GPU offload for a decade and haven't really got anywhere yet. The blocking issue is finding work that is coarse-grained enough to make the context-switch worth it.

gmueckl 7y ago
Efficient CPU implementations and efficient GPU implementations that solve the same problem are typically completely different algorithms for anything that is more complex than a big loop. Programming 64 4-wide or 8-wide extremely powerful cores is very different from programming 400 limited cores, each 16 wide.
apta 7y ago
> this is just the first time code has been contributed for the open source version of Graal, because previously it was an Enterprise-only feature in that compiler. Is the Enterprise-only implementation different from what has been contributed to the open source version?
- repolfx 7y ago
  Yes. This is Twitter's own implementation of the idea. It may not get accepted. The maintainer of Graal asked Twitter some tough questions about whether they'd really be maintaining it over the long run and how much it really helped when they first proposed the patch and didn't seem to get any answers. This seems like a continuation of the problem that has made Java a money hole for first Sun and then Oracle. Any attempt to stabilise its losses by making a better Java with proprietary extensions causes companies like Twitter or Red Hat to duplicate those features and submit competing implementations themselves, rather than simply pay the original team. Those duplicates then have to either both be maintained or one has to be rejected. Given this sort of behaviour it's not clear that there's any sustainable way to fund core Java development, other than indefinite cross-subsidisation by other revenue lines.
  - apta 7y ago
    > Given this sort of behaviour it's not clear that there's any sustainable way to fund core Java development, other than indefinite cross-subsidisation by other revenue lines. Wouldn't it move to a model similar to that of C++? Perhaps have a committee that agree to what features get committed and maintained? There are obviously many large companies with deep pockets who heavily rely on the JVM for their core business.

pjmlp 7y ago

NVidia is already doing contributions for GPGPU programming with Graal. https://github.com/NVIDIA/grcuda https://github.com/NVIDIA/grcuda This is why OpenCL got so little love, Khronos just kept focusing on C until it was too late.

stochastic_monk 7y ago
NVIDIA's emphasis on modern C++ was very smart as well. Compare writing CUDA to OpenCL. I have hopes for ROC.
- pjmlp 7y ago
  Yes, pretty much that, although supporting Fortran, introducing PTX early on, and offering GPGPU debuggers were very clever moves as well.
kllrnohj 7y ago
> This is why OpenCL got so little love, Khronos just kept focusing on C until it was too late. I think the major problem with OpenCL was that the primary author pushing it, Apple, had no presence in either the HPC nor high-end gaming industries (and still doesn't). Keep in mind OpenCL wasn't a Khronos creation, it was Apple's. Khronos adopted it in version 1.1, but the original 1.0 release was Apple-only as part of the Snow Leopard release. As such, nobody that can drive GPU hardware sales was pushing it. Therefore OpenCL support from AMD & Nvidia was slow and bad, because it didn't help their bottom lines. It really wasn't because it was C instead of C++. It was because the drivers were bad and it was made "generic" too early on. There was no good programming guide for it, and it's super critical for GPGPU to know things like warp width and other programming information that OpenCL just doesn't have, because it's generic-ish. This generic-ish constraint is still a problem today, and is a major reason CUDA outperforms OpenCL so badly. By contrast Nvidia drove CUDA hard, because it sold GPUs to the HPC crowd. And HPC crowds adopted it because it actually let them go faster with real programming guides on how to achieve performance from the GPU, along with actual documentation on what you should & shouldn't expect from actual hardware.
- 7y ago

OskarS 7y ago

Does the CLR for .NET do auto-vectorization? I've always found it very snappy as VMs go. What about Mono?

truth_seeker 7y ago
I am not sure about recent enhancement if any but until last year it wasn't possible unless you use Vectors. https://www.codeproject.com/Articles/1223361/Benchmarking-NET-Core-SIMD-performance-vs-Intel-IS https://www.codeproject.com/Articles/1223361/Benchmarking-NE...
chrisseaton 7y ago
.NET has a much simpler kind of JIT than the JVM - it doesn't do any dynamic optimisations or auto-vectorisation.
- pjmlp 7y ago
  RyuJIT does auto-vectorization and .NET Core supports tiered compilation now. On .NET we don't need to wait for the Vector API, it is already here.
  - chrisseaton 7y ago
    > On .NET we don't need to wait for the Vector API, it is already here. I think the point is people don't want an API - they want it to happen automatically for conventional code.
    - pjmlp 7y ago
      Which even for C++ compilers isn't up to what intrisics are capable of.
- merb 7y ago
  I'm not sure if that is still true for .net core 3 Edit: Looks like it still is not done: https://github.com/dotnet/coreclr/issues/20486 https://github.com/dotnet/coreclr/issues/20486

tom_mellior 7y ago

> Similar efforts on JVM in the past - https://astojanov.github.io/blog/2017/12/20/scala-simd.html https://astojanov.github.io/blog/2017/12/20/scala-simd.html If I understand this correctly, this doesn't do any autovectorization. It introduces a clever way to allow programmers to use vector intrinsics without the JVM having to understand them. But the programmer does have to use those intrinsics manually, there is no automatic vectorization.

truth_seeker 7y ago
> It introduces a clever way to allow programmers to use vector intrinsics without the JVM having to understand them. But the programmer does have to use those intrinsics manually, there is no automatic vectorization. Yep but that CLEVER WAY/LAYER can be very central to build higher-level API. So for a programmer point of view, who deals directly with higher-level (scala API as shown in example) dont have to bother about low-level intrinsics.
- truth_seeker 7y ago
  Also, have a look at the paper on LMS: Explicit SIMD instructions into JVM using LMS - https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/166022/Toskov_Ivaylo.pdf https://www.research-collection.ethz.ch/bitstream/handle/20....

lidHanteyk 7y ago

PyPy/RPython too: https://pypyvecopt.blogspot.com/2015/08/the-end-of-summer-pypy-simd.html https://pypyvecopt.blogspot.com/2015/08/the-end-of-summer-py... https://morepypy.blogspot.com/2015/10/automatic-simd-vectorization-support-in.html https://morepypy.blogspot.com/2015/10/automatic-simd-vectori...

kllrnohj 7y ago

> it must also be possible to forward instructions to GPU/GPGPU in the future with little more effort. Highly unlikely. On most (all?) high-end systems the GPU or other compute accelerator has its own memory, and is connected to the CPU via a teeny tiny straw (PCI-E). The cost of shipping the data over and shipping the results back is astronomical. It's worth paying if you can ideally leave the data there or if the computation is so intense that the execution is just so much faster to pay for the overhead and then some. But a "smart JIT" is unlikely to figure that out. If we were talking a homogeneous shared memory system, like most mobile devices, then maybe. But there's still non-trivial costs & setup associated with that.

imtringued 7y ago

That doesn't actually address the real problem. The constraints of GPU hardware and SIMD instructions limit the type of problems that they can solve. Not every algorithm can be executed on a GPU or via SIMD with good performance. If you want the autovectorizer to optimize your code then it has to obey these restrictions but the problem with autovectorizers is that they do not produce errors if they fail to vectorize. If you make even a tiny mistake in your code then your application will perform slower than expected. Without building a benchmark suite and a scalar version as a reference implementation it's going to be very difficult to keep track of whether vectorization happens as intended or not.