6 ms·
Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compiler
by dragontamer 2y ago
Intel needs to see what has happened to their AVX instructions and why NVidia has taken over.
If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512.
GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long term benefits of rewriting into CUDA are heavily apparent.
Intel ISPC is a great project btw if you need high level code that targets SSE, AVX, AVX512 and even ARM NEON all with one codebase + auto compiling across all the architectures.
-------
Intels AVX512 is pretty good at a hardware level. But software methodology to interact with SIMD using GPU-like languages should be a priority.
Intrinsics are good for maximum performance but they are too hard for mainstream programmers.
- dist-epoch 2y ago> If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs That's not true. For maximum performance you need to tweak the code to a particular GPU model/architecture. Intel has SSE/AVX/AVX2/AVX512, but CUDA has like 10 iterations of this (increasing capabilities). Code written 15 years ago would not use modern capabilities, like more flexible memory access, atomics.
- dragontamer 2y agoMaximum performance? Okay, you'll have to upgrade to ballot instructions or whatever and rearchitect your algorithms. (Or other wavefront / voting / etc. etc. new instructions that have been invented. Especially those 4x4 matrix multiplication AI instructions). But CUDA -> PTX intermediate code has allowed for significantly more flexibility. For crying out loud, the entire machine code (aka SASS) of NVidia GPUs has been cycled out at least 4 times in the past decade (128-bit bundles, changes to instruction formats, acquire/release semantics, etc etc) It's amazing what backwards compatibility NVidia has achieved in the past 15 years thanks to this architecture. SASS changes so dramatically from generation to generation but the PTX intermediate code has stayed highly competitive.
- dist-epoch 2y agoIntel code from 15 years ago also runs today. But it will not use AVX512. Which is the same with PTX, right? If you didn't use the tensor core instructions or wavefront voting in the CUDA code, the PTX generated from it will not either, and NVIDIA will not magically add those capabilities in when compiling to SASS. Maybe it remains competitive because the code is inherently parallel anyway, so it will naturally scale to fill the extra execution units of the GPU, which is where most of the improvement is generation to generation. While AVX code can't automatically scale to use the AVX512 units.
- dragontamer 2y agoIt's not the same. AVX2 instructions haven't changed and never will change. In contrast, NVidia can go from 64-bit instruction bundles to 128-bit machine code (96-bit instruction + 32-bit control information) between Pascal (aka PTX Compute Capacity 5) and Voltage (aka PTX Compute Capacity 7) and all the old PTX code just autocompiles to the new assembly instruction format and takes advantage of all the new memory barriers added in Volta. Having a PTX translation later is a MAJOR advantage for the NVidia workflow.
- deleted 2y ago[deleted]
- ashvardanian 2y agoThere is still a lot of similarity between CPU and GPU programming - between AVX and PTX. Different generations of CPU cores handle the same AVX2 instructions differently. The microcode changes and the schedulers change, but the process is transparent for the user, similar to PTX.
- mmoskal 2y agoI imagine there is and order of magnitude of difference between how much you can translate in software, with large memory and significant time budget to work with, compared to microcode.
- jsheard 2y ago> Intel ISPC is a great project btw if you need high level code that targets SSE, AVX, AVX512 and even ARM NEON It's pretty funny how NEON ended up in there. A former Intel employee decided to implement it for fun and submitted it as a pull request, which Intel quietly ignored for obvious reasons, but then another former Intel employee who still had commit rights merged the PR, and the optics of publicly reverting it would be even worse than stonewalling so Intel begrudgingly let it stand (but they did revoke that devs commit rights). https://pharr.org/matt/blog/2018/04/29/ispc-retrospective https://pharr.org/matt/blog/2018/04/29/ispc-retrospective
- deleted 2y ago[deleted]
- pjmlp 2y agoIt is worse than that, given that AVX is the survivor from Larrabee great plan to kill GPUs. Larrabee was going to take over it all, as I enjoyed its presentation at GDCE 2009.
- dragontamer 2y agoI mean, 288-E Core Xeons are about to ship. Xeon 6900 series, right? (Estimated to ship in Q1 2025) So Larrabee lives on for... some reason. These E cores are well known to be modified Intel Atom cores and those were modified Xeon Phi cores which were Larrabee based. Just with.... AVX512 being disabled. (Lost when Xeon Phi turned into Intel Atoms). Intels technical strategy is completely bonkers. In a bad way. Intel invented all this tech 10 to 20 years ago but fails to have a cohesive strategy to bring it to market. There's clearly smart people there but somehow all the top level decisions are just awful
- ashvardanian 2y agoYes, a lot of weird decisions were made at Intel. Ironically, AMD waited so long to implement AVX-512, but now has it on both server and mobile chips (natively and 256 bit emulation, respectively). Intel started the whole thing, has a very fragmented stack and is now preparing those E cores with even more new extensions. Most importantly for Search and AI, it adds AVX_VNNI, which can be used for faster 8-bit integer dot-products: https://github.com/ashvardanian/SimSIMD/blob/75c426fb190a9d4286a9e9343925ecce8562abc1/include/simsimd/dot.h#L1815 https://github.com/ashvardanian/SimSIMD/blob/75c426fb190a9d4... Would be interesting to see how matrix multiplication throughput will differ between AVX-512-capable P cores and a larger quantity of AVX_VNNI-capable E cores!
- alfiedotwtf 2y agoA former Intel CEO even wrote a book where every product was planned 20+ years in advance. Imagine planning 20 years in advance where Moore’s Law is still going strong. Come to think of it, Moore was also CEO of Intel lol
- 2y ago
- variadix 2y agoHow much of this is because CUDA is designed for GPU execution and because the GPU ISA isn’t a stable interface? E.g. new GPU instructions can be utilized by new CUDA compilers for new hardware because the code wasn’t written to a specific ISA? Also, don’t people fine tune GPU kernels per architecture manually (either by hand or via automated optimizers that test combinations in the configuration space)?
- dragontamer 2y agoNVidia PTX is a very stable interface. And the PTX to SASS compiler DOES a degree of automatic fine tuning between architectures. Nothing amazing or anything, but it's a minor speed boost that has made PTX just a easier 'assembly-like language' to build on top of.
- janwas 2y agoMy understanding is that there is a lot of hand-writing (not just fine-tuning) going on. AFAIK CuDNN and TensorRT are written directly as SASS, not CUDA. And the presence of FP8 in H100, but not A100, would likely require a complete rewrite.
- dragontamer 2y agoCub, thrust and many other libraries that make those kernels possible don't need to be rewritten. When you write a merge sort in CUDA, you can keep it across all versions. Maybe the new instructions can improve a few corner cases, but it's not like AVX to AVX512 where you need to rewrite everything. Ex: https://github.com/NVIDIA/cub/blob/main/cub/device/device_merge_sort.cuh https://github.com/NVIDIA/cub/blob/main/cub/device/device_me...
- janwas 2y agoI agree not everything needs to be rewritten. And neither does code using an abstraction such as Highway, so we can stop beating that dead horse.
- snihalani 2y ago> software methodology to interact with SIMD using GPU-like languages should be a priority. What's your opinion on sycl? https://www.khronos.org/sycl/ https://www.khronos.org/sycl/
- ip26 2y agoIs CUDA not more analogous to using MKL, rather than AVX?
- jabl 2y agoSometimes I wonder about an alternative history scenario where CPU ISA's would have chosen a SIMT style model instead of SIMD. "Just" have something like fork/join instructions to start/stop vector mode, otherwise use the standard scalar instructions in both scalar and vector mode. Would have avoided a lot of combinatorial explosion in instructions. (of course you'd have to do something for cross-lane operations, and later tensor instructions etc.)
- janwas 2y agoNot sure why SIMT would help, it requires more compiler transforms than if the code is written for packets/vectors or whatever we want to call them. As you note, cross-lane is a key part of a good SIMD abstraction. Vulkan calls it "subgroups", but from where I sit it's still SIMD.
- janwas 2y agoMax performance is a stretch - recompilation would not utilize tensor cores, right? "too hard for mainstream programmers" seems overly pessimistic. I've run several workshops where devs have written dot-product kernels using Highway after 30 minutes of introduction.
- kristianp 2y agoThey said intrinsics. Highway is an abstraction on top of intrinsics.
- janwas 2y agoOK :) A thin abstraction, though. If comparing with alternative categories such as domain-specific language or autovectorization, I'd still classify Highway as intrinsics, just portable and easier to use.
- adrian_b 2y agoWhile there is some truth in what you say, it makes seem like writing in the CUDA style is something new and revolutionary invented by NVIDIA, which it is not. The CUDA style of writing parallel programs is nothing else than the use of the so-called "parrallel do" a.k.a. "parrallel for" program structure, which has been already discussed in 1963. Notable later evolutions of this concept have been present in "Communicating Sequential Processes" by C.A.R. Hoare (1978-08: "arrays of processes"), then in the programming language Occam, which was designed based on what Hoare had described, then in the OpenMP extension of Fortran (1997-10), then in the OpenMP extension of C and C++ (1998-10). Programming in CUDA does not bring anything new, except that in comparison e.g. with OpenMP some keywords are implicit and others are different, so the equivalence is not immediately obvious. Programming for CPUs in the much older OpenMP is equivalent with programming in CUDA for GPUs. The real innovation of NVIDIA has been the high quality of the NVIDIA CUDA compiler and CUDA runtime GPU driver, which are able to distribute the work that must be done on the elements of an array over all the available cores, threads and SIMD lanes, in a manner that is transparent for the programmer, so in many cases the programmer is free to ignore which is the actual structure of the GPU that will run the program. Previous compilers for OpenMP or for other such programming language extensions for parallel programming have been much less capable to produce efficient parallel programs without being tuned by the programmer for each hardware variant.
- dragontamer 2y agoOh all of this was being done in the 1980s by Lisp* programmers. I'm not calling it new. I'm just saying that the intrinsics style is much much harder than what Lisp*, DirectX HLSL, CUDA, OpenCL (etc. etc) does. A specialized SIMD language makes writing SIMD easier compared to intrinsic style. Look at any CUDA code today and compare it to the AVX that is in the above article and it becomes readily apparent.
- saagarjha 2y agoI'm not sure what you mean. All CUDA code needs to be aware of the programming model the GPU imposes on them, splitting their code manually into threads and warps and blocks and kernels to match. This isn't really transparent at all.
- synack 2y agoI’ve been playing with a new Lunar Lake laptop and they’ve complicated things even further with the Neural Processing Unit (NPU) Now if your vectors are INT8/FP8 you’re supposed to shovel them into this accelerator via PCIe, rather than packing into registers for AVX512. I wish they’d just pick an interface for vector ops and stick with it.
- david-gpu 2y ago> If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512 NVidia compilers would have compiled your code into something functional, but if you want to approach peak performance you need to at least tweak your kernels, and sometimes rewrite them from scratch. See for example the various MMA instructions that were introduced over time. Edit: I see somebody made a similar comment and you addressed it. Sorry for the churn.