22 ms·

AVX 512 will be the future

203 points by fulafel 4y ago

frozenport 4y ago
These dudes are biased: they don't want to support something they don't use.
- Razengan 4y ago
  The PC dev world in a nutshell.
PaulHoule 4y ago
AVX 512 has a number of instructions that make it useful for UTF-8 parsing, floating point parsing, XML parsing, JSON parsing, things like that. It is tricky coding https://lemire.me/blog/2022/05/25/parsing-json-faster-with-intel-avx-512/ https://lemire.me/blog/2022/05/25/parsing-json-faster-with-i... All things that are good for HFT but also good for speeding up your web browser and maybe even saving power because you can dial down the clock rate. It's a tragedy that Intel is fusing off AVX 512 in consumer parts so they can stuff the chips with thoroughly pizzled phonish low-performance cores.
- eesmith 4y ago
  I think that's what "if you aren't into parsing (but not using) JSON" is supposed to be referencing. And, "It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it."
  - PaulHoule 4y ago
    One thing I learned in grad school is you can do a stupendous number of floating point operations in the time it takes to serialize a matrix to ASCII numbers and deserialize it. Whatever it is you are doing with a JSON document might be slower than parsing it. It's true that autovectorization accomplishes very little but specialized libraries could have a notable effect on perceived performance if they were widely developed and used. Frankly Intel has had less interest in getting you to buy a new computer by making it the best computer you ever bought than it has been in taking as much revenue as it can from the rest of the BOM, for instance the junk integrated graphics that sabotaged the Windows Vista launch and has been making your computer crash faster ever since. Another example is that they don't ship MKL out of the box on Windows or Linux although they do on MacOS. And Intel wonders why their sales are slipping...
    - oxfeed65261 4y ago
      Had to look it up: > Intel oneAPI Math Kernel Library (Intel oneMKL; formerly Intel Math Kernel Library or Intel MKL), is a library of optimized math routines for science, engineering, and financial applications. Core math functions include BLAS, LAPACK, ScaLAPACK, sparse solvers, fast Fourier transforms, and vector math. https://en.m.wikipedia.org/wiki/Math_Kernel_Library https://en.m.wikipedia.org/wiki/Math_Kernel_Library
    - kzrdude 4y ago
      Matrix multiplication and similar is also one of the few operations where algorithms and special case instructions are interesting for floating point on a massive scale. I.e adding two arrays together, computing dot products, those operations are just memory bound when the data grows, but matrix multiplication is dense enough with operations per element that it is limited by arithmetic operations too.
    - ilyt 4y ago
      > One thing I learned in grad school is you can do a stupendous number of floating point operations in the time it takes to serialize a matrix to ASCII numbers and deserialize it. Whatever it is you are doing with a JSON document might be slower than parsing it. > It's true that autovectorization accomplishes very little but specialized libraries could have a notable effect on perceived performance if they were widely developed and used. I mean, sure, but even if we take JSON as an example, in vast majority of cases it gets fed to a giant blob of JS driving even bigger blob of browser code. The cases where you do deserialization -> very little processing -> serialization are pretty rare. Sure, if it is already on chip might as well use it, but realistically savings will be in single digit percents.
      - lmm 4y ago
        > The cases where you do deserialization -> very little processing -> serialization are pretty rare. Actually I've seen a lot of systems that do that - query a datastore, do some minimal processing of the results, feed them back to the caller. Although that tends to get addressed at a higher level, e.g. MongoDB drivers shifting towards using BSON.
- zeusk 4y ago
  They have a replacement coming in 2023/24
  - kristianp 4y ago
    Replacement cpu generation or replacement for avx-512? Will it be variable length?
- jeffbee 4y ago
  I don't think Intel has even decided whether to fuse them off or not. In my newest Intel desktop, a Core i7-13700K, AVX-512 is actually available. It wasn't available on the i7-12700K that was in the same system on the same motherboard a few weeks ago.
  - compsciphd 4y ago
    i thought the reason they wanted to fuse them off was for all cores being completely compatible (just slower), without the need for the OS to force scheduling on specific cores (I'd guess what would have to happen is if an instruction not available on efficiency core was called, it trap, the OS would see it and then mark the process to only be executable on performance core?)
    - jeffbee 4y ago
      OK but that doesn't explain why it reappeared in Raptor Lake.
      - EgoIncarnate 4y ago
        The official specs for Raptor Lake don't include AVX512 support, so most likley this is just something that slipped through on your system. It might be "corrected" in later steppings or updated firmware. Some early 12th gen systems had unfused CPUs with AVX512 available on the performance cores and firmware that didn't disable it. https://www.intel.com/content/www/us/en/products/sku/230500/intel-core-i713700k-processor-30m-cache-up-to-5-40-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/230500/...
        chillfox 4y ago
        So your saying that they have shitty QA which we just happen to benefit from this time?
  - borissk 4y ago
    Are you sure? According to Intel spec AVX-512 is not available - https://ark.intel.com/content/www/us/en/ark/products/230500/intel-core-i713700k-processor-30m-cache-up-to-5-40-ghz.html https://ark.intel.com/content/www/us/en/ark/products/230500/...
- ilyt 4y ago
  Didn't read the post ? THat is explicitly mentioned > Yes, yes, yes, you can use it in random places. It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it.
  - bluestreak 4y ago
    I couldn't agree more. I don't think compiler vectorization is that useful even for columnar (!) database we're building. The specialized JIT doesn't even use AVX512 because too much effort for little to no gain.
    - sagarm 4y ago
      Vectorization (auto or manual) can really help optimize bottlenecks like evaluating simple comparisons once you're out of easy algorithmic wins. It takes so much attention it's only worth doing in the most critical cases IMO. Auto-vectorization can be fragile. Manual vectorization is a ton of work and difficult to maintain.
  - gumby 4y ago
    > And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it. How much of that is due to language semantics? Can't FORTRAN compilers make better use of the AVX instructions than, say, a C or Rust compiler could due to memory reference semantics?
    - dan-robertson 4y ago
      In theory, Rust should have much better memory semantics than C but currently much fighting is required to get llvm to take advantage of the Rust semantics.
- sitkack 4y ago
  Vector instructions (ala Cray) are future proof in ways that SIMD are not. Eventually every high performance CPU will have HBM, I am not sure SIMD is the mechanism we will use in to extract the most efficiency out of the platform. https://people.eecs.berkeley.edu/~krste/papers/yunsup-ms.pdf https://people.eecs.berkeley.edu/~krste/papers/yunsup-ms.pdf https://parlab.eecs.berkeley.edu/sites/all/parlab/files/maven-poster-parlab2009-06-01.pdf https://parlab.eecs.berkeley.edu/sites/all/parlab/files/mave... https://www.cs.uic.edu/~ajayk/c566/VectorProcessors.pdf https://www.cs.uic.edu/~ajayk/c566/VectorProcessors.pdf https://inst.eecs.berkeley.edu/~cs152/sp18/lectures/L16-RISCV-Vectors.pdf https://inst.eecs.berkeley.edu/~cs152/sp18/lectures/L16-RISC...
  - janwas 4y ago
    I feel that SIMD vs vector is not the most useful distinction to make. Yes, at the ISA and binutils level there is a difference in terms of number of instructions and ugly encodings. But the actual application source code can actually look the same for SIMD and vector - vector-length agnostic style is helpful for both. Instead, it seems more interesting to focus on preparing more applications for either SIMD or vector: with data-parallel algorithms, avoiding branches etc.
  - lifthrasiir 4y ago
    Of course when you only have packed instructions (a la SIMD) you have no choice but using them in place of vector instructions, but vector instructions in general do not fully subsume packed instructions. I think packed instructions are better at exploiting opportunistic parallelism from either a short loop (where vector instructions would have higher overhead) or a series of similar but slightly different scalar instructions.
  - fooblaster 4y ago
    There have been attempts to remedy the fixed width simd vector problem that are already shipping: https://developer.arm.com/Architectures/Scalable%20Vector%20Extensions https://developer.arm.com/Architectures/Scalable%20Vector%20...
- torginus 4y ago
  Generally speaking, the idea that the average person uses more than 4 cores is insane. Even as a power user/dev I can count the times when I needed more than 4 cores on one hand. All the benchmarks that these CPUs are evaluated by, test either games, or run the same crazy stuff like fluid simulations that OPs article complains about. Hardware reviewers need a reality check.
  - Iwan-Zotow 4y ago
    good porno will saturate all available cores
  - deleted 4y ago
    [deleted]
  - jl6 4y ago
    Eh, a modern desktop can have a lot of concurrent processes running. A browser, one process per tab, software updates continually spawning in the background… Though I do agree that the benefits of high core count CPUs are oversold as far as most users are concerned.
    - Dylan16807 4y ago
      Many processes yes, but background tasks don't need much power and one core has a lot of power. So one core should be plenty for all background tasks combined.
  - eyegor 4y ago
    I hate to break it to you, but outside of gaming, fluid sim, and battery life / power draw tests, nothing else matters for modern chips. Schmucks only need the kind of horsepower a phone is capable of, or want to know how long it will last. The only people who care about performance either really care because more perf = more dollars (fluid sim, hft, etc.) or they're gamers.
    - brookst 4y ago
      I use CAD for a hobby. My 5950x has 16 cores and I desperately wish I could afford a 64 core desktop because each model takes 60 - 240 seconds to compute. That’s a pain when it takes multiple iterations to get a good design, and it leads to more complicated programming to offer quick vs detailed model creation, and that complexity leads to bugs. So yeah I care, and not for more dollars.
- bitcharmer 4y ago
  > All things that are good for HFT (...) Not sure if you're referring to high frequency trading here, but if that's the case then it's just not true. AVX is infamously bad for low latency becuse these instructions result in insane amounts of heat which forces the CPU to get down clocked (sometimes the cores on the neighboring tiles on the SoC too). P-State transitions are a serious source of execution jitter and a big no-no in our business. Many HFT shops actually disable AVX with noxsave. It's fantastically useful for HPC thought.
  - dan-robertson 4y ago
    I think I found there’s also a quite expensive ‘warmup’ which makes much of AVX only worth it for larger loops. But my memory or original understanding might be incorrect and my understanding of why that warmup might be the case (maybe microcode caches in the front end??) is very poor.
37ef_ced3 4y ago
Linus Torvalds: And I claim that that is the real problem with AVX-512 (and pretty much any vectorization). I personally cannot find a single benchmark that does anything I would ever do - not even remotely close. So if you aren't into some chess engine, if you aren't into parsing (but not using) JSON, if you aren't into software raytracing (as opposed to raytracing in games, which is clearly starting to take off thanks to GPU support), what else is there? Answer? Neural net inference, e.g., https://NN-512.com https://NN-512.com If you need a little bit of inference (say, 20 ReNet50s per second per CPU core) as part of a larger system, there's nothing cheaper. If you're doing a small amount of inference, perhaps limited by other parts of the system, you can't keep a GPU fed and the GPU is a huge waste of money. AVX-512, with its masked operations and dual-input permutations, is an expressive and powerful SIMD instruction set. It's a pleasure to write code for, but we need good hardware support (which is literally years overdue).
- quelsolaar 4y ago
  True, but the problem is that that is today better done on vector hardware like a GPU or other ML hardware. The world has sort of diverged int to two camps: vectorizable problems that can be massivly paralleleized (graphics, simulation, ML) and for that we use GPUs, and then everything else is CPU. What i think linus is saying is that there are few reasons to use AVX-512 on a CPU, when the is a GPU much better siuted for those kinds of problems. You could say that the intersecting area in the ven diagram of "Has to run on CPU" and "Can use vector instructions" is small.
  - dtx1 4y ago
    > You could say that the intersecting area in the ven diagram of "Has to run on CPU" and "Can use vector instructions" is small. People are forgetting the "Could run on a GPU but I don't know how" factor. There's tons of Situations where GPU Offloading would be fast or more energy efficient but importing all the libraries, dealing with drivers etc. really is not worth the effort, whereas doing it on a CPU is really just a simple include away.
  - dragontamer 4y ago
superkuh 4y ago
UH... didn't Intel actually stop including AVX 512 in their newest (alder lake) processors? This seems unlikely. ie, https://www.tomshardware.com/news/intel-nukes-alder-lake-avx-512-now-fuses-it-off-in-silicon https://www.tomshardware.com/news/intel-nukes-alder-lake-avx... >"Although AVX-512 was not fuse-disabled on certain early Alder Lake desktop products, Intel plans to fuse off AVX-512 on Alder Lake products going forward." -Intel Spokesperson to Tom's Hardware.
- Kon-Peki 4y ago
  On the other hand, you can now virtually guarantee that a GPGPU is present on Intel consumer chips. So now you can write your code 3 times; no vector acceleration (for the really old stuff), AVX-512 for the servers, and GPU for the consumer chips!
  - convolvatron 4y ago
    or we can finally starting looking into languages with implicit fine grained parallelism and run on whatever
    - anonymoushn 4y ago
      This won't help, most uses of AVX or NEON have nothing to do with parallel processing of a bunch of numbers
    - goalieca 4y ago
      We have had that for 15+ years even in C with fine grained looping like OpenMP. It’s terribly inefficient to communicate across threads. Sometimes you just need SIMD
      - gpderetta 4y ago
        OMP is supposed to support SIMD/GPU offloading theses days. Not sure how good it is though.
      - 4y ago
the8472 4y ago
Some of the AVX512 instructions are nice additions though, even if you only use it on YMM or XMM registers. All the masked instructions make it much simpler to vectorize loops over variable amounts of data without special treatment for the head/tail of the data.
- stephencanon 4y ago
  This is exactly right—-the 512b width is the least interesting thing about AVX512. All the new instructions on 128, 256, and 512b vectors, (finally) good support for unsigned integers, static rounding modes for floating point, and predication are much more exciting.
- anonymoushn 4y ago
  I don't mind head/tail stuff so much but the masked instructions and vcompress make a lot of "do some branchy different stuff to different data or filter a list" easier and faster compared to AVX2, so I'm a big fan
cmrdporcupine 4y ago
SIMD on x86 has become a ridiculous minefield of standards. I honestly pity the compiler writers who have to navigate this nonsense. It was stressful enough trying to just manage the compiler flagging in an application (music synthesizer) I wrote recently. The core amd64 ISA is stable and that's good. I wouldn't call it elegant or nice, but it's there. The patchwork of which processor supports which SIMD extension on top of it, though, has gotten out of control and is unpleasant. I'm sure AVX512 is great. Except as an application author I can't rely on it being there. I am not a MacOS user or MacOS developer, but I envy the consistency of that platform now that they control their own ISA.
- paulmd 4y ago
  > SIMD on x86 has become a ridiculous minefield of standards. people make the AVX-512 compatibility thing more complex than it has to be. there's consumer AVX-512 and server AVX-512 and within those series there has always been strictly increasing compatibility, with a common core between them. and AMD supports the complete thing with Zen4 so there is no differentiation for them. intel didn't want to put bfloat16 ML instructions in consumer chips and people turned that into a whole thing because they like to meme about AVX-512, but that's literally the only difference. it's just not that complex a story though, if you don't turn it into a thing. intel's consumer and server teams have been operating quasi-independently and some of them are on "newer generations" than the other team. currently that's the client team that's ahead of the server team. so client got a couple gens ahead of server, and server has this one extra instruction in their versions. that's it, that's the whole deal. frankly it's actually a simpler story than SSE4 where AMD just kinda went and did its own thing for a while and Intel went and did AVX and there truly was total divergence between what the companies were doing. you can probably turn SSE into a real ugly venn diagram if you try to capture every nuance too! After all, some of them were introduced in K8, K9, Phenom... and there's lots of sub-versions of each. xeon phi is also its own thing but you'll know if you're targeting that. it has a weird 4-wide AVX thing to go along with its 4-wide SMT, and it's its own specific thing.
  - hajile 4y ago
    Based on Wikipedia's compatibility chart, it sure looks like a minefield to me. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512
Aardwolf 4y ago
> Seriously, go look at random code on github - I dare you. Very little of it is clearly delineated loops over arrays with high repeat counts. It's not very helpful to be able to vectorize some "search for a value in an array", when people end up using hash tables or other more complex data structures instead of arrays for searching. Sure, but many parts that matter for performance, the inner loops to optimize, may be like that. Multimedia encoding/decoding, emulation, ML, matrix multiplications, ...
- Tuna-Fish 4y ago
  > Multimedia encoding/decoding, Done on special-purpose hardware that is >10 times more efficient than any implementation on GP hardware. > emulation, Only current practical use of AVX-512. > ML, matrix multiplications, ... Much better to do on the GPU. In general wide SIMD on CPU has the problem that to make effective use of it, you have to massage your code enough that you are ~95% away from just running it on the GPU anyway, and you can gain much more performance if you do that. The best niche of AVX-512 would have been as the baseline common target for things that also get optimized for GPUs... except that Intel has eliminated this possibility by heavy product segmentation right from the start.
  - Aardwolf 4y ago
    > Done on special-purpose hardware that is >10 times more efficient than any implementation on GP hardware. That locks users in to only the codecs that have been implemented in this particular hardware, and increases hardware size for particular vendor implementations rather than provide common building blocks that many codecs use (DCT, ...) > Much better to do on the GPU. Yep and that's what I run ML stuff on, not all systems have a GPU available though, and for some applications it's faster to do something immediately on the CPU, than have the overhead of going to/from GPU > In general wide SIMD on CPU has the problem that to make effective use of it, you have to massage your code enough that you are ~95% away from just running it on the GPU anyway, Better tooling and libraries could help with this imho. Note that the situation for GPU is here also not great at all, since the good tools are locked into 1 vendor.
  - 4y ago
ak217 4y ago
Some of the biggest investments into general purpose Linux performance, reliability, and sustainability were made by people with "specialty engines" that Linus is talking about. The specialty applications often turn out to be important because they push the platform to excel in unexpected ways. With that said, the biggest practical problem we saw with AVX-512 was that it's so big, power hungry, and dissimilar to other components of the CPU. One tenant turning that thing on can throttle the whole chip, causing unpredictable performance and even reliability issues for everything on the machine. I imagine it's even worse on mobile/embedded, where turning it on can blow out your power and thermal budget. The alternatives (integrated GPUs/ML cores) seem to be more power efficient.
- cogman10 4y ago
  > and dissimilar to other components of the CPU It's not terribly dissimilar from the SSE instructions before it. The biggest weirdness about AVX is it's CPU throttling. Which, IMO, should not be a thing for the reasons you mention. I imagine that future iterations of AVX will likely ditch CPU throttling. I'm guessing it was a "our current design can't handle this and we need to ship!" sort of solution. (If they haven't already? Do current CPUs throttle AVX-512 instructions?) > I imagine it's even worse on mobile/embedded Fortunately, x86 mobile/embedded isn't terribly common, certainly not with AVX-512 support.
  - Kon-Peki 4y ago
    > I imagine that future iterations of AVX will likely ditch CPU throttling Already happening: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Downclocking https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Dow...
  - paulmd 4y ago
    > I imagine that future iterations of AVX will likely ditch CPU throttling. I'm guessing it was a "our current design can't handle this and we need to ship!" sort of solution. (If they haven't already? Do current CPUs throttle AVX-512 instructions?) exactly what it looks like to me, and it was already fixed in Ice Lake and probably doesn't exist at all in Zen4. Just nobody cares about Ice Lake-SP when Epyc is crushing the server market. https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... and some measurements of Skylake-X displaying the effect, note the voltage swings in particular: https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html the whole thing looks very much to me like "oops we put not just AVX-512 but dual AVX-512 in our cores and on 14nm it can hammer the chip so hard we need more voltage... guess we're gonna have to stop and swing the voltage upwards". not only is that less of a problem with higher-efficiency nodes, but the consumer core designs drop the idea of dual-AVX 512 entirely which reduces the worst-case voltage droop as well...
fulafel 4y ago
The main thing seems to be that we don't have general purpouse languages in common use that fit SIMD auovectorization in semantics, without carefully "code to the compiler" portable assembly mindset. Even after 25 years of x86 SIMD support in hardware. It's just not been an aim even in more recent-ish languages like Rust, Swift, Go etc. Partly because of the unworkably hard task of targeting fragmented, undebuggable, proprietary driver/OS programming interfaces from POV of portable languages.
- cmrdporcupine 4y ago
  Exactly. But also he inadvertently actually touches on this in the post. Talking about people using hash tables and so on. I think if people had the right guidance from language and libraries and from education, they would choose array-based patterns more often. They're often in fact simpler to write. But many languages and common practices push people in a highly incremental piece by piece approach. I'd argue we're basically taught to think in terms of loops, from our earliest days of programming. We could just as easily be taught to think of things in terms of bulk arrays. The win in programming this way is extremely high. I was very disappointed to find the SIMD story in Rust so underdeveloped. Yes, people are not running weather forecasting on their home computers. But they're playing video, synthesizing audio, doing speech recognition, etc. all the time. And more of this ever day. And if there's any chance of rescuing the uses of ML from a hellish privacy violating landscape worth of a Butlerian Jihad... it would by making sure most of this type of computation was done locally on-device. Guess APL has the last laugh.
  - acdha 4y ago
    > Yes, people are not running weather forecasting on their home computers. But they're playing video, synthesizing audio, doing speech recognition, etc. all the time. And more of this ever day. > > And if there's any chance of rescuing the uses of ML from a hellish privacy violating landscape worth of a Butlerian Jihad... it would by making sure most of this type of computation was done locally on-device. I think SIMD support is useful too but most of the examples you gave are things which are handled on-device now using dedicated hardware. Video playback has been hardware accelerated for ages, ML acceleration is multiple hardware generations in, etc.
sorenjan 4y ago
Are there any libraries that allow me to write different versions of the same function (AVX-512, AVX2, SSE, etc) and then automatically choose the best one that the system supports at runtime? Or maybe even better, the compiler generates multiple versions for me. In other words, one binary that takes advantage of new instructions but still runs on older hardware. It doesn't really have to be older either, plenty of brand new CPUs doesn't support AVX-512.
- bobowzki 4y ago
  Libvolk works exactly like you describe. https://www.libvolk.org/ https://www.libvolk.org/
- nynx 4y ago
  Rust has built-in support for this, I believe.
  - 3836293648 4y ago
    Rust's native support for this is super verbose and repetetive. There's a macro library that can deal with all the unsafe and version picking for you though.
- curiousmindz 4y ago
  In C#, you can write (pseudo-code): if (AVX2) { ... } else if (SSE) { ... } else ... Then, when you run the program, the JIT will pick the first supported option and eliminate the other ones.
- janwas 4y ago
  Cool, I wasn't familiar with libvolk. It seems to be a collection of prewritten kernels, so it only helps if the function you want to write is among them. github.com/google/highway seems to be closer to what is requested here. It provides around 200 portable intrinsics using which you can write a wide variety of functions (only a single implementation required). Disclosure: I am the main author.
- mhh__ 4y ago
paulmd 4y ago
> End result: go look for AVX-512 benchmarks. You'll find them. And then ask yourself: how many of these are relevant to what I bought my PC/mac/phone for? ... do you guys not do JSON? cause we're doing json. https://www.phoronix.com/review/simdjson-avx-512 https://www.phoronix.com/review/simdjson-avx-512 or how about ARM emulation? anybody doing ARM development on x86? do you guys have phones, maybe? https://www.reddit.com/r/emulation/comments/lzfpz5/what_are_the_implications_of_avx512_for_emulation/ https://www.reddit.com/r/emulation/comments/lzfpz5/what_are_... seriously, linus is wrong on this issue, and he just keeps digging. lose the ego and admit you were spewing FUD. avx-512 had legit problems in Skylake-SP server environments where it was running alongside non-AVX code (although you could segment servers by AVX and non-AVX if you wanted). None of that has been true for any subsequent generations where the downclocking is essentially nil, and the offset has always been configurable for enthusiasts who get more buttons to push. It also was never a "one single instruction triggers latency/downclocking" like some people think, it always took a critical mass of AVX-512 instructions (pulling down the voltage rail) before it paused and triggered downclocking, and "lighter" operations that did less work would trigger this less. https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... AMD putting AVX-512 in Zen4 sealed the deal. AMD isn't making a billion-dollar bet on AVX-512 because of "benchmarksmanship", they're doing it because it's a massive usability win and a huge win for performance. Over time you will see more AVX-512 adoption than AVX2 - because it adds a bunch of usability stuff that makes it even more broadly applicable, like op-masking and scatter operations, and AVX is already very broadly applicable. Some games won't even launch without it anymore because they use it and they didn't bother to write a non-AVX fallback. It's also a huge efficiency win when used in a constrained-power environment - for your 200W-per-1RU power budget, you get more performance using AVX-512 ops than scalar ones. Or you can run at a lower power budget for a fixed performance target. And you're seeing that on Zen4, when it's not tied to Intel's 14nm bullshit (and really it's probably also apparent on Ice Lake-SP if anybody bothered to benchmark that). That's also a huge win in laptops too, where watts are battery life. Would anybody like more efficiency when parsing JSON web service responses when they're on battery? I would. Sorry Linus it's over, just admit you're wrong and move on.
- deleted 4y ago
  [deleted]
rashkov 4y ago
If I understand correctly, this discussion is specifically about auto-vectorization, which is when the compiler automatically re-writes your code to use highly parallel (and performant) vectorized assembly instructions. Linus acknowledges that vectorized instructions are valuable, so his rant seems to be specifically about auto-vectorization: > Yes, yes, yes, you can use it in random places. It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it. In other words.. the people writing foundational parsing libraries (json / strchr / strcmp) are writing their vectorized instructions by hand.
- cmrdporcupine 4y ago
  Well, also in other words, extant languages and CS education is failing people and they're writing these things the wrong way.
  - kolbe 4y ago
    I use intrinsics by hand all the time. It's very easy to make a problem too complicated to autovectorize. And even if you do get it to autovectorizie, it's not exactly future proof against compiler changes.
    - cmrdporcupine 4y ago
      I have definitely done the same. BTW I really like this library https://github.com/vectorclass/ https://github.com/vectorclass/ I wish I had a similar thing for Rust. And for AArch64/NEON.
      - kolbe 4y ago
        I use Agner as well. I started up my own version for Rust specifically targeting avx512[1], but I've been hitting enough snags to where I think I'll abandon it. It's super green at the moment, and I haven't pushed it to Cargo. But if I'm going to dedicate time to it, then I need it to work for my purposes, and there's a thread-parallel problem that makes this unusable for me at the moment. [1] https://github.com/matthewkolbe/lit_math https://github.com/matthewkolbe/lit_math
anonymoushn 4y ago
Linus is correct that auto-vectorization is basically useless, but I have some minor disagreements. JSON is obviously not the only format or protocol that can be parsed using SIMD instructions. In general computers spend a lot of time parsing byte streams, and almost all of these could be parsed at gigabytes per second if someone familiar with the protocol, the intel intrinsics guide, and the tricks used in simdjson spent an afternoon on them. It's sort of unfortunate that our current software ecosystem makes it difficult for many developers to adopt these sorts of specialized parsers written in low-level languages and targeting specific vector extensions. If someone wants to use a hash table or TLS, they may find AES-NI useful.
- jackmott42 4y ago
  One of the specific design properties of AVX-512 is to make auto-vectorization less useless. If Intel had committed to more predictable and wide support, we might be seeing the fruits of that now. Whether it would have been a "Game changer" or just a slight improvement, I don't know, but even slight improvements in cpu performance are worth it these days.
  - dathinab 4y ago
    > "Game changer" made me smirk, as games is what likely would have profited from it games are also what tends to be some of the more havy compute applications outside of servers through it likely would be mostly just some code here and there deep withing the game engine but then this is also an example where auto-vectorization can matter as it means writing platform independent code which happens to be fast on the platforms you care about most but also happens to compile and run with some potential (but only potential) performance penalty when porting to other platforms. Not only makes this porting easier, it also for some games might be fine if the engine is a bit slower. Main drawback is that writing such "auto vectorized" code isn't easy and can be brittle.
- eklitzke 4y ago
  Part of his argument though was that for most cases where you're parsing JSON (or some other protocol), the parsing time is small compared to the amount of time you spend actually doing things with the data. And for most of the examples where the code is mostly parsing (say, an application doing RPC routing, where the service just decodes an RPC and then forwards it to another host) you can often use tricks to avoid parsing the whole document anyway. For example, if you have a protocol based on protobuf (like gRPC) you can put the fields needed for routing in the lowest number fields, and then use protobuf APIs to decode the protobuf lazily/incrementally; this way you will usually just need to decode the first (or perhaps the first few) fields, not the entire protobuf. Making code faster and more efficient is always better, it's just that you might not see much improvement in end-to-end latency times if you make your protocol parsing code faster.
cat_plus_plus 4y ago
It's called GPUs and TPUs? Turned out it's more practical to design hardware for known problems comprising big parts of computing workload than "build it and they will come" Itanium/AVX 512 plans. Also, GPUs have been repurposed for science, machine learning and mining crypto and, as additional useful applications are discovered, new hardware can support them more effectively.
alkonaut 4y ago
The problem with AVX isn’t that compilers find it difficult to emit enough of them or that people don’t often work on problems that are easily transformed to simd at all. It’s enough if a small number of libraries use these instructions, and a lot larger number of programs can benefit. The problem with AVX-512 in my opinion is the sparse hardware support, power/thermal problems and noisy neighbor phenomenon. To be useful simd instructions need to be widely available and not have adverse power effects such as making other programs go much slower.
- mhh__ 4y ago
  Be warned that a lot of the power issues are basically Chinese whispers from processors that are getting on for a decade old now. There are still measurable effects but not like on skylake, to my knowledge at least.
  - alkonaut 4y ago
    In the latest iteration, they removed AVX-512 iirc. So was there even as window when it worked well and shipped on a majority of new CPUs?
    - adgjlsfhk1 4y ago
      No, but that's mainly because Intel has been totally incompetent for the last half decade. On Zen4/Intel chips post 2020 that have it, AVX-512 is really good.
  - jmoss20 4y ago
    No, 512 still fries skylake.
    - mhh__ 4y ago
      Read my comment again
      - 4y ago
gok 4y ago
Firstly, there are in fact a lot of problems that benefit from parallelization, but are not quite wide enough to justify dealing with (say) a GPU. For example, even in graphics you often have medium-sized rasterization tasks that are much more convenient to run on CPUs. Audio processing is similar. Often you have chunks of vectorizable work inter-mixed with CPU-friendly pointer chasing, and the ability to dispatch SIMD work within nanoseconds of SISD work is very helpful. Secondly, AVX-512 (and Arm's SVE) support predication and scatter/gather ops, both of which open the door to much more aggressive forms of automatic vectorization.
oconnor663 4y ago
AVX-512 is fabulous for cryptography, especially software-oriented algorithms like ChaCha and BLAKE3 that lean on general-purpose SIMD features rather than dedicated hardware acceleration. Most of the added value is just the larger registers, but the native bit-rotate instructions are also great. (But agreed with Linus that autovectorization isn't very helpful for this stuff.)
- Havoc 4y ago
  Requiring special hardware to avoid other more different special hardware seems a little questionable
  - dathinab 4y ago
    Not if a few docent different "special cases" all can use the same special hardware instead of a few docent different hardwares. And the thing with a lot of this "special cases" is that while most code doesn't write them a lot of systems have them running somewhere.
    - Havoc 4y ago
      Yea you’re right. My comment was admittedly a bit cheeky
raggi 4y ago
trigger warning: possible rust advocacy. I hope the warning can help some folks relax, and stave off the usual repeating meta discussion. I've looked at a lot of rust generated outputs from iterator usage in godbolt. If you have tried this (and if not, go look), you'll know that you almost certainly never want to look at debug output for iterator usage. In release mode builds of rust programs using the iterator package you'll often see SIMD. The author doesn't need to do special things to get this, the combination of aggressive inlining and encouragement to use owned and intrusive structures goes a long way to making this viable more often in practical code - so much so that I see it working _a lot_. It's not a magic bullet, and I'm not advocating that it's some be-all end-all, but it works well there. If you'd like to see an example, search for "godbolt rust iterator" and look at the optimized output. I'm sure one of the near top results will do.
- fulafel 4y ago
  I think the interesting question is that how big a speed delta does a typical Rust app enjoy when built with autovectorization enabled vs not. You get points for going faster, not for sometimes emitting the instructions.
- joshfee 4y ago
  Rust or otherwise, the more semantics that can be expressed in a language, the more optimizations that can be done for you (at the cost of lacking full control in certain scenarios).
- nitwit005 4y ago
  A lot of people in C++ land probably just have the vector optimizations off. If you use clang, you will get vector instructions at -O2. Gcc seems to need -O3, or the specific flags enabled.
protastus 4y ago
SSE, NEON and AVX are fundamental for anything reducible to matrix and vector arithmetic (e.g., signal processing and ML inference on the CPU). > It is hard to find simple enough code to vectorize at all. It's hard to find code simple enough for the compiler to efficiently auto-vectorize. Anything that reduces to GEMM will parallelize in practice extremely well, and there are many excellent libraries with SIMD support (MKL, BLAS, ATLAS, Eigen, etc.). However, these libraries rely on kernels carefully written by experts and benchmarked extensively over decades. They're not the output of running naively written code through a super smart compiler. All of this is extremely relevant to what you bought your PC or phone for. It's also not in the kernel, and therefore Linus seems to be unaware of their pervasiveness and utility.
andromeduck 4y ago
Wish RWT had better UI.
vgatherps 4y ago
I'm freshly out of the weeds of implementing some fixed-point decimal parsers, and only found ~1 cycle that I could save (at best) per iteration using avx512. I largely agree with linus here. It's still a huge pain to do anything non-trivially-parallelizable with simd even in an avx512 world 1. I have NEVER been satisfied with the results of a compiler autovectorization, outside of the most trivial examples. Even when they can autovectorize, they fail to do basic transformations that greatly unlock ILP (to be fair these aren't allowed by FP rules since it changes operation ordering) 2. Gather/scatter still have awful performance, worse than manually doing the loads and piecemeal assembling the vector. The fancy 'results-partially-delivered-in-order-on-trap' semantics probably don't help. Speeding up these operations would instantly unlock SO much more performance (see my username) 3. The whole 'extract-upper-bit-to-bitmask' -> 'find-first-set-in-bitmask' really deserves an instruction to merge the last two (return index of first byte with uppermost bit set, etc). It's such a core series of instructions and ime often the bottleneck (each of those is 3 cycles) 4. The performance of some of the nicer avx512 intrinsics leaves a lot to be desired. Compress for example is neat in theory but is too slow for anything I've tried in real life. Masked loads are another example, cool in theory but in practice MUCH faster to replicate manually. 5. It still feels like there's a ton of really random/arbitrary holes in the instruction set. AVX512 closed some of these but introduced many more with the limited operations you can do on masks and expensive mask <-> integer conversions.
gonzo 4y ago
AVX 2 and AVX 512 are very good for VPN transforms. 10gbps on a single core.
singularity2001 4y ago
Abstract vectorization instructions in wasm will make life a lot easier https://github.com/WebAssembly/flexible-vectors/blob/main/proposals/flexible-vectors/FlexibleVectors.md https://github.com/WebAssembly/flexible-vectors/blob/main/pr... great proposal! Mapping to whatever hardware is available as some sort of micro library

203 points by fulafel 4y ago

frozenport 4y ago

These dudes are biased: they don't want to support something they don't use.

Razengan 4y ago
The PC dev world in a nutshell.

PaulHoule 4y ago

AVX 512 has a number of instructions that make it useful for UTF-8 parsing, floating point parsing, XML parsing, JSON parsing, things like that. It is tricky coding https://lemire.me/blog/2022/05/25/parsing-json-faster-with-intel-avx-512/ https://lemire.me/blog/2022/05/25/parsing-json-faster-with-i... All things that are good for HFT but also good for speeding up your web browser and maybe even saving power because you can dial down the clock rate. It's a tragedy that Intel is fusing off AVX 512 in consumer parts so they can stuff the chips with thoroughly pizzled phonish low-performance cores.

eesmith 4y ago
I think that's what "if you aren't into parsing (but not using) JSON" is supposed to be referencing. And, "It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it."
- PaulHoule 4y ago
  One thing I learned in grad school is you can do a stupendous number of floating point operations in the time it takes to serialize a matrix to ASCII numbers and deserialize it. Whatever it is you are doing with a JSON document might be slower than parsing it. It's true that autovectorization accomplishes very little but specialized libraries could have a notable effect on perceived performance if they were widely developed and used. Frankly Intel has had less interest in getting you to buy a new computer by making it the best computer you ever bought than it has been in taking as much revenue as it can from the rest of the BOM, for instance the junk integrated graphics that sabotaged the Windows Vista launch and has been making your computer crash faster ever since. Another example is that they don't ship MKL out of the box on Windows or Linux although they do on MacOS. And Intel wonders why their sales are slipping...
  - oxfeed65261 4y ago
    Had to look it up: > Intel oneAPI Math Kernel Library (Intel oneMKL; formerly Intel Math Kernel Library or Intel MKL), is a library of optimized math routines for science, engineering, and financial applications. Core math functions include BLAS, LAPACK, ScaLAPACK, sparse solvers, fast Fourier transforms, and vector math. https://en.m.wikipedia.org/wiki/Math_Kernel_Library https://en.m.wikipedia.org/wiki/Math_Kernel_Library
  - kzrdude 4y ago
    Matrix multiplication and similar is also one of the few operations where algorithms and special case instructions are interesting for floating point on a massive scale. I.e adding two arrays together, computing dot products, those operations are just memory bound when the data grows, but matrix multiplication is dense enough with operations per element that it is limited by arithmetic operations too.
  - ilyt 4y ago
    > One thing I learned in grad school is you can do a stupendous number of floating point operations in the time it takes to serialize a matrix to ASCII numbers and deserialize it. Whatever it is you are doing with a JSON document might be slower than parsing it. > It's true that autovectorization accomplishes very little but specialized libraries could have a notable effect on perceived performance if they were widely developed and used. I mean, sure, but even if we take JSON as an example, in vast majority of cases it gets fed to a giant blob of JS driving even bigger blob of browser code. The cases where you do deserialization -> very little processing -> serialization are pretty rare. Sure, if it is already on chip might as well use it, but realistically savings will be in single digit percents.
    - lmm 4y ago
      > The cases where you do deserialization -> very little processing -> serialization are pretty rare. Actually I've seen a lot of systems that do that - query a datastore, do some minimal processing of the results, feed them back to the caller. Although that tends to get addressed at a higher level, e.g. MongoDB drivers shifting towards using BSON.
zeusk 4y ago
They have a replacement coming in 2023/24
- kristianp 4y ago
  Replacement cpu generation or replacement for avx-512? Will it be variable length?
jeffbee 4y ago
I don't think Intel has even decided whether to fuse them off or not. In my newest Intel desktop, a Core i7-13700K, AVX-512 is actually available. It wasn't available on the i7-12700K that was in the same system on the same motherboard a few weeks ago.
- compsciphd 4y ago
  i thought the reason they wanted to fuse them off was for all cores being completely compatible (just slower), without the need for the OS to force scheduling on specific cores (I'd guess what would have to happen is if an instruction not available on efficiency core was called, it trap, the OS would see it and then mark the process to only be executable on performance core?)
  - jeffbee 4y ago
    OK but that doesn't explain why it reappeared in Raptor Lake.
    - EgoIncarnate 4y ago
      The official specs for Raptor Lake don't include AVX512 support, so most likley this is just something that slipped through on your system. It might be "corrected" in later steppings or updated firmware. Some early 12th gen systems had unfused CPUs with AVX512 available on the performance cores and firmware that didn't disable it. https://www.intel.com/content/www/us/en/products/sku/230500/intel-core-i713700k-processor-30m-cache-up-to-5-40-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/230500/...
      - chillfox 4y ago
        So your saying that they have shitty QA which we just happen to benefit from this time?
- borissk 4y ago
  Are you sure? According to Intel spec AVX-512 is not available - https://ark.intel.com/content/www/us/en/ark/products/230500/intel-core-i713700k-processor-30m-cache-up-to-5-40-ghz.html https://ark.intel.com/content/www/us/en/ark/products/230500/...
ilyt 4y ago
Didn't read the post ? THat is explicitly mentioned > Yes, yes, yes, you can use it in random places. It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it.
- bluestreak 4y ago
  I couldn't agree more. I don't think compiler vectorization is that useful even for columnar (!) database we're building. The specialized JIT doesn't even use AVX512 because too much effort for little to no gain.
  - sagarm 4y ago
    Vectorization (auto or manual) can really help optimize bottlenecks like evaluating simple comparisons once you're out of easy algorithmic wins. It takes so much attention it's only worth doing in the most critical cases IMO. Auto-vectorization can be fragile. Manual vectorization is a ton of work and difficult to maintain.
- gumby 4y ago
  > And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it. How much of that is due to language semantics? Can't FORTRAN compilers make better use of the AVX instructions than, say, a C or Rust compiler could due to memory reference semantics?
  - dan-robertson 4y ago
    In theory, Rust should have much better memory semantics than C but currently much fighting is required to get llvm to take advantage of the Rust semantics.
sitkack 4y ago
Vector instructions (ala Cray) are future proof in ways that SIMD are not. Eventually every high performance CPU will have HBM, I am not sure SIMD is the mechanism we will use in to extract the most efficiency out of the platform. https://people.eecs.berkeley.edu/~krste/papers/yunsup-ms.pdf https://people.eecs.berkeley.edu/~krste/papers/yunsup-ms.pdf https://parlab.eecs.berkeley.edu/sites/all/parlab/files/maven-poster-parlab2009-06-01.pdf https://parlab.eecs.berkeley.edu/sites/all/parlab/files/mave... https://www.cs.uic.edu/~ajayk/c566/VectorProcessors.pdf https://www.cs.uic.edu/~ajayk/c566/VectorProcessors.pdf https://inst.eecs.berkeley.edu/~cs152/sp18/lectures/L16-RISCV-Vectors.pdf https://inst.eecs.berkeley.edu/~cs152/sp18/lectures/L16-RISC...
- janwas 4y ago
  I feel that SIMD vs vector is not the most useful distinction to make. Yes, at the ISA and binutils level there is a difference in terms of number of instructions and ugly encodings. But the actual application source code can actually look the same for SIMD and vector - vector-length agnostic style is helpful for both. Instead, it seems more interesting to focus on preparing more applications for either SIMD or vector: with data-parallel algorithms, avoiding branches etc.
- lifthrasiir 4y ago
  Of course when you only have packed instructions (a la SIMD) you have no choice but using them in place of vector instructions, but vector instructions in general do not fully subsume packed instructions. I think packed instructions are better at exploiting opportunistic parallelism from either a short loop (where vector instructions would have higher overhead) or a series of similar but slightly different scalar instructions.
- fooblaster 4y ago
  There have been attempts to remedy the fixed width simd vector problem that are already shipping: https://developer.arm.com/Architectures/Scalable%20Vector%20Extensions https://developer.arm.com/Architectures/Scalable%20Vector%20...
torginus 4y ago
Generally speaking, the idea that the average person uses more than 4 cores is insane. Even as a power user/dev I can count the times when I needed more than 4 cores on one hand. All the benchmarks that these CPUs are evaluated by, test either games, or run the same crazy stuff like fluid simulations that OPs article complains about. Hardware reviewers need a reality check.
- Iwan-Zotow 4y ago
  good porno will saturate all available cores
- deleted 4y ago
  [deleted]
- jl6 4y ago
  Eh, a modern desktop can have a lot of concurrent processes running. A browser, one process per tab, software updates continually spawning in the background… Though I do agree that the benefits of high core count CPUs are oversold as far as most users are concerned.
  - Dylan16807 4y ago
    Many processes yes, but background tasks don't need much power and one core has a lot of power. So one core should be plenty for all background tasks combined.
- eyegor 4y ago
  I hate to break it to you, but outside of gaming, fluid sim, and battery life / power draw tests, nothing else matters for modern chips. Schmucks only need the kind of horsepower a phone is capable of, or want to know how long it will last. The only people who care about performance either really care because more perf = more dollars (fluid sim, hft, etc.) or they're gamers.
  - brookst 4y ago
    I use CAD for a hobby. My 5950x has 16 cores and I desperately wish I could afford a 64 core desktop because each model takes 60 - 240 seconds to compute. That’s a pain when it takes multiple iterations to get a good design, and it leads to more complicated programming to offer quick vs detailed model creation, and that complexity leads to bugs. So yeah I care, and not for more dollars.
bitcharmer 4y ago
> All things that are good for HFT (...) Not sure if you're referring to high frequency trading here, but if that's the case then it's just not true. AVX is infamously bad for low latency becuse these instructions result in insane amounts of heat which forces the CPU to get down clocked (sometimes the cores on the neighboring tiles on the SoC too). P-State transitions are a serious source of execution jitter and a big no-no in our business. Many HFT shops actually disable AVX with noxsave. It's fantastically useful for HPC thought.
- dan-robertson 4y ago
  I think I found there’s also a quite expensive ‘warmup’ which makes much of AVX only worth it for larger loops. But my memory or original understanding might be incorrect and my understanding of why that warmup might be the case (maybe microcode caches in the front end??) is very poor.

37ef_ced3 4y ago

Linus Torvalds: And I claim that that is the real problem with AVX-512 (and pretty much any vectorization). I personally cannot find a single benchmark that does anything I would ever do - not even remotely close. So if you aren't into some chess engine, if you aren't into parsing (but not using) JSON, if you aren't into software raytracing (as opposed to raytracing in games, which is clearly starting to take off thanks to GPU support), what else is there? Answer? Neural net inference, e.g., https://NN-512.com https://NN-512.com If you need a little bit of inference (say, 20 ReNet50s per second per CPU core) as part of a larger system, there's nothing cheaper. If you're doing a small amount of inference, perhaps limited by other parts of the system, you can't keep a GPU fed and the GPU is a huge waste of money. AVX-512, with its masked operations and dual-input permutations, is an expressive and powerful SIMD instruction set. It's a pleasure to write code for, but we need good hardware support (which is literally years overdue).

quelsolaar 4y ago
True, but the problem is that that is today better done on vector hardware like a GPU or other ML hardware. The world has sort of diverged int to two camps: vectorizable problems that can be massivly paralleleized (graphics, simulation, ML) and for that we use GPUs, and then everything else is CPU. What i think linus is saying is that there are few reasons to use AVX-512 on a CPU, when the is a GPU much better siuted for those kinds of problems. You could say that the intersecting area in the ven diagram of "Has to run on CPU" and "Can use vector instructions" is small.
- dtx1 4y ago
  > You could say that the intersecting area in the ven diagram of "Has to run on CPU" and "Can use vector instructions" is small. People are forgetting the "Could run on a GPU but I don't know how" factor. There's tons of Situations where GPU Offloading would be fast or more energy efficient but importing all the libraries, dealing with drivers etc. really is not worth the effort, whereas doing it on a CPU is really just a simple include away.
- dragontamer 4y ago

superkuh 4y ago

UH... didn't Intel actually stop including AVX 512 in their newest (alder lake) processors? This seems unlikely. ie, https://www.tomshardware.com/news/intel-nukes-alder-lake-avx-512-now-fuses-it-off-in-silicon https://www.tomshardware.com/news/intel-nukes-alder-lake-avx... >"Although AVX-512 was not fuse-disabled on certain early Alder Lake desktop products, Intel plans to fuse off AVX-512 on Alder Lake products going forward." -Intel Spokesperson to Tom's Hardware.

Kon-Peki 4y ago
On the other hand, you can now virtually guarantee that a GPGPU is present on Intel consumer chips. So now you can write your code 3 times; no vector acceleration (for the really old stuff), AVX-512 for the servers, and GPU for the consumer chips!
- convolvatron 4y ago
  or we can finally starting looking into languages with implicit fine grained parallelism and run on whatever
  - anonymoushn 4y ago
    This won't help, most uses of AVX or NEON have nothing to do with parallel processing of a bunch of numbers
  - goalieca 4y ago
    We have had that for 15+ years even in C with fine grained looping like OpenMP. It’s terribly inefficient to communicate across threads. Sometimes you just need SIMD
    - gpderetta 4y ago
      OMP is supposed to support SIMD/GPU offloading theses days. Not sure how good it is though.
    - 4y ago

the8472 4y ago

Some of the AVX512 instructions are nice additions though, even if you only use it on YMM or XMM registers. All the masked instructions make it much simpler to vectorize loops over variable amounts of data without special treatment for the head/tail of the data.

stephencanon 4y ago
This is exactly right—-the 512b width is the least interesting thing about AVX512. All the new instructions on 128, 256, and 512b vectors, (finally) good support for unsigned integers, static rounding modes for floating point, and predication are much more exciting.
anonymoushn 4y ago
I don't mind head/tail stuff so much but the masked instructions and vcompress make a lot of "do some branchy different stuff to different data or filter a list" easier and faster compared to AVX2, so I'm a big fan

cmrdporcupine 4y ago

SIMD on x86 has become a ridiculous minefield of standards. I honestly pity the compiler writers who have to navigate this nonsense. It was stressful enough trying to just manage the compiler flagging in an application (music synthesizer) I wrote recently. The core amd64 ISA is stable and that's good. I wouldn't call it elegant or nice, but it's there. The patchwork of which processor supports which SIMD extension on top of it, though, has gotten out of control and is unpleasant. I'm sure AVX512 is great. Except as an application author I can't rely on it being there. I am not a MacOS user or MacOS developer, but I envy the consistency of that platform now that they control their own ISA.

paulmd 4y ago
> SIMD on x86 has become a ridiculous minefield of standards. people make the AVX-512 compatibility thing more complex than it has to be. there's consumer AVX-512 and server AVX-512 and within those series there has always been strictly increasing compatibility, with a common core between them. and AMD supports the complete thing with Zen4 so there is no differentiation for them. intel didn't want to put bfloat16 ML instructions in consumer chips and people turned that into a whole thing because they like to meme about AVX-512, but that's literally the only difference. it's just not that complex a story though, if you don't turn it into a thing. intel's consumer and server teams have been operating quasi-independently and some of them are on "newer generations" than the other team. currently that's the client team that's ahead of the server team. so client got a couple gens ahead of server, and server has this one extra instruction in their versions. that's it, that's the whole deal. frankly it's actually a simpler story than SSE4 where AMD just kinda went and did its own thing for a while and Intel went and did AVX and there truly was total divergence between what the companies were doing. you can probably turn SSE into a real ugly venn diagram if you try to capture every nuance too! After all, some of them were introduced in K8, K9, Phenom... and there's lots of sub-versions of each. xeon phi is also its own thing but you'll know if you're targeting that. it has a weird 4-wide AVX thing to go along with its 4-wide SMT, and it's its own specific thing.
- hajile 4y ago
  Based on Wikipedia's compatibility chart, it sure looks like a minefield to me. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512

Aardwolf 4y ago

> Seriously, go look at random code on github - I dare you. Very little of it is clearly delineated loops over arrays with high repeat counts. It's not very helpful to be able to vectorize some "search for a value in an array", when people end up using hash tables or other more complex data structures instead of arrays for searching. Sure, but many parts that matter for performance, the inner loops to optimize, may be like that. Multimedia encoding/decoding, emulation, ML, matrix multiplications, ...

Tuna-Fish 4y ago
> Multimedia encoding/decoding, Done on special-purpose hardware that is >10 times more efficient than any implementation on GP hardware. > emulation, Only current practical use of AVX-512. > ML, matrix multiplications, ... Much better to do on the GPU. In general wide SIMD on CPU has the problem that to make effective use of it, you have to massage your code enough that you are ~95% away from just running it on the GPU anyway, and you can gain much more performance if you do that. The best niche of AVX-512 would have been as the baseline common target for things that also get optimized for GPUs... except that Intel has eliminated this possibility by heavy product segmentation right from the start.
- Aardwolf 4y ago
  > Done on special-purpose hardware that is >10 times more efficient than any implementation on GP hardware. That locks users in to only the codecs that have been implemented in this particular hardware, and increases hardware size for particular vendor implementations rather than provide common building blocks that many codecs use (DCT, ...) > Much better to do on the GPU. Yep and that's what I run ML stuff on, not all systems have a GPU available though, and for some applications it's faster to do something immediately on the CPU, than have the overhead of going to/from GPU > In general wide SIMD on CPU has the problem that to make effective use of it, you have to massage your code enough that you are ~95% away from just running it on the GPU anyway, Better tooling and libraries could help with this imho. Note that the situation for GPU is here also not great at all, since the good tools are locked into 1 vendor.
- 4y ago

ak217 4y ago

Some of the biggest investments into general purpose Linux performance, reliability, and sustainability were made by people with "specialty engines" that Linus is talking about. The specialty applications often turn out to be important because they push the platform to excel in unexpected ways. With that said, the biggest practical problem we saw with AVX-512 was that it's so big, power hungry, and dissimilar to other components of the CPU. One tenant turning that thing on can throttle the whole chip, causing unpredictable performance and even reliability issues for everything on the machine. I imagine it's even worse on mobile/embedded, where turning it on can blow out your power and thermal budget. The alternatives (integrated GPUs/ML cores) seem to be more power efficient.

cogman10 4y ago
> and dissimilar to other components of the CPU It's not terribly dissimilar from the SSE instructions before it. The biggest weirdness about AVX is it's CPU throttling. Which, IMO, should not be a thing for the reasons you mention. I imagine that future iterations of AVX will likely ditch CPU throttling. I'm guessing it was a "our current design can't handle this and we need to ship!" sort of solution. (If they haven't already? Do current CPUs throttle AVX-512 instructions?) > I imagine it's even worse on mobile/embedded Fortunately, x86 mobile/embedded isn't terribly common, certainly not with AVX-512 support.
- Kon-Peki 4y ago
  > I imagine that future iterations of AVX will likely ditch CPU throttling Already happening: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Downclocking https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Dow...
- paulmd 4y ago
  > I imagine that future iterations of AVX will likely ditch CPU throttling. I'm guessing it was a "our current design can't handle this and we need to ship!" sort of solution. (If they haven't already? Do current CPUs throttle AVX-512 instructions?) exactly what it looks like to me, and it was already fixed in Ice Lake and probably doesn't exist at all in Zen4. Just nobody cares about Ice Lake-SP when Epyc is crushing the server market. https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... and some measurements of Skylake-X displaying the effect, note the voltage swings in particular: https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html the whole thing looks very much to me like "oops we put not just AVX-512 but dual AVX-512 in our cores and on 14nm it can hammer the chip so hard we need more voltage... guess we're gonna have to stop and swing the voltage upwards". not only is that less of a problem with higher-efficiency nodes, but the consumer core designs drop the idea of dual-AVX 512 entirely which reduces the worst-case voltage droop as well...

fulafel 4y ago

The main thing seems to be that we don't have general purpouse languages in common use that fit SIMD auovectorization in semantics, without carefully "code to the compiler" portable assembly mindset. Even after 25 years of x86 SIMD support in hardware. It's just not been an aim even in more recent-ish languages like Rust, Swift, Go etc. Partly because of the unworkably hard task of targeting fragmented, undebuggable, proprietary driver/OS programming interfaces from POV of portable languages.

cmrdporcupine 4y ago
Exactly. But also he inadvertently actually touches on this in the post. Talking about people using hash tables and so on. I think if people had the right guidance from language and libraries and from education, they would choose array-based patterns more often. They're often in fact simpler to write. But many languages and common practices push people in a highly incremental piece by piece approach. I'd argue we're basically taught to think in terms of loops, from our earliest days of programming. We could just as easily be taught to think of things in terms of bulk arrays. The win in programming this way is extremely high. I was very disappointed to find the SIMD story in Rust so underdeveloped. Yes, people are not running weather forecasting on their home computers. But they're playing video, synthesizing audio, doing speech recognition, etc. all the time. And more of this ever day. And if there's any chance of rescuing the uses of ML from a hellish privacy violating landscape worth of a Butlerian Jihad... it would by making sure most of this type of computation was done locally on-device. Guess APL has the last laugh.
- acdha 4y ago
  > Yes, people are not running weather forecasting on their home computers. But they're playing video, synthesizing audio, doing speech recognition, etc. all the time. And more of this ever day. > > And if there's any chance of rescuing the uses of ML from a hellish privacy violating landscape worth of a Butlerian Jihad... it would by making sure most of this type of computation was done locally on-device. I think SIMD support is useful too but most of the examples you gave are things which are handled on-device now using dedicated hardware. Video playback has been hardware accelerated for ages, ML acceleration is multiple hardware generations in, etc.

sorenjan 4y ago

Are there any libraries that allow me to write different versions of the same function (AVX-512, AVX2, SSE, etc) and then automatically choose the best one that the system supports at runtime? Or maybe even better, the compiler generates multiple versions for me. In other words, one binary that takes advantage of new instructions but still runs on older hardware. It doesn't really have to be older either, plenty of brand new CPUs doesn't support AVX-512.

bobowzki 4y ago
Libvolk works exactly like you describe. https://www.libvolk.org/ https://www.libvolk.org/
nynx 4y ago
Rust has built-in support for this, I believe.
- 3836293648 4y ago
  Rust's native support for this is super verbose and repetetive. There's a macro library that can deal with all the unsafe and version picking for you though.
curiousmindz 4y ago
In C#, you can write (pseudo-code): if (AVX2) { ... } else if (SSE) { ... } else ... Then, when you run the program, the JIT will pick the first supported option and eliminate the other ones.
janwas 4y ago
Cool, I wasn't familiar with libvolk. It seems to be a collection of prewritten kernels, so it only helps if the function you want to write is among them. github.com/google/highway seems to be closer to what is requested here. It provides around 200 portable intrinsics using which you can write a wide variety of functions (only a single implementation required). Disclosure: I am the main author.
mhh__ 4y ago

paulmd 4y ago

> End result: go look for AVX-512 benchmarks. You'll find them. And then ask yourself: how many of these are relevant to what I bought my PC/mac/phone for? ... do you guys not do JSON? cause we're doing json. https://www.phoronix.com/review/simdjson-avx-512 https://www.phoronix.com/review/simdjson-avx-512 or how about ARM emulation? anybody doing ARM development on x86? do you guys have phones, maybe? https://www.reddit.com/r/emulation/comments/lzfpz5/what_are_the_implications_of_avx512_for_emulation/ https://www.reddit.com/r/emulation/comments/lzfpz5/what_are_... seriously, linus is wrong on this issue, and he just keeps digging. lose the ego and admit you were spewing FUD. avx-512 had legit problems in Skylake-SP server environments where it was running alongside non-AVX code (although you could segment servers by AVX and non-AVX if you wanted). None of that has been true for any subsequent generations where the downclocking is essentially nil, and the offset has always been configurable for enthusiasts who get more buttons to push. It also was never a "one single instruction triggers latency/downclocking" like some people think, it always took a critical mass of AVX-512 instructions (pulling down the voltage rail) before it paused and triggered downclocking, and "lighter" operations that did less work would trigger this less. https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... AMD putting AVX-512 in Zen4 sealed the deal. AMD isn't making a billion-dollar bet on AVX-512 because of "benchmarksmanship", they're doing it because it's a massive usability win and a huge win for performance. Over time you will see more AVX-512 adoption than AVX2 - because it adds a bunch of usability stuff that makes it even more broadly applicable, like op-masking and scatter operations, and AVX is already very broadly applicable. Some games won't even launch without it anymore because they use it and they didn't bother to write a non-AVX fallback. It's also a huge efficiency win when used in a constrained-power environment - for your 200W-per-1RU power budget, you get more performance using AVX-512 ops than scalar ones. Or you can run at a lower power budget for a fixed performance target. And you're seeing that on Zen4, when it's not tied to Intel's 14nm bullshit (and really it's probably also apparent on Ice Lake-SP if anybody bothered to benchmark that). That's also a huge win in laptops too, where watts are battery life. Would anybody like more efficiency when parsing JSON web service responses when they're on battery? I would. Sorry Linus it's over, just admit you're wrong and move on.

deleted 4y ago
[deleted]

rashkov 4y ago

If I understand correctly, this discussion is specifically about auto-vectorization, which is when the compiler automatically re-writes your code to use highly parallel (and performant) vectorized assembly instructions. Linus acknowledges that vectorized instructions are valuable, so his rant seems to be specifically about auto-vectorization: > Yes, yes, yes, you can use it in random places. It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it. In other words.. the people writing foundational parsing libraries (json / strchr / strcmp) are writing their vectorized instructions by hand.

cmrdporcupine 4y ago
Well, also in other words, extant languages and CS education is failing people and they're writing these things the wrong way.
- kolbe 4y ago
  I use intrinsics by hand all the time. It's very easy to make a problem too complicated to autovectorize. And even if you do get it to autovectorizie, it's not exactly future proof against compiler changes.
  - cmrdporcupine 4y ago
    I have definitely done the same. BTW I really like this library https://github.com/vectorclass/ https://github.com/vectorclass/ I wish I had a similar thing for Rust. And for AArch64/NEON.
    - kolbe 4y ago
      I use Agner as well. I started up my own version for Rust specifically targeting avx512[1], but I've been hitting enough snags to where I think I'll abandon it. It's super green at the moment, and I haven't pushed it to Cargo. But if I'm going to dedicate time to it, then I need it to work for my purposes, and there's a thread-parallel problem that makes this unusable for me at the moment. [1] https://github.com/matthewkolbe/lit_math https://github.com/matthewkolbe/lit_math

anonymoushn 4y ago

Linus is correct that auto-vectorization is basically useless, but I have some minor disagreements. JSON is obviously not the only format or protocol that can be parsed using SIMD instructions. In general computers spend a lot of time parsing byte streams, and almost all of these could be parsed at gigabytes per second if someone familiar with the protocol, the intel intrinsics guide, and the tricks used in simdjson spent an afternoon on them. It's sort of unfortunate that our current software ecosystem makes it difficult for many developers to adopt these sorts of specialized parsers written in low-level languages and targeting specific vector extensions. If someone wants to use a hash table or TLS, they may find AES-NI useful.

jackmott42 4y ago
One of the specific design properties of AVX-512 is to make auto-vectorization less useless. If Intel had committed to more predictable and wide support, we might be seeing the fruits of that now. Whether it would have been a "Game changer" or just a slight improvement, I don't know, but even slight improvements in cpu performance are worth it these days.
- dathinab 4y ago
  > "Game changer" made me smirk, as games is what likely would have profited from it games are also what tends to be some of the more havy compute applications outside of servers through it likely would be mostly just some code here and there deep withing the game engine but then this is also an example where auto-vectorization can matter as it means writing platform independent code which happens to be fast on the platforms you care about most but also happens to compile and run with some potential (but only potential) performance penalty when porting to other platforms. Not only makes this porting easier, it also for some games might be fine if the engine is a bit slower. Main drawback is that writing such "auto vectorized" code isn't easy and can be brittle.
eklitzke 4y ago
Part of his argument though was that for most cases where you're parsing JSON (or some other protocol), the parsing time is small compared to the amount of time you spend actually doing things with the data. And for most of the examples where the code is mostly parsing (say, an application doing RPC routing, where the service just decodes an RPC and then forwards it to another host) you can often use tricks to avoid parsing the whole document anyway. For example, if you have a protocol based on protobuf (like gRPC) you can put the fields needed for routing in the lowest number fields, and then use protobuf APIs to decode the protobuf lazily/incrementally; this way you will usually just need to decode the first (or perhaps the first few) fields, not the entire protobuf. Making code faster and more efficient is always better, it's just that you might not see much improvement in end-to-end latency times if you make your protocol parsing code faster.

cat_plus_plus 4y ago

It's called GPUs and TPUs? Turned out it's more practical to design hardware for known problems comprising big parts of computing workload than "build it and they will come" Itanium/AVX 512 plans. Also, GPUs have been repurposed for science, machine learning and mining crypto and, as additional useful applications are discovered, new hardware can support them more effectively.

alkonaut 4y ago

The problem with AVX isn’t that compilers find it difficult to emit enough of them or that people don’t often work on problems that are easily transformed to simd at all. It’s enough if a small number of libraries use these instructions, and a lot larger number of programs can benefit. The problem with AVX-512 in my opinion is the sparse hardware support, power/thermal problems and noisy neighbor phenomenon. To be useful simd instructions need to be widely available and not have adverse power effects such as making other programs go much slower.

mhh__ 4y ago
Be warned that a lot of the power issues are basically Chinese whispers from processors that are getting on for a decade old now. There are still measurable effects but not like on skylake, to my knowledge at least.
- alkonaut 4y ago
  In the latest iteration, they removed AVX-512 iirc. So was there even as window when it worked well and shipped on a majority of new CPUs?
  - adgjlsfhk1 4y ago
    No, but that's mainly because Intel has been totally incompetent for the last half decade. On Zen4/Intel chips post 2020 that have it, AVX-512 is really good.
- jmoss20 4y ago
  No, 512 still fries skylake.
  - mhh__ 4y ago
    Read my comment again
    - 4y ago

gok 4y ago

Firstly, there are in fact a lot of problems that benefit from parallelization, but are not quite wide enough to justify dealing with (say) a GPU. For example, even in graphics you often have medium-sized rasterization tasks that are much more convenient to run on CPUs. Audio processing is similar. Often you have chunks of vectorizable work inter-mixed with CPU-friendly pointer chasing, and the ability to dispatch SIMD work within nanoseconds of SISD work is very helpful. Secondly, AVX-512 (and Arm's SVE) support predication and scatter/gather ops, both of which open the door to much more aggressive forms of automatic vectorization.

oconnor663 4y ago

AVX-512 is fabulous for cryptography, especially software-oriented algorithms like ChaCha and BLAKE3 that lean on general-purpose SIMD features rather than dedicated hardware acceleration. Most of the added value is just the larger registers, but the native bit-rotate instructions are also great. (But agreed with Linus that autovectorization isn't very helpful for this stuff.)

Havoc 4y ago
Requiring special hardware to avoid other more different special hardware seems a little questionable
- dathinab 4y ago
  Not if a few docent different "special cases" all can use the same special hardware instead of a few docent different hardwares. And the thing with a lot of this "special cases" is that while most code doesn't write them a lot of systems have them running somewhere.
  - Havoc 4y ago
    Yea you’re right. My comment was admittedly a bit cheeky

raggi 4y ago

trigger warning: possible rust advocacy. I hope the warning can help some folks relax, and stave off the usual repeating meta discussion. I've looked at a lot of rust generated outputs from iterator usage in godbolt. If you have tried this (and if not, go look), you'll know that you almost certainly never want to look at debug output for iterator usage. In release mode builds of rust programs using the iterator package you'll often see SIMD. The author doesn't need to do special things to get this, the combination of aggressive inlining and encouragement to use owned and intrusive structures goes a long way to making this viable more often in practical code - so much so that I see it working _a lot_. It's not a magic bullet, and I'm not advocating that it's some be-all end-all, but it works well there. If you'd like to see an example, search for "godbolt rust iterator" and look at the optimized output. I'm sure one of the near top results will do.

fulafel 4y ago
I think the interesting question is that how big a speed delta does a typical Rust app enjoy when built with autovectorization enabled vs not. You get points for going faster, not for sometimes emitting the instructions.
joshfee 4y ago
Rust or otherwise, the more semantics that can be expressed in a language, the more optimizations that can be done for you (at the cost of lacking full control in certain scenarios).
nitwit005 4y ago
A lot of people in C++ land probably just have the vector optimizations off. If you use clang, you will get vector instructions at -O2. Gcc seems to need -O3, or the specific flags enabled.

protastus 4y ago

SSE, NEON and AVX are fundamental for anything reducible to matrix and vector arithmetic (e.g., signal processing and ML inference on the CPU). > It is hard to find simple enough code to vectorize at all. It's hard to find code simple enough for the compiler to efficiently auto-vectorize. Anything that reduces to GEMM will parallelize in practice extremely well, and there are many excellent libraries with SIMD support (MKL, BLAS, ATLAS, Eigen, etc.). However, these libraries rely on kernels carefully written by experts and benchmarked extensively over decades. They're not the output of running naively written code through a super smart compiler. All of this is extremely relevant to what you bought your PC or phone for. It's also not in the kernel, and therefore Linus seems to be unaware of their pervasiveness and utility.

andromeduck 4y ago

Wish RWT had better UI.

vgatherps 4y ago

I'm freshly out of the weeds of implementing some fixed-point decimal parsers, and only found ~1 cycle that I could save (at best) per iteration using avx512. I largely agree with linus here. It's still a huge pain to do anything non-trivially-parallelizable with simd even in an avx512 world 1. I have NEVER been satisfied with the results of a compiler autovectorization, outside of the most trivial examples. Even when they can autovectorize, they fail to do basic transformations that greatly unlock ILP (to be fair these aren't allowed by FP rules since it changes operation ordering) 2. Gather/scatter still have awful performance, worse than manually doing the loads and piecemeal assembling the vector. The fancy 'results-partially-delivered-in-order-on-trap' semantics probably don't help. Speeding up these operations would instantly unlock SO much more performance (see my username) 3. The whole 'extract-upper-bit-to-bitmask' -> 'find-first-set-in-bitmask' really deserves an instruction to merge the last two (return index of first byte with uppermost bit set, etc). It's such a core series of instructions and ime often the bottleneck (each of those is 3 cycles) 4. The performance of some of the nicer avx512 intrinsics leaves a lot to be desired. Compress for example is neat in theory but is too slow for anything I've tried in real life. Masked loads are another example, cool in theory but in practice MUCH faster to replicate manually. 5. It still feels like there's a ton of really random/arbitrary holes in the instruction set. AVX512 closed some of these but introduced many more with the limited operations you can do on masks and expensive mask <-> integer conversions.

gonzo 4y ago

AVX 2 and AVX 512 are very good for VPN transforms. 10gbps on a single core.

singularity2001 4y ago

Abstract vectorization instructions in wasm will make life a lot easier https://github.com/WebAssembly/flexible-vectors/blob/main/proposals/flexible-vectors/FlexibleVectors.md https://github.com/WebAssembly/flexible-vectors/blob/main/pr... great proposal! Mapping to whatever hardware is available as some sort of micro library