8 ms·
Zen5's AVX512 Teardown and More
- PaulHoule 2y agoIntel's handling of SIMD is representative of Intel's value-subtracting principles that have caused Intel to stagnate in the past 15 years or so. It is an intrinsic problem with SIMD as we know it that you have to recode your application to support new instructions, which is a big hassle. Most people and companies will give up on supporting the latest and greatest and will ship binaries that meet the lowest common denominator. For instance it took forever for Microsoft to rely on instructions that were available almost 15 years ago. As Charlie Demerjian has pointed out for years consumers are waking up to the fact that a 2024 craptop isn't much better than a 2014 crapbook and there is zero credibility in claims about "Ultrabooks", "AI PCs", etc. What could make a difference is a coordinated effort end-to-end to widely deploy the latest developments as quickly as possible across as much of the product line as possible, to get tooling support for them, and drive developers to adopt them as quickly as possible. As it is Intel will boast about how the national labs are blowing up H-bombs in VR faster than they ever had, and Facebook is profiling users more efficiently than before and not realize that customers don't believe these advances are going to make a difference for them so instead of buying a new PC which might deliver better performance when software (maybe) catches up in 7-8 years they are going to hold on to old machines longer.
- happycube 2y agoI'd say 2018 ~= 2024 instead of 2014 (those craptops all too often had dual-core i5's and i7's, 15" TN 768 screens, and HDDs), but yeah things have slowed down a bit.
- jsheard 2y agoWorse still is Intels rollout of AVX512 specifically, which started nearly a decade ago but to this day it's still not available across their whole product stack, so the countdown to it becoming ubiquitous hasn't even started yet. They painted themselves into a corner by making 512bit vectors a mandatory feature, which they then decided isn't feasible to support in their small E-cores, so now they're walking it all back with a new "AVX10" spec which is just a redux of AVX512 except 512bit vectors are optional this time. Then we'll have to wait a another decade or so for AVX10 to become baseline, so AVX2 will probably be old enough to drink (in the US) before it's fully phased out.
- Vecr 2y agoI don't think you can phase out AVX2, it's the base of AVX512, because you can't always go 512 wide, and you'd have no backwards compatibility.
- jsheard 2y agoI know AVX2 will continue to exist in hardware forever for backwards compatibility, by "fully phased out" I mean the eventual point when software no longer has to maintain a dedicated path for hardware which supports AVX2 but doesn't support AVX10, because all relevent hardware supports AVX10.
- the8472 2y agoEVEX prefix can address XMM/YMM/ZMM registers. So you can apply the AVX512 instruction set to 128bit and 256bit registers too.
- xeeeeeeeeeeenu 2y agoWhile it does seem that AVX10 was mainly designed for consumer CPUs so they could use modern vector instructions without 512-bit vectors, the upcoming Arrow Lake will not have it.[1] I guess we will have to wait for at least one more generation. [1] - According to Intel® Architecture Instruction Set Extensions Programming Reference: https://cdrdv2-public.intel.com/826290/architecture-instruction-set-extensions-programming-reference.pdf https://cdrdv2-public.intel.com/826290/architecture-instruct...
- jsheard 2y agoIntel is making Xeons out of E-cores (up to 288 of them on one chip) so I assume those will also be motivating the rollout of AVX10, not just their consumer parts.
- xeeeeeeeeeeenu 2y agoWell, they don't support it either. According to the document I linked, neither the just-released Sierra Forest, nor the planned Clearwater Forest support AVX10.
- jeffbee 2y agoI don't think you can make a credible case that a 2024 PC is equal to a 2014 one. In 2014 you could get 4 Haswell cores in a 65W TDP, for 410 2014 US dollars. For the same power and less money in 2024 you get a web browser platform that is around 6x faster, or a code compilation platform that is 5-20x faster.
- antisthenes 2y ago> In 2014 you could get 4 Haswell cores in a 65W TDP, for 410 2014 US dollars. Is there a typo here? I bought a 4-core Haswell in 2014 for around $200. It definitely didn't cost anywhere near $410, except maybe for the 6-core HEDT cpu?
- jeffbee 2y agoI was just looking at the launch MSRP of one of the higher-end hyperthreaded ones, but sure there was a range including $182 for the i5-4460S without hyperthreading, $224 for the i5-4690S, etc.
- jauntywundrkind 2y agoThe n100 that's out is on par-ish both in performance and only a little bit more power saving than a i5-6500t. A 2013 core. https://www.intel.com/content/www/us/en/products/sku/88183/intel-core-i56500t-processor-6m-cache-up-to-3-10-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/88183/i... The price point is far better at least (iff we look at new inventory only).
- michaelt 2y agoA lot of the performance gains of the past 10 years aren't obvious from the headline specs. The desktop PC I built in 2015 for £1082.98 i7-4790K (4 cores, "4.00 GHz base 4.40 GHz Turbo") 32GB DDR3 RAM Samsung 250GB SSD nvidia GTX 960 2GB GPU The desktop PC Dell will sell me, today, for £1,174.80 [1] i7-14700 (8 performance cores, 12 efficient cores, "2.1 GHz Base, 5.4 GHz Turbo") 32GB DDR5 RAM 512 GB SSD Intel Integrated Graphics Sure, it's better. But on paper those spec changes don't look game-changing, considering it's been an entire decade. Especially if you're mistrustful of efficiency cores. But what those headline specs don't mention is that the RAM is 4x faster, the SSD is now nvme and faster, the pcie lanes are 4x faster, and the CPU cache has quadrupled. [1] https://www.dell.com/en-uk/shop/desktop-computers/new-optiplex-micro-form-factor/spd/optiplex-7020-plus-micro/s010o7020mffp_vp https://www.dell.com/en-uk/shop/desktop-computers/new-optipl...
- pjmlp 2y agoThe only reason I no longer use my craptops from 2010, is that they died. For the kind of stuff I do at home outside work, even programming, there were perfectly fine, and had they not died, I would still be using them.
- kolbe 2y agoMystical (the author) does such fantastic work for the CS community. I really like that guy. A compilation of his stack overflow answers on SIMD would be better than any available book. His intelligence and openness, despite no one paying him for it, shines such a bad light on the terrible state of academia. That he was considered a "bad student" is near-proof in and of itself that our system judges people catastrophically poorly.
- RMarcus 2y agoLooks like he got a master's degree from UIUC and did some research on FFT implementations. Seems to have been successful. What makes you say he was 'considered a "bad student"'? (This is a genuine question. I've never met Alex in person, but if an applicant to my lab spent their free time diving into SIMD implementations and breaking records for computing mathematical constants, I'd rush to hire them. Not that either of those two things is a requirement, of course.)
- kolbe 2y agoIt's in his bio. "However, ever since grade school, I've always sucked in terms of grades and standardized tests. I graduated from Palo Alto High School in the bottom quartile among all the college-bound students. My GPA was barely a 3.0 at graduation, so it was somewhat miraculous that I got accepted into Northwestern University at all."[1] I know he has a masters and all, but he is spectacular. Hundreds of thousands of people have masters degrees. He is more impressive than 99% of professors, and academia doesn't even acknowledge him as a peer of them. [1] http://www.numberworld.org/about/ayee/ http://www.numberworld.org/about/ayee/
- nocoiner 2y agoThe fact that he got into Northwestern is proof in itself that the “system” didn’t consider him to be a bad student. It actually seems like the system did a pretty good job of identifying sheer intellectual horsepower and potential despite the self-professed low GPA and standardized test scores.
- ComputerGuru 2y agoGreat article. It really drives home what a damn shame Intel's persistent mishandling of AVX512 has been ever since its introduction. I don't even know if it has a future outside of extremely niche libraries given how scattered hardware support for it is on Intel's side. On a completely different topic: I wasn't expecting the redacted portion of the article due to AMD's embargo to take away too much of the article but the first half (up until the discussion about AVX512) would clearly be much more interesting with the censored out parts. I guess someone will have to resubmit this come August 14th!
- gpapilion 2y agoMishandling aside, the issue I've seen is there really isn't consumer demand for this. Prior to AMD having AVX512, most of the comments were around wasting the silicon on SIMD, rather than improving other aspects of the CPU. I'm pretty sure there was good reason to think it was largely a dark area of the chip. From what I've seen, but haven't heard discussed much, the naive implementation vs AVX512 is a huge gain, but AVX2 vs AVX512 was not very impressive for the application I was looking at. The complexity this code added, and the cases where we needed it to run on AMD (for other reasons), basically made taking advantage of the feature undesirable for a single digit gain. Things like VNNI or AMX are better wins, but they are only needed in very specific cases. VNNI in particular looked to be a 30% improvement in a BERT workload.
- Avamander 2y agoIsn't it a bit weird to expect consumer demand for CPU instruction set extensions? Obviously there's very little of that, but what should matter is the developer uptake and thus better end-user experience that can be delivered? (I'd also hope for even better autovectorization in compilers.) It's in my opinion kind-of insane that we're still building so much software for ancient baselines and leaving quite a bit of performance on the table across the entire system. (How much has Apple won in terms of performance by forcing everyone to build for new ARM targets using new toolchains?)
- gpapilion 2y ago
- drewg123 2y agoThe most interesting bit about this article for me is the "transition time" to get the power needed use AVX-256 or AVX-512 which is present on Intel, but not AMD zen4/zen5. It explains some behavior that I saw years ago when implementing kTLS on FreeBSD, and validates our design of having per-core kTLS crypto worker threads, rather than doing the crypto in the context of sosend() or sendfile's tcp_usr_ready().
- alberth 2y agoAny chance you’d move to using Intel for your content servers in the foreseeable future? Or this further cements the use of AMD?
- altairprime 2y agoUgh, half of this article is > This section has been redacted until August 14. Could you repost it then?
- kolbe 2y agoI think you need to wait one year before being able to post an exact link again. You can ask deng to reset it if you want, but maybe take some initiative for yourself rather than giving other people attitude, then asking them to do you favors?
- ipsum2 2y agoThe part of the article I found most amusing: "Intel added AVX512-VP2INTERSECT to Tiger Lake. But it was really slow. (microcoded ~25 cycles/46 uops) It was so slow that someone found a better way to implement its functionality without using the instruction itself. Intel deprecates the instruction and removes it from all processors after Tiger Lake. (ignoring the fact that early Alder Lake unofficially also had it) AMD adds it to Zen5. So just as Intel kills off VP2INTERSECT, AMD shows up with it. Needless to say, Zen5 had probably already taped out by the time Intel deprecated the instruction. So VP2INTERSECT made it into Zen5's design and wasn't going to be removed. But how good is AMD's implementation? Let's look at AIDA64's dumps for Granite Ridge: AVX512_VP2INTERSECT :VP2INTERSECTQ k1+1, zmm, zmm L: [diff. reg. set] T: 0.23ns= 1.00c Yes, that's right. 1 cycle throughput. ONE cycle. I can't... I just can't... Intel was so bad at this that they dropped the instruction. And now AMD finally appears and shows them how it's done - 2 years too late."
- fuhsnn 2y agoIt's in fact very common to microcode instructions at early iterations of an arch. https://uops.info/table.html https://uops.info/table.html is a nice place if certain instructions being slow brings joy to your life.
- Remnant44 2y agoAMD's avx512 implementation is just lovely and they seem to be firing on all cylinders for it. Zen4 was already great, 'double pumped' or no. It looks like Zen5's support is essentially the dream - all EUs and load/store are expanded to 512 bit, so you can sustain 2 512 FMAs and 2 512 Adds every cycle. There also appears to be essentially no transition penalty to the full-power state which is incredible. The only thing sad here is that all this work to enable full-width AVX512 is going to be mostly wasted as approximately 0% of all client software will get recompiled to an AVX512 baseline for decades if ever. But if you can compile for your own targets, or JIT for it.. it looks really good.
- kardos 2y agoMaybe the CPython JIT can make use of it. That might move the needle past 1%...
- aseipp 2y ago> The only thing sad here is that all this work to enable full-width AVX512 is going to be mostly wasted as approximately 0% of all client software will get recompiled to an AVX512 baseline for decades if ever. Well, the other thing is that for workloads where you're cramming 512b vectors through 2xFMAs every cycle -- there's a good chance you can (and have been) just buying GPUs to handle that problem. So, I think that space has been eaten up a bit in recent times. I don't think it will be decades of waiting though. AVX2 is a practical baseline today IMO and Haswell is what, barely 10 years old? Intel dragged their feet like crazy of course, but "decades" from now is a bit much. And AVX-512's best feature -- it's much more coherent and regular design -- means a lot of vectorization opportunities might be easier to do, even automatically (e.g. universal masking and gather/scatters make loop optimizations more straightforward.) We'll have to see how it shakes out.
- adrian_b 2y agoThe GPUs that can do FP64 operations are priced out of the range acceptable for small businesses or individuals. The consumer GPUs are suitable only for games, graphics and ML/AI. There are also other applications, like in engineering, where the only cost-effective way is to use CPUs with good AVX-512 support, like Zen 5. A 9950X has a similar FP64 throughput like the last GPUs that still had acceptable prices, from 5 years ago (Radeon VII). Even for FP32, the throughput of a 9950X is similar to that of a very good integrated GPU (512 FP32 FMA per cycle, but at a double clock frequency, so equivalent with a GPU doing 1024 FP32 FMA per cycle), even if it is no match for a discrete GPU. There are also applications where the latency of transferring the data to the GPU, then doing only short computations, can reduce the performance below what can be achieved on the CPU. Obviously, there are things better done on a GPU, but there are enough cases where a high throughput CPU like a desktop Zen 5 is better.
- stagger87 2y agoSeeing 2 consumer CPU generations in a row not only support but improve AVX512 capabilities will hopefully go a long ways towards regaining the confidence of the developers that use AVX512 in the consumer space. I know I personally have been holding back as I watched Intel fumble AVX512 for the last 10 years. With their even more recent fumbles there could be a near future where AMD CPUs have majority market share in both desktop and mobile. Great news for developers that can use AVX512.
- Remnant44 2y agoAgreed. My current best-case-scenario hope is that the success of the Zen4/5/etc processors will force Intel to adapt their strategy towards AMDs, and finally move us out of the avx512 mess they've segmented us into.
- vardump 2y agoAssuming Intel is changing direction right now, unfortunately they will face 2-3 years of latency to implement that.
- Manabu-eo 2y agoAMD fixed vpcompressd in Zen5: > Hazards Fixed: > V(P)COMPRESS store to memory is fixed. (3 cycles/store to non-overlapping addresses) > The super-alignment hazard is fixed. I initially tried searching for the string, but the () thwarted that. It used to be 142 cycles/instruction for "vpcompressd [mem]{k}, zmm" in zen4.
- pixelpoet 2y agoWhat an excellent writeup, thx for sharing
- andy_xor_andrew 2y agoout of curiosity, what applications might I see this used for?
- Cold_Miserable 2y agoDoes Zen 5 have cldemote or senduipi ?
- adrian_b 2y agoProbably not. The gcc Zen 5 patch adds as new instructions only AVX-VNNI, MOVDIRI, MOVDIR64B, AVX-512-VP2INTERSECT and PREFETCHI. Zen 4 already had AVX-512 VNNI (for ML/AI), AVX VNNI is only an alternate encoding for the programs that use only AVX (because they have been compiled for Intel Alder Lake/Raptor Lake). MOVDIRI and MOVDIR64B can be very useful in some device drivers, less often in general-purpose programs.
- jiggawatts 2y agoSomething I've wondered about is why CPUs have separate scalar and vector ALUs. Would it be possible to simply have 16x scalar ALUs that can be used either individually or ganged together in groups of 4, 8, or 16 to execute the various vector instructions?
- dzaima 2y agoIntel has shared ports across scalar and SIMD for a long time (only recently they've split them apart). I don't know whether they share silicon though, quite likely that they don't. The complexity of using parts of vector ALUs might not be worth the saved silicon (even for something as expensive as float division, which has both SIMD and scalar versions (both operate in the same register file, just differing in the number of elements processed), both Intel and AMD have the same throughput for the scalar version and 128-bit SIMD version[0]). It additionally allows separating bypass networks, e.g. the output of a SIMD add will never be directly used as a memory address (or, in the case of a gather, an extra cycle or two of latency is acceptable considering the amount of loads that will follow and in general the throughput-centric nature of SIMD), whereas for a scalar op that's an extremely important path. Or, in general, it allows making different tradeoffs between scalar & SIMD. [0]: https://uops.info/table.html?search=div%20xmm)&cb_lat=on&cb_tp=on&cb_HSW=on&cb_SKL=on&cb_ADLP=on&cb_ZEN2=on&cb_ZEN4=on&cb_measurements=on&cb_sse=on https://uops.info/table.html?search=div%20xmm)&cb_lat=on&cb_...
- magicalhippo 2y agoGood question. I imagine that for instruction encoding density one could for example encode instructions so that "add4 ra0, rb4" would do a 4-wide addition ra0+rb4 up to ra3+rb7. Possibly with restrictions like only "aligned" register access, ie an 8-way op can only use regs 0-7 or 8-15. Swizzle instructions can then handle reordering ala SSE. Don't forget about 2-way ops as well.
- jiggawatts 2y agoI was just thinking of the back end ALUs but you make a good point: it might also make sense to unify scalar and vector instructions. If the right encoding is used it might allow scalar code to retain its efficiencies.
- babra4151 2y ago[dead]
- adrian_b 2y agoI want to add a fact about which the author was not aware. While as he says, after Tiger Lake Intel has deprecated VP2INTERSECT, then they have changed their mind and they have added it again in the server CPU Granite Rapids, which will be launched in a few months. Moreover, the ISA of Granite Rapids is considered to be AVX10.1 and all its instructions, including VP2INTERSECT, will be a mandatory part of the ISA of all future Intel CPUs from 2026 on. Therefore it is good that AMD has achieved an excellent implementation of VP2INTERSECT, which they will be able to carry into their future designs. It remains to be seen whether the Intel Granite Rapids implementation of VP2INTERSECT is also good.