25 ms·
Why Skylake CPUs Are Sometimes 50% Slower
- gchokov 8y agoIt's high time for somebody to disrupt Intel and their model. I really hope this can happen soon.
- growlist 8y agoAMD are doing a pretty stunning job recently.
- DCKing 8y agoAMD is not trying to "disrupt Intel's model", it's trying to outcompete them on mostly the same model. I have no idea what "disrupting Intel's model" would actually mean. But I would be fine if Intel is just outcompeted at their own game, and this already seems to be happening.
- lallysingh 8y agoARM is doing well on most fronts.
- ethbro 8y agoDisrupting the model would prolly look more like Transmeta or Cavium.
- DCKing 8y agoAlthough I think Intel derives substantial value from x86 lockin, I don't think "doing things like Intel but with a different CPU architecture" really qualifies as a disruption of Intel's model. I'd say disruption of Intel's model would mean different ways of selling CPUs, fast storage or other silicon. All I see is Intel's competitors trying to mimic Intel to some extent because the enterprise market is the most lucrative. AMD, Qualcomm, Cavium and IBM all consider Intel a main and direct competitor. So I'd be interested if anyone has an idea on disrupting Intel's way of doing business.
- dragontamer 8y agoNote that Power9 and ARM both support for relaxed atomics, which would be a substantial difference from x86's stronger cache coherence policies. That would likely make "properly coded" Power9 or ARM synchronization to be more scalable. Possibly a major advantage as 32-cores and higher are beginning to hit mainstream (or at least, high-end mainstream)
- ethbro 8y agoTo that point, cloud is probably the biggest disruption. It may run on Intel now, but once "runs on AWS/GCP/Azure" becomes more important than running on an ISA, there's no reason compute couldn't transition much faster.
- paulie_a 8y agoAmd has had better chips at a lower price point since always. You can get a 8 year year old octocore that still destroys anything Intel has put out recently. Intel relies on inertia. Amd consistently beats them on actual performance at a lower price. I remember my 386 dx 2 had better performance than an Intel 486. And let's not forget the p4 issues with their garbage ram. Plus they backdoor their chips with stupid insecure crap
- uhhhhhhh 8y agoPerformance per watt/cost vs performance AMD has not been consistently beating them. There is a reason multiple companies continue to get intel after reviewing AMD's current offerings every few years (I'm in one of them to remain nameless) I absolutely don't believe an 8 year old AMD octacore will "destroy" anything intel has. Sure I imagine a single highly specific and biased test could give AMD the advantage in that one instance, but overall performance AMD has been behind since the pentium M/core2 with only a few products that initially compete or win, only to fall behind again.
- ethbro 8y agoK10 v Nehalem? Bulldozer v Sandy Bridge? We may be remembering a different history, but Intel has had single threaded performance locked up for awhile. AMD prices aggressively and glues more cores together, but hyperbole isn't doing your point any favors.
- paulie_a 8y agoI personally don't give a shit about single thread performance. That is a metric that is meaningless. And Sandy bridge was pretty awful.
- mywittyname 8y agoI'm not sure where you get this impression, but Intel held the performance crown over AMD since releasing the Core architecture. In fact, things were looking pretty dire for AMD during the Phenom days when their highest-tier offering could only match mid-tier Intel chips in performance, so they had to compete solely on price. [1] AMD's stock price during this era was the lowest it's been since like 1980. It's only with Ryzen that AMD reached performance parity with Intel once again. [1] https://www.tomshardware.com/reviews/amd-phenom-ii-x6-1090t-890fx,2613-7.html https://www.tomshardware.com/reviews/amd-phenom-ii-x6-1090t-...
- coldtea 8y agoWell, if there's no one (or very few) out there doing any CPUs that are as good, then why consider Intel as doing a bad job (as you seem to imply)? Bad compared to whom?
- rahul24 8y agoHi
- beagle3 8y agoAround the year 2000, Intel decided that the future is Itanium. It didn't matter that the market hated the idea - there was no alternative to Intel, so Intel completely ignored the market's plea for an x86 compatible 64-bit processor that would be able to run old x86 code at native speed. Intel's 32 bit offering at the time wasn't any good either; the pentium had such a deep pipeline (and not so smart prefetch/speculative) that the chips ran at a high frequency, ate a lot of power, but pipeline stages were idling or working on useless stuff a large amount of the time. Luckily for the world, AMD managed to produce that alternative, the AMD64 architecture, which kicked intel's ass so hard that eventually they had to adopt it. If it weren't for AMD's AMD64, and their cross-licensing deal with Intel (that allowed them to make an x86 compatible chip without specifically asking for Intel's permission), we'd be in an Itanium dominated world right now (or .. ARM would have stepped up, it's hard to predict an alternate reality) but surely with less efficient and less effective processors. History does not repeat, but it rhymes. Intel has no Itanic this time, but it did become numb enough to let AMD outmaneuver them again. It's not that Intel is doing a bad job, as much as they have too little incentive to do a good job.
- acdha 8y ago> If it weren't for AMD's AMD64 … we'd be in an Itanium dominated world right now This is far from certain: Itanium shipped years later and much, much below the advertised performance thresholds. In the meantime even Intel’s own x86 had closed the raw performance gap and since IA-64 was such a huge bet on brilliant compilers that the early benchmarks for most business code favored the P-III even before you factored in cost. Similarly, the original claim that it’d offer competitive x86 performance turned out to be half a decade behind years behind when it shipped so it wasn’t even remotely competitive for legacy code and businesses back then had even more weight of code which was difficult to recompile. It’s equally plausible to me that Intel would have continued to see strong x86 sales and weak demand for Itanium and ended up shipping their own 64-bit extension a few years later with less pressure from AMD simply because businesses weren’t keen on the “recompile everything, make a bunch of expensive optimizations, and it might be faster than the system it’s replacing”
- DC-3 8y agoHopefully at some point in the vaguely near future we'll see RISC-V come to the fore as a credible desktop architcture.
- blitmap 8y ago:-) https://www.crowdsupply.com/sifive/hifive-unleashed https://www.crowdsupply.com/sifive/hifive-unleashed https://www.crowdsupply.com/microsemi/hifive-unleashed-expansion-board https://www.crowdsupply.com/microsemi/hifive-unleashed-expan... https://www.crowdsupply.com/sifive/hifive1 https://www.crowdsupply.com/sifive/hifive1
- adrianN 8y agoIt looks like Intel's manufacturing advantage will slowly disappear over the next couple of years. So it seems likely that alternative CPUs will become more competitive, ARM for example.
- walshemj 8y agoAMD Seems do be doing well.
- m_eiman 8y agoWhat kind of code is reasonably written as 40+ cores spinlocking the same lock? Seems like a strange design, IMHO…
- xpuente 8y agoThis is a PITA for coherence controllers.
- mickronome 8y agoThere shouldn't really be any outside of the kernel, and preferably none inside.
- dragontamer 8y agoOn the contrary: Spinlocks are faster than any other lock. In fact, they're faster than many atomics-based lock-free datastructures (!!!) in many cases. https://www.youtube.com/watch?v=9hJkWwHDDxs https://www.youtube.com/watch?v=9hJkWwHDDxs Atomics-based lock-free data-structures scale beyond 40+ core counts. But note that x86's "lock" instruction is fundamental to the system. x86 doesn't have relaxed atomics or barriers under most circumstances. As such, the spinlock best represents what the x86 instruction set offers.
- _wmd 8y agoYou could easily find examples of this - for example in any textbook threaded socket server design (thread pool contending on accept())
- lallysingh 8y agoYeah, those textbooks aren't talking about good implementations that scale up to that many cores.
- spacenick88 8y agoBut wouldn't that use a non spinning OS lock. I.e. the scheduler knows which threads are waiting on the accept and can randomly pick one to directly wake up?
- rahul24 8y agoHi
- mickronome 8y agoUnsure what I'm more astonished about, that the correct case is that it would spin 29ms, or that a CPU instruction has changed it expected latency and prder of magnitude upwards. I would expect any spinning lock to spin at most some slight fraction of the scheduling granularity, but almost always far less than the context switch time except maybe in some specific case of latency sensitive code. Maybe thr long spin tine is for some reason trying to account for the very long scheduling quantas of windows server? Superficially it seems misdirected in anything but real-time code since most schedulers would allow the thread to keep its quanta until the next scheduling if the thread wait time was low. The only case I can see this as not being the case is in a system that is overloaded and all CPU's have multiple threads that are all starved.
- dragontamer 8y ago"Pause" spins give all their resources to the hyperthread-sibling. So unless the hyperthread-sibling is also stalled out for some reason, the "pause" instruction is definitely preferred. So its not like resources are lost (at least, under high CPU utilization). But still, you're right in that a 29ms spin-lock sounds outrageous to me. That's definitely long.
- cma 8y agoPause can be t he best choice even without hyperthreading because it can lower power utilization. And even on desktop that can mean less heat and more time for other stuff to run at a higher-clocked 'Turbo-boost'.
- marcoperaza 8y agoGreat investigation. Does anyone know or can anyone intelligently speculate what would lead to this tradeoff in the CPU design?
- walshemj 8y agoIts more why their code was written that way.
- titzer 8y agoThe pause instruction is supposed to be used as a "yield hint" for programs to signal that they are waiting for a concurrent action in another CPU to unblock, e.g. the contended case in a spinlock implementation, where the current thread needs another thread to release the lock before it can make progress. In Skylake, they have changed the microarchitecture so that pause is a much stronger hint; executing a pause will "yield" the current hyperthread, allowing the other hyperthread on the same core to use more of the CPU's resources (ROB slots). This improves performance for well-tuned synchronization, but hurts patterns where pause is executed too often. Unrolling the spinlock loop would help in this case, probably.
- marcoperaza 8y agoThanks! Very informative.
- dragontamer 8y agoWell, first you have to understand exactly what "Pause" is. "Pause" is a hint to the CPU that the current thread is spinning in a spinlock. You've ALREADY have tested the lock, but it was being held by some other processor. So why do you care about latency? In fact, you probably want to free up more processor resources as much as possible. Indeed, there's not actually any resources wasted when you do a "pause" instruction. In a highly-threaded environment, the hyperthread-brother of the thread picks up the slack (you give all your resources to that thread, so it executes quicker). 10-cycles is probably a poor choice for modern processors. 10-cycles is 3.3 nanoseconds, which is way faster than even the L3 cache. So by the time a single pause instruction is done on older architectures, the L3 cache hasn't updated yet and everything is still locked!! 140-cycles is a bit on the long side, but we're also looking at a server-chip which might be dual socket. So if the processor is waiting for main-memory to update, then 140-cycles is reasonable (but really, it should be ~40 cycles so that it can coordinate over L3 cache if possible). So I can see why Intel would increase the pause time above 10-cycles. But I'm unsure why Intel increased the timing beyond that. I'm guessing it has something to do with pipelining?
- DoofusOfDeath 8y agoI really think the article summary is misleading. The object code in question uses an instruction that's basically a microarchitecture-specific hack. IMO the real issue here is that the object code is optimized for an earlier microarchitecture and needs an update. Disclaimer: I work for Intel, but not on their CPUs and I'm not a fanbois.
- int0x80 8y agoWhat amazes me is that they do not test this extremely specific uarch code when a new uarch is released. On the other hand, I think the hack is not in using the pause instruction itself, but that crazy backoff algo. For example Linux uses pause too (via rep; nop) in lots of places. But not in a exponential loop (that I'm aware of).
- jstarks 8y agoThe title is misleading — no code is broken, though it is true that .NET programs that exhibit lock contention will be slower. The root of the problem is that .NET implemented a rather baroque spin loop with several assumptions about the CPU and OS architecture. Unlike any other spin loop I have seen or written, they execute multiple pause instructions per loop. Arguably this was a poor choice. It would be interesting to see the history of this design. It was probably done years ago when .NET was closed source.
- _wmd 8y agoThe stuff he quoted seems to make a strong case that the behaviour they implement was the intended behaviour prior to Skylake, why else specifically call it out in the newer manuals? Would be interested to know why you think the code was always broken. Perhaps Intel define a preferred algorithm they failed to implement?
- ptero 8y agoNot the parent, but I do not see that Intel is wrong here. New architecture makes new choices and, according to the redlined doc, they expect 1-2% performance improvements on threaded apps, which is a measurable improvement. They also explicitly call for possible degradation cases so folks who care can adjust. .NET apparently hits such a degradation case, but parent says this is due to .NET implementing spin loops in a way (almost) no one else does. If so, in this particular case Intel is doing the greatest good for the greatest number, which is reasonable. My 2c.
- arcticbull 8y agoAnd of course it's really easy for .Net to fix it for everyone in one simple patch, which it looks like they already have. I'm with you on this one.
- eloff 8y agoMy experience is to the contrary. Every spin lock I have seen uses more than one pause instruction. Some may start with one and then execute more. They all involve a loop over pause instruction(s). I don't think that's the problem per se. The problem is more likely the back-off logic.
- tzahola 8y agoTime for a rewrite in Rust? /s
- amelius 8y agoI was expecting CPUs to become slower because of the Meltdown/Spectre problems, not this.
- lucb1e 8y agoTL;DR: There exists a "pause" CPU instruction, which signals to the CPU that the code is in a spinlock wait loop. In Skylake this instruction sleeps longer than before. If your code heavily uses this, then it'll wait longer for things like locks to be released. More info on the purpose of the pause instruction: https://stackoverflow.com/questions/4725676/how-does-x86-pause-instruction-work-in-spinlock-and-can-it-be-used-in-other-sc https://stackoverflow.com/questions/4725676/how-does-x86-pau... Info on what's a "spinlock" (I thought it was another word for busy wait, but there's a subtle difference): https://stackoverflow.com/questions/38124337/spinlock-vs-busy-wait https://stackoverflow.com/questions/38124337/spinlock-vs-bus...
- what-the-grump 8y agoGood summary at bottom of the first link. TLDR; don’t use it unless you really understand why you are pausing the cpu.
- geezerjay 8y agoGreat tldr. This should be the top post in this discussion.
- davidhyde 8y agoThere is also benefit to be had in HOW the author found the root cause of the problem. That was more interesting to me than the problem itself.
- geezerjay 8y agoI agree that's also interesting, but in a discussion with such a clickbaity title it's important to correctly identify the source of the so called problem right upfront just to avoid wild speculations and conspiracies.
- Dylan16807 8y ago> but in a discussion with such a clickbaity title it's important to correctly identify the source of the so called problem right upfront That's fair, but consider it closer to a subtitle than a tl;dr, which is supposed to be a summary that lets you skip the main body entirely. > just to avoid wild speculations and conspiracies The burden there is on people to not comment if all they know about the article is a wild guess.
- al2o3cr 8y agoThe Intel docs mention that PAUSE isn't the right choice for waits of "thousands of cycles or more" - even on the old architecture, 10ms is waaaaaay longer than recommended. I also have to quibble with this conclusion: This is not a .NET issue. It affects all Spinlock implementations which use the pause instruction. The instruction isn't the issue, the exponential backoff that .NET includes is. At the extreme (last iteration, on a 64-core CPU) it will spin for 1.2 million pause instructions...
- phkahler 8y agoRight. The whole time I was thinking the hardware change probably exacerbated an existing software issue. Then I saw this: "The issue has been fixed with .NET Core 2.1 and .NET Framework 4.8 Preview contains also the fixes for it." And by the end of the article I was thinking their code might just run faster as a single thread on a ~4GHz processor. Another thought was why do they have so much resource contention in the first place? That seems fishy. Another thought: Perhaps they should write it in Rust with all of its "fearless concurrency" claims. While I'm skeptical of those claims, The exercise would probably highlight some of the real issues. Any way I look at it, I keep thinking it's a software design problem.
- smaddox 8y agoRust doesn't magically fix resources contention. It just provides safe concurrency primitives (and the surrounding language semantics) that make it difficult -to-impossible to misuse in such a way as to cause a deadlock or race condition.
- phkahler 8y agoRight, so if the language forces you to eliminate deadlocks and race conditions you may find less need for spinlocks which are the problem here. Speculation on my part. I'm not actually advocating Rust but I am thinking about a project to give it a try.
- 8y ago
- userbinator 8y agoFrom the screenshot of the Intel documented linked: The increased latency ... has a small positive performance impact of 1-2% on highly threaded applications. Making an instruction more than 10x slower overall, for a 1-2% gain (who wants to bet it's some "industry standard" benchmark...?) in a very specific situation sounds like a textbook example of premature microoptimisation. Of course they don't want to give the impression that they severely slowed things down, but the wording of the last sentence is quite amusing: "some performance loss"? More like "a lot". It reminds me of what they said about the Spectre/Meltdown workarounds.
- zaarn 8y agoOn the other hand, if 99% of highly threaded applications run 1% faster and 1% of applications run 10 times slower, maybe it's worth it? Maybe?? Tbh I think this is more of a sign that Intel's architecture, the Core i generation of CPUs, has been optimized as much as you reasonably can and Intel is squeezing out a few more cpu cycles here and there.
- ethbro 8y agoOne of the dark arts of chip design is getting a pulse on existing code in the wild, and evaluating potential microarch changes against it. Someone else can pipe in, but the impression I got from friends in that industry was that Intel had a fairly advanced org set up to evaluate that. Or to look at it another way, they didn't want to make an $x billion bet without due diligence.
- smaddox 8y agoThat's what happens when you let simple metrics dictate engineering decisions (and career advancement).
- dpark 8y agoHighly threaded applications are not “a very specific situation”. This is a huge class of important applications. This describes most popular services. It’s not “premature optimization” to gain 1-2% for likely trillions of application executions.
- metalliqaz 8y agoInteresting article but, man, the prose was difficult to follow.
- TimJYoung 8y agoWhat we do with the spin lock classes that we use in our products is a SwitchToThread call: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686352(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms6... If the SwitchToThread call returns False, then the code reverts to a Sleep(0) call instead. Credit to StackOverflow and Joe Duffy's excellent article on this: https://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1 https://stackoverflow.com/questions/1383943/switchtothread-v... http://joeduffyblog.com/2006/08/22/priorityinduced-starvation-why-sleep1-is-better-than-sleep0-and-the-windows-balance-set-manager/ http://joeduffyblog.com/2006/08/22/priorityinduced-starvatio... However, as mentioned in one of the SO replies, you still need to make sure that you don't use locks that use loops on calls like this when those locks will have to wait on anything long-running - you're just going to burn CPU/battery needlessly. So, only use such locks when you know that the protected code doesn't involve any long wait states, or modify the code to back out even further into a proper kernel wait after looping X times. Personally, I agree with second SO answer and don't like such constructs, and would personally say just go with a straight-up kernel wait if there's any question about whether such conditions could exist now or in the future.
- EmilyHealth 8y agohttps://vk.com/away.php?to=http%3A%2F%2Fthepowerofhealthyfood.com%2Fgirls-know-purpose-pocket-underwear-bobble-hat%2F&post=479855050_24&el=snippet https://vk.com/away.php?to=http%3A%2F%2Fthepowerofhealthyfoo...
- tsh56 8y agoThere is another architectural change made in Skylake which is related to this article. In skylake the L3 cache is now non-inclusive, which means that a core trying to acquire the lock will have to read the cache line through a cross core cache read. Making things worse when contention is high the cache-line will frequently be dirty. The latency for a cross core dirty hit of the L1/L2 cache is documented to be 60 cycles which is much high that the L3 cache latency of past architectures. This will increase overall contention which will magnify the frequency the pause instruction is hit. I'm wondering if the pause instruction cycle count was increased to reduce the impact of cross core cache reads.
- rocky1138 8y agoExcellent research into the problem. Very helpful that this person also included a test program to see if one is affected by the problem.
- codedokode 8y agoThis is a kind of bug that can be difficult to reproduce. Imagine if someone reports this bug to you but your own tests show that nothing has changed (because you use an older CPU).
- retrack 8y ago2 elements are fuzzy: - clock speed is way different from 3.5GHz to 2.6. This can lead to huge differences in some computations and no new instruction set can compete against brute force. - changing between those 2 cpus is not just a socket swap but a full platform change: can we get the other details of the 2 platforms ?