23 ms·
Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower
- schoen 8y agoIn case people don't like the headline style (leaving out the reference to the answer), the answer given at the end of the article is new Spectre mitigations. We new that speculative-execution mitigations would sometimes reduce performance, and evidently they do. :-(
- _cs2017_ 8y ago1.3-1.4x slowdown is a lot more than I expected (I know it's for synthetic benchmarks but still...) Can someone explain (or link to an article) how a tweak to HT branch prediction heuristic can have such a huge impact on performance?
- janoc 8y agoIt is unlikely the branch prediction heuristics that is the problem (that in the Intel's microcode). The problem is in the mitigations necessary to make it impossible/more difficult to exploit these side channel attacks. And that is costly because memory and needs to be moved around constantly. So that adds a ton of extra overhead whenever a context switch is made.
- BeeOnRope 8y agoThe impact is big enough that one would suspect the microcode simply disables indirect branch prediction, so you pay a 16-20 cycle penalty per branch. Indirect branches just aren't frequent enough to explain such a regression via say a simple reduction in prediction resources. I can test it once I get the new firmware.
- jcelerier 8y ago> Indirect branches just aren't frequent enough aren't vtable calls / function pointer calls indirect branches ?
- BeeOnRope 8y agoYes, they are (at least when the compiler cannot devirtualize them) - but they make up a fairly small fraction of the total instructions in a typical program - and probably very small in something like cinebench, which also showed a big regression.
- jcelerier 8y ago> but they make up a fairly small fraction of the total instructions in a typical but if their cost increases by a large factor... besides, in any large compiled program, the core would certainly be based around some kind of programmable pipeline, and these would generally be implemented like this unless they wrote their own JIT compiler.
- BeeOnRope 8y agoYes - but for their cost to increase by such a large factor, the only obvious thing I can think of is that their prediction is disabled. I didn't follow your comment about a "programmable pipeline". I don't think many or any of the Phoronix benchmarks are based on a pipeline with indirect branches at their core.
- jcelerier 8y ago
- kim0 8y agoIs Intel already selling CPUs which don't need these patches?
- anticensor 8y agoNot in x86 architecture.
- eugeniub 8y agoThen in which architecture?
- smitty1110 8y agoThey still sell Itanian, I don’t think Spectre/Meltdown affect that product line.
- hermitdev 8y ago*Itanium, and they would likely suffer from the same attacks (maybe not due to being a completely different architecture, but being from thw same company, not sure if there isn't crosspolination between the two), but dont think anyone has looked at them due to small market pressense.
- lmkg 8y agoI would expect the opposite. Meltdown and Spectre both make use of peeking at implicit state that chips hang on to in order to optimize microcode; the entire goal of Itanium was to offload this optimization from the chip to the compiler. A little Googling found an analysis from someone more knowledgeable than myself that backs this up: https://secure64.com/not-vulnerable-intel-itanium-secure64-sourcet/ https://secure64.com/not-vulnerable-intel-itanium-secure64-s... tl;dr Meltdown takes advantage of out-of-order execution, which Itanium simply doesn't have. Spectre makes use of speculative execution, which Itanium only has in a version too limited to support the attack. I'm not an expert in this field so I can't attest to the credibility of this analysis. The Spectre part in particular sounds a little hand-wavy but it might just be over my head. But it aligns with my intuition, which is that the simpler and more explicit architecture doesn't have as many places where data can accidentally end up.
- neopallium 8y agoIs there a comparison table of CPU vulnerabilities? I would like to know which CPU has the least vulnerabilities or the best performance after patching. From what I have read so far is seems that AMD CPUs have had the fewest vulnerabilities/slowdowns? But I can't be sure since I haven't seen a complete comparison (including these new vulnerabilities).
- roblabla 8y agoI don't know of any comparison table. There are two family of vulnerabilities at play when talking about Spectre and co: - The general Spectre family, which affects most (if not all) CPUs built with speculative execution. - Meltdown and L1TF, which only affects Intel CPUs due to them delaying security checks until after speculation has taken place. AMDs, ARMs, etc. that use speculative execution are going to be vulnerable to at least some variant of Spectre (there are 4 variants known right now). ARM published a table[0] explaining which of their CPUs are vulnerable to which variants. I'm not aware of any such table for Intel or AMD. Microsoft published some interesting tables[1] explaining which mitigation protect against which Spectre variant, and under which thread models they operate. [0]: https://developer.arm.com/support/arm-security-updates/speculative-processor-vulnerability https://developer.arm.com/support/arm-security-updates/specu... [1]: https://blogs.technet.microsoft.com/srd/2018/05/21/analysis-and-mitigation-of-speculative-store-bypass-cve-2018-3639/ https://blogs.technet.microsoft.com/srd/2018/05/21/analysis-... (scroll down for the tables)
- bonzini 8y agoThis was known. https://lwn.net/Articles/765837/ https://lwn.net/Articles/765837/ reports 21% performance impact and the plan was to handle STIBP as an opt-in security feature. I am not sure why that was not furthered in the patches that have actually been committed to Linux, as I did not follow them closely.
- rwmj 8y agoApparently the way to turn this and other mitigations off is the following mouthful: pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier Would it make sense to have a single flag to "run insecure but fast" that we can use on pure development machines, test servers and the like? My Intel development server only runs code I choose.
- pixl97 8y agoHey boss, I don't understand why the code works in devel but fails in production?
- CyberShadow 8y agoYou're implying (with unnecessary snark) that: 1. Development machines should be configured identically to production machines. (Do you install a GUI and development tools on your production servers?) Occasional differences in behavior between development/production are par for the course, and is why staging environments are commonly used. 2. The mitigations affect the execution result of code likely to be developed/executed at a typical software shop. AFAIU the attacks are timing-based, and won't affect valid code that's not specifically looking to exploit them.
- manquer 8y agotry convincing a corp sec dept in large org to sign off on disabling anywhere.
- quickben 8y ago
- detaro 8y agoprevious: https://news.ycombinator.com/item?id=18474289 https://news.ycombinator.com/item?id=18474289
- azahk 8y agoI assume this is not a problem if I disable hyperthreading?
- vbezhenar 8y agoProbably you'll lose more performance.
- drb91 8y agoAs always, this depends on your workload.
- azahk 8y agoAs of today still most of the software I use seems to get stuck in one core of my CPU when calculating stuff so I'd argue that by enabling HT I'm losing performance.
- vbezhenar 8y agoThat's not really clear from the article, but I suspect that performance drop is from multithreaded software. And sure, if you are observing performance degradation with hyper-threading, there's 0 reason to keep it enabled. OpenBSD guys suggest to turn it off after all.
- hermanradtke 8y agoThe article claims a 30% perf slowdown with PHP. The majority of PHP workloads are single threaded. That causes me to suspect this impacts single core workloads too.
- bonzini 8y agoYes, on some processors basically it's as if all userspace was recompiled to use retpolines. Enabling STIBP promises that one thread cannot influence the indirect branch predictor's operation when the sibling runs; on those processors the indirect branch predictor is completely disabled, which is an inefficient but valid way to implement STIBP semantics.
- paavoova 8y agoThis is somewhat unfortunate for older-gen hardware. The difference between chugging-along but usable vs 20% reduced performance is crippling.
- jammygit 8y agoI was thinking that it might be an overall large boon to Intel due to all the new computers everyone will buy...
- KozmoNau7 8y agoI am a little fearful at how hard it will impact my X220i with its Sandy Bridge i3. Hopefully it won't be too bad.
- black-tea 8y agoYou can always disable the mitigations or even just stay on old kernels.
- pella 8y agoLatest test ( 17 November 2018 ) : "The Spectre/Meltdown Performance Impact On Linux 4.20, Decimating Benchmarks With New STIBP Overhead" https://www.phoronix.com/scan.php?page=article&item=linux-420-stibp&num=1 https://www.phoronix.com/scan.php?page=article&item=linux-42...
- pcdoodle 8y agoI get slower during 4.20 moments too...
- stinos 8y agoIf only the kernel went more creative..
- tedunangst 8y agoDoes the article not actually mention the reason? (I mean, I know it's STIBP, but it never says that?
- detaro 8y agoit's on page two. Why a short article like this has pages though...
- richrichardsson 8y agoI saw the social buttons and assumed end of article, very bad design.
- jccalhoun 8y agome too!
- kbenson 8y agoI was going to say because it's Phoronix, and they've traditionally had very heavy advertising, but I decided to load it up and turn off ublock to check because I haven't visited Phoronix in quite a while, and it seemed to have little to no big advertising. Nice! Now I assume it's because while even if the textual content is quite small, all the images mean there's quite a bit of vertical length to the article. I had to scroll down quite a bit to get to the next page button. The second page is quite small though, so maybe a better page break algorithm in is order. e.g. break at size X unless total length is X*1.25, and if only two pages, try to make them somewhat even in size (erroring on the side of a larger first page).
- userbinator 8y agoIf you could turn off speculative execution completely I bet that would eliminate all the security concerns, but performance would be abysmal. The Linux kernel really can't be a "one size fits all" type of thing --- an environment where no mutually untrusting code is run has very different security requirements from shared cloud hosts, for example. Personally I think it's all a bit overblown, probably due to owners of the latter environments.
- deleted 8y ago[deleted]
- dev_dull 8y agoThe lede is literally the last line in the article: > But why is it slower? More work on f&#!#(# Spectre!
- orf 8y agoThere is a second page: https://www.phoronix.com/scan.php?page=article&item=linux-420-bisect&num=2 https://www.phoronix.com/scan.php?page=article&item=linux-42...
- esotericn 8y agoNo mention of the fact that AMD seem unaffected?
- rayiner 8y agoI don't like how these Scepter mitigations are being rolled out with no cost/benefit analysis. On a client machine, I'd rather just disable Javascript than pay a 30-50% performance penalty for mitigating these vulnerabilities.
- xxpor 8y agoDo you expect Grandpa to do that analysis? The mitigations are probably behind KCONFIG flags anyway.
- garaetjjte 8y agoNo need to modify build config, this and other spectre mitigations can be disabled in cmdline (spectre_v2=off)
- stefan_ 8y agoA system running a browser probably has other problems, but just take every supercomputer ever or the vast majority of embedded systems. Running other peoples code (on other peoples machines) is a cloud thing, but they have an iron grip on the kernel and other software.
- CamperBob2 8y agoI would rather wait for reports of dangerous JavaScript exploits in the wild, myself. Considering that no real-world exploits of Spectre and related attacks have actually been reported, the odds that I'll fall victim to such an exploit without hearing about it in plenty of time to mitigate it are in the millions to one. Instead, everyone has to pay the performance tax proactively, regardless of their individual threat model. That is not how this sort of thing should be handled.
- deleted 8y ago[deleted]
- frostburg 8y agoWould you like a nuclear powerplant to be ran with this kind of approach to safety? Given the amount of possible targets and how unreliable people can be at assessing what risk they're exposed to, having mitigation active by default is obviously the only sane answer.
- bronzeage 8y agoI'm just sitting here waiting for all the security fiasco to stop ruining performance for ridiculous unpractical attacks. Just a matter of time, the pressure for better performance in a world where moore's law is getting stuck will eventually cause every sane person to throw these ridiculous patches out of the window. If the choice is between unsecure but fast to secure and slow, the default should be fast. Let applications who care about side channel deal with it their way, why the fuck should everything be so slow just so that few apps which actually care about side channel will be secure?
- amaranth 8y agoThis is just the mitigations at the kernel level, to protect the kernel from applications and applications from each other. Without this applications can't even attempt to secure themselves from these issues. I think to allow applications a chance to opt into being secure from these attacks without applying the fixes system-wide you'd need a way to disable speculative memory reads per page which would nerf performance even harder for the applications that turned that option on.
- gizmo686 8y agoAnd I'm sitting here waiting for the industry to start taking security seriously. These are not ridiculously unpractical attacks. These are attacks that have been hypothesized for years and are only now getting attention because people finally started to create (public) demonstrations. I would be shocked if there were not already non-public exploits based on these vulnerabilities. The only reason these attacks even have the appearance of being unpractical is because there are so many other areas of our computing systems that are even easier to attack. The only reason we are in this current Spectre et. al. mess is that CPU vendors choose to prioritize performance over security, and we now have to live with their bluff getting called for another hardware generation or two. I suspect that the performance impact will be far less when the fixes are done during the hardware design process, instead of figuring out how to bolt them on after the fact. Ultimately, I think we are overdo for a new computer architecture (both in terms of security and performance, we are being encumbered by the need to keep new architectures backwards compatible with old ones).
- 8y ago
- maccio92 8y agoBig win for AMD!!!
- beatgammit 8y agoIt's times like this when I'm glad I went with a Ryzen chip instead of whatever Intel has on offer. Not only do I get a ton of cores, but now my chip performs better than the competition! The only real problem I've had has been with soft lockups, which I was able to solve by disabling a power saving feature on the CPU. If AMD doesn't mess things up, they can really catch up to Intel in the next couple years while Intel is working on fixing their issues.
- seiferteric 8y agoThink I will wait at least 1-2 more years for all this to shake out before I buy a new computer.
- ams6110 8y agoYou'll get a much better deal anyway if you do that. I never buy the current generation of hardware. Much better value per dollar to buy one or two generations old. Unless you absolutely need bleeding edge performance.
- w323898 8y agoAll the security changes, plus 7nm chips should be widely available and will hopefully bring a price decrease.
- Aardwolf 8y agoAccording to this [1], the coffee lake refresh has hardware mitigation to meltdown variant 3 and variant 5 (which are only a few of all the things). Would that hardware mitigation reduce this performance loss? [1]: https://www.anandtech.com/show/13400/intel-9th-gen-core-i9-9900k-i7-9700k-i5-9600k-review/2 https://www.anandtech.com/show/13400/intel-9th-gen-core-i9-9...
- ToFab123 8y agoWhy is so much code being added to a kernel? <terms of lines of code changed with more than 354 thousand lines of new code added at the end of October when this merge window opened.
- justapassenger 8y agoDrivers
- azahk 8y agoSomeone once had the great idea that drivers should be in the same codebase as the kernel. Crazy, I know.
- umanwizard 8y agoIt doesn’t seem crazy to me that something that runs in kernel mode and uses kernel APIs should be in the kernel codebase. What are the downsides?
- gizmo686 8y agoThe main downside is that a bad/buggy driver can take down the entire system.
- black-tea 8y agoThat's a problem with the architecture but a reason for having the drivers in the kernel. It means such a case will be handled as a regression and fixed as opposed to the driver just never being available again.
- quotemstr 8y agoWhy should drivers necessarily run in the kernel? With appropriate abstraction for interrupts, DMA, etc. much of the work could be done in userland.
- 8y ago
- xbmcuser 8y agoIntel was supposed to have 20-30% advantage on single thread performance over Amd. Now I guess single or multiple thread AMD is better on Linux. I don't know why it makes me giddy thinking of the coming consumer wars between the 2.
- rdtsc 8y agoThis is how it should be. And this is going to net positive thing for AMD. If performance takes a hit by about 20% it is going to be noticeable. Next up I think people will start looking to upgrade to a faster CPU just to get back to last week's performance metrics. They might now consider AMD. If AMD finds this affects them less they should sponsor security researchers writing proof of concept exploits to convince people that that turning off the mitigation is an absolute no-no, so it will for force people to take a performance hit and go shopping for CPUs.