30 ms·
Linux 5.8 Set to Optionally Flush the L1d Cache on Context Switch
- enitihas 6y agoThe performance implications of this would be huge, and I hope it remains opt in for long time.
- lmilcin 6y agoI don't see how the implications will be huge. This is L1d cache which is just 48kB for Ice Lake. We are also talking about context switches which are not happening very frequently. Applications that are generating load don't context switch all the time because they are busy doing work. Then, when you context switch it is likely the context to which you are switching would like to use that cache for something. By the time we switch to your original thread it is very likely L1d has already been filled with something else. I am pretty sure you would not notice anything except for very special, rare situations.
- deleted 6y ago[deleted]
- robocat 6y agoDo interrupt handlers cause context switches? Presumably more of a problem if all cores are busy, which is more likely if there are few cores. Also dependent on the number of interrupts (e.g. high network traffic of small packets etc). Presumably not a problem if there is an idle core that can run the interrupt code.
- namibj 6y agoIf you have many interrupts due to network packet load, you're doing something wrong. Interrupt-based handling is slower and less efficient (than polling) after some throughput that's iirc about a few hundred Mbit/s/core.
- robocat 6y agoI think you are confusing Linux’s epoll with the hardware network interface. Some hardware offloads a lot of processing to dedicated network card processors, other network hardware might just use hardware interrupts for the driver module. Either way, I am sure there are plenty of devices that can cause a lot of interrupts (USB?), not just network IO. Presumably there is a way to monitor the count of interrupts per second in Linux?
- twic 6y agoIf the 48 kB of cache has 64-byte lines, then it has 768 lines. If a line takes 5.3 ns to fetch from the L2 cache [1], then that's ~4 microseconds to fetch all of them. It's not as if the processor will stop and do that after a context switch (and it can overlap the loads with other work etc), but that's roughly the order of magnitude of the cost of an L1 cache flush [1] https://stackoverflow.com/a/4087331/116639 https://stackoverflow.com/a/4087331/116639
- lmilcin 6y agoNope. The cost would be right if the cache was usable after context switch. Since it is likely stale, the new context will be pulling new data into cache as if nothing really happened.
- monocasa 6y agoWell, the question is how many of those lines are already stale because of the work done by the other processes during the context switch.
- brigade 6y agoThroughput isn't the inverse of latency; the throughput of L1 <-> L2 is 1 line per cycle. If IA32_FLUSH_CMD exists, probably a better order-of-magnitude estimate is ~200ns for writing back dirty lines to L2 during the switch.
- RedShift1 6y agoUh what? Context switches happen all the time and it's not the applications that decide that, it's the kernel. It will preempt processes at its own discretion and the more that are running the more context switches will happen. So as more processes are running (or the same processes start doing more work) and/or interrupts increase, the more performance will be affected by the extra work having to be done each context switch. As a desktop user you might not notice but if you just invested into some new server iron and it suddenly performs 10% worse, I wouldn't take that lightly.
- saagarjha 6y agoHigh-performance applications are typically run with anticipation of how the kernel does context switching and are generally designed to accommodate this; on the flip side, the kernel's job is spend the most time executing as it can and it will try to not switch when it can avoid it.
- ajross 6y agoEven on loaded servers, modern systems tend to be running with at least one idle core virtually always. True context switches are very rare -- CPU-bound processes tend to keep their processor for seconds at a time. Obviously there are software architectures that are exceptions, but all the big ones tend not to switch much. (Which isn't surprising, as cache flush or no, switchin has always been a slow process that software has tried to avoid).
- RedShift1 6y agoIf context switches are not happening, then what are your processes doing then? Shuffling memory around? Every time you do I/O a context switch happens (disk, network, ...). If your processes are not hitting disk or network, what are they doing then? Calculating something but keeping the results for itself?
- ajross 6y agoSome of this is a terminology problem: properly a "context switch" refers to the kernel switching control between two user processes on the same CPU. If all you're doing is taking an interrupt in the kernel and returning to whatever was interrupted, that's about half the work of a "context switch" on most architectures (though still expensive, obviously!). But FWIW: most HPC computing is, in fact, "shuffling memory around", yeah. Very few architectures are actually interrupt bound, and the ones that are work very hard to address that (because hardware interrupt parallelism is an even harder nut to crack than context switch overhead).
- ollien 6y agoWhat do you mean by "not happening very frequently?" The default timeslice is something on the order of 100ms, isn't it? And that's if the process doesn't yield. Clearing L1d every 100ms (at worst) seems pretty frequent to me.
- lmilcin 6y agoThe cost of clearing cache on context switch has to be put in context (hey, pun intended:) 100ms is a huge amount of time and 48kB is a tiny, tiny part of what processor does during 100ms. Gigabytes of data can be transferred during that time, 48kB isn't really much. As I have pointed out, that cache has very little value over context switch anyway. The cost is removing data from cache that would be usable after we have returned to the original context. But it is already very likely the data in the cache is already for a completely different context and hence completely unusable. Say you have apps A and B and OS. You are running A which has 48kB of data in L1d. It switches to OS which causes some of L1d to be evicted and puts its own data there. Then it switches to B which is likely another process, this causes very likely entire L1d to be evicted unless this is extremely small process. Then we come to OS and again to A. By the time you are at A, there is no data from the original L1d state. Cleaning L1d upfront on context switch is likely not hurting anything.
- MaulingMonkey 6y agoTo further back this up with some math - L2 cache hits (what you'll hit on a L1D cache miss caused by clearing L1D cache) - are still in the mid/low single digit nanosecond ranges[1]. Say flushing the L1D causes another 1000 L2 cache misses[2] - maybe we got really lucky and the next thread was hashing all the exact same data at the exact same time or something equally unlikely? That'd still put us in the mid/low single digit microseconds range. On par with DDR4-1600 (12.8GB/s)'s 3.75us to read 48KB [3][4]. Let's more than double that and say it takes 10 microseconds = 0.01 milliseconds = 0.01% of 100 milliseconds. Any noticable perf overhead is going to be from the act of cache flushing taking some super slow path for some reason, or much more frequent context switching than 100ms timeslices. [1] https://stackoverflow.com/a/4087331 https://stackoverflow.com/a/4087331 [2] 1000x 32-128B cachelines = 32-128KB, definitely in the ballpark to completely refill a 48KB L1D cache. [3] https://en.wikipedia.org/wiki/DDR4_SDRAM#Modules https://en.wikipedia.org/wiki/DDR4_SDRAM#Modules [4] https://www.wolframalpha.com/input/?i=48KB+%2F+12800+MB%2Fs https://www.wolframalpha.com/input/?i=48KB+%2F+12800+MB%2Fs
- zelly 6y ago> Then, when you context switch it is likely the context to which you are switching would like to use that cache for something. By the time we switch to your original thread it is very likely L1d has already been filled with something else. The other thread may have been doing work with memory on a GPU. The other thread may already have a hot cache at another layer. It's definitely not an edge case, or else the L1d cache would not have been designed to maintain state between context switches in the first place. There are going to be consequences to this.
- rayiner 6y agoI mean it depends on when you need to do it, right? If there are vulnerabilities that lead to private kernel data leaking to userspace through the L1D, you’re talking about needing to wipe out your data cache on every system call, which might need to happen millions of times per second. Also, context switches can be very frequent in some designs. For example, in micro kernel systems you often have ping-ponging with processes communicating with servers via RPC. Wiping out your whole L1D every time that happens could be pretty unpleasant.
- deleted 6y ago[deleted]
- dirtydroog 6y ago> We are also talking about context switches which are not happening very frequently That depends on how your software is written. If, for example, you're running a web server that uses a thread-per-connection, you'll be context switching all over. Hi Apache!
- lmilcin 6y agoWell, you are right on that one. I may have stressed this one too much, I guess. The real reason flushing L1d is not going to be noticed is that even without flushing the cache is unusable after context switch. It is highly unlikely the next thread that gets ownership of the core will require exactly the data present in L1d. On a busy web server the two most frequent reasons to switch context will be: 1. The thread is waiting on I/O so it yields the rest of its time share back. 2. The thread has finished processing request. Now, if you imagine a thread that just did a bit of I/O returning its time so that OS is switching context to another thread... it is very unlikely any of the data in L1d has any meaning or worth for the other thread. Anything that the next thread will do will require fresh data at least from L3. So L1d is practically worthless and blanking it isn't going to do anything noticeable. (I have intentionally omitted all the interrupts happening in the meantime and OS also using the cache which is the proverbial nail in the coffin when it comes to usability of L1d after context switch)
- monocasa 6y agoL1D isn't _that_ bad I think. Most cache usage modeling I've seen assumes that L1 is completely stale across context switches anyway. The separate address spaces for user and kernel were probably a way worse hit on platforms that don't have ASIDs. L2 would be absolute crazy town though.
- Iwan-Zotow 6y agoIdeally one would desire an indefinitely large memory capacity … We are therefore forced to recognize the possibility of constructing a hierarchy of memories, each of which has greater capacity than the preceding but which is less quickly accessible. ”Burks, Goldstine, and von Neumann, "Preliminary discussionof the logical design of an electronic computing instrument," 1946.
- acchow 6y agoHow often do you not overwrite all of L1 cache in a new context anyway?
- vbezhenar 6y agoIf you're doing some quick system call and that system call does not use much RAM, then you'll have your cache almost ready after its return. Now if that's in some tight loop, that might influence performance. Probably in single digits of percents, but it's still degradation.
- ajsnigrutin 6y agoSo.... when will we get a part of our money back, due to performance losses? (i'm talking about a case like Volkswagen)
- jzwinck 6y agoVW broke the law by deliberately defeating regulations which applied to them. Is there a regulation that applies to Intel and AMD which requires them to allocate die space and performance to improving security? I'm not even sure they could have delivered equal performance at the high end if they had included these mitigations earlier. Whether to produce processors which are safer or faster depends on what customers prefer, even now. Not everyone needs ultimate security or wants to pay for it (in money or performance). So unless the law says less-secure processors must never be sold, this situation was and still is inevitable.
- ajsnigrutin 6y agoIntel knew their security was broken, and they were still selling those affected CPUs. Products were recalled for smaller issues than that.
- ken 6y agoNot CPUs -- they're rarely recalled. The Xeon in my computer has 50 pages of errata. It's common knowledge that CPUs are terribly buggy.
- ajsnigrutin 6y agoIf a car lock failes, those cars get recalled, and the locks fixed. https://www.recallmasters.com/mercedes-recalls-vehicles-defective-door-locks/ https://www.recallmasters.com/mercedes-recalls-vehicles-defe... Also, intel did replace CPUs with issues before: https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug > On December 20, 1994, Intel offered to replace all flawed Pentium processors on the basis of request, in response to mounting public pressure.[5] Although it turned out that only a small fraction of Pentium owners bothered to get their chips replaced, the financial impact on the company was significant.[citation needed] On January 17, 1995, Intel announced "a pre-tax charge of $475 million against earnings, ostensibly the total cost associated with replacement of the flawed processors."[1] Some of the defective chips were later turned into key rings by Intel.[6]
- saagarjha 6y ago> This flushing does address CVE-2020-0550 for snoop-assisted L1 data sampling but the main emphasis seems to be on the "yet to be discovered vulnerabilities." I am unsure if this is just idle speculation (heh) that there may be issues in this area or there are issues that have been disclosed to vendors but not the public yet?
- snazz 6y agoYes, this is probably another coordinated disclosure where we see evidence in Linux kernel commits first. I noticed that Apple hasn't released the security credits page for iOS 13.5 yet; is that a normal delay or are we waiting on the disclosure of another processor bug (that presumably also affects ARM)?
- saagarjha 6y agoInterestingly, the security content for Xcode 11.5 is out already.
- snazz 6y agoIt looks like that's just updating Git to a new version to address an already-public bug. I guess they figured that there's no advantage to waiting to release that information.
- 0x0 6y agoMaybe, the notes will probably be published when macOS 10.15.5 drops (this week? But who knows if the new 0day in the unc0ver jailbreak will shake up things)
- rasz 6y agoThanks Intel.
- rayiner 6y agoI'm not that old, but I've been closely following these trends since the late 1990s, and it seems to me like we are descending into madness now. We are wiping out years of single-core performance gains--which have been hard to come by over the last decade to begin with--through all of these mitigations. It seems to me like maybe the mental model is broken. If untrusted code is running on the same core/package/what have you, your security has already been breached.
- ghshephard 6y agoYou run untrusted code every day you browse the web.
- deleted 6y ago[deleted]
- asveikau 6y agoOne of the recent times where I pointed this out somebody retorted that the browsers got rid of js access to high resolution timers, making a timing attack infeasible. Is this true?
- myopenid 6y agoThese vulnerabilities feel like Coronavirus. They sound scary but has practically little threat, but we all pay dearly for the mitigation.
- pestaa 6y agoPlease consider the idea that the coronavirus threat may be practically little precisely because of the ongoing mitigation efforts.
- Trellmor 6y agoLast week Microsoft also rereleased the Intel microcode updates package [1][2]. I kinda expect to see a new CPU flaw in the next few days. [1]https://support.microsoft.com/en-us/help/4497165/kb4497165-intel-microcode-updates https://support.microsoft.com/en-us/help/4497165/kb4497165-i... [2]https://www.windowslatest.com/2020/05/21/windows-10-kb4497165-update-released/ https://www.windowslatest.com/2020/05/21/windows-10-kb449716...
- Zenst 6y agoSo with more cores and associated L1 cache, context switching would be potentially less I would of thought, small but maybe measurable. Interestingly enough: https://www.theregister.co.uk/2020/05/24/linus_torvalds_adopts_amd_threadripper/ https://www.theregister.co.uk/2020/05/24/linus_torvalds_adop...
- Zenst 6y agoMaybe time to redo the ring model with multi core cpu's now the norm. Why one or two CPU cores couldn't be dedicated to the OS aspect and locked out of user-space of any form, certainly would be something worth exploring.
- wtallis 6y agoIn general, dedicating some cores to userspace and some to kernel would mean sending a lot more data up to L3 for core-to-core communication, and I'm not sure that would be any better than flushing L1d. The exception would be with SMT, but then you're locked into a 1:1 ratio for user and kernel virtual cores, and still have to worry about side channel vulnerabilities.
- Zenst 6y agoThat is true, which does somewhat put a whole new perspective upon this, will they flush l2,l3 next! Be nice though to have a proper isolated core or two for the OS, after all - that is exactly what is done for enclave based security and management systems. Though some not all a great track record.
- monocasa 6y agoOr more rings. The problem is that we have this idea of running trusted and untrusted code and data in the same context as far as the CPU is concerned. Add more fine grained memory partitions to let the memory hierarchy in on what you're doing. Make Rings 1 & 2 Great Again