18 ms·
I'm unsure of the security of simultaneous multithreading on modern x86 CPUs
- daneel_w 5y agoIf these servers are "general multi-user environments", are you sure you're really getting any benefit at all out of SMT? It's never a magical 200% performance silver bullet, not even for highly specialized workloads.
- phkahler 5y agoI've found SMT to give 20-25 percent performance improvement on parallel workloads using a lot of the same data, so threads shouldn't be polluting each others cache. Different users would obviously not get the same improvement. I'm not sure 20-25 percent is even worth having SMT given that is' like going from 4 cores to 5.
- daneel_w 5y agoFor highly optimized workloads (e.g. x264 video encoding or raytracers) I've seen about the same 20-25% increase at most. On general heavy and diverse workloads the number is rarely ever above 5%, while the power consumption and the security concerns run off all the same. I personally disable SMT on everything I can.
- dragontamer 5y ago> I'm not sure 20-25 percent is even worth having SMT given that is' like going from 4 cores to 5. Can you think of another 25% bonus that costs nothing in terms of chip area? SMT doesn't add any new registers (the register files and retirement buffers are the same). SMT doesn't add cache, or pipelines, or anything. SMT just requires the chip's resources to be partitioned up in some cases, so its a complication to add to a chip but doesn't seem to take up much area at all. Normally to get a 25% bonus, you'd have to double the L2 cache or something similar (and as the 3x L3 cache tests from Microsoft proves for Milan-X, even 300% L3 cache isn't anywhere close to 300% faster, despite taking up significantly more die area... albeit cheap die area in the form of 3d-stacked chiplets but still, a lot more silicon is being thrown at the problem there... compared to nearly-zero-silicon in the case of SMT)
- trasz 5y agoDoesn't SMT require a second register file to keep the "other logical CPU" context in?
- dragontamer 5y agoCPUs lie to you. AMD Zen 2 has 180 registers in its 64-bit register file (an additional set of registers, also nearly 200ish, is also for the 256-bit vector YMM registers). The 16 architectural registers (RAX, RBX, etc. etc.) are mapped to these 180 registers for out-of-order execution. SMT is simply two threads splitting this 180-register file between each other. --------- That's how out-of-order execution is accomplished. The "older registers" are remembered so that future code can be executed. In practice, instructions like "xor rax, rax" are implemented as "malloc-new-register from the file, then set new register to zero". As such, all you need is the "architectural file", saying "Thread#0's RAX instruction-pointer #0x80084412 is really RegisterFile[25]", and "Thread#0's RAX instruction pointer #0x80084418 is RegisterFile[26]", and finally "Thread#1's RAX instruction pointer #9001beef is RegisterFile[27]". That allows all 3 contexts to execute out-of-order and concurrently. (In this case, the decoder has decided that Thread#0 has two things to execute in parallel for some reason, while Thread#1 only has one RAX allocated in its stream) The retirement queue puts everything back into the original, programmer intended order before anyone notices whats up. Different threads on different sections of code will in practice use up a different number of registers. A "linked-list" loop, like "while(node) node = node->next;" will be sequential and use almost none of the register file (impossible to extract parallelism, even with all of our Out-Of-Order tricks we have today), meaning Thread#2 can SMT and maybe use all 180-registers in the file for itself. ----- Side note: this is why I think Apple's M1 design is kind of insane/terrible. Apple's register file is something like 380+ registers or somewhere there-abouts (on an ARM architecture only showing 31 registers to the assembly language), but does NOT in fact use SMT. That is to say: Apple expects a singular thread to actually use those 380+ registers (!!!), and not really benefit from additional threads splitting up that absurdly huge register file. I'm sure Apple has done their research / simulations or whatever, and maybe it works out for the programs they've benchmarked / studied. But... I have to imagine that implementing SMT onto the Apple chips would be one of the easier ways to improve multithreaded performance.
- zdw 5y agoGoal of SMT isn't double the performance, it having >1 threads better using execution units within a CPU core. The example frequently given is running an integer heavy and floating point heavy code at the same time - they don't use the same execution units, so they can better utilize available resources. In the real world it's more likely that one thread is waiting on memory access or some other part of the system and the other thread can proceed during that wait.
- OldHand2018 5y agoThe blog entry states: "(For reasons beyond the scope of this entry, we can assume that SMT is worthwhile for us.)" This is also coming from utoronto.ca - so... I guess the question I would have is actually more along the lines of whether there are actually adverse effects from exploiting SMT to snoop on other processes? Is this going to be a whole lot of engineering students doing finite analysis homework? If so, what do you get out of an SMT exploit? There are probably a lot easier ways to cheat on your homework. Are they doing analysis of medical data that hasn't yet been anonymized? Perhaps that's a bigger deal. Maybe. 1. There is no such thing as perfect security 2. Security is an economic decision. You have to know what you stand to lose before you decide if buying 2x the number of processors and disabling SMT is worth it.
- mdriley 5y agoEric Brandwine (VP/DE @ AWS) said publicly in 2019 that EC2 had never scheduled different tenants on the same physical core at the same time, even before we learned about things like MDS. https://youtu.be/kQ4H6XO-iao?t=2485 https://youtu.be/kQ4H6XO-iao?t=2485
- hn_throwaway_69 5y agoThat's a really good talk. Wish I'd watched it around the time the speculative execution vulnerabilities were all the talk. I'm not qualified in computer science or programming but I could easily follow along with his explanation about challenging CPU architectural concepts. Gives me a great deal of confidence that AWS not only know what they're doing, but can communicate it well to management.
- deleted 5y ago[deleted]
- gruez 5y agoHow does that work for burstable VMs? Surely they must be oversubscribed on a per-core basis? Are they sharing cores, but doing scheduling at a very fine granularity (eg. millisecond)?
- FartyMcFarter 5y agoI guess they can only schedule hyperthreads if the same tenant requests at least two virtual cores?
- mochomocha 5y agoI don't think it really matters, the goal post will move to LLC attacks. As long as you have caching involved, I wouldn't bet anything can run "safely".
- codefined 5y agoI wonder if per-tenant cache structures might become more popular in the future. It isn't unimaginable that different tenants will be running different things anyway, so split caches might still keep a reasonable amount of effectiveness.
- gautamcgoel 5y agoA lot of these discussions focus on servers. What about us Plebs, who only run desktop machines? What should we do?
- modzu 5y agoif you arent running vms, you can chill.
- 4ad 5y agoNo you can't, attacks like spectre are exploitable from javascript.
- Filligree 5y agoHas anyone seen an attack in the wild, though? Ever?
- crazysim 5y agoPeople were doing POCs before the browser makers nerfed the hell out of timing APIs especially from web assembly and autoupdate distributed the nerfs out.
- davidw 5y agoWhat does 'nerfed' mean in this context?
- mekkkkkk 5y ago'Nerf' is a term used for when game developers update a game and reduce the effectiveness of something. 'My sword attack ability was nerfed in the latest patch'. In this context I suppose the API capabilities was reduced in browser updates.
- 5y ago
- ldng 5y agoI'm unsure of the security of simultaneous multithreading on modern Apple CPUs
- MiguelHudnandez 5y agoCan you share some of your research about it?
- eyelidlessness 5y agoI’m unsure of the existence of simultaneous multithreading on Apple CPUs.
- garren 5y agoApple cpus specifically, or arm64 in general? The same class of bug (i.e., speculative execution) or something else altogether? I’ve found myself with more and more arm hardware these days, and I’d assumed (probably incorrectly) that it wasn’t as prone to these kinds of issues.
- sam0x17 5y agoI'm unsure of the security of anything on modern Apple operating systems or hardware, since I can't review the source code or hardware schematics.
- emteycz 5y agoCan you send me the Intel/AMD/Qualcomm/Samsung sources and schematics?
- mhh__ 5y agoIntel at very least do publish a pretty hefty optimization manual. We do have a bunch of leaked intel firmware, or so I'm told...
- macdice 5y agoI haven't followed it or tried it, but supposedly you can boot up a pure open source Darwin system. Certainly it has been very useful for me as a cross-platform developer to be able to browse its source and understand quirks of system calls etc. Are you saying that key parts you'd like to review are closed source? Hmm, the new file system is closed source, which is a clue, but I don't know to what extent Darwin is "different" from what macOS is really running...
- londons_explore 5y agoIf you are trying to prevent attacker code communicating with another piece of attacker code on another core: No. It isn't secure. There are a lot of side channels. If you are trying to prevent attacker code figuring out what code is running on another core: No. Patterns of L3 cache eviction and memory bus contention will probably leak sufficient information to have a good guess what another core is up to. If you are trying to prevent attacker code figuring out data in another cores memory or registers: Probably, yes. As long as the core is doing something with no data dependent branches, or instructions with data dependent timings, it should be safe. Very little code is written to have no data dependent branches, or instructions with data dependent timings. Some well written cryptography code has that, but thats about it.
- omegalulw 5y ago> Very little code is written to have no data dependent branches, or instructions with data dependent timings Ooh I have some idea on how to use data dependent branches in your own code to leak critical information through side channels (Spectre) but how do you leak information from others' data dependent branches and instructions?
- londons_explore 5y agoImagine code you want to attack has an array lookup based on some secret data: x = array[secretdata]. The entry in the array will be loaded into the cpu core via the L3 cache. If the L3 cache is already full, then one entry will have to be evicted from the cache. The evicted entry will depend on the memory address that is being accessed. Another CPU, running an attackers code, can repeatedly try to load stuff from L3, and detect what was evicted and when to learn the secretdata.
- acdha 5y agoI've been thinking along similar lines, and also wondering what percentage of workloads still see significant benefits from SMT. It made a lot more sense in 2002 when it was still common to find servers with only one core but now even a phone has half a dozen cores and laptops are pushing into the tens of cores range while servers are pushing an order of magnitude greater. I remember disabling SMT in the mid-2000s because it was a performance degradation for some memory-sensitive code one of my users was running. I'd be really curious what percentage of workloads are not limited by I/O but still stay below the level where resource contention between threads lowers performance.
- dragontamer 5y agoDDR4 RAM has the same latency as DDR2 RAM. DDR5 will probably have the same latency as well. In 2002, our processors were so thin they could only execute maybe 2-instructions per clock tick. Today, we've got CPUs that hit 4 (x86 L1 cache), 6 (x86 uop cache), or 8 (Apple M1) instructions per tick. With many problems still memory-latency bound (aka: any Java code going through blah->foo()->bar()... look at all those pointer indirections!!), it makes sense to "find other work to do" while the CPU stalls on memory latency. The RAM supports higher bandwidth. The CPU supports more instructions-per-tick. Spending effort to find work to do just makes sense.
- acdha 5y agoOh, I'm aware of that concern but I was wondering how many benchmarks show a significant win. After you consider the impacts on the processor caches, and the fraction of apps which are performance sensitive but still end up limited on cache-unfriendly random memory access, it seems like it could be an increasingly niche concern where most of the code people run either isn't performance sensitive or has gotten better about memory access patterns after decades of hardware trending in that direction. Apple's highly-competitive performance with processors which don't support SMT suggests that this is not such a clear win, especially given the non-trivial cost of the security mitigations.
- dragontamer 5y ago
- stefan_ 5y agoThis is a red herring. Where are all the people that are unsure of the security of virtualization? There is some Xen vulnerability every year that is the equivalent of the house burning down, but where is the Azure support article that suggests disabling Xen? (Oh right, you can't.) I mean, the article is totally right of course, you shouldn't run untrusted code - THAT INCLUDES THE FREAKING HYPERVISOR.
- api 5y agoThere's definitely a bias in security toward "sexy" vulnerabilities that are novel and creative but hard to exploit in practice vs boring "forgot to bounds check" or "off by one logic error" type vulnerabilities that are pervasive. SMT (a.k.a. hyperthreading) is kind of a security minefield, but so is out of order execution and crufty instruction sets like x86_64. But vulnerabilities of this sort tend to be hard to exploit in the wild. Not saying you shouldn't be concerned about them, but how concerned you are should depend on your threat model. If I were really paranoid and could pick I'd choose something like a low-power simple ARM or RISC-V core to do critical computation. Alternatively you could run your secure compute on any system as long as nobody is allowed to run any other code on that system or have physical access to it. BTW I am deeply surprised that there has never been a cloud hypervisor doomsday vulnerability that has brought down AWS or some other huge provider. Back when virtualization in the cloud got big I would have bet serious money that this would have happened by now.
- melolife 5y agoTo quote James Mickens, "large swathes of the security community are fixated on avant-garde horrors such as the fact that, during solar eclipses, pacemakers can be remotely controlled with a garage door opener and a Pringles can."
- sonicanatidae 5y agoYeah, making a note of this quote. Thanks!
- mwint 5y ago
- vlovich123 5y agoLinux added core scheduling recently which is a way to turn on HT/SMT in unsafe environments (but requires SW changes to make sure you stay safe) https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/core-scheduling.html https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/c...
- abotsis 5y agoRecent Linux patches allow the scheduler to only schedule processes with the same seed on the same core. The “seed” would be something tied to the user, or something else… I don’t think they’re merged yet… but that might be a solution.
- phkahler 5y agoFrom TFA: >> We're planning to get some high core count machines to be new compute machines in our environment of general multi-user Unix login servers I think the "general multi-user Ubix login" thing assumes different users on the same CPU. Unless they're actually going to have more CPUs than users. It seems like sharing a CPU is actually less secure than sharing one via SMT because of all those SMT-specific exploits (in addition to whatever exploits might be possible due to CPU sharing).
- toast0 5y agoSharing an SMT is IMHO less secure, because the adversary can probe while the victim is running. Sharing a CPU means you can only probe the effects. It would make sense to me for a traditional multiuser shell server to have a scheduling policy prohibiting SMT sharing between users. As long as the scheduler work is not expensive, it would seem almost no-cost vs disabling SMT in case all users only run a single thread, but you can still get whatever benefits SMT provides within a user's workload.
- cozzyd 5y agoAnd... can each browser tab run as a different user? (or more likely, there's a way to have the browser opt in to some security policy that would effectively do the same thing).
- bjoli 5y agoI disabled SMT in 2007 after reading colin percivals thoughts on intels hyperthreading. I have disabled it on every computer I have had since, since the fixes didn't inspire much confidence back then. I have done benchmarks from time to time and it doesn't really matter much for my workloads.
- tryauuum 5y agoI'm a simple guy, I trust the kernel and its devs. If kernel says to me via /sys/ that on this CPU I have to disable SMT if I want to get rid of "Vulnerable" messages, I do so. Kernel considers recent Intel CPUs not vulnerable even with SMT, it's good enough for me
- tenebrisalietum 5y agoif /sys/kernel/jump_off_a_cliff was 1 would you go ahead and do it?
- ByThyGrace 5y agoWhat if sys/kernel/jump_off_a_cliff returns .9999?
- monopoledance 5y agoUnderneath the cliff will be a water body, but the kernel prepared accordingly.
- yjftsjthsd-h 5y agoIf the kernel developers had collectively decided that it was safer to jump off the cliff then stay, then yes, absolutely. https://xkcd.com/1170/ https://xkcd.com/1170/
- jasonhansel 5y agoActually, that's a common misconception. When "jump_off_a_cliff_ratio" is set to a value other than "-1", then "jump_off_a_cliff" is ignored. The ratio is divided by 20000, so if you set it to 10000 your system will only jump off a cliff if 50% of friends do so. The default setting is "15000", but some distros change it (e.g. Arch sets it to "0" so that your system always jumps first--more performant, but less safe). Conveniently, none of this is documented anywhere, all the other information you can find online is years out of date, and most sysadmins just accept the risk of cliff-jumping rather than trying to tune it manually.
- manpagereader 5y agoOpenBSD defaults to turning SMT off these days because the core team is of the same opinion. Theo actually expressed a similar opinion over 15 years ago.