7 ms·
Does that mean hyperthreading is effectively unpatchably insecure? Cloud Providers are gonna have a bad time if this is true.
by Roritharr 8y ago
Does that mean hyperthreading is effectively unpatchably insecure?
Cloud Providers are gonna have a bad time if this is true.
- fermienrico 8y agoQuestion: If I rent 4 core AWS instance, does it mean 4 physical cores or 4 hyper threaded cores? Is there a standard to this definition of “cores” across GCP, DO, Linode, etc. I don’t have the experience or knowledge about cloud computing but just have a DO instance running a web server. I’m curious.
- Roritharr 8y agoI asked this a while ago: a cloud core is always just a hyperthread.
- djsumdog 8y agoYou mean it's random then, right? I mean let's talk about what a hyperthread really is: It's the left over functional units (or execution units). Say you have 100 adders, the processor tries to schedule as many instructions as it can on those 100 adders, but eventually it will run into data dependencies. The left over units can go to a hyperthread. My understanding (let me know if I'm wrong) is that Hyperthread aware OSes (which is like what, everything since WinXP/Linux kernel 2.4?) will schedule lower priority tasks to the logical cores and higher priority tasks to the real cores. So when it comes to a hosted provider (a.k.a cloud provider, a.k.a somebody else's computer), what you get pretty much depends on the virtualisation layer they use: Vmware, KVM, Xen, Hyper-V, etc. Do hypervisors typically peg VMs to a real physical core? I was always under the impression they over-provision on most hosts, so you're getting part of a core and the vCPUs the list in the product documents just indicates your priority and how may vCPUs appear to your particular VM.
- rosser 8y agoFrom my understanding, you can't (necessarily) even rely on your guest's CPUs mapping to the host's actual CPUs, which makes spending time twiddling NUMA actively useless. Assuming that's actually the case, I very much doubt the guest's scheduler has the ability to schedule tasks between logical and physical cores, based on priority.
- gaius 8y agoMy understanding (let me know if I'm wrong) is that Hyperthread aware OSes (which is like what, everything since WinXP/Linux kernel 2.4?) will schedule lower priority tasks to the logical cores and higher priority tasks to the real cores That is not how I understand it. The OS sees two identical logical cores per physical core and the CPU manages which is which internally. Also it's not really high and low priority - it's two queues multiplexing between the available execution units. If one queue is using the FPU then the other is free to execute integer instructions, but a thousand cycles later they might have switched places. Or if one queue stalls while fetching from main memory, the other gets exclusive use of the execution units until it gets unstuck. In my floating-point heavy tests on i7 however, there is still a small advantage in leaving HT on, the common wisdom is if you are doing FP, HT is pointless and may actually harm performance, but that doesn't match my observations if your working set doesn't fit into L2 cache. YMMV. A semi-modern OS will try to keep a process on the same physical core if it can, so it may be flipflopping between two logicals, but should still see the same cache. Disabling HT means the OS still sees logical cores, but half as many of them, with a 1:1 correspondence between logicals and physicals.
- Sohcahtoa82 8y ago> In my floating-point heavy tests on i7 however, there is still a small advantage in leaving HT on, the common wisdom is if you are doing FP, HT is pointless and may actually harm performance, but that doesn't match my observations if your working set doesn't fit into L2 cache. YMMV. I benchmarked this myself using POV-Ray (which is extremely heavy on floating point) when I first got my i7-3770k (4 cores, 8 threads). Using two rendering threads was double the speed of one, four was double the speed of two, but eight was only about 15% faster than four. I don't think I've ever actually seen an example of real-world tasks that get slowed down by HT. Every example I've seen was contrived, built specifically to be slower with HT.
- wmf 8y agoA cloud "vCPU" is a hyperthread and in good providers (EC2/GCE) they are properly pinned to the hardware such that, for example, a 4-vCPU VM would be placed on two dedicated physical cores. This was probably done for performance originally but now it also has security benefits. You can get hints of this by running lstopo on VMs and similar bare metal servers. On second and third tier cloud providers, the vCPUs tend to be dynamically scheduled so that they may share cores with other VMs.
- oavdeev 8y agoHere's lstopo output on a 4-core AWS instance to illustrate your point: https://instaguide.io/info.html?type=c5.xlarge#tab=lstopo https://instaguide.io/info.html?type=c5.xlarge#tab=lstopo
- Roritharr 8y agoThinking about this, they're probably gonna introduce "insecure but cheap" instances for customers that don't mind the chance of data leaks and takeovers...
- pvarangot 8y agoWhich is going to be everyone except customers who already have issues with cloud and need special instances because of regulations. Then we'll wee the occasional "30.000 credit cards stolen" hack every three years because of this issue and that'll be it. It's another situation like what happened with WEP WiFi encryption ten years ago.
- ofrzeta 8y agoThat would be hard to market.
- fermienrico 8y agoIndeed, the idea of "Security" is not negotiable when marketing. Imagine buying a car that says, "Save $5000 for a less safe version without airbags." Yes, I know Airbags are DOT requirement, just trying to make a point. Edit: I think people are missing my point. I am not saying they don't sell cheaper models that are less safe. My point is that they don't ACTIVELY market them as such. Point me to an advertisement that says "Save $5000 for a less safe car!". This is in the spirit of what the GP was talking about whether cloud providers can market as "Less secure but cheaper HT option".
- ISL 8y agoThere are absolutely situations in which a substantially cheaper but less-secure/safe solution to a problem can make economic sense. Suppose you have $5k, you need a car in order to feed your family, and that only the following two options are available: You can buy the safe car for $10k or a less safe car for $5k. In that situation, less safety can be a reasonable choice. Indeed, there was a long period of time in which Volvos were demonstrably more safe than other lower-cost vehicles, yet people bought the lower-cost vehicles. In the cloud-offering world, instead of marketing servers as "less-secure", they can simply offer "more-secure" options that run on non-HT hardware. HIPAA-compliant cloud-buyers will have to upgrade, and then the cloud vendors can slowly lower the prices on both, making the less-secure option lower cost than the present day.
- Aardwolf 8y agoIs it a feasible solution to enable hyperthreading only for threads or forks of the same process? Then they can use this ability, but other processes cannot do timing attacks on this process in this core... I think
- Roritharr 8y agoI'm guessing someone at Amazon is looking at this right now.
- gruez 8y ago>Is it a feasible solution to enable hyperthreading only for threads or forks of the same process? how does that work on unix systems when processes are all forked from 1 process? even if you get past that issue, how do you prevent less privileged processes that use other security mechanisms (cgroups, pledge, selinux, croot, sandboxing)?
- tinus_hn 8y agoYou could allow processes that have ptrace rights on each other to run simultaneously which would cover most issues, but you’d still run into trouble with JavaScript engines running untrusted code.
- LinuxBender 8y agoProbably in most cases. Vmware have disclaimers in the mitigation options that preclude turning off HT, meaning, use at your own risk. [1] I am still waiting on a comment from Linode [2] Openstack have some knobs you can adjust, but it really depends on your workloads and what risk you are willing to accept. [3] AWS have their own custom hypervisor and are said to have worked around the issue. [4] Amazon had info on this before others. It appears they have a special relationship with Intel? I have not found any hardware or OS vendors that are willing to say that you can leave HT enabled. It is a very heated topic because folks will have to increase their VM infrastructure anywhere from 5% to 50% depending on their workload profiles. For public clouds, you can't predict workload profiles. Edit: Oops I left out the main site for L1TF [5] [1] - https://kb.vmware.com/s/article/55806 https://kb.vmware.com/s/article/55806 [2] - https://blog.linode.com/2018/08/16/intels-l1tf-cpu-vulnerabilities-linode/ https://blog.linode.com/2018/08/16/intels-l1tf-cpu-vulnerabi... [3] - https://access.redhat.com/articles/3569281 https://access.redhat.com/articles/3569281 [4] - https://aws.amazon.com/security/security-bulletins/AWS-2018-019/ https://aws.amazon.com/security/security-bulletins/AWS-2018-... [5] - https://foreshadowattack.eu/ https://foreshadowattack.eu/
- zitterbewegung 8y agoAWS is able to get custom Intel Processors due to their size (c5 instances have a custom Intel processor).
- LinuxBender 8y agoMakes sense. I sure would like some custom processors. :-)
- exikyut 8y agoWell, fabbing using reasonably respectable processes only runs circa $2000/mm2 or so, and using crazy old process nodes like CMOS et al gets you down to $300/mm2, so you could very well make something. Technically. (I want to...)
- 8y ago
- geofft 8y agoYou can give both hyperthreads in a physical core to the same tenant, no? Scheduling different VMs to run on the same hyperthreaded core at once seems like it can't be good for either VM's performance, even if there were no security concerns. Hyperthreading is much more useful for running multiple threads of the same app, accessing similar instruction caches etc. (There's also a question of safety within the VM, but a huge number of cloud users are running effectively one user within their VM.)
- d1zzy 8y agoYes, you can isolate hyperthread siblings to the same VM but you also need to ensure no host code (userspace or kernel) runs on that core, or the untrusted guest may be able to read values stored in L1 by that code. This is harder to do and likely would result in large performance drops for some workloads (because you are essentially disabling the advantage of locality for data that needs to be accessed from both guest and host environment).
- Roritharr 8y agoThe latter question is very important indeed. If you for instance render websites in your vm they, if i understand correctly, can potentially read secrets from other processes, like db credentials and other stuff... If the only real solution is to turn off HT/SMT that, seen positively, should net us a lot faster VMs then...
- geofft 8y agoIf you render websites that run code in your VM (e.g., you're running a traditional shared hosting infrastructure where mutually-untrusted users can upload PHP scripts, or you're doing something serverless / FaaS / Cloudflare Workers / etc. where mutually-untrusted users can upload functions), then yes. If you're rendering websites in the sense of hosting WordPress for lots of people but not permitting custom plugins, then no.
- Roritharr 8y agoI thought more about Rendering and executing their js for screenshotting purposes for example.
- crunchlibrarian 8y agoHyperthreading is fine on its own, but yes in combination with other CPU features it is effectively impossible to secure. Turn it off or sell your Intel chips.
- shittyadmin 8y agoIs there any risk this also impacts browsers executing JS?
- paulie_a 8y agoYes. you are executing JavaScript after all.
- _JamesA_ 8y agohttps://blog.digitalocean.com/a-message-about-l1tf/ https://blog.digitalocean.com/a-message-about-l1tf/