4 ms·

The Intel paper link is dead, this seems to be the right one: https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/t

The Intel paper link is dead, this seems to be the right one: https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/gather-data-sampling.html?wapkw=gather%20data%20sampling https://www.intel.com/content/www/us/en/developer/articles/t... General caveats: are there many clouds that still run workloads from different users on the same physical core? I thought most had changed their schedulers years ago so you can't get cross-domain leaks between hyperthreads anymore. Claiming that it affects all users on the internet seems like a massive over-exaggeration, as he hasn't demonstrated any kind of browser based exploit and even if such a thing did exist, it'd affect only a tiny minority of targeted users, as AFAIK many years after the introduction of Spectre nobody has ever found a specex attack in the wild (or have they?) I think the more interesting thing here is that it continues the long run of speculation bugs that always seem to be patchable in microcode. When this stuff first appeared there was the looming fear that we'd have to be regularly junking and replacing the physical chips en masse, but has that ever been necessary? AFAIK all of the bugs could be addressed via a mix of software and microcode changes, sometimes at the cost of some performance in some cases. But there's never been a bug that needed new physical silicon (except for the early versions of AMD SEV, which were genuinely jailbroken in unpatchable ways).

winternewt 3y ago
The Spectre attack had to be patched in the kernel in a way that significantly slowed down execution on Intel CPU:s: https://www.notebookcheck.net/Spectre-v2-mitigation-wreaks-havoc-on-the-performance-of-some-Intel-CPUs-as-AMD-chips-come-out-largely-unscathed.607925.0.html https://www.notebookcheck.net/Spectre-v2-mitigation-wreaks-h...
- mike_hearn 3y ago
  Yes, I think that's what I said? Every attack no matter how deep it seemed to be has been patchable in microcode, sometimes at a cost in performance. But so far nobody had to toss the physical silicon, at least not with Intel. The malleability of these chips is quite fascinating.
  - rocqua 3y ago
    The meltdown and spectre mitigations weren't patched in microcode, they were patched by changing the internal calling methods of the kernel.
    - mike_hearn 3y ago
      I think it was both. There were initial software only patches, but those were quickly superseded by microcode+kernel patches, where microcode added features the kernel enabled. IBRS or something like that.
      - cbsmith 3y ago
        It was both, and none of them completely addressed the problem. They did need new hardware for that.
  - sigotirandolas 3y ago
    On some Skylake CPUs to get full mitigations you are taking a 30% performance penalty _and_ you need to disable SMT which is often another double digit penalty. It's not a literal "toss the physical silicon", but it's getting there. In fact my thesis is that it's never a literal "toss the physical silicon" because the kernel is able to take control when switching between tasks so (with help from the microcode) it's able to wipe all potential speculation vectors (at an arbitrarily expensive cost) before switching to the next task. This also explains why SMT is unfixably broken on some processors, since by design the kernel does not intervene in the task switching on the virtual cores.
- dmatech 3y ago
  What's interesting is that the FDIV bug from 1994 could also be worked around, but Intel recalled and wrote off those processors[1]. For their latest several problems, their response was more of a "sucks to be you". While they provided microcode updates and worked with OS vendors, there were performance impacts that materially affected the value of the chips. 1. https://www.intel.com/content/www/us/en/history/history-1994-annual-report.html https://www.intel.com/content/www/us/en/history/history-1994...
  - phire 3y ago
    The software workaround for the FDIV bug required the actual userspace software to be modified and recompiled. There was a decade of pre-existing software out there that would be hard to fix, especially in the days before most people had the internet. There was nothing the OS could do to work around the bug, short of disabling the entire FPU and falling back to expensive software emulation of all floating point math. The workarounds for all these speculation bugs can be mostly applied at the operating system and/or microcode level, and is comparably cheap.
calibas 3y ago
> Claiming that it affects all users on the internet seems like a massive over-exaggeration, as he hasn't demonstrated any kind of browser based exploit and even if such a thing did exist He's saying it likely affects "everyone on the Internet" because most servers are vulnerable.
- Dylan16807 3y ago
  Most servers being vulnerable to a local attack is generally pretty boring news.
- CanaryLayout 3y ago
  If you throw in the same vulnerability that AMD has with the list from Intel I think it pretty much covers every server available for rent at Quadra.
amarshall 3y ago
> same physical core…between hyperthreads These are not the same thing. Afaik, most “vCPU” are hyperthreads, not physical cores. > I thought most had changed their schedulers years ago so you can't get cross-domain leaks between hyperthreads anymore It would be great to have a source on this.
- BeeOnRope 3y ago
  > These are not the same thing. Afaik, most “vCPU” are hyperthreads, not physical cores. OP didn't say otherwise. They are saying that public clouds do not let work from different tenants run on the same physical core (on different hyperthreads) at the same time. This doesn't prevent you from selling 1 hyperthread as 1 vCPU, it just means there are some scheduling restrictions and your smallest instance type will probably have 2 vCPUs if you have SMT-2 hardware (and that's exactly what you see on AWS outside of the burstable instance types).
vladvasiliu 3y ago
> General caveats: are there many clouds that still run workloads from different users on the same physical core? I thought most had changed their schedulers years ago so you can't get cross-domain leaks between hyperthreads anymore. Isn't this the whole point of AWS' t instances? It's my understanding that they are "shared" at the core level, or else there wouldn't be a reason for the CPU credit balance thing.
- BeeOnRope 3y ago
  They are definitely time-sliced among tenants and very possibly two tenants may run at the same time on two hardware threads on the same core: but you could have a viable burstable instance with time-slicing alone.
  - BeeOnRope 3y ago
    Nevermind, AWS explicitly documents that all instance types, including burstable, never co-locate different tenants on the same physical core at the same time: https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/the-ec2-approach-to-preventing-side-channels.html https://docs.aws.amazon.com/whitepapers/latest/security-desi...
    - anarazel 3y ago
      It only seems to document that there's group scheduling for SMT cores. But that doesn't prevent issues due to switching between customers on the same physical core, no? "It is possible, however, for two burstable performance EC2 instances to run sequentially (not simultaneously) on the same core. It is also possible for physical memory pages to be reused, remapped, and swapped in and out as virtual memory pages. However, even burstable instances never share the same core at the same time, and virtual memory pages are never shared across instances. " I only started reading the paper - so I very well might be wrong here - but it doesn't look to me like you need victim/attacker to be scheduled simultaneously on two SMT threads, but that a single core sequentially executing victim / attacker code would be vulnerable. It's possible that the cross-customer "context switch" is larger than the the vulnerable window, but I'd not want to bet on it.
pwarner 3y ago
I think most if not all cloud VMs dedicate a core to you. Well, there are some that share like the T series on AWS and I think other clouds have similar, but my bet is they can put in an extra "flush" between users to prevent cross tenant leakage. Of course cross process leakage for a single tenant is an issue, in cloud or on prem, and folks will have to decide how much they trust the processes on their machine to not become evil...
- CanaryLayout 3y ago
  This concern I also share and it's probably worth converting into layman's terms so that all computer users understand what it is. Basically the job scheduler Behavior in the OS needs to surface to the user with understandable language they can read so they can make the trade-off decision.
tedunangst 3y ago
Does Digital Ocean count as a major cloud player?
- CanaryLayout 3y ago
  Yes. And Linode. And Quadra. And OVH. A lot of people on YC are enterprisey-brained and only think there are 3 possible clouds, and then there is the rest of the planet who can't afford to park their cash at AWS and set it on fire.
kiririn 3y ago
>are there many clouds that still run workloads from different users on the same physical core? There are a vast number of VPS providers out there that aren’t AWS/GCP/Azure/etc where the answer is yes. Even the ones that sell ‘dedicated’ cores, which really just means unmetered cpu
- H8crilA 3y ago
  What about burstable instances on AWS, and whatever is the equivalent in other clouds? Hard to imagine those having a dedicated core, would probably defeat the purpose.
  - watermelon0 3y ago
    Not just burstable instances. AWS Fargate, container as a service, allows specifying 0.25 or 0.5 CPU, and I would be surprised if those weren't shared. Same probably? also applies to AWS Lambda.
    - tracker1 3y ago
      My guess is there's likely less value in trying to target those kinds of environments... Just poking random data out of lambda or low end vps neighbors is a needle in a haystack the size of the moon in terms of finding anything useful. It's more likely useful as part of a group of exploits to hit an individual, targeted system.
      - TrapLord_Rhodo 3y ago
        >Just poking random data out of lambda or low end vps neighbors is a needle in a haystack the size of the moon in terms of finding anything useful. LLM's might change that. getting a firehouse of data and asking it to classify controls against NIST 53- rev5 produces interesting results.
    - c2h5oh 3y ago
      Worse: by default it's not CPU but VCPU - a single core with multithreading = 2 VCPU

winternewt 3y ago

The Spectre attack had to be patched in the kernel in a way that significantly slowed down execution on Intel CPU:s: https://www.notebookcheck.net/Spectre-v2-mitigation-wreaks-havoc-on-the-performance-of-some-Intel-CPUs-as-AMD-chips-come-out-largely-unscathed.607925.0.html https://www.notebookcheck.net/Spectre-v2-mitigation-wreaks-h...

mike_hearn 3y ago
Yes, I think that's what I said? Every attack no matter how deep it seemed to be has been patchable in microcode, sometimes at a cost in performance. But so far nobody had to toss the physical silicon, at least not with Intel. The malleability of these chips is quite fascinating.
- rocqua 3y ago
  The meltdown and spectre mitigations weren't patched in microcode, they were patched by changing the internal calling methods of the kernel.
  - mike_hearn 3y ago
    I think it was both. There were initial software only patches, but those were quickly superseded by microcode+kernel patches, where microcode added features the kernel enabled. IBRS or something like that.
    - cbsmith 3y ago
      It was both, and none of them completely addressed the problem. They did need new hardware for that.
- sigotirandolas 3y ago
  On some Skylake CPUs to get full mitigations you are taking a 30% performance penalty _and_ you need to disable SMT which is often another double digit penalty. It's not a literal "toss the physical silicon", but it's getting there. In fact my thesis is that it's never a literal "toss the physical silicon" because the kernel is able to take control when switching between tasks so (with help from the microcode) it's able to wipe all potential speculation vectors (at an arbitrarily expensive cost) before switching to the next task. This also explains why SMT is unfixably broken on some processors, since by design the kernel does not intervene in the task switching on the virtual cores.
dmatech 3y ago
What's interesting is that the FDIV bug from 1994 could also be worked around, but Intel recalled and wrote off those processors[1]. For their latest several problems, their response was more of a "sucks to be you". While they provided microcode updates and worked with OS vendors, there were performance impacts that materially affected the value of the chips. 1. https://www.intel.com/content/www/us/en/history/history-1994-annual-report.html https://www.intel.com/content/www/us/en/history/history-1994...
- phire 3y ago
  The software workaround for the FDIV bug required the actual userspace software to be modified and recompiled. There was a decade of pre-existing software out there that would be hard to fix, especially in the days before most people had the internet. There was nothing the OS could do to work around the bug, short of disabling the entire FPU and falling back to expensive software emulation of all floating point math. The workarounds for all these speculation bugs can be mostly applied at the operating system and/or microcode level, and is comparably cheap.

calibas 3y ago

> Claiming that it affects all users on the internet seems like a massive over-exaggeration, as he hasn't demonstrated any kind of browser based exploit and even if such a thing did exist He's saying it likely affects "everyone on the Internet" because most servers are vulnerable.

Dylan16807 3y ago
Most servers being vulnerable to a local attack is generally pretty boring news.
CanaryLayout 3y ago
If you throw in the same vulnerability that AMD has with the list from Intel I think it pretty much covers every server available for rent at Quadra.

amarshall 3y ago

> same physical core…between hyperthreads These are not the same thing. Afaik, most “vCPU” are hyperthreads, not physical cores. > I thought most had changed their schedulers years ago so you can't get cross-domain leaks between hyperthreads anymore It would be great to have a source on this.

BeeOnRope 3y ago
> These are not the same thing. Afaik, most “vCPU” are hyperthreads, not physical cores. OP didn't say otherwise. They are saying that public clouds do not let work from different tenants run on the same physical core (on different hyperthreads) at the same time. This doesn't prevent you from selling 1 hyperthread as 1 vCPU, it just means there are some scheduling restrictions and your smallest instance type will probably have 2 vCPUs if you have SMT-2 hardware (and that's exactly what you see on AWS outside of the burstable instance types).

vladvasiliu 3y ago

> General caveats: are there many clouds that still run workloads from different users on the same physical core? I thought most had changed their schedulers years ago so you can't get cross-domain leaks between hyperthreads anymore. Isn't this the whole point of AWS' t instances? It's my understanding that they are "shared" at the core level, or else there wouldn't be a reason for the CPU credit balance thing.

BeeOnRope 3y ago
They are definitely time-sliced among tenants and very possibly two tenants may run at the same time on two hardware threads on the same core: but you could have a viable burstable instance with time-slicing alone.
- BeeOnRope 3y ago
  Nevermind, AWS explicitly documents that all instance types, including burstable, never co-locate different tenants on the same physical core at the same time: https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/the-ec2-approach-to-preventing-side-channels.html https://docs.aws.amazon.com/whitepapers/latest/security-desi...
  - anarazel 3y ago
    It only seems to document that there's group scheduling for SMT cores. But that doesn't prevent issues due to switching between customers on the same physical core, no? "It is possible, however, for two burstable performance EC2 instances to run sequentially (not simultaneously) on the same core. It is also possible for physical memory pages to be reused, remapped, and swapped in and out as virtual memory pages. However, even burstable instances never share the same core at the same time, and virtual memory pages are never shared across instances. " I only started reading the paper - so I very well might be wrong here - but it doesn't look to me like you need victim/attacker to be scheduled simultaneously on two SMT threads, but that a single core sequentially executing victim / attacker code would be vulnerable. It's possible that the cross-customer "context switch" is larger than the the vulnerable window, but I'd not want to bet on it.

pwarner 3y ago

I think most if not all cloud VMs dedicate a core to you. Well, there are some that share like the T series on AWS and I think other clouds have similar, but my bet is they can put in an extra "flush" between users to prevent cross tenant leakage. Of course cross process leakage for a single tenant is an issue, in cloud or on prem, and folks will have to decide how much they trust the processes on their machine to not become evil...

CanaryLayout 3y ago
This concern I also share and it's probably worth converting into layman's terms so that all computer users understand what it is. Basically the job scheduler Behavior in the OS needs to surface to the user with understandable language they can read so they can make the trade-off decision.

tedunangst 3y ago

Does Digital Ocean count as a major cloud player?

CanaryLayout 3y ago
Yes. And Linode. And Quadra. And OVH. A lot of people on YC are enterprisey-brained and only think there are 3 possible clouds, and then there is the rest of the planet who can't afford to park their cash at AWS and set it on fire.

kiririn 3y ago

>are there many clouds that still run workloads from different users on the same physical core? There are a vast number of VPS providers out there that aren’t AWS/GCP/Azure/etc where the answer is yes. Even the ones that sell ‘dedicated’ cores, which really just means unmetered cpu

H8crilA 3y ago
What about burstable instances on AWS, and whatever is the equivalent in other clouds? Hard to imagine those having a dedicated core, would probably defeat the purpose.
- watermelon0 3y ago
  Not just burstable instances. AWS Fargate, container as a service, allows specifying 0.25 or 0.5 CPU, and I would be surprised if those weren't shared. Same probably? also applies to AWS Lambda.
  - tracker1 3y ago
    My guess is there's likely less value in trying to target those kinds of environments... Just poking random data out of lambda or low end vps neighbors is a needle in a haystack the size of the moon in terms of finding anything useful. It's more likely useful as part of a group of exploits to hit an individual, targeted system.
    - TrapLord_Rhodo 3y ago
      >Just poking random data out of lambda or low end vps neighbors is a needle in a haystack the size of the moon in terms of finding anything useful. LLM's might change that. getting a firehouse of data and asking it to classify controls against NIST 53- rev5 produces interesting results.
  - c2h5oh 3y ago
    Worse: by default it's not CPU but VCPU - a single core with multithreading = 2 VCPU