5 ms·

These CPU flaws make it seem as if virtualization in the data center is becoming really, really dangerous. If these exploits continue to appear, the only way fo

These CPU flaws make it seem as if virtualization in the data center is becoming really, really dangerous. If these exploits continue to appear, the only way forward would be dedicated machines for each application of each customer. Essentially, this might be killing the cloud by 1000 papercuts because it loses efficiency and cost effectiveness and locally hosted hardware does not necessarily have to have all mitigations applied (no potential of a unknown 3rd party code deployed to the same server).

ljlolel 7y ago
it increases cloud revenues because of slow downs in CPU, and people can't move off cloud because they're locked in and can't hire datacenter engineers anyway
- gnode 7y ago
  Ultimately cloud providers don't want revenue, they want profit. Except in some perverse cases (like cost-plus-percentage contracts), it's not generally in a business's interests for their costs to go up. Even if there's no opportunity to switch away, eventually you bleed your customers dry and put them out of business. You will typically always aim to price your offering at the equilibrium point where loss of custom increases faster than the increase in profit, and vice versa. One situation in which an increase in your costs can be good, is if the same increase applies more to your competition. But, in this case multi-tenant cloud is hit harder than the competing alternative of private infrastructure.
- chatmasta 7y ago
  This is an important point that I noticed around the time of Spectre/Meltdown as well. The mitigation for those bugs caused an average of 30% CPU slowdown, meaning it took 30% more CPU cycles to perform the same work as it did prior to the mitigation. If a cloud provider rolls that out to every server, then every customer's bill for CPU usage should increase by roughly 30%. Am I totally misunderstanding this? Someone please correct me if I'm wrong.
gnode 7y ago
> dedicated machines for each application of each customer. I don't think you don't need to go this far. You can probably get away with circuit switching small blocks of hardware, and fully resetting them between handovers. Although you'd have to ensure sufficient randomisation / granularity to destroy side channels in the switching logic.
Tharkun 7y ago
Many years ago, OpenBSD's Theo De Raadt made a sneer at virtualization, saying something the lines of "they can't even build a secure system, let alone a secure virtualized system". I can't remember who he was referring to specifically, but we've certainly been seeing a lot of similar vulnerabilities.
- gmueckl 7y ago
  Would be cool to see a source for that one. Theo de Raadt is a hardliner whom I don't always agree with, but I'd like to know how visionary he actually was in this case (quite a bit by the looks of it).
  - t1amat 7y ago
    I believe this is the source: https://marc.info/?l=openbsd-misc&m=119318909016582 https://marc.info/?l=openbsd-misc&m=119318909016582
  - jancsika 7y ago
    I'm not a security person but I wanna practice trying to sum up his points: 1. There's no way in hell that a bunch of VMs running on one physical server is more secure than a bunch of different physical servers each running an OS. If there were architectural hooks for those VMs to provide additional security beyond what the host OS provides, then an OS like OpenBSD would already be making use of it. 2. Running a bunch of VMs on a single physical machine is certainly cheaper. 3. People who are in favor of the cost-cutting are claiming that there's a security benefit to sell more stuff. Am I right? If so, how does that stance jibe with the research that Qubes is based on?
    - fwip 7y ago
      I think the argument VM-sellers make is that it's more secure than running a bunch of colocated code on the same machine without VMs, not that it's more secure than distinct physical systems.
      - 7y ago
dooglius 7y ago
Well, dedicated machines for each security domain for each customer, a lot of the time it's fine for many applications to be in the same security domain.
- gnode 7y ago
  Even this isn't enough. Sometimes mutually untrusted parties must exchange data (say you're running a trading platform, or a social network). You have to ensure every point of interaction between such parties is immune to timing attacks.
  - gmueckl 7y ago
    In theory, yes. But getting statistically meaningful data on sub-ms timing variations on a jittery connection with both round trip and jitter orders of magnitudes larger is hard... it would be a very, very slow attack and probably impractical in most cases.
dchest 7y ago
Except most companies don't care about security of the cloud apart from the magical "compliance" and that they are on the same cloud as everyone else.
Thaxll 7y ago
You should use large instance that don't share the same CPU socket, on AWS for example it would be c5.9xlarge and above.
peterwwillis 7y ago
The point of virtualization isn't to add security. It gives you functionality you just cannot have otherwise, and the cloud enables you to scale in a way that is impossible otherwise. If there are security holes, they get patched and the market moves on. It's not just going to abandon either the cloud or virtualization.
- ct520 7y ago
  I don't agree with that statement. As a stand-alone physical machine is expected to be secure in its own enclave an I think it's within reason a virtual one would have the same expectations.
NathanKP 7y ago
Interesting to note that AWS has been working on their own custom silicon, such as the announced Arm based AWS Graviton powered machines. We will most likely see a continued divergence between "consumer silicon" which is designed for speed in a single tenant environment on your local desktop or laptop, and "cloud silicon" which is optimized to protect virtualization, be power efficient, etc. I'd predict that this will actually lead to increased efficiency and lower prices of cloud resources rather than the "death by a 1000 cuts" that you are proposing.
dboreham 7y ago
There are bare metal "cloud" providers such as Packet.net where you get the click n deploy convenience of the cloud but a physical machine. They have quite small machines in the inventory that are close to price competitive with VMs. Even Amazon has this bare metal capability FWIW, but afaik only for big expensive machines.

ljlolel 7y ago

it increases cloud revenues because of slow downs in CPU, and people can't move off cloud because they're locked in and can't hire datacenter engineers anyway

gnode 7y ago
Ultimately cloud providers don't want revenue, they want profit. Except in some perverse cases (like cost-plus-percentage contracts), it's not generally in a business's interests for their costs to go up. Even if there's no opportunity to switch away, eventually you bleed your customers dry and put them out of business. You will typically always aim to price your offering at the equilibrium point where loss of custom increases faster than the increase in profit, and vice versa. One situation in which an increase in your costs can be good, is if the same increase applies more to your competition. But, in this case multi-tenant cloud is hit harder than the competing alternative of private infrastructure.
chatmasta 7y ago
This is an important point that I noticed around the time of Spectre/Meltdown as well. The mitigation for those bugs caused an average of 30% CPU slowdown, meaning it took 30% more CPU cycles to perform the same work as it did prior to the mitigation. If a cloud provider rolls that out to every server, then every customer's bill for CPU usage should increase by roughly 30%. Am I totally misunderstanding this? Someone please correct me if I'm wrong.

gnode 7y ago

> dedicated machines for each application of each customer. I don't think you don't need to go this far. You can probably get away with circuit switching small blocks of hardware, and fully resetting them between handovers. Although you'd have to ensure sufficient randomisation / granularity to destroy side channels in the switching logic.

Tharkun 7y ago

Many years ago, OpenBSD's Theo De Raadt made a sneer at virtualization, saying something the lines of "they can't even build a secure system, let alone a secure virtualized system". I can't remember who he was referring to specifically, but we've certainly been seeing a lot of similar vulnerabilities.

gmueckl 7y ago
Would be cool to see a source for that one. Theo de Raadt is a hardliner whom I don't always agree with, but I'd like to know how visionary he actually was in this case (quite a bit by the looks of it).
- t1amat 7y ago
  I believe this is the source: https://marc.info/?l=openbsd-misc&m=119318909016582 https://marc.info/?l=openbsd-misc&m=119318909016582
- jancsika 7y ago
  I'm not a security person but I wanna practice trying to sum up his points: 1. There's no way in hell that a bunch of VMs running on one physical server is more secure than a bunch of different physical servers each running an OS. If there were architectural hooks for those VMs to provide additional security beyond what the host OS provides, then an OS like OpenBSD would already be making use of it. 2. Running a bunch of VMs on a single physical machine is certainly cheaper. 3. People who are in favor of the cost-cutting are claiming that there's a security benefit to sell more stuff. Am I right? If so, how does that stance jibe with the research that Qubes is based on?
  - fwip 7y ago
    I think the argument VM-sellers make is that it's more secure than running a bunch of colocated code on the same machine without VMs, not that it's more secure than distinct physical systems.
    - 7y ago

dooglius 7y ago

Well, dedicated machines for each security domain for each customer, a lot of the time it's fine for many applications to be in the same security domain.

gnode 7y ago
Even this isn't enough. Sometimes mutually untrusted parties must exchange data (say you're running a trading platform, or a social network). You have to ensure every point of interaction between such parties is immune to timing attacks.
- gmueckl 7y ago
  In theory, yes. But getting statistically meaningful data on sub-ms timing variations on a jittery connection with both round trip and jitter orders of magnitudes larger is hard... it would be a very, very slow attack and probably impractical in most cases.

dchest 7y ago

Except most companies don't care about security of the cloud apart from the magical "compliance" and that they are on the same cloud as everyone else.

Thaxll 7y ago

You should use large instance that don't share the same CPU socket, on AWS for example it would be c5.9xlarge and above.

peterwwillis 7y ago

The point of virtualization isn't to add security. It gives you functionality you just cannot have otherwise, and the cloud enables you to scale in a way that is impossible otherwise. If there are security holes, they get patched and the market moves on. It's not just going to abandon either the cloud or virtualization.

ct520 7y ago
I don't agree with that statement. As a stand-alone physical machine is expected to be secure in its own enclave an I think it's within reason a virtual one would have the same expectations.

NathanKP 7y ago

Interesting to note that AWS has been working on their own custom silicon, such as the announced Arm based AWS Graviton powered machines. We will most likely see a continued divergence between "consumer silicon" which is designed for speed in a single tenant environment on your local desktop or laptop, and "cloud silicon" which is optimized to protect virtualization, be power efficient, etc. I'd predict that this will actually lead to increased efficiency and lower prices of cloud resources rather than the "death by a 1000 cuts" that you are proposing.

dboreham 7y ago

There are bare metal "cloud" providers such as Packet.net where you get the click n deploy convenience of the cloud but a physical machine. They have quite small machines in the inventory that are close to price competitive with VMs. Even Amazon has this bare metal capability FWIW, but afaik only for big expensive machines.