4 ms·

Still not seen a discussion of how they are dealing with the huge security nightmare of direct customer hardware access.

by cmsj 12y ago

pfg 12y ago
Is this any different from your regular dedicated server provider? (Honest question, I don't know much about OpenStack.)
- hyperliner 12y ago
  I am sure a regular dedicated server provider would take a long time to get you the server. Maybe days? Not sure. I think their point is they provision a "dedicated" server in "minutes."
  - wmf 12y ago
    Does the reduced provisioning time create a huge security nightmare? I don't really see how it does.
    - hyperliner 12y ago
      Most hosters or clouds already provide you servers in seconds / minutes.
    - cmsj 12y ago
      The time to provision doesn't really affect the size of the security problem, but it does make it easier to deploy malicious firmware across a fleet of hardware, because everything is behind an API.
      - _delirium 12y ago
        It's also much cheaper, since it has per-minute billing rather than per-month billing, and no setup fee. Getting temporary access to 100 Hetzner servers would cost you ~€10,000, because you have to pay a setup fee plus a minimum one month rental for each one, while it looks like it'd only cost you ~€100 to get 100x OnMetal servers for an hour. If someone stands to benefit enough, you'd be screwed in either situation, but it lowers the cost of opportunistically sowing some malware.
old-gregg 12y ago
What makes you think it's "huge"? How do you think the dedicated hosting industry has been operating for more than a decade? You can re-flush the bios, you can secure-erase the storage, etc.
- cmsj 12y ago
  Just because the dedicated server hosting industry hasn't been dealing with the problem, doesn't mean it's not there :) If I flash the BIOS (or the network card firmware or the LOM device firmware or the disk controller firmware or the individual disk firmware, etc) with malicious firmware, it can lie to you when you try to reflash it later, leaving my malicious code running against later customers.
wmf 12y ago
This was discussed at OpenStack Summit Atlanta: https://www.openstack.org/summit/openstack-summit-atlanta-2014/session-videos/presentation/multi-tenant-bare-metal-provisioning-with-ironic https://www.openstack.org/summit/openstack-summit-atlanta-20... The main concern seems to be customers reflashing the firmware which they want to prevent with firmware signing.
- cmsj 12y ago
  The question about that (at about 19:30 in the video) was asked by one of the Ironic developers on my team :) The answer is basically "we don't do it, we'd like to see it happen". I wasn't at that session, but I was at the Atlanta Summit and so far I haven't managed to get a good answer from Rackspace people about how they're actually tackling this in production today.
  - cmsj 12y ago
    I'd also add that firmware folk tend to be pretty conservative about updates and it's an area of machines that's not used to being hardened. you only need to own it once and you can lie forever about the attempts to reflash it with a legit firmware.
    - contingencies 12y ago
      It's even worse with IPMI. AFAIK you can own either the IPMI BMC or the BIOS, then if one is flashed, the other can replace it to stay resident in denial of your attempts to remove it. You have to coordinate flashing both at 'the same time' (which is possibly impossible) in order to get any peace of mind. That includes situations in which the IPMI BMC isn't even plugged in to a network, because it still has constantly running firmware and backdoor access to the main system. Of course, you can do half-way mitigations like dumping firmware contents and comparing checksums over time, but no single savior probably exists. A potentially strong tool here is a hardware jumper on the mainboards that says "disallow flashing at all, full stop" and another one that says "disallow reconfiguration at all, full stop" that is honored at a level of circuitry unenslaved to software. Both the BIOS and the IPMI BMC need protections on this level.

Still not seen a discussion of how they are dealing with the huge security nightmare of direct customer hardware access.

by cmsj 12y ago

Still not seen a discussion of how they are dealing with the huge security nightmare of direct customer hardware access.

pfg 12y ago

Is this any different from your regular dedicated server provider? (Honest question, I don't know much about OpenStack.)

hyperliner 12y ago
I am sure a regular dedicated server provider would take a long time to get you the server. Maybe days? Not sure. I think their point is they provision a "dedicated" server in "minutes."
- wmf 12y ago
  Does the reduced provisioning time create a huge security nightmare? I don't really see how it does.
  - hyperliner 12y ago
    Most hosters or clouds already provide you servers in seconds / minutes.
  - cmsj 12y ago
    The time to provision doesn't really affect the size of the security problem, but it does make it easier to deploy malicious firmware across a fleet of hardware, because everything is behind an API.
    - _delirium 12y ago
      It's also much cheaper, since it has per-minute billing rather than per-month billing, and no setup fee. Getting temporary access to 100 Hetzner servers would cost you ~€10,000, because you have to pay a setup fee plus a minimum one month rental for each one, while it looks like it'd only cost you ~€100 to get 100x OnMetal servers for an hour. If someone stands to benefit enough, you'd be screwed in either situation, but it lowers the cost of opportunistically sowing some malware.

old-gregg 12y ago

What makes you think it's "huge"? How do you think the dedicated hosting industry has been operating for more than a decade? You can re-flush the bios, you can secure-erase the storage, etc.

cmsj 12y ago
Just because the dedicated server hosting industry hasn't been dealing with the problem, doesn't mean it's not there :) If I flash the BIOS (or the network card firmware or the LOM device firmware or the disk controller firmware or the individual disk firmware, etc) with malicious firmware, it can lie to you when you try to reflash it later, leaving my malicious code running against later customers.

wmf 12y ago

This was discussed at OpenStack Summit Atlanta: https://www.openstack.org/summit/openstack-summit-atlanta-2014/session-videos/presentation/multi-tenant-bare-metal-provisioning-with-ironic https://www.openstack.org/summit/openstack-summit-atlanta-20... The main concern seems to be customers reflashing the firmware which they want to prevent with firmware signing.

cmsj 12y ago
The question about that (at about 19:30 in the video) was asked by one of the Ironic developers on my team :) The answer is basically "we don't do it, we'd like to see it happen". I wasn't at that session, but I was at the Atlanta Summit and so far I haven't managed to get a good answer from Rackspace people about how they're actually tackling this in production today.
- cmsj 12y ago
  I'd also add that firmware folk tend to be pretty conservative about updates and it's an area of machines that's not used to being hardened. you only need to own it once and you can lie forever about the attempts to reflash it with a legit firmware.
  - contingencies 12y ago
    It's even worse with IPMI. AFAIK you can own either the IPMI BMC or the BIOS, then if one is flashed, the other can replace it to stay resident in denial of your attempts to remove it. You have to coordinate flashing both at 'the same time' (which is possibly impossible) in order to get any peace of mind. That includes situations in which the IPMI BMC isn't even plugged in to a network, because it still has constantly running firmware and backdoor access to the main system. Of course, you can do half-way mitigations like dumping firmware contents and comparing checksums over time, but no single savior probably exists. A potentially strong tool here is a hardware jumper on the mainboards that says "disallow flashing at all, full stop" and another one that says "disallow reconfiguration at all, full stop" that is honored at a level of circuitry unenslaved to software. Both the BIOS and the IPMI BMC need protections on this level.