4 ms·

Leaving “gifts” behind on dedicated server hosts

18 points by tbodt 9y ago

switch007 9y ago
When I was a junior sysadmin at a hosting company I raised this point in passing. It was acknowledged as a problem, but apparently so hard to fix that it was just ignored (and I didn't care enough to pursue it further). I haven't given it much thought as I've not worked with physical kit since then. I wonder if virtualization is good enough now that they could take a security stance to only deploy VMs, and deprecate the root "physical servers" product offering (i.e. you can spec out a physical server but you will always get a VM).
- db48x 9y ago
  No, they're not good enough. If you can read what's in the memory or disk images, then you can find all the interesting things there are to find.
  - richardwhiuk 9y ago
    Huh? Memory and disk should be independent for each VM? Short answer - yes using VMs should be good enough.
    - db48x 9y ago
      If I install a custom bios, it doesn't matter where the VM images are stored, or what format they are stored in. The bios can read them all.
      - switch007 9y ago
        I'm quite confused about the situation and attack vector you're describing. Could you explain? If you had direct (root) access to the hardware, you can replace firmwares/BIOS, which is what we are discussing. However, as I'm suggesting, if that machine was only ever provisioned with a VM, the bad actor is not given access to the hardware directly (hypervisor bugs aside)
        db48x 9y ago
        You can't count on the hypervisor being bug-free, and hardware moves from machine to machine. If compromised hardware gets into your machine, regardless of the route, the fact that your programs are running in a VM won't save you.
saltcured 9y ago
This issue was always taken very seriously in other circles, including recognizing that virtualization is no cure-all. Physical separation of the resource pools into equivalence classes or trust zones is just about the only sane way to recycle equipment. You might shelve and reuse equipment for the same customer or set of customers bound by mutually-covering legal agreements and mutual existential risks. You really have to think twice before putting non-trusting tenants together and consider the worst case. A proper service offering for sensitive workloads should involve decommissioning plans. Promoting or demoting hardware between classes is difficult. You have to be very confident in scrubbing writable storage to demote hardware to a less trusted class, so you don't leak privileged information. But you have to be even more confident to go the other way, so you don't allow injection of malware as postulated in the blog post. There was a time when the provider might be able to strip a machine to its bare bones, re-flashing firmware and replacing peripherals which couldn't be sanely verified, to reinitialize it as a new trusted machine. But, there are so many bits of writable firmware storage and different embedded controllers in modern machines, so it becomes futile to imagine scrubbing it all.
jason_slack 9y ago
I'm always amazed at what "rachelbythebay" writes. I've gone from occasionally reading one of her posts to wanting to read them. The topics are always so interesting and about things I don't ever find myself thinking about. I wonder if the company I work for could use such a "smart cookie"...
jason_slack 9y ago
To extend this a bit further, I once was tasked with "re-purposing" an old EMC Clariion and found that none of the data had ever been wiped. Completely left intact.

18 points by tbodt 9y ago

When I was a junior sysadmin at a hosting company I raised this point in passing. It was acknowledged as a problem, but apparently so hard to fix that it was just ignored (and I didn't care enough to pursue it further). I haven't given it much thought as I've not worked with physical kit since then. I wonder if virtualization is good enough now that they could take a security stance to only deploy VMs, and deprecate the root "physical servers" product offering (i.e. you can spec out a physical server but you will always get a VM).

db48x 9y ago
No, they're not good enough. If you can read what's in the memory or disk images, then you can find all the interesting things there are to find.
- richardwhiuk 9y ago
  Huh? Memory and disk should be independent for each VM? Short answer - yes using VMs should be good enough.
  - db48x 9y ago
    If I install a custom bios, it doesn't matter where the VM images are stored, or what format they are stored in. The bios can read them all.
    - switch007 9y ago
      I'm quite confused about the situation and attack vector you're describing. Could you explain? If you had direct (root) access to the hardware, you can replace firmwares/BIOS, which is what we are discussing. However, as I'm suggesting, if that machine was only ever provisioned with a VM, the bad actor is not given access to the hardware directly (hypervisor bugs aside)
      - db48x 9y ago
        You can't count on the hypervisor being bug-free, and hardware moves from machine to machine. If compromised hardware gets into your machine, regardless of the route, the fact that your programs are running in a VM won't save you.

saltcured 9y ago

This issue was always taken very seriously in other circles, including recognizing that virtualization is no cure-all. Physical separation of the resource pools into equivalence classes or trust zones is just about the only sane way to recycle equipment. You might shelve and reuse equipment for the same customer or set of customers bound by mutually-covering legal agreements and mutual existential risks. You really have to think twice before putting non-trusting tenants together and consider the worst case. A proper service offering for sensitive workloads should involve decommissioning plans. Promoting or demoting hardware between classes is difficult. You have to be very confident in scrubbing writable storage to demote hardware to a less trusted class, so you don't leak privileged information. But you have to be even more confident to go the other way, so you don't allow injection of malware as postulated in the blog post. There was a time when the provider might be able to strip a machine to its bare bones, re-flashing firmware and replacing peripherals which couldn't be sanely verified, to reinitialize it as a new trusted machine. But, there are so many bits of writable firmware storage and different embedded controllers in modern machines, so it becomes futile to imagine scrubbing it all.

jason_slack 9y ago

I'm always amazed at what "rachelbythebay" writes. I've gone from occasionally reading one of her posts to wanting to read them. The topics are always so interesting and about things I don't ever find myself thinking about. I wonder if the company I work for could use such a "smart cookie"...

jason_slack 9y ago

To extend this a bit further, I once was tasked with "re-purposing" an old EMC Clariion and found that none of the data had ever been wiped. Completely left intact.