6 ms·

This is one of these things where virtualisation can help even more. For example, VMware can dynamically put servers in standby mode when demand is low and powe

This is one of these things where virtualisation can help even more. For example, VMware can dynamically put servers in standby mode when demand is low and power them up again when needed: http://www.vmware.com/products/vsphere/features/drs-dpm http://www.vmware.com/products/vsphere/features/drs-dpm

SEJeff 12y ago
You know that would be trivial to do with bare metal and out of band management cards like Dell Dracs, IBM RSA cards, HP ILOs, or generic IPMI BMCs, right? Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance. Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial. In fact, the same ideas are behind Rackspace's "OnMetal" initiative: http://www.rackspace.com/cloud/servers/onmetal/ http://www.rackspace.com/cloud/servers/onmetal/
- wmf 12y ago
  If you want to power manage your pets, VMware makes sense. But yeah, there's a reason Facebook doesn't use virtualization.
  - reeze_xia 12y ago
    I am curious why Facebook don't use? If they do use virtualization, the fact, 0 request lead to low power does not correct any more, for example linux container, other container may accept requests, if so they have to dispatch request cross all containers and servers.
    - otterley 12y ago
      The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.
      - reeze_xia 12y ago
        How about lightweight virtualization like linux container, docker?
        otterley 12y ago
        That's not virtualization; it's namespace isolation. There's a small performance impact if you're using NAT, but otherwise the kernel networking stack is used, so there's no performance penalty.
        reeze_xia 12y ago
        Yes, indeed. I mean the optimization of energy saving seems not suitable for the condition of namespace isolation, since you can't control the other containers requests. if we have to, we need to dispatch request from server perspective but not container.
      - vicaya 12y ago
        This is wrong as well. VMware standard and distributed virtual switch is in hypervisor kernel.
        otterley 12y ago
        How's VMWare's TCP new-session-switching-rate compared to Xen's OVS? Any benchmarks showing a significant improvement?
      - wmf 12y ago
        We just did some benchmarks on this topic: https://news.ycombinator.com/item?id=8146536 https://news.ycombinator.com/item?id=8146536
- devicenull 12y ago
  Getting IPMI controllers to do what you want is anything but trivial :)
  - deleted 12y ago
    [deleted]
  - thrownaway2424 12y ago
    Seriously. A person who would advocate using IPMI at scale has either never owned an IPMI card or has never worked at scale or both. The just don't work, and they erase whatever power savings you're trying to achieve.
    - wmf 12y ago
      Real servers can't be bought without IPMI and AFAIK the BMC cannot be turned off, so it's probably not worth worrying about BMC power if there's nothing you can do about it.
      - thrownaway2424 12y ago
        Sure, but as you're aware facebook, google, et al don't buy "real servers", they buy servers that actually meet their requirements. That's why "real vendors" like HP have missed the boat on selling millions of servers into the cloud.
        wmf 12y ago
        Speaking of Facebook specifically, the evolution is interesting. They replaced BMCs with the reboot-on-LAN hack but then their next motherboard version had BMCs again. It would be interesting to hear the story behind that.
    - SEJeff 12y ago
      I actually advocated using the vendor specific ones with the BMC being if you use shitty hardware. It works ok, but is absolutely suboptimal.
- acdha 12y ago
  > Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial. You do, I'm sure, because you've invested a lot of time learning it and building tooling around it. For anyone who isn't a full-time sysadmin, debugging all of the many and various quirks in management hardware is a major time sink versus scripting a VM server's API.
- vicaya 12y ago
  "Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance." This is BS. What if you have 3 physical servers with 30% utilization? DRS can _seamlessly_ consolidate _arbitrary_ application VMs to one server and shutdown the rest. With bare metal, only certain specifically designed workloads (stateless web farms and some distributed systems etc.) can be moved easily. It seems that dismissing virtualization out of sheer ignorance is a fad these days. Virtualization provides important hardware abstraction to a much wider variety of workloads.

SEJeff 12y ago

You know that would be trivial to do with bare metal and out of band management cards like Dell Dracs, IBM RSA cards, HP ILOs, or generic IPMI BMCs, right? Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance. Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial. In fact, the same ideas are behind Rackspace's "OnMetal" initiative: http://www.rackspace.com/cloud/servers/onmetal/ http://www.rackspace.com/cloud/servers/onmetal/

wmf 12y ago
If you want to power manage your pets, VMware makes sense. But yeah, there's a reason Facebook doesn't use virtualization.
- reeze_xia 12y ago
  I am curious why Facebook don't use? If they do use virtualization, the fact, 0 request lead to low power does not correct any more, for example linux container, other container may accept requests, if so they have to dispatch request cross all containers and servers.
  - otterley 12y ago
    The Achilles heel of virtualization is networking. All of the hypervisors out there (VMWare, Xen, KVM) have user-space software switch implementations that dramatically reduce the throughput of TCP session creation. As a consequence you lose a significant amount of hardware potential to serve HTTP connections.
    - reeze_xia 12y ago
      How about lightweight virtualization like linux container, docker?
      - otterley 12y ago
        That's not virtualization; it's namespace isolation. There's a small performance impact if you're using NAT, but otherwise the kernel networking stack is used, so there's no performance penalty.
        reeze_xia 12y ago
        Yes, indeed. I mean the optimization of energy saving seems not suitable for the condition of namespace isolation, since you can't control the other containers requests. if we have to, we need to dispatch request from server perspective but not container.
    - vicaya 12y ago
      This is wrong as well. VMware standard and distributed virtual switch is in hypervisor kernel.
      - otterley 12y ago
        How's VMWare's TCP new-session-switching-rate compared to Xen's OVS? Any benchmarks showing a significant improvement?
    - wmf 12y ago
      We just did some benchmarks on this topic: https://news.ycombinator.com/item?id=8146536 https://news.ycombinator.com/item?id=8146536
devicenull 12y ago
Getting IPMI controllers to do what you want is anything but trivial :)
- deleted 12y ago
  [deleted]
- thrownaway2424 12y ago
  Seriously. A person who would advocate using IPMI at scale has either never owned an IPMI card or has never worked at scale or both. The just don't work, and they erase whatever power savings you're trying to achieve.
  - wmf 12y ago
    Real servers can't be bought without IPMI and AFAIK the BMC cannot be turned off, so it's probably not worth worrying about BMC power if there's nothing you can do about it.
    - thrownaway2424 12y ago
      Sure, but as you're aware facebook, google, et al don't buy "real servers", they buy servers that actually meet their requirements. That's why "real vendors" like HP have missed the boat on selling millions of servers into the cloud.
      - wmf 12y ago
        Speaking of Facebook specifically, the evolution is interesting. They replaced BMCs with the reboot-on-LAN hack but then their next motherboard version had BMCs again. It would be interesting to hear the story behind that.
  - SEJeff 12y ago
    I actually advocated using the vendor specific ones with the BMC being if you use shitty hardware. It works ok, but is absolutely suboptimal.
acdha 12y ago
> Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial. You do, I'm sure, because you've invested a lot of time learning it and building tooling around it. For anyone who isn't a full-time sysadmin, debugging all of the many and various quirks in management hardware is a major time sink versus scripting a VM server's API.
vicaya 12y ago
"Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance." This is BS. What if you have 3 physical servers with 30% utilization? DRS can _seamlessly_ consolidate _arbitrary_ application VMs to one server and shutdown the rest. With bare metal, only certain specifically designed workloads (stateless web farms and some distributed systems etc.) can be moved easily. It seems that dismissing virtualization out of sheer ignorance is a fad these days. Virtualization provides important hardware abstraction to a much wider variety of workloads.