6 ms·

OnMetal Performance In Early Benchmarks

64 points by jayofdoom 12y ago

cmsj 12y ago
Still not seen a discussion of how they are dealing with the huge security nightmare of direct customer hardware access.
- pfg 12y ago
  Is this any different from your regular dedicated server provider? (Honest question, I don't know much about OpenStack.)
  - hyperliner 12y ago
    I am sure a regular dedicated server provider would take a long time to get you the server. Maybe days? Not sure. I think their point is they provision a "dedicated" server in "minutes."
    - wmf 12y ago
      Does the reduced provisioning time create a huge security nightmare? I don't really see how it does.
      - hyperliner 12y ago
        Most hosters or clouds already provide you servers in seconds / minutes.
      - cmsj 12y ago
        The time to provision doesn't really affect the size of the security problem, but it does make it easier to deploy malicious firmware across a fleet of hardware, because everything is behind an API.
        _delirium 12y ago
        It's also much cheaper, since it has per-minute billing rather than per-month billing, and no setup fee. Getting temporary access to 100 Hetzner servers would cost you ~€10,000, because you have to pay a setup fee plus a minimum one month rental for each one, while it looks like it'd only cost you ~€100 to get 100x OnMetal servers for an hour. If someone stands to benefit enough, you'd be screwed in either situation, but it lowers the cost of opportunistically sowing some malware.
- old-gregg 12y ago
  What makes you think it's "huge"? How do you think the dedicated hosting industry has been operating for more than a decade? You can re-flush the bios, you can secure-erase the storage, etc.
  - cmsj 12y ago
    Just because the dedicated server hosting industry hasn't been dealing with the problem, doesn't mean it's not there :) If I flash the BIOS (or the network card firmware or the LOM device firmware or the disk controller firmware or the individual disk firmware, etc) with malicious firmware, it can lie to you when you try to reflash it later, leaving my malicious code running against later customers.
- wmf 12y ago
  This was discussed at OpenStack Summit Atlanta: https://www.openstack.org/summit/openstack-summit-atlanta-2014/session-videos/presentation/multi-tenant-bare-metal-provisioning-with-ironic https://www.openstack.org/summit/openstack-summit-atlanta-20... The main concern seems to be customers reflashing the firmware which they want to prevent with firmware signing.
  - cmsj 12y ago
    The question about that (at about 19:30 in the video) was asked by one of the Ironic developers on my team :) The answer is basically "we don't do it, we'd like to see it happen". I wasn't at that session, but I was at the Atlanta Summit and so far I haven't managed to get a good answer from Rackspace people about how they're actually tackling this in production today.
    - cmsj 12y ago
      I'd also add that firmware folk tend to be pretty conservative about updates and it's an area of machines that's not used to being hardened. you only need to own it once and you can lie forever about the attempts to reflash it with a legit firmware.
      - contingencies 12y ago
        It's even worse with IPMI. AFAIK you can own either the IPMI BMC or the BIOS, then if one is flashed, the other can replace it to stay resident in denial of your attempts to remove it. You have to coordinate flashing both at 'the same time' (which is possibly impossible) in order to get any peace of mind. That includes situations in which the IPMI BMC isn't even plugged in to a network, because it still has constantly running firmware and backdoor access to the main system. Of course, you can do half-way mitigations like dumping firmware contents and comparing checksums over time, but no single savior probably exists. A potentially strong tool here is a hardware jumper on the mainboards that says "disallow flashing at all, full stop" and another one that says "disallow reconfiguration at all, full stop" that is honored at a level of circuitry unenslaved to software. Both the BIOS and the IPMI BMC need protections on this level.
abrahamrhoffman 12y ago
Wanna see if we can salt-call the servers to spin-up on demand. O_o
- jayofdoom 12y ago
  If it can do it for normal Rackspace Cloud instances, it can do it for OnMetal instances.
jderekw 12y ago
SoftLayer has been doing this since 2006 and also offers virtual compute instances as well. Can be purchased hourly or monthly http://www.softlayer.com/bare-metal-servers http://www.softlayer.com/bare-metal-servers
- old-gregg 12y ago
  I've been SL customer and have using that prior to joining Rackspace and building OnMetal. The point isn't about "lets get rid of hypervisor", the point was to: * Lets provision as quickly as VMs (as opposed to 1hr+ on SL) * Lets engineer hardware to deliver maximum uptime via no-moving-parts design (as opposed to vanilla SuperMicro on SL) * Lets design hardware to deliver maximum unit of work per dollar (like DB transactions/second per dollar, or requests/second/dollar) as opposed to average value elsewhere. * Deriving from the above, lets just give RAM away nearly for free, and put dual 10Gig network in place, because modern apps should be mostly RAM-based. * Lets adopt standard OpenStack provisioning API, with myriad of pre-existing tools, community and ecosystem (like auto-scaling, orchestration, etc) as opposed to proprietary API. The end result is a completely different infrastructure, something akin to what OpenCompute pioneers use internally. This is how running at scale should be like. As always, I encourage skeptics to spend more than 10 seconds on a product page, because most of the time there're humans behind it, and - in this case for sure - they are way too ambitious to be spending their lives simply cloning old designs. Enjoy OnMetal, dear jberekw, it's built for critical thinkers (and skeptics! :-) like you and it's awesome - it's going to rock your world.
  - toomuchtodo 12y ago
    > * Lets provision as quickly as VMs (as opposed to 1hr+ on SL) How many Rackspace customers need a full machine in under an hour? > * Lets engineer hardware to deliver maximum uptime via no-moving-parts design (as opposed to vanilla SuperMicro on SL) So just replace spinning disk with SSDs. Unless you're replacing the last moving parts (CPU, PSU, Chassis Fans) with something solid state. > * Lets design hardware to deliver maximum unit of work per dollar (like DB transactions/second per dollar, or requests/second/dollar) as opposed to average value elsewhere. This is fair for RAM intensive workloads. Everyone else is already giving you instance SSD access. > * Deriving from the above, lets just give RAM away nearly for free, and put dual 10Gig network in place, because modern apps should be mostly RAM-based. Again, perfect for RAM intensive workloads. > * Lets adopt standard OpenStack provisioning API, with myriad of pre-existing tools, community and ecosystem (like auto-scaling, orchestration, etc) as opposed to proprietary API. Also another fair point. OpenStack (and its open platform design) is all Rackspace has to compete against AWS and Google. I don't want to say OnMetal is their "Hail Mary", but Rackspace is exploring their options in the marketplace with regards to an acquisition: http://www.bloomberg.com/news/2014-05-15/rackspace-hires-morgan-stanley-to-evaluate-options.html http://www.bloomberg.com/news/2014-05-15/rackspace-hires-mor...
snewman 12y ago
I want to want this, but the pricing just doesn't seem competitive with EC2. Am I missing something? For example, compare the "I/O" server with an EC2 i2.4xlarge instance. The Rackspace server has 128GB RAM, 3.2TB disk, and 20 cores; i2.4xlarge has 122GB, 3.2TB, and 16 -- nearly comparable. On EC2, I can buy a 3-year Light Utilization Reserved Instance for $3884, and then pay $828/month (based on 720 hours per month). After 12 months, my average cost has been $1152/month. I still have two years left on my reservation, which I can keep using, or possibly sell, so the effective cost is even lower. If I'm more certain of a 12-month server lifetime, I can buy a 1-year Heavy Utilization Reserved Instance for $7280, and then pay $447/month, for a total cost of $1054/month. On Rackspace, list price is $1800/month. Suppose my total spend is $10,000/month (list price) and I commit to 12 months. I get a 15% discount, or $1530/month. That's quite a bit more expensive than EC2, and with EC2 I'm committing less up front. A longer commitment would help, but it would also bring down the EC2 price. Can anyone poke holes in my analysis?
- hyperliner 12y ago
  You are not too far off. I would use 730 hours = 365 * 24 / 12 instead of 720. You could add little (or not so little) things like the cost of IOPS, bandwidth differential, etc. You could add a little more for the core diffs. But all that won't close the gap. Regardless, if you want the cheapest, I would go with Digital Ocean, though it is a different kind of hosting. For Rackspace, you really go when you need their support people and not just a server. I hear a lot more developers this year going to DO than last year, but I have never used them though, trying to stick with the "devil I know."
  - snewman 12y ago
    Digital Ocean is interesting, but they don't really come out cheaper for large, continuous-duty instances. Their tiny servers look great, if you don't need much storage. But on the larger instances, it's a straight $1/GB/month for SSD. With some commitment, Rackspace is more like $0.50, and Amazon around $0.35. Digital Ocean looks roughly comparable to Amazon for RAM, and slightly ahead on price-per-core (if you assume all cores are equal), but way behind on SSD. Again, this is assuming you want large instances and are going to run them 24/7 for at least 6 to 12 months -- otherwise, DO pricing starts to look a lot better.
brendangregg 12y ago
SysBench is ok when used properly, but UnixBench? They provide instructions on how you can patch and run it, using "./Run", but no warnings about what this is actually doing. See http://www.brendangregg.com/blog/2014-05-02/compilers-love-messing-with-benchmarks.html http://www.brendangregg.com/blog/2014-05-02/compilers-love-m... . I'd like to write a lot more about UnixBench, but I really don't have the time. It takes a lot of energy to refute this stuff. If I were benchmarking OnMetal vs HW virt, I'd be showing a spectrum of micro-benchmarks, from equal performance (CPU) to network I/O. I'd expect some of my results to show a ~10x difference. You would then choose/weight them depending on what matters for your intended application.
mrinterweb 12y ago
The pricing of OnMetal appears relatively competitive if not on the expensive side. I think the best way to make the case that OnMetal is worth the premium is with a series of system benchmarks comparing their solution to comparably priced AWS instances.
ColinCera 12y ago
As near as I can figure, the OnMetal servers are 3x-5x what I'd pay for dedicated servers from, say, a Hurricane reseller, and the virtual servers are more than 2x as much as comparable Linode servers, with the Linode servers offering way more SSD storage, more bandwidth, etc. Not to mention, Rackspace charges you $120 per terabyte of data transfer, while you'll get several terabytes of transfer free/included when buying from Linode or most dedicated server resellers. That seems like a hefty premium for...what? The Rackspace name? Is there any reason to believe that Rackspace has better uptime/reliability than Linode? Any reason to believe Rackspace has better hardware than Linode? Given that neither company — as with most cloud providers — provides any truly meaningful transparency, it's impossible to say. For my money, I think I'd rather spend less of it, or spend the same amount and get more/redundant servers. I'm genuinely struggling to understand the value of Rackspace.
- deleted 12y ago
  [deleted]
- quacker 12y ago
  I'm genuinely struggling to understand the value of Rackspace. See [1]: "We don't offer raw infrastructure without service." That is, support is included in the cost. Lots of people don't need support or don't want to pay for it, particularly the startup-oriented crowd here on HN. But I would be interested in a comparison of Rackspace's support to others'. Rackspace charges you $120 per terabyte of data transfer Is this in the fine print somewhere? I wouldn't be surprised, but I don't see it mentioned in [1]. 1: http://www.rackspace.com/cloud/servers/onmetal/ http://www.rackspace.com/cloud/servers/onmetal/
  - ColinCera 12y ago
    Actually, I didn't notice that support was required. I was basing my cost comparisons on Rackspaces's "Raw Infrastructure" pricing — the fact you are required to purchase support at additional cost makes their pricing even less competitive. They have a more detailed pricing page, with bandwidth charges listed near the bottom, here: http://www.rackspace.com/cloud/servers/ http://www.rackspace.com/cloud/servers/
tszming 12y ago
So is rackspace now giving up the virtualized cloud market? (Because I remember they didn't follow the recent price drop of Google/AWS/Azure?)

64 points by jayofdoom 12y ago

cmsj 12y ago

Still not seen a discussion of how they are dealing with the huge security nightmare of direct customer hardware access.

pfg 12y ago
Is this any different from your regular dedicated server provider? (Honest question, I don't know much about OpenStack.)
- hyperliner 12y ago
  I am sure a regular dedicated server provider would take a long time to get you the server. Maybe days? Not sure. I think their point is they provision a "dedicated" server in "minutes."
  - wmf 12y ago
    Does the reduced provisioning time create a huge security nightmare? I don't really see how it does.
    - hyperliner 12y ago
      Most hosters or clouds already provide you servers in seconds / minutes.
    - cmsj 12y ago
      The time to provision doesn't really affect the size of the security problem, but it does make it easier to deploy malicious firmware across a fleet of hardware, because everything is behind an API.
      - _delirium 12y ago
        It's also much cheaper, since it has per-minute billing rather than per-month billing, and no setup fee. Getting temporary access to 100 Hetzner servers would cost you ~€10,000, because you have to pay a setup fee plus a minimum one month rental for each one, while it looks like it'd only cost you ~€100 to get 100x OnMetal servers for an hour. If someone stands to benefit enough, you'd be screwed in either situation, but it lowers the cost of opportunistically sowing some malware.
old-gregg 12y ago
What makes you think it's "huge"? How do you think the dedicated hosting industry has been operating for more than a decade? You can re-flush the bios, you can secure-erase the storage, etc.
- cmsj 12y ago
  Just because the dedicated server hosting industry hasn't been dealing with the problem, doesn't mean it's not there :) If I flash the BIOS (or the network card firmware or the LOM device firmware or the disk controller firmware or the individual disk firmware, etc) with malicious firmware, it can lie to you when you try to reflash it later, leaving my malicious code running against later customers.
wmf 12y ago
This was discussed at OpenStack Summit Atlanta: https://www.openstack.org/summit/openstack-summit-atlanta-2014/session-videos/presentation/multi-tenant-bare-metal-provisioning-with-ironic https://www.openstack.org/summit/openstack-summit-atlanta-20... The main concern seems to be customers reflashing the firmware which they want to prevent with firmware signing.
- cmsj 12y ago
  The question about that (at about 19:30 in the video) was asked by one of the Ironic developers on my team :) The answer is basically "we don't do it, we'd like to see it happen". I wasn't at that session, but I was at the Atlanta Summit and so far I haven't managed to get a good answer from Rackspace people about how they're actually tackling this in production today.
  - cmsj 12y ago
    I'd also add that firmware folk tend to be pretty conservative about updates and it's an area of machines that's not used to being hardened. you only need to own it once and you can lie forever about the attempts to reflash it with a legit firmware.
    - contingencies 12y ago
      It's even worse with IPMI. AFAIK you can own either the IPMI BMC or the BIOS, then if one is flashed, the other can replace it to stay resident in denial of your attempts to remove it. You have to coordinate flashing both at 'the same time' (which is possibly impossible) in order to get any peace of mind. That includes situations in which the IPMI BMC isn't even plugged in to a network, because it still has constantly running firmware and backdoor access to the main system. Of course, you can do half-way mitigations like dumping firmware contents and comparing checksums over time, but no single savior probably exists. A potentially strong tool here is a hardware jumper on the mainboards that says "disallow flashing at all, full stop" and another one that says "disallow reconfiguration at all, full stop" that is honored at a level of circuitry unenslaved to software. Both the BIOS and the IPMI BMC need protections on this level.

abrahamrhoffman 12y ago

Wanna see if we can salt-call the servers to spin-up on demand. O_o

jayofdoom 12y ago
If it can do it for normal Rackspace Cloud instances, it can do it for OnMetal instances.

jderekw 12y ago

SoftLayer has been doing this since 2006 and also offers virtual compute instances as well. Can be purchased hourly or monthly http://www.softlayer.com/bare-metal-servers http://www.softlayer.com/bare-metal-servers

old-gregg 12y ago
I've been SL customer and have using that prior to joining Rackspace and building OnMetal. The point isn't about "lets get rid of hypervisor", the point was to: * Lets provision as quickly as VMs (as opposed to 1hr+ on SL) * Lets engineer hardware to deliver maximum uptime via no-moving-parts design (as opposed to vanilla SuperMicro on SL) * Lets design hardware to deliver maximum unit of work per dollar (like DB transactions/second per dollar, or requests/second/dollar) as opposed to average value elsewhere. * Deriving from the above, lets just give RAM away nearly for free, and put dual 10Gig network in place, because modern apps should be mostly RAM-based. * Lets adopt standard OpenStack provisioning API, with myriad of pre-existing tools, community and ecosystem (like auto-scaling, orchestration, etc) as opposed to proprietary API. The end result is a completely different infrastructure, something akin to what OpenCompute pioneers use internally. This is how running at scale should be like. As always, I encourage skeptics to spend more than 10 seconds on a product page, because most of the time there're humans behind it, and - in this case for sure - they are way too ambitious to be spending their lives simply cloning old designs. Enjoy OnMetal, dear jberekw, it's built for critical thinkers (and skeptics! :-) like you and it's awesome - it's going to rock your world.
- toomuchtodo 12y ago
  > * Lets provision as quickly as VMs (as opposed to 1hr+ on SL) How many Rackspace customers need a full machine in under an hour? > * Lets engineer hardware to deliver maximum uptime via no-moving-parts design (as opposed to vanilla SuperMicro on SL) So just replace spinning disk with SSDs. Unless you're replacing the last moving parts (CPU, PSU, Chassis Fans) with something solid state. > * Lets design hardware to deliver maximum unit of work per dollar (like DB transactions/second per dollar, or requests/second/dollar) as opposed to average value elsewhere. This is fair for RAM intensive workloads. Everyone else is already giving you instance SSD access. > * Deriving from the above, lets just give RAM away nearly for free, and put dual 10Gig network in place, because modern apps should be mostly RAM-based. Again, perfect for RAM intensive workloads. > * Lets adopt standard OpenStack provisioning API, with myriad of pre-existing tools, community and ecosystem (like auto-scaling, orchestration, etc) as opposed to proprietary API. Also another fair point. OpenStack (and its open platform design) is all Rackspace has to compete against AWS and Google. I don't want to say OnMetal is their "Hail Mary", but Rackspace is exploring their options in the marketplace with regards to an acquisition: http://www.bloomberg.com/news/2014-05-15/rackspace-hires-morgan-stanley-to-evaluate-options.html http://www.bloomberg.com/news/2014-05-15/rackspace-hires-mor...

snewman 12y ago

I want to want this, but the pricing just doesn't seem competitive with EC2. Am I missing something? For example, compare the "I/O" server with an EC2 i2.4xlarge instance. The Rackspace server has 128GB RAM, 3.2TB disk, and 20 cores; i2.4xlarge has 122GB, 3.2TB, and 16 -- nearly comparable. On EC2, I can buy a 3-year Light Utilization Reserved Instance for $3884, and then pay $828/month (based on 720 hours per month). After 12 months, my average cost has been $1152/month. I still have two years left on my reservation, which I can keep using, or possibly sell, so the effective cost is even lower. If I'm more certain of a 12-month server lifetime, I can buy a 1-year Heavy Utilization Reserved Instance for $7280, and then pay $447/month, for a total cost of $1054/month. On Rackspace, list price is $1800/month. Suppose my total spend is $10,000/month (list price) and I commit to 12 months. I get a 15% discount, or $1530/month. That's quite a bit more expensive than EC2, and with EC2 I'm committing less up front. A longer commitment would help, but it would also bring down the EC2 price. Can anyone poke holes in my analysis?

hyperliner 12y ago
You are not too far off. I would use 730 hours = 365 * 24 / 12 instead of 720. You could add little (or not so little) things like the cost of IOPS, bandwidth differential, etc. You could add a little more for the core diffs. But all that won't close the gap. Regardless, if you want the cheapest, I would go with Digital Ocean, though it is a different kind of hosting. For Rackspace, you really go when you need their support people and not just a server. I hear a lot more developers this year going to DO than last year, but I have never used them though, trying to stick with the "devil I know."
- snewman 12y ago
  Digital Ocean is interesting, but they don't really come out cheaper for large, continuous-duty instances. Their tiny servers look great, if you don't need much storage. But on the larger instances, it's a straight $1/GB/month for SSD. With some commitment, Rackspace is more like $0.50, and Amazon around $0.35. Digital Ocean looks roughly comparable to Amazon for RAM, and slightly ahead on price-per-core (if you assume all cores are equal), but way behind on SSD. Again, this is assuming you want large instances and are going to run them 24/7 for at least 6 to 12 months -- otherwise, DO pricing starts to look a lot better.

brendangregg 12y ago

SysBench is ok when used properly, but UnixBench? They provide instructions on how you can patch and run it, using "./Run", but no warnings about what this is actually doing. See http://www.brendangregg.com/blog/2014-05-02/compilers-love-messing-with-benchmarks.html http://www.brendangregg.com/blog/2014-05-02/compilers-love-m... . I'd like to write a lot more about UnixBench, but I really don't have the time. It takes a lot of energy to refute this stuff. If I were benchmarking OnMetal vs HW virt, I'd be showing a spectrum of micro-benchmarks, from equal performance (CPU) to network I/O. I'd expect some of my results to show a ~10x difference. You would then choose/weight them depending on what matters for your intended application.

mrinterweb 12y ago

The pricing of OnMetal appears relatively competitive if not on the expensive side. I think the best way to make the case that OnMetal is worth the premium is with a series of system benchmarks comparing their solution to comparably priced AWS instances.

ColinCera 12y ago

As near as I can figure, the OnMetal servers are 3x-5x what I'd pay for dedicated servers from, say, a Hurricane reseller, and the virtual servers are more than 2x as much as comparable Linode servers, with the Linode servers offering way more SSD storage, more bandwidth, etc. Not to mention, Rackspace charges you $120 per terabyte of data transfer, while you'll get several terabytes of transfer free/included when buying from Linode or most dedicated server resellers. That seems like a hefty premium for...what? The Rackspace name? Is there any reason to believe that Rackspace has better uptime/reliability than Linode? Any reason to believe Rackspace has better hardware than Linode? Given that neither company — as with most cloud providers — provides any truly meaningful transparency, it's impossible to say. For my money, I think I'd rather spend less of it, or spend the same amount and get more/redundant servers. I'm genuinely struggling to understand the value of Rackspace.

deleted 12y ago
[deleted]
quacker 12y ago
I'm genuinely struggling to understand the value of Rackspace. See [1]: "We don't offer raw infrastructure without service." That is, support is included in the cost. Lots of people don't need support or don't want to pay for it, particularly the startup-oriented crowd here on HN. But I would be interested in a comparison of Rackspace's support to others'. Rackspace charges you $120 per terabyte of data transfer Is this in the fine print somewhere? I wouldn't be surprised, but I don't see it mentioned in [1]. 1: http://www.rackspace.com/cloud/servers/onmetal/ http://www.rackspace.com/cloud/servers/onmetal/
- ColinCera 12y ago
  Actually, I didn't notice that support was required. I was basing my cost comparisons on Rackspaces's "Raw Infrastructure" pricing — the fact you are required to purchase support at additional cost makes their pricing even less competitive. They have a more detailed pricing page, with bandwidth charges listed near the bottom, here: http://www.rackspace.com/cloud/servers/ http://www.rackspace.com/cloud/servers/

tszming 12y ago

So is rackspace now giving up the virtualized cloud market? (Because I remember they didn't follow the recent price drop of Google/AWS/Azure?)