9 ms·

Comparing Nginx performance in bare metal and virtual environments

132 points by el_duderino 5y ago

marginalia_nu 5y ago
I saw quite a significant performance and resource usage benefit from migrating away from a virtualized kubernetes environment to bare metal debian, so their findings align well with my anecdata as well.
blowski 5y ago
I recently moved a smallish business application from two bare-metal servers onto Azure VMs. It's a standard PHP 7.4 application, MySQL, Redis, nginx. Despite the VMs costing more and having twice the spec of the bare-metal servers, it has consistently performed slower and less reliably throughout the stack. The client's budget is too small to spend much time looking into it. Instead, they upped the spec even further, and what used to cost £600 per month now costs £2000.
- hhw 5y ago
  (Disclaimer) As a bare metal provider, I hope more people become aware what I've been saying for years: cloud is great for scaling down, but not that great for scaling up. If you want to have lots of VM's that don't warrant their own hardware that you can conveniently turn up and down, then cloud is fantastic. If you have a big application that needs to scale, you can get further vertically with bare metal, and if you need to scale horizontally, you need to optimize better higher up in the stack anyway, and the much lower cost for equivalent resources (without even taking any virtualization performance hit into account), more flexibility and thus more/better fitted performance of bare metal should have the clear advantage.
  - stingraycharles 5y ago
    FYI I clicked on “get a dedicated server” on your site and ended up getting a 404 https://astuteinternet.com/services/dedicated-servers-order https://astuteinternet.com/services/dedicated-servers-order {"error":"URI Not Found"}
    - grepfru_it 5y ago
      >Starting at $199/mo Sounds pricey :)
      - duskwuff 5y ago
        Canadian dollars, so ~$160 USD. But that's still extremely high for a quad-core CPU from 2013, 8 GB RAM, no solid-state storage, and capped bandwidth.
        hhw 5y ago
        Sorry, the website is pretty outdated. We're almost exclusively rolling out AMD EPYC3's these days, and we'd price any of those older configurations much lower than what the website lists them at. Nobody, other than spammers, ever orders through our website (although to be fair, our website may be to blame for that also). We get all of our business through word of mouth, and keep busy enough on that alone, so the website hasn't been a priority.
        duskwuff 5y ago
        You really ought to take your price lists down if they're that out of date. As it stands, they're probably driving some potential customers away -- even ones who heard of you through word of mouth, but decided to do a bit of research before proceeding.
      - RNCTX 5y ago
        Not really. Before there was a "cloud" in the early 2000s, we were paying anywhere from $100 (cheap, low spec machines) to $199 (new hardware, more ram, faster disks) per month for rented bare metal from places like Servermatrix, Softlayer, etc. The going rate also typically included anywhere from 1TB to 2TB of egress, as well. Another way of looking at it for the low end: How many things do you run on a single bare metal host versus an equivalent amount of discrete "services"?
      - FpUser 5y ago
        Look at Hetzner / OVH. I got incredibly good deals from them on dedicated servers. I think I am paying around $150 Canadian for AMD 16 cores, 128GB ECC RAM, do not remember storage. Updated: I also run some things right from my home office since I have symmetric 1Gbps fiber. For that I paid around $4000 Cdn for off lease 20 core 3.4GHz 512GB RAM server from HP
  - laumars 5y ago
    > what I've been saying for years: cloud is great for scaling down, but not that great for scaling up. Yes and no. The cloud isn’t cheap at running any lift and shift type project. Where the cloud comes into its own is serverless and SaaS. If you have an application that’s a typical VM farm and you want to host it in the cloud, then you should at least first pause and identify how, if at all, your application can be re-architected to be more “cloudy”. Not doing this is usually the first mistake people make when deploying to the cloud.
- speedgoose 5y ago
  Did you throw money at the storage layer? The IOs are ridiculously slow by default.
- stingraycharles 5y ago
  I think “the cloud” never claimed to be cheaper, though. Its promise is mainly that you’ll offset a lot of risks with it against a higher price. And of course the workflow with the cloud is very different, with virtual machine images and whatnot. Whether that’s worth the price is up debate, though. I hope we’ll get more mature bare metal management software, perhaps even standardized in some remote management hardware so you can flash entire bare metal systems. Right now I’m mostly running Debian stable and everything inside Docker on the host machine, making it effectively a Docker “hypervisor”. It can get you quite far, without leaking any application state to the host machine.
  - blowski 5y ago
    Oh I agree, it is easier to mitigate some risks in a cloud solution. But this client - and they’re not unusual in their thinking - believes the cloud is somehow automatically mitigating those risks, when in fact it’s doing nothing because they’re not paying for it.
- RedShift1 5y ago
  Try Google Cloud instead. I've used both Azure and GCP and Azure VMs always feel sluggish, a problem I don't have with GCP.
  - blowski 5y ago
    In this specific case, they chose Azure. They had a consultant in a fancy suit tell them Azure would be safer, and proposed an architecture that ended up not working at all. But they still went with Azure, and it’s difficult to point to any gains they’ve got for the 200% price increase.
- pabs3 5y ago
  The clock is ticking on PHP 7.4 security support, one year left: https://www.php.net/supported-versions https://www.php.net/supported-versions
mgerdts 5y ago
I wonder how many of the extra cpu cycles are in PV network interfaces. It would be interesting to see how this works out with SR-IOV capable NICs with a VF in each VM.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
secondcoming 5y ago
I don't quite understand. The 'bare-metal' webservers returned 128 byte responses, but the Kubernetes version returned 1KB responses
- phillipseamore 5y ago
  Both the configuration files only have 1KB data files so might be an error in the article.
DSingularity 5y ago
I wonder if any of the CPU mitigations are behind this. They have a pretty significant cost in virtualized settings.
- LinuxBender 5y ago
  Another thing to check is how nginx was compiled. Using generic optimizations vs. x86_64 can do interesting things on VM's vs bare metal. nginx and haproxy specifically should be compiled generic for VM's. I don't have any links, just my own performance testing in the past.
  - theevilsharpie 5y ago
    A binary running in a VM is still executing native machine code, so compiler optimizations should have the same effect whether running on bare metal or a VM.
    - LinuxBender 5y ago
      Should being the key word. In truth the implementations of each hypervisor vary. Try it on each hypervisor that you use. I found KVM to have the most parity to bare metal performance.
      - theevilsharpie 5y ago
        I'm struggling to think of a situation when running virtualized vs. bare metal where compiler optimizations would matter. Certain hypervisors have the ability to disable features on the virtual CPU to enable live migration between different generation physical CPUs, in which case a binary that depends on a disabled virtual CPU feature (a.g., AVX-512) will simply crash (or otherwise fail) when it executes an unsupported instruction. Other than that, I'm drawing a blank. Hypervisor performance will vary, but I can't envision any scenario where a binary optimized for the processor's architecture would perform worse than one without any optimizations when running on a VM vs bare metal.
ALittleLight 5y ago
Am I reading correctly that there is a huge difference in http versus SSL requests per second? e.g. in the Bare Metal 1 CPU case it's 48k http to 800 SSL? I had no idea the performance impact of SSL was that huge, if this is correct.
- jcims 5y ago
  >The Ixia client sent a series of HTTPS requests, each on a new connection. The Ixia client and NGINX performed a TLS handshake to establish a secure connection, then NGINX proxied the request to the backend. The connection was closed after the request was satisfied. I honestly struggle to understand why they didn't incorporate keepalives in the testing. Reusing an existing TLS connection, something done far more often than not in the wild, will have a dramatic positive effect on throughput.
  - mlyle 5y ago
    Because, why bother? The symmetric encryption itself is cheap. We have datapoints for A) 0% SSL handshake workload per connection and B) 100% SSL handshake workload per connection. A reasonable step is just to solve the linear system--- 800 (s+r) = 1 48000 r = 1 and go from there for preliminary sizing. And, when you need detailed sizing, you should be using the real data from your real application. E.g., if it's reused 6 times, we'd expect around 4400 reqs/second from one core.
    - jcims 5y ago
      Sure, for a first order approximation that works well. I still think there’s something worth exploring here relative to bare metal vs hypervisors around AES-NI passthrough and cipher suites. My reaction came from the fact that someone that doesn’t understand how much cheaper symmetric encryption is might look at this and just think wow there’s no way I’m using SSL on my back end.
- kevin_thibedeau 5y ago
  You need to breakout full handshakes from reconnects with a ticket.
Thaxll 5y ago
Who actually runs Centos 7 ( Kernel 3.10 ) for benchmarks in 2021? Run something recent like Ubuntu 20 + KVM, you will see big difference. I don't beleive modern virtualization has ~20% overhead ( it should be less than 5% ).
- marginalia_nu 5y ago
  Can you find sources for those "less than 5%"-numbers form people who aren't selling kubernetes or cloud-related services? It's generally pretty easy to construct benchmarks that make look favorable. It's why there's constantly blog posts to the effect of "$INTERPRETED_HIGH_LEVEL_LANGUAGE is faster than C++".
  - grepfru_it 5y ago
    I mean one could say kubernetes has virtualized components, and requires VT-d extensions to operate at an accelerated speed, but I don't think containers are truly virtualized. So you can probably get away with a less than 5% benchmark if the stars aligned. With a hypervisor you're looking at 10-15% overhead, typically. maybe getting down to 7-12% using tricks (paravirtualization, pci-passthrough etc). In my environment i am around 12% OH on a good day
    - merb 5y ago
      i've seen way les than 10% overhead on modern hardware with sr-iov compat devices. it really depends on your network card/raid controller. some servers (and some cpus) are just not optimal for running virtualized stuff.
      - grepfru_it 5y ago
        *using tricks :)
    - w7 5y ago
      Indeed, standard container runtimes neither require VT-x (virtualization instructions) or VT-d (IOMMU for PCIe devices). They're "just" process isolation (cgroups), network stack isolation (netns), and filesystem isolation (overlayfs). Workloads on Kubernetes are not using virtual hardware unless you choose a managed provider or container runtime that states they are.
BiteCode_dev 5y ago
Yes, perfs are better on bare metal, and it's cheaper too. In fact, even a cheap VPS will perform better than the cloud for much less money, and scale vertically to incredible highs. You do have to do a lot of things manually, and even doing that, you won't get the high availability and elasticity you have the potential to get from a cloud offer. Potential being the key word though. But honestly, most projects don't need it. Crashing used to terrify me. One day I started to work for a client that had his service down once a month. Guess what ? Nothing happened. No customers ever complained. No money was lost, in fact, the cash flow kept growing. Most services are not so important they can't get switched off once in a while. Not to mention, monoliths are more robust they are given credit for.
- omegalulw 5y ago
  One thing I don't get is why not have an on prem solution with cloud fallback? It's hard but not super hard. You would just need a cloud data store. And depending on your app, you can have on prem data store that periodically backs up (and no egress under normal operation) if you can design with an eventual consistency models.
- Nextgrid 5y ago
  > you won't get the high availability Debatable. I've seen a fair amount of outages caused by the extra complexity brought on by making a system distributed for the purposes of high availability. Hardware is actually quite reliable nowadays, and I'll trust a hardware single-point of failure running a monolithic application more than a distributed microservice-based system with lots of moving parts. Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware.
  - KronisLV 5y ago
    > Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware. Isn't this because of the rampant introduction of accidental complexity whenever you attempt to make a system horizontally scalable - e.g. for whatever reason the developers or the people in charge suddenly try to cram in as many technological solutions as possible because apparently that's what the large companies are doing? There's no reason why you couldn't think about which data can or cannot be shared, and develop your system as one that's scalable, possibly modular, but with the codebase still being largely monolithic in nature. I'd argue that there's a large gray area between the the opposite ends of that spectrum - monoliths vs microservices. I actually wrote down some thoughts in a blog post of mine, called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to...

132 points by el_duderino 5y ago

I saw quite a significant performance and resource usage benefit from migrating away from a virtualized kubernetes environment to bare metal debian, so their findings align well with my anecdata as well.

blowski 5y ago

I recently moved a smallish business application from two bare-metal servers onto Azure VMs. It's a standard PHP 7.4 application, MySQL, Redis, nginx. Despite the VMs costing more and having twice the spec of the bare-metal servers, it has consistently performed slower and less reliably throughout the stack. The client's budget is too small to spend much time looking into it. Instead, they upped the spec even further, and what used to cost £600 per month now costs £2000.

hhw 5y ago
(Disclaimer) As a bare metal provider, I hope more people become aware what I've been saying for years: cloud is great for scaling down, but not that great for scaling up. If you want to have lots of VM's that don't warrant their own hardware that you can conveniently turn up and down, then cloud is fantastic. If you have a big application that needs to scale, you can get further vertically with bare metal, and if you need to scale horizontally, you need to optimize better higher up in the stack anyway, and the much lower cost for equivalent resources (without even taking any virtualization performance hit into account), more flexibility and thus more/better fitted performance of bare metal should have the clear advantage.
- stingraycharles 5y ago
  FYI I clicked on “get a dedicated server” on your site and ended up getting a 404 https://astuteinternet.com/services/dedicated-servers-order https://astuteinternet.com/services/dedicated-servers-order {"error":"URI Not Found"}
  - grepfru_it 5y ago
    >Starting at $199/mo Sounds pricey :)
    - duskwuff 5y ago
      Canadian dollars, so ~$160 USD. But that's still extremely high for a quad-core CPU from 2013, 8 GB RAM, no solid-state storage, and capped bandwidth.
      - hhw 5y ago
        Sorry, the website is pretty outdated. We're almost exclusively rolling out AMD EPYC3's these days, and we'd price any of those older configurations much lower than what the website lists them at. Nobody, other than spammers, ever orders through our website (although to be fair, our website may be to blame for that also). We get all of our business through word of mouth, and keep busy enough on that alone, so the website hasn't been a priority.
        duskwuff 5y ago
        You really ought to take your price lists down if they're that out of date. As it stands, they're probably driving some potential customers away -- even ones who heard of you through word of mouth, but decided to do a bit of research before proceeding.
    - RNCTX 5y ago
      Not really. Before there was a "cloud" in the early 2000s, we were paying anywhere from $100 (cheap, low spec machines) to $199 (new hardware, more ram, faster disks) per month for rented bare metal from places like Servermatrix, Softlayer, etc. The going rate also typically included anywhere from 1TB to 2TB of egress, as well. Another way of looking at it for the low end: How many things do you run on a single bare metal host versus an equivalent amount of discrete "services"?
    - FpUser 5y ago
      Look at Hetzner / OVH. I got incredibly good deals from them on dedicated servers. I think I am paying around $150 Canadian for AMD 16 cores, 128GB ECC RAM, do not remember storage. Updated: I also run some things right from my home office since I have symmetric 1Gbps fiber. For that I paid around $4000 Cdn for off lease 20 core 3.4GHz 512GB RAM server from HP
- laumars 5y ago
  > what I've been saying for years: cloud is great for scaling down, but not that great for scaling up. Yes and no. The cloud isn’t cheap at running any lift and shift type project. Where the cloud comes into its own is serverless and SaaS. If you have an application that’s a typical VM farm and you want to host it in the cloud, then you should at least first pause and identify how, if at all, your application can be re-architected to be more “cloudy”. Not doing this is usually the first mistake people make when deploying to the cloud.
speedgoose 5y ago
Did you throw money at the storage layer? The IOs are ridiculously slow by default.
stingraycharles 5y ago
I think “the cloud” never claimed to be cheaper, though. Its promise is mainly that you’ll offset a lot of risks with it against a higher price. And of course the workflow with the cloud is very different, with virtual machine images and whatnot. Whether that’s worth the price is up debate, though. I hope we’ll get more mature bare metal management software, perhaps even standardized in some remote management hardware so you can flash entire bare metal systems. Right now I’m mostly running Debian stable and everything inside Docker on the host machine, making it effectively a Docker “hypervisor”. It can get you quite far, without leaking any application state to the host machine.
- blowski 5y ago
  Oh I agree, it is easier to mitigate some risks in a cloud solution. But this client - and they’re not unusual in their thinking - believes the cloud is somehow automatically mitigating those risks, when in fact it’s doing nothing because they’re not paying for it.
RedShift1 5y ago
Try Google Cloud instead. I've used both Azure and GCP and Azure VMs always feel sluggish, a problem I don't have with GCP.
- blowski 5y ago
  In this specific case, they chose Azure. They had a consultant in a fancy suit tell them Azure would be safer, and proposed an architecture that ended up not working at all. But they still went with Azure, and it’s difficult to point to any gains they’ve got for the 200% price increase.
pabs3 5y ago
The clock is ticking on PHP 7.4 security support, one year left: https://www.php.net/supported-versions https://www.php.net/supported-versions

mgerdts 5y ago

I wonder how many of the extra cpu cycles are in PV network interfaces. It would be interesting to see how this works out with SR-IOV capable NICs with a VF in each VM.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

secondcoming 5y ago

I don't quite understand. The 'bare-metal' webservers returned 128 byte responses, but the Kubernetes version returned 1KB responses

phillipseamore 5y ago
Both the configuration files only have 1KB data files so might be an error in the article.

DSingularity 5y ago

I wonder if any of the CPU mitigations are behind this. They have a pretty significant cost in virtualized settings.

LinuxBender 5y ago
Another thing to check is how nginx was compiled. Using generic optimizations vs. x86_64 can do interesting things on VM's vs bare metal. nginx and haproxy specifically should be compiled generic for VM's. I don't have any links, just my own performance testing in the past.
- theevilsharpie 5y ago
  A binary running in a VM is still executing native machine code, so compiler optimizations should have the same effect whether running on bare metal or a VM.
  - LinuxBender 5y ago
    Should being the key word. In truth the implementations of each hypervisor vary. Try it on each hypervisor that you use. I found KVM to have the most parity to bare metal performance.
    - theevilsharpie 5y ago
      I'm struggling to think of a situation when running virtualized vs. bare metal where compiler optimizations would matter. Certain hypervisors have the ability to disable features on the virtual CPU to enable live migration between different generation physical CPUs, in which case a binary that depends on a disabled virtual CPU feature (a.g., AVX-512) will simply crash (or otherwise fail) when it executes an unsupported instruction. Other than that, I'm drawing a blank. Hypervisor performance will vary, but I can't envision any scenario where a binary optimized for the processor's architecture would perform worse than one without any optimizations when running on a VM vs bare metal.

ALittleLight 5y ago

Am I reading correctly that there is a huge difference in http versus SSL requests per second? e.g. in the Bare Metal 1 CPU case it's 48k http to 800 SSL? I had no idea the performance impact of SSL was that huge, if this is correct.

jcims 5y ago
>The Ixia client sent a series of HTTPS requests, each on a new connection. The Ixia client and NGINX performed a TLS handshake to establish a secure connection, then NGINX proxied the request to the backend. The connection was closed after the request was satisfied. I honestly struggle to understand why they didn't incorporate keepalives in the testing. Reusing an existing TLS connection, something done far more often than not in the wild, will have a dramatic positive effect on throughput.
- mlyle 5y ago
  Because, why bother? The symmetric encryption itself is cheap. We have datapoints for A) 0% SSL handshake workload per connection and B) 100% SSL handshake workload per connection. A reasonable step is just to solve the linear system--- 800 (s+r) = 1 48000 r = 1 and go from there for preliminary sizing. And, when you need detailed sizing, you should be using the real data from your real application. E.g., if it's reused 6 times, we'd expect around 4400 reqs/second from one core.
  - jcims 5y ago
    Sure, for a first order approximation that works well. I still think there’s something worth exploring here relative to bare metal vs hypervisors around AES-NI passthrough and cipher suites. My reaction came from the fact that someone that doesn’t understand how much cheaper symmetric encryption is might look at this and just think wow there’s no way I’m using SSL on my back end.
kevin_thibedeau 5y ago
You need to breakout full handshakes from reconnects with a ticket.

Thaxll 5y ago

Who actually runs Centos 7 ( Kernel 3.10 ) for benchmarks in 2021? Run something recent like Ubuntu 20 + KVM, you will see big difference. I don't beleive modern virtualization has ~20% overhead ( it should be less than 5% ).

marginalia_nu 5y ago
Can you find sources for those "less than 5%"-numbers form people who aren't selling kubernetes or cloud-related services? It's generally pretty easy to construct benchmarks that make look favorable. It's why there's constantly blog posts to the effect of "$INTERPRETED_HIGH_LEVEL_LANGUAGE is faster than C++".
- grepfru_it 5y ago
  I mean one could say kubernetes has virtualized components, and requires VT-d extensions to operate at an accelerated speed, but I don't think containers are truly virtualized. So you can probably get away with a less than 5% benchmark if the stars aligned. With a hypervisor you're looking at 10-15% overhead, typically. maybe getting down to 7-12% using tricks (paravirtualization, pci-passthrough etc). In my environment i am around 12% OH on a good day
  - merb 5y ago
    i've seen way les than 10% overhead on modern hardware with sr-iov compat devices. it really depends on your network card/raid controller. some servers (and some cpus) are just not optimal for running virtualized stuff.
    - grepfru_it 5y ago
      *using tricks :)
  - w7 5y ago
    Indeed, standard container runtimes neither require VT-x (virtualization instructions) or VT-d (IOMMU for PCIe devices). They're "just" process isolation (cgroups), network stack isolation (netns), and filesystem isolation (overlayfs). Workloads on Kubernetes are not using virtual hardware unless you choose a managed provider or container runtime that states they are.

BiteCode_dev 5y ago

Yes, perfs are better on bare metal, and it's cheaper too. In fact, even a cheap VPS will perform better than the cloud for much less money, and scale vertically to incredible highs. You do have to do a lot of things manually, and even doing that, you won't get the high availability and elasticity you have the potential to get from a cloud offer. Potential being the key word though. But honestly, most projects don't need it. Crashing used to terrify me. One day I started to work for a client that had his service down once a month. Guess what ? Nothing happened. No customers ever complained. No money was lost, in fact, the cash flow kept growing. Most services are not so important they can't get switched off once in a while. Not to mention, monoliths are more robust they are given credit for.

omegalulw 5y ago
One thing I don't get is why not have an on prem solution with cloud fallback? It's hard but not super hard. You would just need a cloud data store. And depending on your app, you can have on prem data store that periodically backs up (and no egress under normal operation) if you can design with an eventual consistency models.
Nextgrid 5y ago
> you won't get the high availability Debatable. I've seen a fair amount of outages caused by the extra complexity brought on by making a system distributed for the purposes of high availability. Hardware is actually quite reliable nowadays, and I'll trust a hardware single-point of failure running a monolithic application more than a distributed microservice-based system with lots of moving parts. Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware.
- KronisLV 5y ago
  > Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware. Isn't this because of the rampant introduction of accidental complexity whenever you attempt to make a system horizontally scalable - e.g. for whatever reason the developers or the people in charge suddenly try to cram in as many technological solutions as possible because apparently that's what the large companies are doing? There's no reason why you couldn't think about which data can or cannot be shared, and develop your system as one that's scalable, possibly modular, but with the codebase still being largely monolithic in nature. I'd argue that there's a large gray area between the the opposite ends of that spectrum - monoliths vs microservices. I actually wrote down some thoughts in a blog post of mine, called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to...