10 ms·

In the last 5 years we've experimented with Linux on top of hyper-v a few times. The basic I/O, compile & runtime performance was significantly inferior to xen

In the last 5 years we've experimented with Linux on top of hyper-v a few times. The basic I/O, compile & runtime performance was significantly inferior to xen & kvm (didn't bench against vmware, phasing it out due to cost), it wasn't worth any effort to even deploy apps for testing. Therefore I don't see Linux on hyper-v being a compelling option for the cost conscientious technical officer or lead engineer.

pweissbrod 11y ago
I find this interesting. I see many conflicting and biased studies comparing cloud/hypervisor performance. There are a great number of variables that come into play with configuring each hypervisor. Also, considering continual updates, a solid analysis 6 months ago may be stale. Not to mention most of the studies come from the vendors, not from independent parties. Question: does anyone know of an ongoing/reliable performance review of the major hypervisors?
aexaey 11y ago
I'm getting approximately the same CPU & storage performance out of Azure's A0 machine w/Debian8, as from AWS t2.micro instance with spinning rust storage and Debian8 (both are cheapest offer available, and price is quite similar between them). Network on Azure though is a totally wrecked thing: - A0 instance is capped to a miserable 5 megabit/s uplink (downlink is fine); - If you want TCP/UDP ports open, you need to open them one-by-one, going through a 2-page wizard with 8 fields in total to fill in for each port you are opening, and then waiting 15-20 seconds for each "change" to be applied. Good luck opening a 1024-port range for your SIP server; - You can't open ICMP/GRE/IPSEC or any other custom protocol; - There are ton of really weirdly configured network gear between your VM and the Internet - packet drops and private IPs in traceroute, etc... - On a plus side, you can enable so called "direct server return" and you will get incoming packets with your public IP as a destination IP; a really nice feature sorely missing on AWS.
- gruez 11y ago
  >A0 instance is capped to a miserable 5 megabit/s uplink (downlink is fine); I thought it was 10Mb/s? >If you want TCP/UDP ports open, you need to open them one-by-one, going through a 2-page wizard with 8 fields in total to fill in for each port you are opening, and then waiting 15-20 seconds for each "change" to be applied. Good luck opening a 1024-port range for your SIP server; ...or use the powershell cmdlets and do it in one line
  - aexaey 11y ago
    > I thought it was 10Mb/s? Alas... Also, in my experience, throttling is really slow to react, i.e. you get a chunk of data out at 10-20Mb/s, then everything freezes for few seconds until average would drop below 5Mbps, then another speed-up/slow-down cycle. Long-term average is exactly 5Mbps as per their spec-sheet [1]. [1] http://download.microsoft.com/download/4/1/1/411621F0-D0BF-4E44-BAFD-18C7AEA7B6FA/VirtualMachines.pdf http://download.microsoft.com/download/4/1/1/411621F0-D0BF-4... [2] https://www.oaklight.us/2014/06/azure-network-speed-quick-tests/ https://www.oaklight.us/2014/06/azure-network-speed-quick-te...
- ygra 11y ago
  Wouldn't this work for configuring lots of open ports: https://msdn.microsoft.com/library/azure/dn495300.aspx https://msdn.microsoft.com/library/azure/dn495300.aspx? Never used the cmdlets for Azure and it's been a while that I dealt with it, but that sounds like an issue you'd only do via the web interface if you're insane.
  - aexaey 11y ago
    Wouldn't work for me, as the {Get,Add}-Azure* scripts are windows/powershell-only. But then again, existence of CLI tools means there's an API. So, with a bit of luck, we might one day see a portable CLI tool (maybe boto3 [1] ported to azure API... one can dream! :-) Thanks for pointing those out! [1] https://github.com/boto/boto3 https://github.com/boto/boto3
    - weazl 11y ago
      Try their xplat cli [1], it's built on node/npm. I'm on Windows but still prefer this one over the PowerShell one. [1] https://www.npmjs.com/package/azure-cli https://www.npmjs.com/package/azure-cli
    - ppog 11y ago
      There is a cross-platform CLI (based on Node.js), which runs on OS X and Linux as well as Windows - I don't specifically know if it has commands for opening ports though. There's information and links at https://azure.microsoft.com/en-us/documentation/articles/xplat-cli/ https://azure.microsoft.com/en-us/documentation/articles/xpl... if you want to check it out.
- X-Istence 11y ago
  I don't see why private IP's in a trace route is necessarily wrong... Many large providers are numbering their hops with private IP space and routing public space across it because they don't want to waste /30's of public space for each point to point. Yes, /31 is an option, but there were a lot of edge cases or network gear that didn't support it.
  - aexaey 11y ago
    Yes, you are absolutely right that IPv4 addresses are scarce. Which bring a related question - what's up with none of AWS/Azure/GCE supporting IPv6?
    - X-Istence 11y ago
      Unfortunately IPv6 in cloud deployments is still very much lacking. OpenStack is one of those where IPv6 has only recently become something that is being built into OpenStack.
- mobiplayer 11y ago
  Hi aexaey, There are some things wrong on your comment, please let me comment on them: I don't think A0 are capped at all, I don't know where did you get that: user@A0VM:~$ speedtest-cli Retrieving speedtest.net configuration... Retrieving speedtest.net server list... Testing from Microsoft Corporation (40.113.XX.XX)... Selecting best server based on latency... Hosted by KsFiberNet (Wichita, KS) [45.26 km]: 139.703 ms Testing download speed........................................ Download: 44.48 Mbit/s Testing upload speed.................................................. Upload: 22.54 Mbit/s I just had to run one single test to get these results. Multi-threading will probably pump this up (let's say a multi-thread iPerf test between two VMs on two different regions, through their public IP addresses). (Funnily enough this VM is in Dublin, hence the latency to Wichita). You can get access to all your TCP/UDP ports by adding an instance-level IP: https://azure.microsoft.com/en-gb/documentation/articles/virtual-networks-instance-level-public-ip/ https://azure.microsoft.com/en-gb/documentation/articles/vir... (you can later protect it with NSGs --> https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-nsg/ https://azure.microsoft.com/en-us/documentation/articles/vir... ). What you've seen is when you create endpoints on the load balancer that points to the VMs behind it. Now in Azure Resource Manager, spinning up a VM from the new portal will actually get you an ILPIP and no load balancer by default. About that statement about a ton of weirdly configured network gear: Well, for starters you're doing network virtualization here :) but seriously now, the platform doesn't forward ICMPs to/from Internet, so I'm curious about those private IP addresses on a traceroute (and of course the packet drops). If you're talking about hops between you and the VIP (the load balanced public IP in front of your VM), that's probably your provider, other providers and finally Microsoft's Network. As per the rest of the path, you've said that ICMP doesn't go through, so traceroute it's not going to work (tcptraceroute also needs ICMP to work :)).
  - aexaey 11y ago
    ILPIP look promising. I'll try it, thanks! Speed test though... hm... 22Mbit/s is not exactly unrestricted, plus not sure how long is the burst that "speedtest-cli" makes. In my experience A0's uplink is burstable (but unusably slow when recovering from a burst), with long-term average being bang-on 5Mbps. I do of course see exactly one "my" private IP in traceroute, but I'm not talking about that. I meant there are microsoft's private IPs (#12-#15) and dropped packets (#16 and on). And of course, "platform doesn't forward ICMPs" is an annoyance in itself - i can't even ping my own VM. # target IP anonymized, but still have enough granularity to reproduce the traceroute traceroute to xxxxx.cloudapp.net (191.235.128.128), 30 hops max, 60 byte packets 1 192.168.1.1 (192.168.1.1) 0.093 ms 0.117 ms 0.063 ms 2 193.47.232.12 (193.47.232.12) 0.785 ms 0.785 ms 0.776 ms 3 microsoft.mix-it.net (217.29.66.112) 1.219 ms 1.233 ms 1.225 ms 4 xe-0-1-1-0.fra-96cbe-1a.ntwk.msn.net (207.46.42.12) 8.700 ms 8.902 ms 8.430 ms 5 * * * 6 * * * 7 104.44.9.142 (104.44.9.142) 42.112 ms 39.194 ms 38.987 ms 8 * * * 9 be-6-0.ibr01.dub30.network.microsoft.com (104.44.4.142) 39.738 ms 38.266 ms 38.929 ms 10 * * * 11 ae2-0.db3-96c-3b.ntwk.msn.net (204.152.141.81) 36.986 ms 36.984 ms 36.925 ms 12 25.149.64.247 (25.149.64.247) 36.951 ms 36.775 ms 37.152 ms 13 10.10.132.81 (10.10.132.81) 36.917 ms 37.477 ms 37.287 ms 14 10.60.31.77 (10.60.31.77) 37.317 ms 37.699 ms 37.899 ms 15 10.60.31.69 (10.60.31.69) 37.653 ms 37.326 ms 37.441 ms 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * P.S. #12 - lol!
kefka 11y ago
But the game isn't the performance on a machine, or even a few machines. The game are tools like Kubernetes and Apache Mesos. I have X thousand VMs spread out through the world, with a few hundred per datacenter for geo-stability. If a network link fails, no biggie. Still runs. If a rack dies, still no biggie. Redundancy makes it work. Earthquake/tsunami/natural disaster ? Your stuff still runs, with a bit higher latency and slower throughput.
Someone1234 11y ago
With all due respect: I don't buy it. That isn't how the technology works. Compilation in particular would never even hit the hypervisor layer regardless of which one you were utilising. It is CPU/memory bound. The difference between hypervisors for IO is tiny. They're all slower than native, but vary very little between one another. That's why the manufacturers have all almost given up trying to use performance for comparison, and instead argue value adds/management/automation/cost. If you really found one specific hypervisor slower than another (regardless of which two), I'd want to look at your exact test conditions since that has never been the case for me (everything else being equal). And even the manufacturers don't argue that point much. There are plenty of reasons to e.g. pick AWS over Azure, but Hyper-V performance is not one of them. That's just FUD.
- stronglikedan 11y ago
  I have a developer who can show a marked difference in performance between VMWare Fusion (paid) and VirtualBox (free) running Windows 7 on OSX. Same Windows ISO used to install both, and both running on the same hardware and OS, with exact VM configs (as close a you can get between the two). I'm talking a night-and-day difference - VMWare Fusion is usable, but VirtualBox is not. After reading your comment, and admittedly not knowing much about hypervisors (yet), I am genuinely curious as to what else could be causing this drastic difference in performance, if not for the hypervisor itself. Any ideas?
  - Someone1234 11y ago
    I was more talking about actual hypervisors that run on hardware. Client hypervisors (those that run on an underlying OS) are an entirely different thing. Real hypervisors are utilising a lot of hardware features to accomplish jailing. Client hypervisors may or may not be, depending on how they're implemented, and or how they're integrated into the underlying OS (e.g. specialised drivers, just simulating hardware features, etc).
  - 5ilv3r 11y ago
    That's not a very good example since both of those are desktop applications providing virtualization and are expected to play nice with other desktop apps. You can virtualize in a web browser too, but that doesn't make it a good high performance idea. You should be looking at dedicated virtualization hosts to do an apples to apples comparison, like vmware on bare server metal vs hyperv or any of the free tools also on metal. Not on a cpu throttled desktop.
davidgerard 11y ago
Main problem with Hyper-V: getting into the admin console with anything but IE. Yes, really.
- overgryphon 11y ago
  Vmconnect(1) is a local exe. The best experience for a console to your Hyper-V VM is remote desktop, but that requires a Windows guest and virtual networking. If you are trouble shooting networking or boot, or are using Linux guests, vmconnect is very easy to use (no network required) and works great. I'm not sure of any way to get a console through IE. Can you explain? (1) https://technet.microsoft.com/en-us/library/dn282274.aspx https://technet.microsoft.com/en-us/library/dn282274.aspx

pweissbrod 11y ago

I find this interesting. I see many conflicting and biased studies comparing cloud/hypervisor performance. There are a great number of variables that come into play with configuring each hypervisor. Also, considering continual updates, a solid analysis 6 months ago may be stale. Not to mention most of the studies come from the vendors, not from independent parties. Question: does anyone know of an ongoing/reliable performance review of the major hypervisors?

aexaey 11y ago

I'm getting approximately the same CPU & storage performance out of Azure's A0 machine w/Debian8, as from AWS t2.micro instance with spinning rust storage and Debian8 (both are cheapest offer available, and price is quite similar between them). Network on Azure though is a totally wrecked thing: - A0 instance is capped to a miserable 5 megabit/s uplink (downlink is fine); - If you want TCP/UDP ports open, you need to open them one-by-one, going through a 2-page wizard with 8 fields in total to fill in for each port you are opening, and then waiting 15-20 seconds for each "change" to be applied. Good luck opening a 1024-port range for your SIP server; - You can't open ICMP/GRE/IPSEC or any other custom protocol; - There are ton of really weirdly configured network gear between your VM and the Internet - packet drops and private IPs in traceroute, etc... - On a plus side, you can enable so called "direct server return" and you will get incoming packets with your public IP as a destination IP; a really nice feature sorely missing on AWS.

gruez 11y ago
>A0 instance is capped to a miserable 5 megabit/s uplink (downlink is fine); I thought it was 10Mb/s? >If you want TCP/UDP ports open, you need to open them one-by-one, going through a 2-page wizard with 8 fields in total to fill in for each port you are opening, and then waiting 15-20 seconds for each "change" to be applied. Good luck opening a 1024-port range for your SIP server; ...or use the powershell cmdlets and do it in one line
- aexaey 11y ago
  > I thought it was 10Mb/s? Alas... Also, in my experience, throttling is really slow to react, i.e. you get a chunk of data out at 10-20Mb/s, then everything freezes for few seconds until average would drop below 5Mbps, then another speed-up/slow-down cycle. Long-term average is exactly 5Mbps as per their spec-sheet [1]. [1] http://download.microsoft.com/download/4/1/1/411621F0-D0BF-4E44-BAFD-18C7AEA7B6FA/VirtualMachines.pdf http://download.microsoft.com/download/4/1/1/411621F0-D0BF-4... [2] https://www.oaklight.us/2014/06/azure-network-speed-quick-tests/ https://www.oaklight.us/2014/06/azure-network-speed-quick-te...
ygra 11y ago
Wouldn't this work for configuring lots of open ports: https://msdn.microsoft.com/library/azure/dn495300.aspx https://msdn.microsoft.com/library/azure/dn495300.aspx? Never used the cmdlets for Azure and it's been a while that I dealt with it, but that sounds like an issue you'd only do via the web interface if you're insane.
- aexaey 11y ago
  Wouldn't work for me, as the {Get,Add}-Azure* scripts are windows/powershell-only. But then again, existence of CLI tools means there's an API. So, with a bit of luck, we might one day see a portable CLI tool (maybe boto3 [1] ported to azure API... one can dream! :-) Thanks for pointing those out! [1] https://github.com/boto/boto3 https://github.com/boto/boto3
  - weazl 11y ago
    Try their xplat cli [1], it's built on node/npm. I'm on Windows but still prefer this one over the PowerShell one. [1] https://www.npmjs.com/package/azure-cli https://www.npmjs.com/package/azure-cli
  - ppog 11y ago
    There is a cross-platform CLI (based on Node.js), which runs on OS X and Linux as well as Windows - I don't specifically know if it has commands for opening ports though. There's information and links at https://azure.microsoft.com/en-us/documentation/articles/xplat-cli/ https://azure.microsoft.com/en-us/documentation/articles/xpl... if you want to check it out.
X-Istence 11y ago
I don't see why private IP's in a trace route is necessarily wrong... Many large providers are numbering their hops with private IP space and routing public space across it because they don't want to waste /30's of public space for each point to point. Yes, /31 is an option, but there were a lot of edge cases or network gear that didn't support it.
- aexaey 11y ago
  Yes, you are absolutely right that IPv4 addresses are scarce. Which bring a related question - what's up with none of AWS/Azure/GCE supporting IPv6?
  - X-Istence 11y ago
    Unfortunately IPv6 in cloud deployments is still very much lacking. OpenStack is one of those where IPv6 has only recently become something that is being built into OpenStack.
mobiplayer 11y ago
Hi aexaey, There are some things wrong on your comment, please let me comment on them: I don't think A0 are capped at all, I don't know where did you get that: user@A0VM:~$ speedtest-cli Retrieving speedtest.net configuration... Retrieving speedtest.net server list... Testing from Microsoft Corporation (40.113.XX.XX)... Selecting best server based on latency... Hosted by KsFiberNet (Wichita, KS) [45.26 km]: 139.703 ms Testing download speed........................................ Download: 44.48 Mbit/s Testing upload speed.................................................. Upload: 22.54 Mbit/s I just had to run one single test to get these results. Multi-threading will probably pump this up (let's say a multi-thread iPerf test between two VMs on two different regions, through their public IP addresses). (Funnily enough this VM is in Dublin, hence the latency to Wichita). You can get access to all your TCP/UDP ports by adding an instance-level IP: https://azure.microsoft.com/en-gb/documentation/articles/virtual-networks-instance-level-public-ip/ https://azure.microsoft.com/en-gb/documentation/articles/vir... (you can later protect it with NSGs --> https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-nsg/ https://azure.microsoft.com/en-us/documentation/articles/vir... ). What you've seen is when you create endpoints on the load balancer that points to the VMs behind it. Now in Azure Resource Manager, spinning up a VM from the new portal will actually get you an ILPIP and no load balancer by default. About that statement about a ton of weirdly configured network gear: Well, for starters you're doing network virtualization here :) but seriously now, the platform doesn't forward ICMPs to/from Internet, so I'm curious about those private IP addresses on a traceroute (and of course the packet drops). If you're talking about hops between you and the VIP (the load balanced public IP in front of your VM), that's probably your provider, other providers and finally Microsoft's Network. As per the rest of the path, you've said that ICMP doesn't go through, so traceroute it's not going to work (tcptraceroute also needs ICMP to work :)).
- aexaey 11y ago
  ILPIP look promising. I'll try it, thanks! Speed test though... hm... 22Mbit/s is not exactly unrestricted, plus not sure how long is the burst that "speedtest-cli" makes. In my experience A0's uplink is burstable (but unusably slow when recovering from a burst), with long-term average being bang-on 5Mbps. I do of course see exactly one "my" private IP in traceroute, but I'm not talking about that. I meant there are microsoft's private IPs (#12-#15) and dropped packets (#16 and on). And of course, "platform doesn't forward ICMPs" is an annoyance in itself - i can't even ping my own VM. # target IP anonymized, but still have enough granularity to reproduce the traceroute traceroute to xxxxx.cloudapp.net (191.235.128.128), 30 hops max, 60 byte packets 1 192.168.1.1 (192.168.1.1) 0.093 ms 0.117 ms 0.063 ms 2 193.47.232.12 (193.47.232.12) 0.785 ms 0.785 ms 0.776 ms 3 microsoft.mix-it.net (217.29.66.112) 1.219 ms 1.233 ms 1.225 ms 4 xe-0-1-1-0.fra-96cbe-1a.ntwk.msn.net (207.46.42.12) 8.700 ms 8.902 ms 8.430 ms 5 * * * 6 * * * 7 104.44.9.142 (104.44.9.142) 42.112 ms 39.194 ms 38.987 ms 8 * * * 9 be-6-0.ibr01.dub30.network.microsoft.com (104.44.4.142) 39.738 ms 38.266 ms 38.929 ms 10 * * * 11 ae2-0.db3-96c-3b.ntwk.msn.net (204.152.141.81) 36.986 ms 36.984 ms 36.925 ms 12 25.149.64.247 (25.149.64.247) 36.951 ms 36.775 ms 37.152 ms 13 10.10.132.81 (10.10.132.81) 36.917 ms 37.477 ms 37.287 ms 14 10.60.31.77 (10.60.31.77) 37.317 ms 37.699 ms 37.899 ms 15 10.60.31.69 (10.60.31.69) 37.653 ms 37.326 ms 37.441 ms 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * P.S. #12 - lol!

kefka 11y ago

But the game isn't the performance on a machine, or even a few machines. The game are tools like Kubernetes and Apache Mesos. I have X thousand VMs spread out through the world, with a few hundred per datacenter for geo-stability. If a network link fails, no biggie. Still runs. If a rack dies, still no biggie. Redundancy makes it work. Earthquake/tsunami/natural disaster ? Your stuff still runs, with a bit higher latency and slower throughput.

Someone1234 11y ago

With all due respect: I don't buy it. That isn't how the technology works. Compilation in particular would never even hit the hypervisor layer regardless of which one you were utilising. It is CPU/memory bound. The difference between hypervisors for IO is tiny. They're all slower than native, but vary very little between one another. That's why the manufacturers have all almost given up trying to use performance for comparison, and instead argue value adds/management/automation/cost. If you really found one specific hypervisor slower than another (regardless of which two), I'd want to look at your exact test conditions since that has never been the case for me (everything else being equal). And even the manufacturers don't argue that point much. There are plenty of reasons to e.g. pick AWS over Azure, but Hyper-V performance is not one of them. That's just FUD.

stronglikedan 11y ago
I have a developer who can show a marked difference in performance between VMWare Fusion (paid) and VirtualBox (free) running Windows 7 on OSX. Same Windows ISO used to install both, and both running on the same hardware and OS, with exact VM configs (as close a you can get between the two). I'm talking a night-and-day difference - VMWare Fusion is usable, but VirtualBox is not. After reading your comment, and admittedly not knowing much about hypervisors (yet), I am genuinely curious as to what else could be causing this drastic difference in performance, if not for the hypervisor itself. Any ideas?
- Someone1234 11y ago
  I was more talking about actual hypervisors that run on hardware. Client hypervisors (those that run on an underlying OS) are an entirely different thing. Real hypervisors are utilising a lot of hardware features to accomplish jailing. Client hypervisors may or may not be, depending on how they're implemented, and or how they're integrated into the underlying OS (e.g. specialised drivers, just simulating hardware features, etc).
- 5ilv3r 11y ago
  That's not a very good example since both of those are desktop applications providing virtualization and are expected to play nice with other desktop apps. You can virtualize in a web browser too, but that doesn't make it a good high performance idea. You should be looking at dedicated virtualization hosts to do an apples to apples comparison, like vmware on bare server metal vs hyperv or any of the free tools also on metal. Not on a cpu throttled desktop.

davidgerard 11y ago

Main problem with Hyper-V: getting into the admin console with anything but IE. Yes, really.

overgryphon 11y ago
Vmconnect(1) is a local exe. The best experience for a console to your Hyper-V VM is remote desktop, but that requires a Windows guest and virtual networking. If you are trouble shooting networking or boot, or are using Linux guests, vmconnect is very easy to use (no network required) and works great. I'm not sure of any way to get a console through IE. Can you explain? (1) https://technet.microsoft.com/en-us/library/dn282274.aspx https://technet.microsoft.com/en-us/library/dn282274.aspx