17 ms·

Euro-cloud provider Anexia moves 12,000 VMs off VMware to homebrew KVM platform

274 points by xbmcuser 2y ago

depr 2y ago
Seems like a good move and everything but 12,000 doesn't sound like that many VMs? Is that a lot of VMs?
- zamadatix 2y ago
  In terms of VMware customers it isn't a ton of VM but not peanuts either. E.g. the last healthcare place I was at (single customer rather than cloud provider) had ~30k VMware VMs and we were still small fish compared to some others. I've heard of places 10x the VM count as this making the move post acquisition - albeit less publicly. I think the purpose of the article is to highlight companies like this are starting/continuing to migrate post-acquisition rather than this particular customer was impressively large and did so. Particularly with the bits about the relative cost increase even though the customer was willing to walk away if needed.
  - wink 2y ago
    Single customer 30k sounds a lot easier to migrate (or at least to schedule) than 12k with potentially 3k different customers (probably more like several dozens or a couple hundred)
    - zamadatix 2y ago
      Could be, lots of pros and cons to each scenario - doubt either are easy by any measure. E.g. for about 20% we didn't even have a single piece of documentation other than the server name for who might actually care about the VM going down for the migration we wanted to schedule. Let alone how to test the migration, when it is best to do it, what software was actually running on it, if it's actually managed/monitored integrated with/by other systems whoch need to be looked at too, or if it could just be shit down instead (yay healthcare mergers and acqs). Our migration was also to VMware from (mostly) Hyper-V at the time, so not as much custom tooling needed. On the flip side a cloud provider is going to have all of the owner contact info but no direct control of the guest OS to effect the change so the battle is more with trying to get the customers to care enough to do the migration with you but not be so bothered by it all they up and leave your hosting.Not exactly a walk in the park either. In either case - almost never the tech that's the hard part for sure :).
- aaronax 2y ago
  Difficult to say. That could conceivably fit in one rack of 60 compute nodes (1/2U size) at 200VMs per node, leaving 15U for networking and SAN. Maybe $100/year/VM (rough cost of a lower-end cloud VM like EC2, droplet, etc.) in that case so $1.2 million per year cost. Or it could take 10 racks and $50 million per year.
  - Symbiote 2y ago
    The article says, right at the start, > Anexia was founded in 2006, is based in Austria, and provides cloud services from over 100 locations around the world by placing equipment in third party datacenters. From the company's homepage: > The founder and CEO of Anexia [...] recently acquired a small hydropower plant in Kammern in the Liesingtal region of Styria for a “significant seven-figure sum” – i.e. several million euros. The power plant on the River Liesing generates 600 KW of electricity, enough to cover a third of the electricity consumption of Anexia’s Vienna data center so this seems to be a significant operation.
    - quesomaster9000 2y ago
      While average power consumption per rack has been increasing fairly steadily over the past 10 years, the metric I currently use is abound 10kW per rack under reasonable to heavy load - that's about the same as a consumer electric shower. So, this is implying their Vienna data center has 180 racks? With 60 being about a third, if we say each rack has 40 servers... that's ~7k servers total... which is a sizeable chunk of floor space, like 3000m^2, or... 40 tennis courts? But yea, that a non-insignificant operation just for the Vienna data center.
- zipy124 2y ago
  Entirely depends on context, like asking how long is a piece of string. One VM could be 200 cores, or it could be 1 core. It could also be a kubernetes/docker worker as-well so one VM may be thousands of containers. Finally they could instead just be important VM's. You could imagine a small or medium company having maybe 4 VM's each for prod, staging, testing etc... letting CDN's handle scaling (with everything else running on local dev machines) and so that 12,000 could be an entire 3,000 companies whole stacks.
- mrweasel 2y ago
  That depends on your context. 12,000 VMs is enough to a fairly large chunk of a small countries healthcare infrastructure, if not all of it. It a pretty decent amount of VMs, but not close to being unmanageable. I think it's more down to how the rest of your infrastructure looks, if we're talking about ease of migration.
- rwmj 2y ago
  I'm involved with Red Hat's effort to shift customers off VMware (upstream project: https://github.com/libguestfs/virt-v2v https://github.com/libguestfs/virt-v2v). Things have really blown up since the Broadcom acquisition at the end of 2023. For us 12K VMs is a medium to large customer, but definitely not unusual. Think someone like a regional bank. A full conversion of such a customer might involve one consultant on-site, a scoping exercise to classify the VMs into groups and assess which ones are going to be more or less difficult to convert, and perhaps 1-3 months of work to convert them all. Individual VMs would be down from anything between a simple reboot, up to 12 hours, depending on which strategy we used for conversion (there are complicated trade-offs related to storage and network bandwidth).
  - justinclift 2y ago
    Sounds like it's similar to what happened with PostgreSQL after Oracle bought MySQL. ;)
wkat4242 2y ago
Good to see companies calling Broadcom's bluff!
neilv 2y ago
Did Broadcom ever justify price-gouging captive customers after acquiring VMware? ("Sorry, folks, poor banana harvest this year, so we have to pass on the skyrocketing cost of bits licensing, which runs on bananas"?)
- rad_gruchalski 2y ago
  They spent money on the acquisition and they would like to see some back. Isn’t that obvious?
  - toomuchtodo 2y ago
    There is a lot of road between "reasonable return on capital invested" and "we are going to rake you over the coals so badly people are going to publicly flee to not be captive to us."
    - rad_gruchalski 2y ago
      Well, it’s Broadcom and they just spent the money so they want some back. It’s like IBM: acquire and rake. Hey IBM employees, it doesn’t matter how upset you get. It is what it is.
  - 0xedd 2y ago
    [dead]
  - burnte 2y ago
    No one said it wasn't obvious, we're saying spending almost $70bn on a company and announcing that the only way you can make it back is by squeezing 10x more out of the big fish, that might not have been smart.
    - rad_gruchalski 2y ago
      Apparently they are known for that so I'm not sure why acting surprised.
- 2y ago
throw310822 2y ago
A-nexia, does it mean "lack of connectivity"? Great name for a cloud provider :)
- szszrk 2y ago
  It's close to Anoxia and Alexia, which are also "nice" choices to be known for.
- pmontra 2y ago
  Or "to annex" [1], as in all your VMs (are) belong to us [2] [1] https://www.merriam-webster.com/dictionary/annex https://www.merriam-webster.com/dictionary/annex [2] https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us
- kgeist 2y ago
  Anexia means "patience, humility" in Ancient Greek. Maybe that's what they were after?
mrtksn 2y ago
If the claims about AI capabilities are true, it's matter of time before all providers to switch to homegrown solutions.
- cruffle_duffle 2y ago
  The funny part is you can convince an LLM any bullshit you propose is right. It will agree with anything you say unless you encourage it not to and even then… “VMWare costs too much. Our use case is incredibly niche and nothing out there seems like it would fulfill our needs. Can you help me write a new stack using bash scripts and AWK?” “What an exciting idea! Bash Scripting is a clever way to orchestrate 12,000 virtual machines. Here, let me provide you with a bunch of code to copy and paste without doing any requirements gathering whatsoever. <pages of random bash scripts> These bash scripts will let you run 12,000 virtual machines. Let me know if you need any explanation!” “I forgot to me to mention we need to provision these using a rest api. Also these are globally distributed and require extensive monitoring” “Not a problem! REST is an excellent way to communicate with the platform. I won’t bother asking any follow up questions or gathering any more requirements. Here is all the code you need:” … and so on…
  - mrtksn 2y ago
    If you know what are you doing in general, I'm convinced that it can actually help you out but I had 0 success with doing it brainfuck(chatgpt claims that anything beyond hello world is too complicated haha). Just like with the image and video generators, I'm fishing to find evidence of machines building something useful - had very little success so far. If the claims are true and these huge investments are not done in vain, not too far in the future we should be able to tell AI to build the tools we need, for example Microsoft should be expecting to discontinue MS Office either because someone can tell AI to make them an Excel instead of paying a subscription or tell AI to just do the job they will do in Excel. It's very strange that they talk about replacing the developers with AI but somehow still sell software. Something doesn't add up.
    - kube-system 2y ago
0xedd 2y ago
[dead]
homebrewer 2y ago
They only mention KVM, however KVM is a kernel API and not a full hypervisor, and can't do half the things they claim it does (like support for certain disk images). What hypervisor are they actually using, does anybody know? Probably qemu? It doesn't feel like a major enough company to be able to write its own like Google did.
- klooney 2y ago
  They probably mean libvirtd/qemu, when people say "we used kvm" that's generally what they mean.
- freedomben 2y ago
  They are most assuredly using qemu. It's been my experience that when most people say "KVM" they usually are referring to the whole libvirt ecosystem. I assume that's the case here. I think of it very analogously to how people say "linux" to refer to a full OS/distro such as Fedora. Though "linux" is just a kernel, it's become common parlance.
- blueflow 2y ago
  KVM is not only the kernel side - the userspace side of KVM was implemented in a fork of qemu. That fork later got merged into mainline qemu, so qemu is the "original" implementation of a KVM hypervisor.
- deleted 2y ago
  [deleted]
- bertman 2y ago
  Netcup seem to use qemu[0], which is what Anexia will also use according to OP's article [0]https://old.reddit.com/r/VPS/comments/1ghz4fd/i_know_contabo_uses_proxmox_what_does_hetzner_and/lv2amg7/ https://old.reddit.com/r/VPS/comments/1ghz4fd/i_know_contabo...
freedomben 2y ago
I definitely don't want to miniminize the significance of this. This is huge! But, they did have some major benefits that most companies looking to do the same won't: > Anexia therefore resolved to migrate, a choice made easier by its ownership of another hosting business called Netcup that ran on a KVM-based platform. > The hosting company is also a big user of NetApp storage, so customer data was already stored in a resource independent of its VMware rig – any new VMs would just need to be pointed at existing volumes. Again, still a great accomplishment and an exciting milestone for them, but for people still stuck on VMWare that are looking to migrate, it's good to know about the above things.
- tw04 2y ago
  Other than the additional capital outlay, this shouldn't prove a hurdle to anyone looking to do similar. Migrating from VSAN to external storage like NetApp is a non-disruptive process. And frankly VSAN has always been pretty horrible, so you'll likely end up with better performance and storage efficiency when all is said and done.
amluto 2y ago
I found this rather odd: > “We used to pay for VMware software one month in arrears,” he said. “With Broadcom we had to pay a year in advance with a two-year contract.” If your goal is to extract every possible cent from your existing customers, why would you also switch them from net 30 to requiring partial prepayment? VMware wants money in general but should not have a cash flow problem, and forcing a monster early payment seems like it will force customers to notice an immediate problem and make a choice instead of allowing themselves to be slowly and steadily ripped off. If I were a pointy-haired CEO committed to the multiply-pricing-by-five strategy, I would do my best to sweeten the deal: offer generous payment terms, give nice-sounding discounts for up front commitments, give very large discounts for nodes that haven’t yet been leased to a customer, etc.
- shepardrtc 2y ago
  Either their sales team absolutely screwed this up, or they just don't want to bother with the VMware platform anymore. Maybe both?
  - nyrikki 2y ago
    Broadcom aspires to the Oracle model, they said as much when they bought VMware.
- diggan 2y ago
  Perhaps similar strategy as spammers employ today? Where they try to filter away people quickly who wouldn't fall for it when they need it to. So do something slightly outrageous today, so you filter away the ones that won't stick around for the future more outrageous changes.
- nolok 2y ago
  Because they have twelve thousands vms and are themselves a provider that can't afford to have downtime for its customers. So the thinking here was probably "there is no way they can refuse to sign right now and destroy their business in the process, so we might as well take the cake and also force them to stay after so they don't leave in 11 months and 29 days". Turns out that thinking is wrong for that specific customers, but for how many did it work ?
segasaturn 2y ago
Anybody who formalizes their "homebrew KVM platform" into a marketable hypervisor product is going to make a lot of money I suspect. Every IT department I know is scrambling to replace their VMWare stack ASAP, including very large ones.
- formerly_proven 2y ago
  > Every IT department I know is scrambling to replace their VMWare stack ASAP, including very large ones. At least some of the big ones seem to just pay up. Probably because they / their MSP / their relationship with their MSP is so dysfunctional that they know migrating is a pipe dream.
  - stackskipton 2y ago
    As former DevOps who dealt with VMware, it's not relationship with MSP. It's just so many things plugin to VMware that migrating off of VMware is just difficult. Monitoring, Backups, Deployment and so forth are deeply integrated into VMware so companies just look at work involved getting off and go, never mind.
- dehrmann 2y ago
  Most things that would have used VMware 15 years ago are using Kubernetes now. The things that aren't are probably looking at Proxmox or just KVM.
  - taskforcegemini 2y ago
    >VMware 15 years ago are using Kubernetes now. this doesn't sound right, both serve different purposes
- intelVISA 2y ago
  Soon.
- icedchai 2y ago
  Is there a reason Proxmox wouldn't work for many of them?
raincom 2y ago
12000 VMs = 800 to 1200 baremetals. Since KVM can run vmdk images, it is more of how to manage all these barementals and VMs from a central place.
- olavgg 2y ago
  That depends, on a modern Epyc, you can easily run a few thousand VM's on one system.
  - menaerus 2y ago
    How do you run thousands of VM's on 192 physical cores? That must be assuming that large percentage of VM's are idling most of the time.
    - justinclift 2y ago
      That's pretty commonly the case.
      - menaerus 2y ago
        I understand but I'm just curious to learn under what conditions.
- AviationAtom 2y ago
  We have run 8,000 VMs on ~120 servers
rcleveng 2y ago
The phrase "homebrew KVM platform" made me chuckle. All of the hyperscalers have a homebrew, aka proprietary platform using a hypervisor. AWS has Nitro which seems to have been based on KVM but likely quite different by now. If you are selling VMs to customers, I can't understand a good reason to use VMWare. The only reason would be if you are selling VMWare as a service.
- TheNewsIsHere 2y ago
  If you’re providing application services to customers who need to use an application that the vendor only supports in VMware, then you don’t have a choice. Well, I suppose you do. But an EMR/EHR for instance is going to _need_ vendor support, which means requiring VMware even if you’re not selling VMware itself as a service.
  - rcleveng 2y ago
    Absolutely - in enterprise software, it's the certified configurations that are not something you can mess around with, if you like support. I'd still consider this VMware as a service, although not full VMware, but just enough for the checkboxes. Maybe you don't get API access, console access, etc, but the main thing you are selling is vmware (to check the certification boxes), and not a generic VM.
- monocasa 2y ago
  > AWS has Nitro which seems to have been based on KVM but likely quite different by now. It still is relatively stock KVM on the CPU side of things. They've been upstreaming changes they need like lower overhead for emulating Xen's hypercall interface. Most of their special sauce is in the devices though, as those natively provide VM boundaries leaving the hypervisor to not have to manage all that much at runtime.
- dilyevsky 2y ago
  The challenging bits are all outside of the KVM like the VPC networking that has to be implemented using some SDN (e.g OpenVSwitch), block devices, etc VMware had a solution for all of these natively and with support. Not using their hypervisors you have to manage a huge pile of OSS+proprietary integrations and actually have staff who truly understand how everything works down to the lowest level. Doable but probably above the pay grade for most
smetj 2y ago
Congratulations Anexia, well done! Pulling this off means you have experienced leaders and excellent engineers.
pcblues 2y ago
I heard first-hand years ago of a bug in VMWare that was _CPU-Level_ for the company that rolled it out in an effort to replace their PCs that had software locally installed. The failure would have been catastrophic for a company that had just replaced all its PCs. However, they hadn't shipped the old PCs away yet for destruction, and managed to keep the lights on by putting them all back out again until the problem was fixed.
hd4 2y ago
Hard to imagine how other similar companies wouldn't just follow suit even if the cost of staying with VMware wasn't prohibitive (which is unlikely for most cases, probably). Stupid business decisions should be punished by the customer walking out. Homebrew KVM is not that hard.
leoh 2y ago
Incredible to see how much value Broadcom is destroying.

274 points by xbmcuser 2y ago

depr 2y ago

Seems like a good move and everything but 12,000 doesn't sound like that many VMs? Is that a lot of VMs?

zamadatix 2y ago
In terms of VMware customers it isn't a ton of VM but not peanuts either. E.g. the last healthcare place I was at (single customer rather than cloud provider) had ~30k VMware VMs and we were still small fish compared to some others. I've heard of places 10x the VM count as this making the move post acquisition - albeit less publicly. I think the purpose of the article is to highlight companies like this are starting/continuing to migrate post-acquisition rather than this particular customer was impressively large and did so. Particularly with the bits about the relative cost increase even though the customer was willing to walk away if needed.
- wink 2y ago
  Single customer 30k sounds a lot easier to migrate (or at least to schedule) than 12k with potentially 3k different customers (probably more like several dozens or a couple hundred)
  - zamadatix 2y ago
    Could be, lots of pros and cons to each scenario - doubt either are easy by any measure. E.g. for about 20% we didn't even have a single piece of documentation other than the server name for who might actually care about the VM going down for the migration we wanted to schedule. Let alone how to test the migration, when it is best to do it, what software was actually running on it, if it's actually managed/monitored integrated with/by other systems whoch need to be looked at too, or if it could just be shit down instead (yay healthcare mergers and acqs). Our migration was also to VMware from (mostly) Hyper-V at the time, so not as much custom tooling needed. On the flip side a cloud provider is going to have all of the owner contact info but no direct control of the guest OS to effect the change so the battle is more with trying to get the customers to care enough to do the migration with you but not be so bothered by it all they up and leave your hosting.Not exactly a walk in the park either. In either case - almost never the tech that's the hard part for sure :).
aaronax 2y ago
Difficult to say. That could conceivably fit in one rack of 60 compute nodes (1/2U size) at 200VMs per node, leaving 15U for networking and SAN. Maybe $100/year/VM (rough cost of a lower-end cloud VM like EC2, droplet, etc.) in that case so $1.2 million per year cost. Or it could take 10 racks and $50 million per year.
- Symbiote 2y ago
  The article says, right at the start, > Anexia was founded in 2006, is based in Austria, and provides cloud services from over 100 locations around the world by placing equipment in third party datacenters. From the company's homepage: > The founder and CEO of Anexia [...] recently acquired a small hydropower plant in Kammern in the Liesingtal region of Styria for a “significant seven-figure sum” – i.e. several million euros. The power plant on the River Liesing generates 600 KW of electricity, enough to cover a third of the electricity consumption of Anexia’s Vienna data center so this seems to be a significant operation.
  - quesomaster9000 2y ago
    While average power consumption per rack has been increasing fairly steadily over the past 10 years, the metric I currently use is abound 10kW per rack under reasonable to heavy load - that's about the same as a consumer electric shower. So, this is implying their Vienna data center has 180 racks? With 60 being about a third, if we say each rack has 40 servers... that's ~7k servers total... which is a sizeable chunk of floor space, like 3000m^2, or... 40 tennis courts? But yea, that a non-insignificant operation just for the Vienna data center.
zipy124 2y ago
Entirely depends on context, like asking how long is a piece of string. One VM could be 200 cores, or it could be 1 core. It could also be a kubernetes/docker worker as-well so one VM may be thousands of containers. Finally they could instead just be important VM's. You could imagine a small or medium company having maybe 4 VM's each for prod, staging, testing etc... letting CDN's handle scaling (with everything else running on local dev machines) and so that 12,000 could be an entire 3,000 companies whole stacks.
mrweasel 2y ago
That depends on your context. 12,000 VMs is enough to a fairly large chunk of a small countries healthcare infrastructure, if not all of it. It a pretty decent amount of VMs, but not close to being unmanageable. I think it's more down to how the rest of your infrastructure looks, if we're talking about ease of migration.
rwmj 2y ago
I'm involved with Red Hat's effort to shift customers off VMware (upstream project: https://github.com/libguestfs/virt-v2v https://github.com/libguestfs/virt-v2v). Things have really blown up since the Broadcom acquisition at the end of 2023. For us 12K VMs is a medium to large customer, but definitely not unusual. Think someone like a regional bank. A full conversion of such a customer might involve one consultant on-site, a scoping exercise to classify the VMs into groups and assess which ones are going to be more or less difficult to convert, and perhaps 1-3 months of work to convert them all. Individual VMs would be down from anything between a simple reboot, up to 12 hours, depending on which strategy we used for conversion (there are complicated trade-offs related to storage and network bandwidth).
- justinclift 2y ago
  Sounds like it's similar to what happened with PostgreSQL after Oracle bought MySQL. ;)

wkat4242 2y ago

Good to see companies calling Broadcom's bluff!

neilv 2y ago

Did Broadcom ever justify price-gouging captive customers after acquiring VMware? ("Sorry, folks, poor banana harvest this year, so we have to pass on the skyrocketing cost of bits licensing, which runs on bananas"?)

rad_gruchalski 2y ago
They spent money on the acquisition and they would like to see some back. Isn’t that obvious?
- toomuchtodo 2y ago
  There is a lot of road between "reasonable return on capital invested" and "we are going to rake you over the coals so badly people are going to publicly flee to not be captive to us."
  - rad_gruchalski 2y ago
    Well, it’s Broadcom and they just spent the money so they want some back. It’s like IBM: acquire and rake. Hey IBM employees, it doesn’t matter how upset you get. It is what it is.
- 0xedd 2y ago
  [dead]
- burnte 2y ago
  No one said it wasn't obvious, we're saying spending almost $70bn on a company and announcing that the only way you can make it back is by squeezing 10x more out of the big fish, that might not have been smart.
  - rad_gruchalski 2y ago
    Apparently they are known for that so I'm not sure why acting surprised.
2y ago

throw310822 2y ago

A-nexia, does it mean "lack of connectivity"? Great name for a cloud provider :)

szszrk 2y ago
It's close to Anoxia and Alexia, which are also "nice" choices to be known for.
pmontra 2y ago
Or "to annex" [1], as in all your VMs (are) belong to us [2] [1] https://www.merriam-webster.com/dictionary/annex https://www.merriam-webster.com/dictionary/annex [2] https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us https://en.wikipedia.org/wiki/All_your_base_are_belong_to_us
kgeist 2y ago
Anexia means "patience, humility" in Ancient Greek. Maybe that's what they were after?

mrtksn 2y ago

If the claims about AI capabilities are true, it's matter of time before all providers to switch to homegrown solutions.

cruffle_duffle 2y ago
The funny part is you can convince an LLM any bullshit you propose is right. It will agree with anything you say unless you encourage it not to and even then… “VMWare costs too much. Our use case is incredibly niche and nothing out there seems like it would fulfill our needs. Can you help me write a new stack using bash scripts and AWK?” “What an exciting idea! Bash Scripting is a clever way to orchestrate 12,000 virtual machines. Here, let me provide you with a bunch of code to copy and paste without doing any requirements gathering whatsoever. <pages of random bash scripts> These bash scripts will let you run 12,000 virtual machines. Let me know if you need any explanation!” “I forgot to me to mention we need to provision these using a rest api. Also these are globally distributed and require extensive monitoring” “Not a problem! REST is an excellent way to communicate with the platform. I won’t bother asking any follow up questions or gathering any more requirements. Here is all the code you need:” … and so on…
- mrtksn 2y ago
  If you know what are you doing in general, I'm convinced that it can actually help you out but I had 0 success with doing it brainfuck(chatgpt claims that anything beyond hello world is too complicated haha). Just like with the image and video generators, I'm fishing to find evidence of machines building something useful - had very little success so far. If the claims are true and these huge investments are not done in vain, not too far in the future we should be able to tell AI to build the tools we need, for example Microsoft should be expecting to discontinue MS Office either because someone can tell AI to make them an Excel instead of paying a subscription or tell AI to just do the job they will do in Excel. It's very strange that they talk about replacing the developers with AI but somehow still sell software. Something doesn't add up.
  - kube-system 2y ago

0xedd 2y ago

[dead]

homebrewer 2y ago

They only mention KVM, however KVM is a kernel API and not a full hypervisor, and can't do half the things they claim it does (like support for certain disk images). What hypervisor are they actually using, does anybody know? Probably qemu? It doesn't feel like a major enough company to be able to write its own like Google did.

klooney 2y ago
They probably mean libvirtd/qemu, when people say "we used kvm" that's generally what they mean.
freedomben 2y ago
They are most assuredly using qemu. It's been my experience that when most people say "KVM" they usually are referring to the whole libvirt ecosystem. I assume that's the case here. I think of it very analogously to how people say "linux" to refer to a full OS/distro such as Fedora. Though "linux" is just a kernel, it's become common parlance.
blueflow 2y ago
KVM is not only the kernel side - the userspace side of KVM was implemented in a fork of qemu. That fork later got merged into mainline qemu, so qemu is the "original" implementation of a KVM hypervisor.
deleted 2y ago
[deleted]
bertman 2y ago
Netcup seem to use qemu[0], which is what Anexia will also use according to OP's article [0]https://old.reddit.com/r/VPS/comments/1ghz4fd/i_know_contabo_uses_proxmox_what_does_hetzner_and/lv2amg7/ https://old.reddit.com/r/VPS/comments/1ghz4fd/i_know_contabo...

freedomben 2y ago

I definitely don't want to miniminize the significance of this. This is huge! But, they did have some major benefits that most companies looking to do the same won't: > Anexia therefore resolved to migrate, a choice made easier by its ownership of another hosting business called Netcup that ran on a KVM-based platform. > The hosting company is also a big user of NetApp storage, so customer data was already stored in a resource independent of its VMware rig – any new VMs would just need to be pointed at existing volumes. Again, still a great accomplishment and an exciting milestone for them, but for people still stuck on VMWare that are looking to migrate, it's good to know about the above things.

tw04 2y ago
Other than the additional capital outlay, this shouldn't prove a hurdle to anyone looking to do similar. Migrating from VSAN to external storage like NetApp is a non-disruptive process. And frankly VSAN has always been pretty horrible, so you'll likely end up with better performance and storage efficiency when all is said and done.

amluto 2y ago

I found this rather odd: > “We used to pay for VMware software one month in arrears,” he said. “With Broadcom we had to pay a year in advance with a two-year contract.” If your goal is to extract every possible cent from your existing customers, why would you also switch them from net 30 to requiring partial prepayment? VMware wants money in general but should not have a cash flow problem, and forcing a monster early payment seems like it will force customers to notice an immediate problem and make a choice instead of allowing themselves to be slowly and steadily ripped off. If I were a pointy-haired CEO committed to the multiply-pricing-by-five strategy, I would do my best to sweeten the deal: offer generous payment terms, give nice-sounding discounts for up front commitments, give very large discounts for nodes that haven’t yet been leased to a customer, etc.

shepardrtc 2y ago
Either their sales team absolutely screwed this up, or they just don't want to bother with the VMware platform anymore. Maybe both?
- nyrikki 2y ago
  Broadcom aspires to the Oracle model, they said as much when they bought VMware.
diggan 2y ago
Perhaps similar strategy as spammers employ today? Where they try to filter away people quickly who wouldn't fall for it when they need it to. So do something slightly outrageous today, so you filter away the ones that won't stick around for the future more outrageous changes.
nolok 2y ago
Because they have twelve thousands vms and are themselves a provider that can't afford to have downtime for its customers. So the thinking here was probably "there is no way they can refuse to sign right now and destroy their business in the process, so we might as well take the cake and also force them to stay after so they don't leave in 11 months and 29 days". Turns out that thinking is wrong for that specific customers, but for how many did it work ?

segasaturn 2y ago

Anybody who formalizes their "homebrew KVM platform" into a marketable hypervisor product is going to make a lot of money I suspect. Every IT department I know is scrambling to replace their VMWare stack ASAP, including very large ones.

formerly_proven 2y ago
> Every IT department I know is scrambling to replace their VMWare stack ASAP, including very large ones. At least some of the big ones seem to just pay up. Probably because they / their MSP / their relationship with their MSP is so dysfunctional that they know migrating is a pipe dream.
- stackskipton 2y ago
  As former DevOps who dealt with VMware, it's not relationship with MSP. It's just so many things plugin to VMware that migrating off of VMware is just difficult. Monitoring, Backups, Deployment and so forth are deeply integrated into VMware so companies just look at work involved getting off and go, never mind.
dehrmann 2y ago
Most things that would have used VMware 15 years ago are using Kubernetes now. The things that aren't are probably looking at Proxmox or just KVM.
- taskforcegemini 2y ago
  >VMware 15 years ago are using Kubernetes now. this doesn't sound right, both serve different purposes
intelVISA 2y ago
Soon.
icedchai 2y ago
Is there a reason Proxmox wouldn't work for many of them?

raincom 2y ago

12000 VMs = 800 to 1200 baremetals. Since KVM can run vmdk images, it is more of how to manage all these barementals and VMs from a central place.

olavgg 2y ago
That depends, on a modern Epyc, you can easily run a few thousand VM's on one system.
- menaerus 2y ago
  How do you run thousands of VM's on 192 physical cores? That must be assuming that large percentage of VM's are idling most of the time.
  - justinclift 2y ago
    That's pretty commonly the case.
    - menaerus 2y ago
      I understand but I'm just curious to learn under what conditions.
AviationAtom 2y ago
We have run 8,000 VMs on ~120 servers

rcleveng 2y ago

The phrase "homebrew KVM platform" made me chuckle. All of the hyperscalers have a homebrew, aka proprietary platform using a hypervisor. AWS has Nitro which seems to have been based on KVM but likely quite different by now. If you are selling VMs to customers, I can't understand a good reason to use VMWare. The only reason would be if you are selling VMWare as a service.

TheNewsIsHere 2y ago
If you’re providing application services to customers who need to use an application that the vendor only supports in VMware, then you don’t have a choice. Well, I suppose you do. But an EMR/EHR for instance is going to _need_ vendor support, which means requiring VMware even if you’re not selling VMware itself as a service.
- rcleveng 2y ago
  Absolutely - in enterprise software, it's the certified configurations that are not something you can mess around with, if you like support. I'd still consider this VMware as a service, although not full VMware, but just enough for the checkboxes. Maybe you don't get API access, console access, etc, but the main thing you are selling is vmware (to check the certification boxes), and not a generic VM.
monocasa 2y ago
> AWS has Nitro which seems to have been based on KVM but likely quite different by now. It still is relatively stock KVM on the CPU side of things. They've been upstreaming changes they need like lower overhead for emulating Xen's hypercall interface. Most of their special sauce is in the devices though, as those natively provide VM boundaries leaving the hypervisor to not have to manage all that much at runtime.
dilyevsky 2y ago
The challenging bits are all outside of the KVM like the VPC networking that has to be implemented using some SDN (e.g OpenVSwitch), block devices, etc VMware had a solution for all of these natively and with support. Not using their hypervisors you have to manage a huge pile of OSS+proprietary integrations and actually have staff who truly understand how everything works down to the lowest level. Doable but probably above the pay grade for most

smetj 2y ago

Congratulations Anexia, well done! Pulling this off means you have experienced leaders and excellent engineers.

pcblues 2y ago

I heard first-hand years ago of a bug in VMWare that was _CPU-Level_ for the company that rolled it out in an effort to replace their PCs that had software locally installed. The failure would have been catastrophic for a company that had just replaced all its PCs. However, they hadn't shipped the old PCs away yet for destruction, and managed to keep the lights on by putting them all back out again until the problem was fixed.

hd4 2y ago

Hard to imagine how other similar companies wouldn't just follow suit even if the cost of staying with VMware wasn't prohibitive (which is unlikely for most cases, probably). Stupid business decisions should be punished by the customer walking out. Homebrew KVM is not that hard.

leoh 2y ago

Incredible to see how much value Broadcom is destroying.