9 ms·

I think this only solves the cost problem. VMware is damn expensive for what it is. I'm not really a fan of virtualization myself. We canned it at our organisa

by hp50g 14y ago

I think this only solves the cost problem. VMware is damn expensive for what it is. I'm not really a fan of virtualization myself. We canned it at our organisation a couple of years ago. We had 30 ESX/vSphere managed hosts across 3 data centers which hosted app servers, web front end servers, virtual load balancers and some other minor infrastructure. The final straw was the upgrade costs. Not only that we had problems with volume size limits on our SAN which would require more expense and hackery to work around. Also, we couldn't host our database servers on top of it due to performance problems so we had to have dedicated machines there anyway. The whole thing at the end of the day just added complexity, expense and didn't improve reliability, security or load distribution as our architecture was sound on that front already. We've gone back to the original virtualization system: processes and sensible distribution of them across multiple machines. Performance, cost and sanity have improved. I can understand that virtualization is useful when your resource requirement is less than one machine but above that, I doubt there are any real benefits. It's snake oil.

IheartApplesDix 14y ago
Virtualization is a requirement for any kind of serious dynamic infrastructure.
- hp50g 14y ago
  Processes are a form of virtualisation are they not? That's enough abstraction.
  - wmf 14y ago
    Originally, yes. Unfortunately Unix/Windows has accumulated a ton of global state (libfoo 3.7.1 not parallel installable with libfoo 3.6.4) and enterprise software is now so fragile that any alternative to VM sprawl is unthinkable.
    - hp50g 14y ago
      We run our software (and other "enterprise" software) multiple instances per node and they are fully isolated. Most is in JVMs but we have C++ stuff with no problems.
  - IheartApplesDix 14y ago
    Virtualization is a form of process so yes.
- Swannie 14y ago
  Except that Google, Facebook, etc. etc. successfully use baremetal deployments without a VM in sight...
hhw 14y ago
Exactly. Virtualization/Cloud are only good for scaling down, not scaling up. The 20-30% hit to IO, additional latency of SAN vs local storage, management overhead, and added complexity to infrastructure more than negate any potential benefits of improved load distribution when you're dealing with applications that use anywhere near the capacity of physical servers.
- druiid 14y ago
  20-30% hit in IO? It hasn't been at that level for a long time. With new KVM versions and good Intel processors there is a performance hit of as low as 5% these days. That's not to necessarily say your particular workload will see that, but 10% is generally an 'at-worst' level at this point.
  - hp50g 14y ago
    Its definitely 20-30% for realistic workloads using VM based tech on top I.e. CLR/JVM or a database engine which is realistic. This is on top of VMware. I can't speak for Xen. The outcome is pretty grim.
    - rektide 14y ago
      Ah, VMware is the problem, that is explains it. Parties that care muchly about fine performance margins apparently need to be using Xen or KVM or Illuminos then.
      - acdha 14y ago
        Can't speak for the parent poster's company but the numbers don't match my experience with VMware many years back. It's possible they've had a sharp regression but we were maxing out gigabit ethernet and local RAID arrays in 2006.
        druiid 14y ago
        Well, don't confuse I/O with throughput here. You can look at performance numbers for just about anything and tweak one direction or the other. For instance it's easy to make a benchmark showing huge throughput to any given storage solution (and many NAS providers sell on this basis), but your I/O might be terrible because to get that throughput you're maxing the CPU (etc.). Likewise, you can change your benchmark and show high I/O, but the throughput is 'terrible'.
incision 14y ago
>It's snake oil. As opposed to a miracle elixir that cures all ills? It's a tool. Like most tools, it's best used when you thoroughly understand the challenge and how/why to apply that tool to it. The problems you've noted here seem more indicative of poor planning and understanding of the solution than any intrinsic deficiency of virtualization.
- hp50g 14y ago
  Its a shit tool that creates issues and costs fuckwads of cash. That is it. We understand it. The numbers sold are not the numbers gained. The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead.
  - incision 14y ago
    >Its a shit tool that creates issues and costs fuckwads of cash. That is it. Calm down please. >The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead. That might say something about VMWare's specific solution and you being taken for a ride, but it's nothing intrinsic to virtualization as a tool.
  - druiid 14y ago
    VMWare sales from what I have heard, has a history of over-selling, over-promising and ultimately getting people to spend a crap-ton of money on their products. Honestly I have to say, it sounds a bit like you guys got taken for a ride. I'd suggest sitting back, relaxing and taking a look at what other VM options there are out there, some of which don't have license fees attached to them. When I first deployed virtualization there were really only two 'enterprise' ready solutions out there which were essentially VMWare or XenServer. Knowing about VMWare and their 'history' I chose XenServer. I haven't regretted the choice but they wouldn't be a fit for everyone. These days there are multiple numbers of solutions out there to choose from, which Openstack may or may not be one for you... and that aren't going to whisper into your ear about how great their product is and how much money it will save you.
tsmith 14y ago
> I can understand that virtualization is useful when your resource requirement is less than one machine but above that, I doubt there are any real benefits. It's snake oil. Tell that to Netflix - or any of AWS' customers, really.
- hp50g 14y ago
  AWS is a prime example. EC2 unreliable, doesn't perform well and is expensive (and fairly weather dependent by the looks too). If you want anything that can actually shift anything, dedicated servers are cheaper (but you might actually have to commit - oh dear if your margin is that tight, you don't have a product worth it). The only use case I can see for it really is rapid scaling but that seems only viable for content delivery as your data back end architecture is way harder to scale than click the magic cloud button. Back in the old days we used to do content via CDN (Akamai etc) which actually works out cheaper per GiB. Then we approach things like S3 which is terribly unreliable (some of our clients use it for storage as it's cheaper than a SAN but they suffer for that). Drop outs, unreliable network requests, latency, rubbish throughput and buggy as fuck SDK. Its like consuming value line products from a supermarket. Sure there's quantity but its lacking in quality.
  - riotingpscifis 14y ago
    If you setup you're stack wrong then some of you're points are valid. But many amazon services offer you redundant infrastructure at a reasonable cost. But the key advantage over traditional hosted solution is that you can run reproducible test and dev stacks for 8/12 hours a day without having to pay for hardware that sits there doing nothing while you sleep. Sure cloud computing has it's faults but the ability to run you a HA test stack in 3/4 datacenters in minutes, and pay for just the time you are testing it, enables people to move forward and develop more and more impressive technology. OTOH I'm not sure why you would run cloud on your own hardware as you're already paying for the HW, I suppose it simplifies the management significantly for PayPal or they wouldn't be doing it
  - vacri 14y ago
    Take a deep breath, relax, and realise that plenty of people have use cases that are different to yours.
Peaker 14y ago
What was your SAN?
simplexion 14y ago
Volume size limits on your SAN... Is 16TB too restrictive for your volume size limit?
- hp50g 14y ago
  2TiB per LUN on ESX until recently actually. We have a single filesystem that is 19TiB.
  - simplexion 14y ago
    So... 19TiB and you're only using iSCSI instead of NFS and you're complaining about volume size restrictions?
    - hp50g 14y ago
      We have to use iSCSI. Not all our kit talks NFS plus we use SAN block replication between DCs which really doesn't play nice with NFS (we did test it and discovered that not all NFS implementations are good). This whole thing has to play nice with a windows DFS cluster as well.
      - simplexion 14y ago
        You said you're using ESXi? The Windows virtual servers won't care whether they are sitting on a LUN or NFS. Why do you have to use iSCSI? Or SAN block replication for that matter? If you think you need to use iSCSI, why don't you use it for some of your storage and NFS for parts that require storage beyond 2TB?

by hp50g 14y ago

IheartApplesDix 14y ago

Virtualization is a requirement for any kind of serious dynamic infrastructure.

hp50g 14y ago
Processes are a form of virtualisation are they not? That's enough abstraction.
- wmf 14y ago
  Originally, yes. Unfortunately Unix/Windows has accumulated a ton of global state (libfoo 3.7.1 not parallel installable with libfoo 3.6.4) and enterprise software is now so fragile that any alternative to VM sprawl is unthinkable.
  - hp50g 14y ago
    We run our software (and other "enterprise" software) multiple instances per node and they are fully isolated. Most is in JVMs but we have C++ stuff with no problems.
- IheartApplesDix 14y ago
  Virtualization is a form of process so yes.
Swannie 14y ago
Except that Google, Facebook, etc. etc. successfully use baremetal deployments without a VM in sight...

hhw 14y ago

Exactly. Virtualization/Cloud are only good for scaling down, not scaling up. The 20-30% hit to IO, additional latency of SAN vs local storage, management overhead, and added complexity to infrastructure more than negate any potential benefits of improved load distribution when you're dealing with applications that use anywhere near the capacity of physical servers.

druiid 14y ago
20-30% hit in IO? It hasn't been at that level for a long time. With new KVM versions and good Intel processors there is a performance hit of as low as 5% these days. That's not to necessarily say your particular workload will see that, but 10% is generally an 'at-worst' level at this point.
- hp50g 14y ago
  Its definitely 20-30% for realistic workloads using VM based tech on top I.e. CLR/JVM or a database engine which is realistic. This is on top of VMware. I can't speak for Xen. The outcome is pretty grim.
  - rektide 14y ago
    Ah, VMware is the problem, that is explains it. Parties that care muchly about fine performance margins apparently need to be using Xen or KVM or Illuminos then.
    - acdha 14y ago
      Can't speak for the parent poster's company but the numbers don't match my experience with VMware many years back. It's possible they've had a sharp regression but we were maxing out gigabit ethernet and local RAID arrays in 2006.
      - druiid 14y ago
        Well, don't confuse I/O with throughput here. You can look at performance numbers for just about anything and tweak one direction or the other. For instance it's easy to make a benchmark showing huge throughput to any given storage solution (and many NAS providers sell on this basis), but your I/O might be terrible because to get that throughput you're maxing the CPU (etc.). Likewise, you can change your benchmark and show high I/O, but the throughput is 'terrible'.

incision 14y ago

>It's snake oil. As opposed to a miracle elixir that cures all ills? It's a tool. Like most tools, it's best used when you thoroughly understand the challenge and how/why to apply that tool to it. The problems you've noted here seem more indicative of poor planning and understanding of the solution than any intrinsic deficiency of virtualization.

hp50g 14y ago
Its a shit tool that creates issues and costs fuckwads of cash. That is it. We understand it. The numbers sold are not the numbers gained. The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead.
- incision 14y ago
  >Its a shit tool that creates issues and costs fuckwads of cash. That is it. Calm down please. >The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead. That might say something about VMWare's specific solution and you being taken for a ride, but it's nothing intrinsic to virtualization as a tool.
- druiid 14y ago
  VMWare sales from what I have heard, has a history of over-selling, over-promising and ultimately getting people to spend a crap-ton of money on their products. Honestly I have to say, it sounds a bit like you guys got taken for a ride. I'd suggest sitting back, relaxing and taking a look at what other VM options there are out there, some of which don't have license fees attached to them. When I first deployed virtualization there were really only two 'enterprise' ready solutions out there which were essentially VMWare or XenServer. Knowing about VMWare and their 'history' I chose XenServer. I haven't regretted the choice but they wouldn't be a fit for everyone. These days there are multiple numbers of solutions out there to choose from, which Openstack may or may not be one for you... and that aren't going to whisper into your ear about how great their product is and how much money it will save you.

tsmith 14y ago

> I can understand that virtualization is useful when your resource requirement is less than one machine but above that, I doubt there are any real benefits. It's snake oil. Tell that to Netflix - or any of AWS' customers, really.

hp50g 14y ago
AWS is a prime example. EC2 unreliable, doesn't perform well and is expensive (and fairly weather dependent by the looks too). If you want anything that can actually shift anything, dedicated servers are cheaper (but you might actually have to commit - oh dear if your margin is that tight, you don't have a product worth it). The only use case I can see for it really is rapid scaling but that seems only viable for content delivery as your data back end architecture is way harder to scale than click the magic cloud button. Back in the old days we used to do content via CDN (Akamai etc) which actually works out cheaper per GiB. Then we approach things like S3 which is terribly unreliable (some of our clients use it for storage as it's cheaper than a SAN but they suffer for that). Drop outs, unreliable network requests, latency, rubbish throughput and buggy as fuck SDK. Its like consuming value line products from a supermarket. Sure there's quantity but its lacking in quality.
- riotingpscifis 14y ago
  If you setup you're stack wrong then some of you're points are valid. But many amazon services offer you redundant infrastructure at a reasonable cost. But the key advantage over traditional hosted solution is that you can run reproducible test and dev stacks for 8/12 hours a day without having to pay for hardware that sits there doing nothing while you sleep. Sure cloud computing has it's faults but the ability to run you a HA test stack in 3/4 datacenters in minutes, and pay for just the time you are testing it, enables people to move forward and develop more and more impressive technology. OTOH I'm not sure why you would run cloud on your own hardware as you're already paying for the HW, I suppose it simplifies the management significantly for PayPal or they wouldn't be doing it
- vacri 14y ago
  Take a deep breath, relax, and realise that plenty of people have use cases that are different to yours.

Peaker 14y ago

What was your SAN?

simplexion 14y ago

Volume size limits on your SAN... Is 16TB too restrictive for your volume size limit?

hp50g 14y ago
2TiB per LUN on ESX until recently actually. We have a single filesystem that is 19TiB.
- simplexion 14y ago
  So... 19TiB and you're only using iSCSI instead of NFS and you're complaining about volume size restrictions?
  - hp50g 14y ago
    We have to use iSCSI. Not all our kit talks NFS plus we use SAN block replication between DCs which really doesn't play nice with NFS (we did test it and discovered that not all NFS implementations are good). This whole thing has to play nice with a windows DFS cluster as well.
    - simplexion 14y ago
      You said you're using ESXi? The Windows virtual servers won't care whether they are sitting on a LUN or NFS. Why do you have to use iSCSI? Or SAN block replication for that matter? If you think you need to use iSCSI, why don't you use it for some of your storage and NFS for parts that require storage beyond 2TB?