3 ms·

For my last start-up, I moved off EC2 to a dedicated vSphere cluster with a hosted provider. vSphere has an API, so adapting existing provisioning & deployment

For my last start-up, I moved off EC2 to a dedicated vSphere cluster with a hosted provider. vSphere has an API, so adapting existing provisioning & deployment code was quite straightforward (I used the fog gem in Ruby). I basically treated the vSphere cluster very similarly to EC2 (small root stores, attached volumes, etc). Granted, I did have to give up the benefit of being in different regions. I found the maintenance burden dropped substantially. It may have just been that I was running on newer hardware, but vSphere has built-in HA features such that VMs will just migrate between hosts when hardware degrades. In the two years I ran that setup, I never lost a VM. I also had dedicated, modern hardware -- no need to worry about CPU steal. I could create instances/VMs of any size I want. If I needed to resize a VM, I could do so without losing all the data on it. As long as I had capacity, I could add new VMs at no extra cost. When I needed more capacity, I'd just call up and have them add a new blade to the cluster, which both increased my redundancy and gave me extra capacity. Essentially, the cost curve, while tied to usage, was much more favorable than linear growth once you get over the base setup costs. On top of that, I had a real console to each of the VMs should anything go wrong. And if I couldn't fix something myself, there was dedicated staff at the colo facility I could just call up. It's not for everyone, but there are options out there that give you an EC2-like environment with many of the benefits of having your own hardware.

throwaway2016a 11y ago
The tradeoff for CPU stealing is that now you have multiple services running on the same machine so if the machine goes down you lose them all. On AWS it is very unlikely that two of your VMs end up on the same physical machine. Also, I actually can't recall (in 8 years of AWS) a time where I saw CPU being stolen. Perhaps I've been lucky. I also don't watch for it constantly. We actually did this with our dev servers. Although switch VSphere for VirtualBox on Linux and colo for a server room in our office. It worked great and saved a TON of cost. But we don't need pesky things like 24hr uptime and dedicated bandwidth for dev servers.
- toomuchtodo 11y ago
  > The tradeoff for CPU stealing is that now you have multiple services running on the same machine so if the machine goes down you lose them all. On AWS it is very unlikely that two of your VMs end up on the same physical machine. Yes. But its also very likely that when AWS has issues, the entire region is going to be having problems, like last Saturday (IAM, EC2, Autoscaling, etc broken badly for 6 hours). > But we don't need pesky things like 24hr uptime and dedicated bandwidth for dev servers. You're not getting this SLA unless you're managing everything yourself and are globally redundant. AWS doesn't provide bandwidth guarantees, nor will you get 100% uptime.
  - throwaway2016a 11y ago
    True. You have to design for failure with AWS. Which does have a large amount of cognitive overhead. If you don't design for failure you're going to have a bad time. However, designing for strict SLAs is not impossible with AWS. You just need to have multi-region redundancy and you can get very very good with multi-availability zone redundancy. I have no excuse for the IAM outage, it sucked for our ops team. I guess my only two consultations are: 1. We haven't had a customer-visible outage due to AWS in years because we follow best practices (which does cost more - but see my previous point on more smaller machines vs many large machines) 2. If we were running our own authentication and access control system similar to IAM, it too could have an outage. But I agree, that is a bad thing about AWS.
- nirvdrum 11y ago
  Well, this is what I was referring to with the HA features of vSphere. It works basically like a compute level RAID. The VMs were stored on a SAN and vSphere monitored each of the compute blades. If one went down, the VMs were seamlessly migrated to a hot spare blade. The vSphere docs claim this can be achieved with zero packet loss -- a claim I was never able to test or verify. If you're worried about losing multiple machines, just add multiple hot spares. Of course, this doesn't help if you lose an entire rack or the data center. I concede this was a trade-off. But given how many times an entire region went down when I was on EC2, I was satisfied with the risk based on the colo environment's uptime record. The provider did offer another facility, but the latency between the two was too high to be of practical use in a failover without keeping a completely mirrored configuration in both locations. It sounds like you have some experience with vSphere, so I don't intend to be patronizing. But there's a huge difference between "enterprise" virtualization and what you get with an ad hoc setup using desktop virtualization tools.

throwaway2016a 11y ago

The tradeoff for CPU stealing is that now you have multiple services running on the same machine so if the machine goes down you lose them all. On AWS it is very unlikely that two of your VMs end up on the same physical machine. Also, I actually can't recall (in 8 years of AWS) a time where I saw CPU being stolen. Perhaps I've been lucky. I also don't watch for it constantly. We actually did this with our dev servers. Although switch VSphere for VirtualBox on Linux and colo for a server room in our office. It worked great and saved a TON of cost. But we don't need pesky things like 24hr uptime and dedicated bandwidth for dev servers.

toomuchtodo 11y ago
> The tradeoff for CPU stealing is that now you have multiple services running on the same machine so if the machine goes down you lose them all. On AWS it is very unlikely that two of your VMs end up on the same physical machine. Yes. But its also very likely that when AWS has issues, the entire region is going to be having problems, like last Saturday (IAM, EC2, Autoscaling, etc broken badly for 6 hours). > But we don't need pesky things like 24hr uptime and dedicated bandwidth for dev servers. You're not getting this SLA unless you're managing everything yourself and are globally redundant. AWS doesn't provide bandwidth guarantees, nor will you get 100% uptime.
- throwaway2016a 11y ago
  True. You have to design for failure with AWS. Which does have a large amount of cognitive overhead. If you don't design for failure you're going to have a bad time. However, designing for strict SLAs is not impossible with AWS. You just need to have multi-region redundancy and you can get very very good with multi-availability zone redundancy. I have no excuse for the IAM outage, it sucked for our ops team. I guess my only two consultations are: 1. We haven't had a customer-visible outage due to AWS in years because we follow best practices (which does cost more - but see my previous point on more smaller machines vs many large machines) 2. If we were running our own authentication and access control system similar to IAM, it too could have an outage. But I agree, that is a bad thing about AWS.
nirvdrum 11y ago
Well, this is what I was referring to with the HA features of vSphere. It works basically like a compute level RAID. The VMs were stored on a SAN and vSphere monitored each of the compute blades. If one went down, the VMs were seamlessly migrated to a hot spare blade. The vSphere docs claim this can be achieved with zero packet loss -- a claim I was never able to test or verify. If you're worried about losing multiple machines, just add multiple hot spares. Of course, this doesn't help if you lose an entire rack or the data center. I concede this was a trade-off. But given how many times an entire region went down when I was on EC2, I was satisfied with the risk based on the colo environment's uptime record. The provider did offer another facility, but the latency between the two was too high to be of practical use in a failover without keeping a completely mirrored configuration in both locations. It sounds like you have some experience with vSphere, so I don't intend to be patronizing. But there's a huge difference between "enterprise" virtualization and what you get with an ad hoc setup using desktop virtualization tools.