3 ms·
Seems to me that keeping an EC2 instance spun up and idle might become a sensible approach for people who run downtime-sensitive apps with a single AWS region a
by bigiain 14y ago
Seems to me that keeping an EC2 instance spun up and idle might become a sensible approach for people who run downtime-sensitive apps with a single AWS region architecture. A lot of recent outages have been exacerbated for many people by the flood of control plane traffic and contention for new instances - at some stage paying for enough idle instances (which might be just one) spread over availability zones to allow you to at least have some chance of surviving an entire datacenter going dark has to be at least worth running the numbers on.
I also predict the control plane contention problem is going to get worse - I'm sure I'm not the only one working on a system to spin up and grab a few extra instances as soon as my monitoring detects the beginnings of a problem. It seems to be "the right thing" to do is start spinning up and configuring replacement instances immediately you suspect a problem - even if it turns out not to be a problem most of the time, when it _is_ a problem at least you're well ahead in the queue of the people who chose to wait 5 or 10 mins on the assumption that perhaps it's only a temporary network glitch.
(And there's a weird negative incentive for Amazon here. If they fail to address the poor heavy load performance of the control plane/management console/provisioning api, they may end up with more money in their pockets from people keeping spare idle instances or provisioning additional instances speculatively at the first sign of trouble.)
- bigiain 14y ago"I'm sure I'm not the only one working on a system to spin up and grab a few extra instances as soon as my monitoring detects the beginnings of a problem" Evil idea of the day. Instead of actually spending money and buying possibly unneeded services from AWS, a somewhat less ethical person might choose to ddos the provisioning API at the first sign of trouble - then if it turns out that you _do_ need extra provisioning, you'll be first to know when the API becomes available (since you call the ddos off yourself when it suits you)…
- jacques_chester 14y agoIt's a classic tragedy of the commons. The normal response in this situation is to do away with the commons; a floating price for control plane requests, for example. I don't think that would be very popular, though. Like it or not, people designing for AWS must now assume that the control plane will simply be unavailable during partial outages.