3 ms·
Wow, that's a brutal under-the-bus-tossing. But well deserved. Most of the points are spot-on. I've operated a several dozen machine fleet for 2+ years on EC2
by newobj 16y ago
Wow, that's a brutal under-the-bus-tossing. But well deserved. Most of the points are spot-on.
I've operated a several dozen machine fleet for 2+ years on EC2 and I can tell you that the number of times boxes have gone down never to come back (which most people tend to think happens regularly on EC2) is incredibly small.
We actually run a Hadoop cluster (including DFS) on a spot instances. We never lose the spot bid. We pay way less than the going rate for the compute time. Less than the reserved instance rate. It's awesome. Yes, you obviously need a plan to deal with your cluster vanishing in the blink of an eye. It's not too hard.
I would second another commenter's caution on EBS though. We never put it into production. Personally I never experienced an ephemeral drive failure that had repercussions - when they (rarely) occurred, the drive was RAIDed or in our Hadoop cluster (e.g. redundant). We made two experiments into using EBS with our DB's, and both times, literally within 24 hours, we experienced a catastrophic failure of the EBS volume, one time unrecoverable. So that put me pretty well off EBS.
I can't say that the performance is necessarily the best, and we do experience the occasional odd asymmetrical inter-machine latencies (e.g. 300ms to establish a tcp connection in one direction, but normal <1ms in the other), but for the most part AWS is just awesome.
- gruseom 16y agoYes, you obviously need a plan to deal with your cluster vanishing in the blink of an eye. How long do the spot instances typically run before they vanish?
- cperciva 16y agoA spot instance should stay around until the auction price is higher than what you bid. That price has been quite stable recently, so as long as you bid slightly higher than that rate you'll probably have your instance for months.
- gruseom 16y agoThat's better than I was expecting. I'm surprised more people don't take advantage of this (or maybe they do, I don't know). Use S3 for permanent storage, ephemeral disk and RAM for cache, and pay spot rates. Sounds pretty good if one's design allows for it.
- count 16y agoI actually did a market analysis of the EC2 spot price (they give you a year of price history in JSON format). The spot price was under the reserved price for all but a few hours since it came out. The savings in general were around 30-50% (depending on instance size), with higher savings coming from using spot market pricing in the non-VA markets (CA, EU, Asia), where prices were higher to start with. Why don't people just bid the reserve price on the spot market, and never lose the bid? You'll still pay less...
- cperciva 16y agoWhy don't people just bid the reserve price on the spot market, and never lose the bid? Because you will lose the bid if a disaster occurs. A lot of reserved instances have been purchased for disaster-recovery purposes but aren't actively being used; those are instead being sold as spot instances. If a disaster occurs, the supply of spot instances will drop dramatically.
- newobj 16y agoThey only vanished once (the entire bid and therefore the entire cluster), and I think it was actually due to a glitch; the price never actually went over our bid. That's the only time we ever lost our spot instances. I think we actually bid over even the normal (non-reserved pricing), because I've (almost) never seen prices cross the reserve price, and even if we had to pay full price for a day, that's fine -- it's still tremendous savings in the long run.