5 ms·

I wrote a list of jobs I could perhaps pipe to this method of instance procurement but the fact that the process could be terminated at any moment really messes

by dotBen 17y ago

I wrote a list of jobs I could perhaps pipe to this method of instance procurement but the fact that the process could be terminated at any moment really messes things up. I'm not even talking about "high-availability", but the cessation of a key node in a dependency tree of instances messes stuff up (at least for my problem space). What is missing here is a "5 minute warning" API call before shutdown that would let me engineer in a) the opportunity to save any long-duration calculation to disk and b) have my software make a decision to instantiate a full-price instance if my queue is too long and having no cheap-o priced instances running is going to mess things up. Thoughts?

cperciva 17y ago
b) have my software make a decision to instantiate a full-price instance if my queue is too long and having no cheap-o priced instances running is going to mess things up. If having your instance die is going to cause problems, why not just bid higher in the first place? You're only going to pay the spot price, regardless of what you bid.
- dotBen 17y ago
  That's true. but thinking that out... I guess if Amazon allowed everyone to programatically do that via an API warning then I guess everyone would simply build their software to bid up 1c at the time of warning... and at scale, everyone doing that would simply inflate the spot price higher and high to the point it would approach the full cost.
  - kevinpet 17y ago
    That's assuming people need to keep things running at all times, whatever the cost. This is for those people who want to do their calculations only when CPU is cheap.
    - dotBen 17y ago
      No because this could become a hack... using the mechanism I describe above I could keep instances going at below market rate. I can see some people gaming the system in this way if the system allows it
      - bdonlan 17y ago
        The instance always runs at market rate. You set a _maximum_ price but are charged less than that if the market rate is lower. So there's no sense in doing such a reactive system when amazon does it for you.
tb 17y ago
Run the key nodes on ordinary EC2 nodes, and only use Spot nodes for things that you can cope with disappearing i.e. things that can be frequently checkpointed and won't disturb other nodes when they disappear.
fragmede 17y ago
Some sort of warning system sounds nice, but since your program should be saving as often as it can anyway, a 5 minute warning wouldn't actually be useful because if there was a save point, you've saved, and if there wasn't, well, there wasn't and you couldn't have saved given a 5 minute warning anyway. Amazon also, of course, pushes all their other products to help deal w/ this problem. They do also mention that it is 'by the hour', which I'd double check that it doesn't include fractional hours, but after the instance has been up for ~55 minutes, you have your 5 minute warning. You do, however have a really good point with the up-sell off the 5-minute warning. I wonder if that could be too easily gamed. When that 5 minute warning comes the spot instances tell the normal-instance key node, which then requests a bunch of spot instances at 2x the current spot-instance price to be able to finish the current 'set'. (But then also request a bunch more spot instances at below the current spot price, in an attempt to drive down spot instance pricing.) The other thing is that it is region specific; which means that you could target the EU region from the US west coast to get work done on spot-instances during the day.
- wmf 17y ago
  after the instance has been up for ~55 minutes, you have your 5 minute warning. Nope; Amazon can terminate spot instances at any time. Thought experiment: Assume EC2 is full (which it will be from now on) and someone requests a regular instance. To create the instance promptly, EC2 will have to shut down a spot instance. (But then also request a bunch more spot instances at below the current spot price, in an attempt to drive down spot instance pricing.) That's not how auctions work. The number of bids below the winning price is irrelevant.
- dotBen 17y ago
  <i>a 5 minute warning wouldn't actually be useful because if there was a save point, you've saved,</i> Not with cloud storage where disk I/O costs money. You really only want to save at the end of a task rather than during it. Or you save the results off-cloud in a back store hosted elsewhere. (all of which assumes you are working at scale, where the cumulative costs become prohibitive)
wmf 17y ago
have my software make a decision to instantiate a full-price instance if my queue is too long and having no cheap-o priced instances running is going to mess things up. Maybe auto scaling can do this for you; if it detects that the group has no instances it will start a new (full-priced) one.

by dotBen 17y ago

cperciva 17y ago

b) have my software make a decision to instantiate a full-price instance if my queue is too long and having no cheap-o priced instances running is going to mess things up. If having your instance die is going to cause problems, why not just bid higher in the first place? You're only going to pay the spot price, regardless of what you bid.

dotBen 17y ago
That's true. but thinking that out... I guess if Amazon allowed everyone to programatically do that via an API warning then I guess everyone would simply build their software to bid up 1c at the time of warning... and at scale, everyone doing that would simply inflate the spot price higher and high to the point it would approach the full cost.
- kevinpet 17y ago
  That's assuming people need to keep things running at all times, whatever the cost. This is for those people who want to do their calculations only when CPU is cheap.
  - dotBen 17y ago
    No because this could become a hack... using the mechanism I describe above I could keep instances going at below market rate. I can see some people gaming the system in this way if the system allows it
    - bdonlan 17y ago
      The instance always runs at market rate. You set a _maximum_ price but are charged less than that if the market rate is lower. So there's no sense in doing such a reactive system when amazon does it for you.

tb 17y ago

Run the key nodes on ordinary EC2 nodes, and only use Spot nodes for things that you can cope with disappearing i.e. things that can be frequently checkpointed and won't disturb other nodes when they disappear.

fragmede 17y ago

Some sort of warning system sounds nice, but since your program should be saving as often as it can anyway, a 5 minute warning wouldn't actually be useful because if there was a save point, you've saved, and if there wasn't, well, there wasn't and you couldn't have saved given a 5 minute warning anyway. Amazon also, of course, pushes all their other products to help deal w/ this problem. They do also mention that it is 'by the hour', which I'd double check that it doesn't include fractional hours, but after the instance has been up for ~55 minutes, you have your 5 minute warning. You do, however have a really good point with the up-sell off the 5-minute warning. I wonder if that could be too easily gamed. When that 5 minute warning comes the spot instances tell the normal-instance key node, which then requests a bunch of spot instances at 2x the current spot-instance price to be able to finish the current 'set'. (But then also request a bunch more spot instances at below the current spot price, in an attempt to drive down spot instance pricing.) The other thing is that it is region specific; which means that you could target the EU region from the US west coast to get work done on spot-instances during the day.

wmf 17y ago
after the instance has been up for ~55 minutes, you have your 5 minute warning. Nope; Amazon can terminate spot instances at any time. Thought experiment: Assume EC2 is full (which it will be from now on) and someone requests a regular instance. To create the instance promptly, EC2 will have to shut down a spot instance. (But then also request a bunch more spot instances at below the current spot price, in an attempt to drive down spot instance pricing.) That's not how auctions work. The number of bids below the winning price is irrelevant.
dotBen 17y ago
<i>a 5 minute warning wouldn't actually be useful because if there was a save point, you've saved,</i> Not with cloud storage where disk I/O costs money. You really only want to save at the end of a task rather than during it. Or you save the results off-cloud in a back store hosted elsewhere. (all of which assumes you are working at scale, where the cumulative costs become prohibitive)

wmf 17y ago

have my software make a decision to instantiate a full-price instance if my queue is too long and having no cheap-o priced instances running is going to mess things up. Maybe auto scaling can do this for you; if it detects that the group has no instances it will start a new (full-priced) one.