3 ms·

The documentation says that you get 30 seconds warning before your VM is forcibly terminated. Does anyone know if that's enough time to spin up a new instance t

The documentation says that you get 30 seconds warning before your VM is forcibly terminated. Does anyone know if that's enough time to spin up a new instance to take over from the one being preempted? I haven't messed with GCE much, so I don't know what the typical startup latency is.

boulos 11y ago
That's just about our median time to "create to ssh". I wouldn't encourage betting on being able to spin up a replacement when you're getting shot: that's precisely when we might be running out of capacity! If you're looking to checkpoint and save state though, you can mount your persistent disks separately and store whatever you need. You can also start your shot VM back up later when capacity is available again.
- toomuchtodo 11y ago
  Your team might want to consider extending the warning. AWS gives two minutes notice: https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termination-notices/ https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termi...
  - boulos 11y ago
    Thanks for the feedback. We explained the reasoning behind 30 seconds vs "longer" in the InfoQ interview I mentioned in a different thread. But basically if you're getting preempted it's often because some regular VM is trying to land on the box and needs to be kicked off. Our goal here is to make sure we don't impact even our 95th-percentile VM creation time for regular VMs. We create and boot VMs so much faster (I think even p90 is 36s right now) that "2 minutes" is more than it sounds like. Finally, one of the things we tracked carefully during Beta is how quickly people actually shut down. Median on Linux was something like 5 seconds. Even the 90th percentile was something like 10 or 15 seconds. There might be people avoiding the product because they "need" more time, but honestly most of those use cases aren't really prepared for preemption and should probably stick with a different solution.
    - toomuchtodo 11y ago
      > There might be people avoiding the product because they "need" more time, but honestly most of those use cases aren't really prepared for preemption and should probably stick with a different solution. I really have to say it, and I'm sorry to say so, but its things like this that keep people on AWS. Google Compute still has the feel of an experiment versus a business, and you don't want your critical infrastructure on someone's experiment. 2 minutes is provided by AWS to give you time to checkpoint your process(es), get logs off the box, and to remove from an ELB if its attached to one. Doesn't matter if you get a new VM for me spun up fast if I can't finish up housekeeping on the ones being terminated.

boulos 11y ago

That's just about our median time to "create to ssh". I wouldn't encourage betting on being able to spin up a replacement when you're getting shot: that's precisely when we might be running out of capacity! If you're looking to checkpoint and save state though, you can mount your persistent disks separately and store whatever you need. You can also start your shot VM back up later when capacity is available again.

toomuchtodo 11y ago
Your team might want to consider extending the warning. AWS gives two minutes notice: https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termination-notices/ https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termi...
- boulos 11y ago
  Thanks for the feedback. We explained the reasoning behind 30 seconds vs "longer" in the InfoQ interview I mentioned in a different thread. But basically if you're getting preempted it's often because some regular VM is trying to land on the box and needs to be kicked off. Our goal here is to make sure we don't impact even our 95th-percentile VM creation time for regular VMs. We create and boot VMs so much faster (I think even p90 is 36s right now) that "2 minutes" is more than it sounds like. Finally, one of the things we tracked carefully during Beta is how quickly people actually shut down. Median on Linux was something like 5 seconds. Even the 90th percentile was something like 10 or 15 seconds. There might be people avoiding the product because they "need" more time, but honestly most of those use cases aren't really prepared for preemption and should probably stick with a different solution.
  - toomuchtodo 11y ago
    > There might be people avoiding the product because they "need" more time, but honestly most of those use cases aren't really prepared for preemption and should probably stick with a different solution. I really have to say it, and I'm sorry to say so, but its things like this that keep people on AWS. Google Compute still has the feel of an experiment versus a business, and you don't want your critical infrastructure on someone's experiment. 2 minutes is provided by AWS to give you time to checkpoint your process(es), get logs off the box, and to remove from an ELB if its attached to one. Doesn't matter if you get a new VM for me spun up fast if I can't finish up housekeeping on the ones being terminated.