4 ms·

Would this help in the case when sending shutdown to an EC2 instance sometimes takes 10+ minutes before the instance is stopped?

slovenlyrobot 7y ago
If I understand the EC2 instance lifecycle correctly, shutdown causes ACPI shutdown, and the machine does not enter stopped state until the OS in turn requests ACPI to power it off. That means shutdown time is dependent on the configuration of your software -- for example, if a single service is configured to gracefully wait up to 10 minutes for, say a persistent database client connection to disconnect, you might pay the full timeout during shutdown. There is also a 'forced shutdown' API, but for whatever reason in testing for me it has always been slower than a graceful shutdown. I have no idea what it does internally
- bostik 7y ago
  > There is also a 'forced shutdown' API, but for whatever reason in testing for me it has always been slower than a graceful shutdown. I have no idea what it does internally I have sort of reverse engineered what the two options ("shutdown", "force shutdown") in AWS do. A regular shutdown causes the hypervisor to issue the equivalent of "init 0" to the VM, over whatever communication channel is used between dom0 and domU. Semantically this is treated as a request. If the VM is completely catatonic, the domU kernel won't be able to process the request and will just fail. A forced shutdown causes the hypervisor to issue a similar request, but also puts the VM on a watchlist. If the domU kernel has not terminated in a given time window, probably 10 or 20 minutes[ß], the hypervisor will nuke the VM. ß: I'm not entirely sure about the time window, 10 and 20 minutes are what I have witnessed most often. I have on one occasion seen a VM take 90 minutes get nuked. And of course it was a legacy snowflake system that we couldn't just respawn on demand. That was fun.

slovenlyrobot 7y ago

If I understand the EC2 instance lifecycle correctly, shutdown causes ACPI shutdown, and the machine does not enter stopped state until the OS in turn requests ACPI to power it off. That means shutdown time is dependent on the configuration of your software -- for example, if a single service is configured to gracefully wait up to 10 minutes for, say a persistent database client connection to disconnect, you might pay the full timeout during shutdown. There is also a 'forced shutdown' API, but for whatever reason in testing for me it has always been slower than a graceful shutdown. I have no idea what it does internally

bostik 7y ago
> There is also a 'forced shutdown' API, but for whatever reason in testing for me it has always been slower than a graceful shutdown. I have no idea what it does internally I have sort of reverse engineered what the two options ("shutdown", "force shutdown") in AWS do. A regular shutdown causes the hypervisor to issue the equivalent of "init 0" to the VM, over whatever communication channel is used between dom0 and domU. Semantically this is treated as a request. If the VM is completely catatonic, the domU kernel won't be able to process the request and will just fail. A forced shutdown causes the hypervisor to issue a similar request, but also puts the VM on a watchlist. If the domU kernel has not terminated in a given time window, probably 10 or 20 minutes[ß], the hypervisor will nuke the VM. ß: I'm not entirely sure about the time window, 10 and 20 minutes are what I have witnessed most often. I have on one occasion seen a VM take 90 minutes get nuked. And of course it was a legacy snowflake system that we couldn't just respawn on demand. That was fun.