4 ms·

> I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance jus

> I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away. Wouldn't just running "halt -f" do the same?

jedberg 6y ago
It would be really close yes, but not exactly the same as ripping the power cord out. It still gives the OS a hint that shutdown is coming.
zyamada 6y ago
Possibly, but how can we be 100% sure without the ability to compare behavior? If I were following this line of research I'd still want to know if there's any difference in the nature of a failure when it comes from within the OS (possibly simulated by half -f) and the situation the parent OP pointed out where the instance just goes poof without sending any kind of signal to the OS itself.
- BillinghamJ 6y ago
  Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. It is a bit better, but hard to say how much better. I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.
  - jsjohnst 6y ago
    > Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.
    - BillinghamJ 6y ago
      The problem is that they can't really realistically simulate that for a single instance unless you've got the whole physical machine to yourself, and of course the worst issues tend to be when it's half-broken, not entirely off/disconnected
      - jsjohnst 6y ago
        Power loss or network disconnect could easily be simulated without needing the whole physical machine.
JulianMorrison 6y ago
To give an example of the difference: does it fsync() before stopping?
0xEFF 6y ago
Kernel panics work very well for this use case. echo c > /proc/sysrq-trigger
- raverbashing 6y ago
  For reference: https://unix.stackexchange.com/a/66205 https://unix.stackexchange.com/a/66205 (you obviously need to be root)}
- derefr 6y ago
  Maybe for testing the crash-resilience of software running on the node; but not necessarily for testing how the SDN autoscaling glop you've got configured responds to the node's death. A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.
  - kl4m 6y ago
    echo o > /proc/sysrq-trigger will shutdown a AWS instance immediately. I've used it a couple of times to make real sure my costly spot instance was terminated. https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html#what-are-the-command-keys https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html...
    - derefr 6y ago
      https://elixir.bootlin.com/linux/latest/source/kernel/power/poweroff.c#L21 https://elixir.bootlin.com/linux/latest/source/kernel/power/... Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off(). This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting. Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.
vetrom 6y ago
Not exactly. "halt -f" depends on some variant of power management working. For extra fun, you could also have partial power-offs, like say a power cut to the motherboard, or any combination of other devices with their own power lines from a power supply. (in a current PC these would typically be PCI supplementary power, motherboard, SATA power, 4-pin 12v molex power, supplemental CPU power.) Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.
- fragmede 6y ago
  To elaborate a bit further - when things are fine, they're, well, fine. Unfortunately, the assumptions of how computers actually function go out of the window when enough things go wrong, like lightning striking the wrong thing. Things go weird sometimes, like the accepting TCP connections, but not sending responses. can't ping but can SSH in, or vice versa. In a cluster with automatic failover, you want to be sure the bad node is dead dead dead because a bad machine can cause harm to all the packets in the rack. (Eg HA with floating IPs and ARP takeover.) Thus the requirement is to forcefully disable the bad node, which is where the acronym STONITH comes from - Shoot The Other Node In The Head. (HA clusters with more than two nodes existed, but were rarer than 2-node clusters simply due to cost.) Before cloud computing was so pervasive, high availability was implemented with serial cables between in physical hardware and when one machine stopped responding, the equipment basically yanked its power cable out.

jedberg 6y ago

It would be really close yes, but not exactly the same as ripping the power cord out. It still gives the OS a hint that shutdown is coming.

zyamada 6y ago

Possibly, but how can we be 100% sure without the ability to compare behavior? If I were following this line of research I'd still want to know if there's any difference in the nature of a failure when it comes from within the OS (possibly simulated by half -f) and the situation the parent OP pointed out where the instance just goes poof without sending any kind of signal to the OS itself.

BillinghamJ 6y ago
Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. It is a bit better, but hard to say how much better. I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.
- jsjohnst 6y ago
  > Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.
  - BillinghamJ 6y ago
    The problem is that they can't really realistically simulate that for a single instance unless you've got the whole physical machine to yourself, and of course the worst issues tend to be when it's half-broken, not entirely off/disconnected
    - jsjohnst 6y ago
      Power loss or network disconnect could easily be simulated without needing the whole physical machine.

JulianMorrison 6y ago

To give an example of the difference: does it fsync() before stopping?

0xEFF 6y ago

Kernel panics work very well for this use case. echo c > /proc/sysrq-trigger

raverbashing 6y ago
For reference: https://unix.stackexchange.com/a/66205 https://unix.stackexchange.com/a/66205 (you obviously need to be root)}
derefr 6y ago
Maybe for testing the crash-resilience of software running on the node; but not necessarily for testing how the SDN autoscaling glop you've got configured responds to the node's death. A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.
- kl4m 6y ago
  echo o > /proc/sysrq-trigger will shutdown a AWS instance immediately. I've used it a couple of times to make real sure my costly spot instance was terminated. https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html#what-are-the-command-keys https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html...
  - derefr 6y ago
    https://elixir.bootlin.com/linux/latest/source/kernel/power/poweroff.c#L21 https://elixir.bootlin.com/linux/latest/source/kernel/power/... Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off(). This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting. Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.

vetrom 6y ago

Not exactly. "halt -f" depends on some variant of power management working. For extra fun, you could also have partial power-offs, like say a power cut to the motherboard, or any combination of other devices with their own power lines from a power supply. (in a current PC these would typically be PCI supplementary power, motherboard, SATA power, 4-pin 12v molex power, supplemental CPU power.) Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.

fragmede 6y ago
To elaborate a bit further - when things are fine, they're, well, fine. Unfortunately, the assumptions of how computers actually function go out of the window when enough things go wrong, like lightning striking the wrong thing. Things go weird sometimes, like the accepting TCP connections, but not sending responses. can't ping but can SSH in, or vice versa. In a cluster with automatic failover, you want to be sure the bad node is dead dead dead because a bad machine can cause harm to all the packets in the rack. (Eg HA with floating IPs and ARP takeover.) Thus the requirement is to forcefully disable the bad node, which is where the acronym STONITH comes from - Shoot The Other Node In The Head. (HA clusters with more than two nodes existed, but were rarer than 2-node clusters simply due to cost.) Before cloud computing was so pervasive, high availability was implemented with serial cables between in physical hardware and when one machine stopped responding, the equipment basically yanked its power cable out.