4 ms·

Fascinating! The mere existence of such an api would be an interesting source of problems when used accidentally/via buggy code ... I wonder the degree to wh

Fascinating! The mere existence of such an api would be an interesting source of problems when used accidentally/via buggy code ... I wonder the degree to which this functionality would end up becoming relied on as a "in worst case hard kill things to recover" behavior that folks utilize for bad engineering reasons as opposed to good ones ...

giancarlostoro 6y ago
I would assume said code should never be part of actual deployments or only part of unit tests, maybe an external project even.
- agravier 6y ago
  Oh you, sweet summer child...
  - drjasonharrison 6y ago
    Totally. I have added wifi power outlets to installations because process restart, OS restart, and hardware restart did not get a system back up and running. Yes it was a startup, and yes it was remotely installed hardware.
    - tacLog 6y ago
      Might be a bit strange as you are saying this is a example of bad planning but my employer was discussing this today. We had a hard time finding options, what did you end up deploying or did you develop your own in house?
      - mey 6y ago
        You want to search for "PDU remote management" Edit: Another term is Switched PDU
      - rcxdude 6y ago
        When I had this issue I just bought some cheap wifi 'smart sockets' which had some open-source firmware available and reflashed them. They even support reporting the voltage and current draw from the socket (though not particularly accurately) and toggling the power through a POST (with a password if set). Was very useful for power cycling various bits of hardware remotely.
odonnellryan 6y ago
Make it hard to accidentally call and hide it in the docs somewhere with items related to testing.
- drjasonharrison 6y ago
  Make it expensive.
  - blaser-waffle 6y ago
    I'd imagine there is only a comparatively small (but earnest) pool of people who would want such an offering. On top of keeping people who don't want it out, you'd need it to be expensive to offset the lower utilization. I'd be okay with a garbage-tier, too. It's not failing on purpose, but only because no one gives a shit. Maybe that's just standard AWS?
- dragonwriter 6y ago
  > Make it hard to accidentally call and hide it in the docs somewhere with items related to testing. This would be more effective if so many other AWS features weren't hidden in non-obvious places in the docs.
- corobo 6y ago
  Make these functions only work in us-chaos-1
eru 6y ago
Compare https://en.wikipedia.org/wiki/Crash-only_software https://en.wikipedia.org/wiki/Crash-only_software and https://lwn.net/Articles/191059/ https://lwn.net/Articles/191059/ In a previous job we actually had to make a decision like this: our server software was generally stable enough, but we reset it whenever we did an update. Every once in a while it would break when running for long enough. I opted for forcefully restarting the software every day, because I'd rather relive the pains of restarting, than to discover in production what it is like for the first time to run the system for the 501st day of uptime. (To be honest, the system in question was mostly written in Erlang, so controlled restarts were already baked into its philosophy.)

giancarlostoro 6y ago

I would assume said code should never be part of actual deployments or only part of unit tests, maybe an external project even.

agravier 6y ago
Oh you, sweet summer child...
- drjasonharrison 6y ago
  Totally. I have added wifi power outlets to installations because process restart, OS restart, and hardware restart did not get a system back up and running. Yes it was a startup, and yes it was remotely installed hardware.
  - tacLog 6y ago
    Might be a bit strange as you are saying this is a example of bad planning but my employer was discussing this today. We had a hard time finding options, what did you end up deploying or did you develop your own in house?
    - mey 6y ago
      You want to search for "PDU remote management" Edit: Another term is Switched PDU
    - rcxdude 6y ago
      When I had this issue I just bought some cheap wifi 'smart sockets' which had some open-source firmware available and reflashed them. They even support reporting the voltage and current draw from the socket (though not particularly accurately) and toggling the power through a POST (with a password if set). Was very useful for power cycling various bits of hardware remotely.

odonnellryan 6y ago

Make it hard to accidentally call and hide it in the docs somewhere with items related to testing.

drjasonharrison 6y ago
Make it expensive.
- blaser-waffle 6y ago
  I'd imagine there is only a comparatively small (but earnest) pool of people who would want such an offering. On top of keeping people who don't want it out, you'd need it to be expensive to offset the lower utilization. I'd be okay with a garbage-tier, too. It's not failing on purpose, but only because no one gives a shit. Maybe that's just standard AWS?
dragonwriter 6y ago
> Make it hard to accidentally call and hide it in the docs somewhere with items related to testing. This would be more effective if so many other AWS features weren't hidden in non-obvious places in the docs.
corobo 6y ago
Make these functions only work in us-chaos-1

eru 6y ago

Compare https://en.wikipedia.org/wiki/Crash-only_software https://en.wikipedia.org/wiki/Crash-only_software and https://lwn.net/Articles/191059/ https://lwn.net/Articles/191059/ In a previous job we actually had to make a decision like this: our server software was generally stable enough, but we reset it whenever we did an update. Every once in a while it would break when running for long enough. I opted for forcefully restarting the software every day, because I'd rather relive the pains of restarting, than to discover in production what it is like for the first time to run the system for the 501st day of uptime. (To be honest, the system in question was mostly written in Erlang, so controlled restarts were already baked into its philosophy.)