5 ms·

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says,

by pinko 10y ago

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

josephorjoe 10y ago
then you need to "sudo take them all offline now!"
- deegles 10y ago
  "We can't because we put a safeguard in for that last week!"
  - phonon 10y ago
    "Just pull the power!"
    - jrgifford 10y ago
      "We've got redundant power - you can't!"
      - TeMPOraL 10y ago
        "So take the emergency chainsaw, and go cut the Internet cables."
        hinkley 10y ago
        We have physically redundant cables and microwave backup.
runevault 10y ago
Which is why there needs to be an override switch, but it needs to be very very explicit that you are going past the safeguards. And only a limited number of people who can use that override.
savanaly 10y ago
By safeguard I meant (and I think Amazon means too) an extra step that is required by the user before they can do the action so they don't do it by accident. Not something that prevents it entirely. Like how an MMO requires you before you delete a character to type the character's name in a box that pops up before you can delete it. That's far outside the realm of usual user interface, but that's so if you are just trying to edit a character it's impossible to accidentally hit that delete key. An analogous system for Amazon that would have prevented this outage: delete 10 nodes, ok. Delete 100 nodes, box pops up saying 'To delete this many nodes you must type the following in to a message box: "I want to take down a dangerously large amount of nodes."'
- hinkley 10y ago
  It occurs to me that having to type the English version of the numbers would probably work in this scenario. s3-shutdown -c "one hundred fifty" But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown
  - rickcnagy 10y ago
    I think the biggest problem with flags like --emergency is if they end up in daily use, such as git --force. Then, they are both sudo-level AND used without a lot of though.
    - geeio 10y ago
      Maybe more like --force-with-lease
    - hinkley 10y ago
      I was in fact thinking of the --force problem which is why I went another way.
  - traek 10y ago
    Facebook uses a `--clowntown` flag to let people do things without normal checks. [0] [0] https://www.quora.com/When-people-who-work-at-Facebook-say-clowntown-what-do-they-mean/answer/Girish-Patangay?srid=2bu7 https://www.quora.com/When-people-who-work-at-Facebook-say-c...
1_2__3 10y ago
I'm remembering tools I've worked with where the cheeky dev required things like type the sentence "I know what I am doing and wish to proceed." in order to perform unsafe operations. I've always wondered why ops hasn't adopted some of the best practices that have been around for years to avoid fat finger errrors. Like why don't we have systems where to do something dangerous requires two separate people run the command, or there's an approval step, or whatever.
- fixermark 10y ago
  The most common explanation: "cute" interactions make it harder to script the command-line tools because you have to account for the extra layer of indirection or write a bit of screen-scrape logic to get the command prompt input. I've always found that explanation a little threadbare.
  - TeMPOraL 10y ago
    I think the hoops you have to jump through should scale with severity. Then if you find yourself writing scripts that screen-scrape interactive output to get around some safeguards, you're doing something very wrong.
accountface 10y ago
Instead of rate limiting you'd be better off making it a two-key operation once you hit X threshold. Jr admin can delete 10, but the boss needs to confirm a deletion of 100.
perlgeek 10y ago
You can always build the safeguard to require approval from a peer (or superior) to an action that is normally considered dangerous, at least for overriding the throttle.
- romaniv 10y ago
  Anything that requires human approval for routine operations quickly devolves into bureaucracy that adds a lot of manual steps without any real safety. Now, you may argue definition of "routine operation", but the thing in the article didn't sound like they were doing anything crazy.
mrep 10y ago
There are people who can still do fleet wide root access commands and I don't think that type of thing will ever be removed (just very restricted) for this exact type of situation.

by pinko 10y ago

josephorjoe 10y ago

then you need to "sudo take them all offline now!"

deegles 10y ago
"We can't because we put a safeguard in for that last week!"
- phonon 10y ago
  "Just pull the power!"
  - jrgifford 10y ago
    "We've got redundant power - you can't!"
    - TeMPOraL 10y ago
      "So take the emergency chainsaw, and go cut the Internet cables."
      - hinkley 10y ago
        We have physically redundant cables and microwave backup.

runevault 10y ago

Which is why there needs to be an override switch, but it needs to be very very explicit that you are going past the safeguards. And only a limited number of people who can use that override.

savanaly 10y ago

By safeguard I meant (and I think Amazon means too) an extra step that is required by the user before they can do the action so they don't do it by accident. Not something that prevents it entirely. Like how an MMO requires you before you delete a character to type the character's name in a box that pops up before you can delete it. That's far outside the realm of usual user interface, but that's so if you are just trying to edit a character it's impossible to accidentally hit that delete key. An analogous system for Amazon that would have prevented this outage: delete 10 nodes, ok. Delete 100 nodes, box pops up saying 'To delete this many nodes you must type the following in to a message box: "I want to take down a dangerously large amount of nodes."'

hinkley 10y ago
It occurs to me that having to type the English version of the numbers would probably work in this scenario. s3-shutdown -c "one hundred fifty" But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown
- rickcnagy 10y ago
  I think the biggest problem with flags like --emergency is if they end up in daily use, such as git --force. Then, they are both sudo-level AND used without a lot of though.
  - geeio 10y ago
    Maybe more like --force-with-lease
  - hinkley 10y ago
    I was in fact thinking of the --force problem which is why I went another way.
- traek 10y ago
  Facebook uses a `--clowntown` flag to let people do things without normal checks. [0] [0] https://www.quora.com/When-people-who-work-at-Facebook-say-clowntown-what-do-they-mean/answer/Girish-Patangay?srid=2bu7 https://www.quora.com/When-people-who-work-at-Facebook-say-c...

1_2__3 10y ago

I'm remembering tools I've worked with where the cheeky dev required things like type the sentence "I know what I am doing and wish to proceed." in order to perform unsafe operations. I've always wondered why ops hasn't adopted some of the best practices that have been around for years to avoid fat finger errrors. Like why don't we have systems where to do something dangerous requires two separate people run the command, or there's an approval step, or whatever.

fixermark 10y ago
The most common explanation: "cute" interactions make it harder to script the command-line tools because you have to account for the extra layer of indirection or write a bit of screen-scrape logic to get the command prompt input. I've always found that explanation a little threadbare.
- TeMPOraL 10y ago
  I think the hoops you have to jump through should scale with severity. Then if you find yourself writing scripts that screen-scrape interactive output to get around some safeguards, you're doing something very wrong.

accountface 10y ago

Instead of rate limiting you'd be better off making it a two-key operation once you hit X threshold. Jr admin can delete 10, but the boss needs to confirm a deletion of 100.

perlgeek 10y ago

You can always build the safeguard to require approval from a peer (or superior) to an action that is normally considered dangerous, at least for overriding the throttle.

romaniv 10y ago
Anything that requires human approval for routine operations quickly devolves into bureaucracy that adds a lot of manual steps without any real safety. Now, you may argue definition of "routine operation", but the thing in the article didn't sound like they were doing anything crazy.

mrep 10y ago

There are people who can still do fleet wide root access commands and I don't think that type of thing will ever be removed (just very restricted) for this exact type of situation.