51 ms·

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidenta

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

robertlagrant 6y ago
Who automates the automators? :)
papito 6y ago
Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.
- ignoramous 6y ago
  Yes but that was eons ago. The safeguards are well and truly in-place now. Not just one, several in fact.
  - qz2 6y ago
    Apart from the ones that they haven't worked out yet :)
    - zentiggr 6y ago
      When I meet the engineer who can design for the unknown unknowns, I will bow to them.
      - papito 6y ago
        The trick is to be paranoid. You literally sit down and think exclusively about what COULD go wrong.
        GreenWatermelon 6y ago
        Anxiety is a bitch.
      - sitkack 6y ago
        Formal methods for your formal methods. And never shipping on Friday.
- marcus_holmes 6y ago
  colorblind (red/green) person here - 5% of the male population just don't see color enough for it to be an important visual clue. So sure, color-code your environments, but if you find someone about to do something to a red environment that they clearly should only be doing to a green environment, just check if they're seeing what you're seeing before you sack them ;)
  - warrenm 6y ago
    My primary customer right now color codes both with "actual color", and with words - ie the RED environment is tagged with red color bars, and also big [black] letters in the [red] bars reading "RED"
- deleted 6y ago
  [deleted]
ahachete 6y ago
Doesn't AWS (and every big cloud/enterprise) follow best-practices for production operation like FIT-ACER? https://pythian.com/uncategorized/fit-acer-dba-checklist/ https://pythian.com/uncategorized/fit-acer-dba-checklist/ That's even more surprising to me.
ayoubElk 6y ago
Irrelevant to the discussion, but I just wanted to say thank you for the categorized list of users I can follow on your profile!
- rptr 6y ago
  wow, that list is crazy. thanks OP.
- laddng 6y ago
  Which tool do you use to follow users on HN?
  - ignoramous 6y ago
    There used to be hystry.com [0] but it isn't functional anymore. Another workflow, though cumbersome, is: Search for a username on hn.algolia, select "comments" and "past months" as filters, then press enter. Ex: https://hn.algolia.com/?dateRange=pastMonth&query=nostrademons&page=0&prefix=true&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=pastMonth&query=nostrademo... [0] https://news.ycombinator.com/item?id=71827 https://news.ycombinator.com/item?id=71827
    - laddng 6y ago
      Thanks for sharing those tips!
znpy 6y ago
It reminds me of this: https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE -- "GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill"
ardy42 6y ago
> At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases). It seems like a design flaw for actions like that to be so easy. E.g. > Hey, we detected you want to delete an AWS region. Please have an authorized coworker enter their credentials to second your command.
- ignoramous 6y ago
  If it was indeed in-production, I'd never in a million years have had access rights to delete the stack. Those are gated just the way one would imagine they should be. The service stack for the region (and not an entire region itself) looked like prod, but wasn't. It made me feel like shit anyway.

robertlagrant 6y ago

Who automates the automators? :)

papito 6y ago

Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.

ignoramous 6y ago
Yes but that was eons ago. The safeguards are well and truly in-place now. Not just one, several in fact.
- qz2 6y ago
  Apart from the ones that they haven't worked out yet :)
  - zentiggr 6y ago
    When I meet the engineer who can design for the unknown unknowns, I will bow to them.
    - papito 6y ago
      The trick is to be paranoid. You literally sit down and think exclusively about what COULD go wrong.
      - GreenWatermelon 6y ago
        Anxiety is a bitch.
    - sitkack 6y ago
      Formal methods for your formal methods. And never shipping on Friday.
marcus_holmes 6y ago
colorblind (red/green) person here - 5% of the male population just don't see color enough for it to be an important visual clue. So sure, color-code your environments, but if you find someone about to do something to a red environment that they clearly should only be doing to a green environment, just check if they're seeing what you're seeing before you sack them ;)
- warrenm 6y ago
  My primary customer right now color codes both with "actual color", and with words - ie the RED environment is tagged with red color bars, and also big [black] letters in the [red] bars reading "RED"
deleted 6y ago
[deleted]

ahachete 6y ago

Doesn't AWS (and every big cloud/enterprise) follow best-practices for production operation like FIT-ACER? https://pythian.com/uncategorized/fit-acer-dba-checklist/ https://pythian.com/uncategorized/fit-acer-dba-checklist/ That's even more surprising to me.

ayoubElk 6y ago

Irrelevant to the discussion, but I just wanted to say thank you for the categorized list of users I can follow on your profile!

rptr 6y ago
wow, that list is crazy. thanks OP.
laddng 6y ago
Which tool do you use to follow users on HN?
- ignoramous 6y ago
  There used to be hystry.com [0] but it isn't functional anymore. Another workflow, though cumbersome, is: Search for a username on hn.algolia, select "comments" and "past months" as filters, then press enter. Ex: https://hn.algolia.com/?dateRange=pastMonth&query=nostrademons&page=0&prefix=true&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=pastMonth&query=nostrademo... [0] https://news.ycombinator.com/item?id=71827 https://news.ycombinator.com/item?id=71827
  - laddng 6y ago
    Thanks for sharing those tips!

znpy 6y ago

It reminds me of this: https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE -- "GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill"

ardy42 6y ago

> At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases). It seems like a design flaw for actions like that to be so easy. E.g. > Hey, we detected you want to delete an AWS region. Please have an authorized coworker enter their credentials to second your command.

ignoramous 6y ago
If it was indeed in-production, I'd never in a million years have had access rights to delete the stack. Those are gated just the way one would imagine they should be. The service stack for the region (and not an entire region itself) looked like prod, but wasn't. It made me feel like shit anyway.