4 ms·

They just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.

• DaiPlusPlus 10y ago
  The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!") There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UPDATE without a WHERE) - but given what I know about Amazon's internal practices (i.e. the brutality) it wouldn't surprise me if they did terminate the unfortunate operator - not because they want to, but because AWS simply has too many large-scale customers who would demand immediate action like that.
  • unit91 10y ago
    Maybe so. I didn't take it that way. I took it as, "nobody panic, we weren't hacked."
    • fixermark 10y ago
      It's both. This isn't "our system was compromised by attack;" it's "SNAFU." Everyone who's had operations experience knows that there will be, as time approaches infinity, more than zero SNAFU. That's why companies offer five nines of uptime, not 100% uptime.
  • ckozlowski 10y ago
    Not at all. The "an authorized S3 team member using an established playbook..." is to establish that: 1. They were authorized to be doing it. 2. They were following an established process, and not winging it. The reason for that wording is to illustrate that operational changes are made in accordance with established CM. Mistakes happen. The system didn't catch the error. That's why the mitigation is "fix the system", and not "fire the team member". =)