6 ms·
What unholy thing did they do that broke it across 12 different datacenters, good lord.
by kyledrake 7y ago
What unholy thing did they do that broke it across 12 different datacenters, good lord.
- bluedino 7y agoProbably the old "one command ran on everything"
- astrodust 7y agotmux is a dangerous tool in the wrong hands.
- rubbingalcohol 7y agoto be fair, it's dangerous even in the best hands. mistakes happen but business processes need to be in place to prevent catastrophes... every time i see something like this, my inclination is to blame the CTO, not the engineer who pulled the trigger.
- toomuchtodo 7y agoA post mortem should always be a place to highlight deficiencies in processes and communicate necessary improvements put into place, not to blame. Blame should only occur if the cadence of outages becomes excessive. Complex systems are tricky, and to err is human. Disclaimer: Ops/infra engineer in a previous life.
- astrodust 7y agoI wonder how many outages these days start with something like "kubectl apply" and then things go horribly awry.
- deleted 7y ago[deleted]
- solotronics 7y agoWe can blame whoever we want but you better believe shit rolls downhill at most places.
- GhettoMaestro 7y agoUntil it is a big enough F-up that an executive's head must roll.
- mdaniel 7y agoThere's a famous corollary to that approach: "Fire him? Why, I just spent 10 million dollars _educating_ him" (regrettably I can't find any evidence it's a true (quote|story), but I enjoy the sentiment)
- alexeldeib 7y agoThis does seem to indicate a notable lack of isolation for the blast radius between DO datacenters. Would be interesting to see the post mortem.
- protomyth 7y agoI get the feeling that whoever writes the post-mortem is going to have a bit of pressure to assure folks that there is isolation going forward.
- swsieber 7y agoWhoever broke it is going to feel significant pressure to actually isolate things too.
- klodolph 7y agoThat would be a bad sign that there’s something wrong with the culture. I would hope for a postmortem that identified flaws that genuinely needed to be fixed.
- viraptor 7y agoThose are not mutually exclusive and actually a good idea. You want to fix this specific issue, but also ensure that whatever process took down one DC doesn't affect other DCs. That's scaling and redundancy 101 - not sure why it would be something wrong.
- klodolph 7y ago> Those are not mutually exclusive and actually a good idea. The goals “assuring folks that there is isolation” and “identifying flaws that need to be fixed” are somewhat contrary to each other. The post-mortem should identify flaws in systems, processes, and thinking. It should not try to assure people that there is isolation when there is evidence to the contrary. > You want to fix this specific issue, but also ensure that whatever process took down one DC doesn't affect other DCs. This was a multi-regional failure. So, this specific issue is also an isolation problem, among other things. You will want to ensure that this problem doesn’t happen again but you shouldn’t assure that it won’t.
- mdellavo 7y agodee ennn esss
- hinkley 7y agoA bug that has no obvious side effects that only became visible once all data centers were upgraded? Happens. Statistics are hard.
- usernametologin 7y agoThankfully they fired the VP last year who is a complete idiot when it comes to infrastructure. Probably related to decisions he made no doubt. Companies prior to DO are still fixing his bad decisions. And now he's in charge of blockchain stuff. wcpgw????
- pmlnr 7y agoBad puppet/ansible/etc commit is the most probably explanation.
- nonbirithm 7y agoIt could be DNS. Azure has had an all-region failure due to a single DNS provider outage. It was possible that same DNS provider's outage was also causing problems for GCE and AWS at the same time. https://news.ycombinator.com/item?id=19812919 https://news.ycombinator.com/item?id=19812919
- dc352 7y agoThat wouldn't be at the top of my list. We have "Volumes" for databases and they were inaccessible for like 6 hours. I don't think any DNS is involved in mounting these. But hey, there's always a lot of crap hidden behind the scenes :)
- markonen 7y agoI would be absolutely amazed if DNS was not involved in mounting a block storage volume.