6 ms·
I just want to make clear that for me, write ups containing this level of detail analyzing the error that occurred give me greater confidence in the company rat
by mcherm 6y ago
I just want to make clear that for me, write ups containing this level of detail analyzing the error that occurred give me greater confidence in the company rather than less confidence. Greater even than if I had never heard of the original bug. There are many bugs I have never heard about, but any company that uses this level of care and openness is likely to be very reliable.
- Complexicate 6y agoAgreed. However, I smirked a bit when I read your comment, since this is ultimately coming from Microsoft. Let's hope some of this mentality seeps into other Microsoft projects
- thitcanh 6y agoGitHub is owned by Microsoft but as far as I know it operates pretty independently. Let’s hope it stays that way.
- craftinator 6y agoAt this point we're all just waiting for that last E.
- edoceo 6y agoOr, pro-actively moving to GitLab - either self-hosted or paid and using GH as a simple mirror for discovery.
- arthurcolle 6y agoI don't understand this reference, can you elucidate
- artificial 6y agoMost likely a reference to Embrace, Extend, Extinguish. A strategy Microsoft employed in the past.
- arthurcolle 6y agoAh thanks, yeah. Familiar with the strategy, but didn't get the connection - been a long day over here
- m463 6y agoWe should Eulogize
- scintill76 6y agohttps://en.m.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.m.wikipedia.org/wiki/Embrace,_extend,_and_extingu...
- xxqp 6y agoelucidate, extend, extinguish
- deleted 6y ago[deleted]
- johannes1234321 6y agoWhat would be the benefit for Microsoft in that case? - They are transforming themselves from a platform company based around windows, it a platform company based around Azure. GitHub is a relevant part there. This doesn't mean they are only doing good™ with GitHub, but destroying it and open source vendors would be counted to their interests.
- petters 6y agoI agree with you in general, but in this particular case they knowingly deployed thread-unsafe code in production while increasing the threaded concurrency. That should have broken internal development rules.
- eli 6y agoNow imagine what happens behind the scenes at other service providers. I’d bet the ones that have policies that bury the details of incidents, on average, have worse problems.
- simonw 6y agoIt sounds to me like they created new internal development rules as a result, which is a smart response to the issue.
- yuliyp 6y agoYes. Writing code with bugs breaks internal development rules. The issue wasn't thread-unsafe code. It was confusion about object lifecycle. An env object was assumed to no longer be used by the web framework after a request terminated, while in reality it was recycled for future requests as a performance optimization.
- TedShiller 6y agoThis is such a n00b mistake though, the "greater confidence" is quickly canceled out. I end up having less confidence in this company.
- SilverRed 6y agoEvery bug is a "n00b mistake" when you are looking right at it. There is not a single large project that is not filled with or at least has had trivial bugs with serious repercussions.
- sellyme 6y ago> Every bug is a "n00b mistake" when you are looking right at it. Except the ones that are just the universe conspiring against you, like bit flipping in the SM-1800 because of heavily irradiated cattle being transported on a nearby train line.
- TedShiller 6y agoIf you’re a noob then yes every mistake is a noob mistake
- ShroudedNight 6y agoI think it's more a statement of the quality and clarity of the postmortem that this problem appear so obvious in hindsight.
- m463 6y agoI have to say the presentation is exceptional, including very good diagrams of the problem. But there's a thought in the back of my head wondering what kind of stuff hit the fan, and wondering who had to create the presentation. Was it the person who checked in the broken code, was it the bug fixer or his manager, or was it a pm that got stuck having to write up what happened?
- fotta 6y agoMy guess is that someone close to the remediation wrote it (or it was written by the eng team) and it went through 5 levels of editing before publication.
- m463 6y agoI wonder about the email thread. how many recipients on the TO, how many on the CC (how many on BCC?)
- codetrotter 6y ago> how many on BCC? I guess the only way we will know that is if there’s ever a vulnerability in the session handling code of the mail server that they use also xD