9 ms·
While I don't have anything smart to say about stuff like this, I'd love to see their postmortem on this one. Would be somewhat hilarious if it's another bad co
by keehun 13y ago
While I don't have anything smart to say about stuff like this, I'd love to see their postmortem on this one. Would be somewhat hilarious if it's another bad configuration push across their infrastructure. Github has had so many of them and especially with the recent article of the stock brokerage firm going bankrupt due to bad configuration/code push.
- yeukhon 13y agoMaybe. They have been working on migrating some of the repositories over to new hardware lately.
- unistdh 13y agoA firm going bankrupt due to a bad code/config? Do you have a link to the article you're referring to? Sounds interesting...
- keehun 13y ago$400 million down the drain http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ http://dougseven.com/2014/04/17/knightmare-a-devops-cautiona...
- rietta 13y agoWow! Bookmarking that one. What a great cautionary tale both for developers and devops. I may well need to use that as a teaching aide. Though a security principle, I cannot tell you how many times I have to point of the need for defense in depth in the design of software.
- frugalfirbolg 13y agoJust the fact that they didn't have a way to record and verify whether the deployment was done properly boggles my mind. When I worked at a bank we had package management to do deployments, a separate tool for taking inventory of installed software (in case of users managing to sneak third party programs on to their system), and on top of that a web framework for tracking milestones during projects that allowed for manual entry by technicians and automated input from scripts so tasks that had to be done by hand like replacing hardware could be coordinated with build scripts and management could monitor the whole thing from a dashboard.
- deleted 13y ago[deleted]
- Hoff 13y agoA reference to Knight Capital Group, most likely.
- kevinburke 13y agoI doubt they are pushing new configuration at 6am on a Sunday morning.
- keehun 13y agoI doubted that Github would push new (bad) production code mid-day unannounced, and it still happened. To be fair, Github I think pushes new production code several times a day every day?
- kevinburke 13y agoIf you are pushing new configuration at 6am on a Sunday, and things immediately stop working, you revert the configuration change.
- keehun 13y agoI don't think it's that simple, especially if the new bad configuration has already run amok with the machine/data. Backups are a thing, but at that scale, is probably not completely current every minute.
- akerl_ 13y agoIf you work in an environment where such rollbacks are that simple, you're in the rare minority. The reality of working with large-scale distributed systems is that rolling back becomes much more complicated. Push out new code and the accompanying DB schema change? Good luck rolling back to the older schema when you find the bug.
- rietta 13y agoI do not know about that. In a lot of environments deployments on the weekend are common, even in full devops/automatic deploy situations.
- deleted 13y ago[deleted]