4 ms·
From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult
by yazboo 5y ago
From their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot.
I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide whether to tune the doublewrite buffer or redo log, or both, or neither.
- llamataboot 5y agoI agree that getting deep into the weeds on some of that stuff can be taxing on a smaller development team with a few senior generalists (of which I tend to be one) but I'm quite sure that companies at github scale have deep levels of performance expertise - still not always easy of course, because lots of these types of things only come up at some certain scale
- outworlder 5y agoDo not immediately assume that they do. Only those who have management that recognizes the need to have those experts even have a shot at getting them. If they do have the required staff it still might not be readily available due to org chart boundaries. Unclear if GitHub has the staff or if they are able to draw from the larger Microsoft pool. If they keep having issues I expect Microsoft to push them to move everything to MSSQL
- akshayshah 5y agoGitHub probably _had_ deep levels of performance expertise, but getting acquired by a megacorporation comes with a big shift in culture. I’d bet that many tenured GitHubbers left, and that there are relatively few people remaining who understand the core systems deeply. Medium-term, the more closely aligned with the rest of Microsoft’s technology they can become, the better - not many MSFT folks understand the ins and outs of sharded Percona, but many of them do understand SQL Server and .NET.
- jen20 5y agoI wouldn't assume SQL Server is the only database MS have core understanding of - they bought Citus Data not so long ago.
- akshayshah 5y agoSure, but the pool of Citus folks at Microsoft is relatively small. I’d assume that many Microsoft teams have worked with SQL Server for years, so it’s likely a widely-available skillset internally.
- kragen 5y agoThis is what happens when your company (notoriously) moves away from having meritocracy as a core value: you put people in charge of things who don't have the expertise to run them very well. https://www.businessinsider.com/githubs-ceo-ditches-meritocracy-rug-2014-1 https://www.businessinsider.com/githubs-ceo-ditches-meritocr...
- lucasmullens 5y agoPretty sure that rug wouldn't have prevented downtime.
- kragen 5y agoNot if they didn't uphold the values written on it. But publicly repudiating some values diminishes the chance of upholding them, and failing to uphold those values in particular is what causes downtime.
- nwh5jg56df 5y agoWhat did they move away to?
- kragen 5y agoThe alternatives to meritocracy are organizing relations of dominance and submission around some other criteria other than competence (typically reproducing established hierarchies of privilege from one generation to the next) or anarchy (in the very general sense of having nobody in charge). We know they didn't choose anarchy — it's not even clear how that would be possible inside a shareholder-owned corporation — and we can see from the results that they didn't put the most competent people in charge, but people outside the company can only guess who they did choose to put in charge.
- infamouscow 5y agoThere are lots of exceptional individual contributors with no desire to ever go into management.