4 ms·
Former tech lead on Packages here (no longer at GitHub). > New services like Packages and Codespaces rely on the `mysql1` primary db. A lot of this is due to
by taywrobel 5y ago
Former tech lead on Packages here (no longer at GitHub).
> New services like Packages and Codespaces rely on the `mysql1` primary db.
A lot of this is due to that DB handling core data to GitHub (notably organization and repository data), which the integrated nature of the new feature offerings forces them to interact with (syncing permissions from repos for packages, publishing from actions, etc.). The link between repos and codespaces is even more unavoidable.
We were careful with Packages to keep as few service dependencies as possible in the critical path, especially for things like the anonymous read path, so serving project packages to users for open source projects or package managers such as homebrew are as insulated as they can be (and I suspect were unaffected by this incident).
But at the end of the day, there is some data that is central to most everything at the company, unfortunately.
- rattray 5y agoThanks, that makes sense. I would have imagined that most of what you describe could hit the `mysql1` replica (not primary) but certainly it's imaginable that this would not be possible or wise in all cases. And of course it's worth applauding that I think most read traffic (whether to Packages or other services) worked just fine through these incidents, if I understand correctly.
- foolfoolz 5y agothis sounds backwards. i’ve worked at multiple places with the one big legacy mysql db. all of them had blanket rule no new features live in this db. whatever this complicates you would design around it
- MrStonedOne 5y ago
- taywrobel 5y agoLive in the DB and use data from the DB are two separate things. Packages has its own dedicated MySQL cluster for all package related info. Auth also lives in a separate database. But it still needs to access org or repo information in many use cases.
- 8n4vidtmkvmk 5y agoat my company we just migrate away from old tech at ridiculous cost. back to back 2 years migrations. weeee
- foolfoolz 5y agowhat would you rather have, 2 year of migrations or having production incidents daily?
- 8n4vidtmkvmk 5y agoI'm not sure they're increasing production stability.. every migration results in several post mortems
- masklinn 5y agoThe new features don’t live in that cluster, but they have to interact with it as it holds all of the core stuff. E.g. codespaces is literally a web IDE, so extensively interacting with repositories and accounts would be involved in pretty much all if its tasks.
- eek2121 5y agoThey still could have designed around it. At my current job, all of our services are in separate clusters, with none of them being largely interdependent on the other. It's possible our front-end could be bottlenecked, but as the front-end is read only, that is an easier problem to fix. While we obviously have our own unique issues, this could never happen (in its entirety) at our company.
- ripper1138 5y agoSeparate clusters? That was not the issue here. It makes complete sense that a GitHub feature would have to interact with their core organization data/service. It’s crazy that you immediately assume that you’re smarter than the actual tech lead on the project.
- deleted 5y ago[deleted]
- seer 5y agoI've not been part of designing systems at that scale (yet!) but isn't event sourcing kinda envisioned for just these kinds of problems in mind? Like you would have all of that "core" data as Kafka topics and can safely interact with them without affecting the core services? I know the answer is always "legacy" and "its been designed that way from the beginning" but I was wondering what do you think would have been the right way to design something like this to mitigate risk of current / future problems?