3 ms·

They had massive problems with their main database cluster (MySQL). If you read through their engineering blog, most of the outages were related to their growth

They had massive problems with their main database cluster (MySQL). If you read through their engineering blog, most of the outages were related to their growth and the main database cluster. They moved workloads for some features to different clusters, but that's only to buy more time. Eventually they'll do proper shredding (by user or org I guess, not by feature) but that takes time. Their engineering blog is full of articles about MySQL and the main "mysql1" database cluster, e.g. https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale/ https://github.blog/2021-09-27-partitioning-githubs-relation...

throwusawayus 5y ago
i've noticed this too .. the real head-scratcher is how a solid chunk of github's db & infra folks left to join a database startup, one of them even becoming its ceo!! if they had made github db/infra super-stable before this, it would be a vote of confidence in their new company, but instead imho it is the opposite
- dimitrisnl 5y ago
  Which one are you referring to?
  - throwusawayus 5y ago
    maybe i shouldn't have mentioned it, i don't want to name names and have this to come off as an off-topic attack subthread about a different company, sorry! it's a db company that has raised a lot of money and is mentioned on hn a lot, there are only a handful of these
  - tedmiston 5y ago
    my guess is: rot13 cynargfpnyr
- avar 5y ago
  DB and infra folks are often tasked with shoveling shit uphill, and aren't in total control over how data or schemas get organized.
  - throwusawayus 5y ago
    that's fair. i am just raising an eyebrow to github's apparent lack of sharding, as described in their incident reports -- while these engineers all left to join a db company that focuses specifically on sharding -- it seems like an experience mismatch. if they were all sharding experts why wasn't github sharded properly. other large mysql shops have solved this, all the way back to the days of yahoo and flickr and livejournal

throwusawayus 5y ago

i've noticed this too .. the real head-scratcher is how a solid chunk of github's db & infra folks left to join a database startup, one of them even becoming its ceo!! if they had made github db/infra super-stable before this, it would be a vote of confidence in their new company, but instead imho it is the opposite

dimitrisnl 5y ago
Which one are you referring to?
- throwusawayus 5y ago
  maybe i shouldn't have mentioned it, i don't want to name names and have this to come off as an off-topic attack subthread about a different company, sorry! it's a db company that has raised a lot of money and is mentioned on hn a lot, there are only a handful of these
- tedmiston 5y ago
  my guess is: rot13 cynargfpnyr
avar 5y ago
DB and infra folks are often tasked with shoveling shit uphill, and aren't in total control over how data or schemas get organized.
- throwusawayus 5y ago
  that's fair. i am just raising an eyebrow to github's apparent lack of sharding, as described in their incident reports -- while these engineers all left to join a db company that focuses specifically on sharding -- it seems like an experience mismatch. if they were all sharding experts why wasn't github sharded properly. other large mysql shops have solved this, all the way back to the days of yahoo and flickr and livejournal