8 ms·

It’s interesting to read that so many systems and activities are dependent on a single point of failure : the main primary MySQL node at GitHub.

eatonphil 5y ago
I can imagine what they're in a rush to refactor.
cyberpunk 5y ago
I mean, they have $MEGABUCKS, they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted, and they still have a giant single db node doing writes and struggle to fail it over. We're doomed >_< You would think it wouldn't be THAT hard to shard something like GitHub effectively. I mean, all user accounts/repos starting with the letter 'a' go to the 'a' cluster and so on seems not exactly science-fiction levels of technology.
- sillysaurusx 5y ago
  > they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted The Mythical Man Month has a few things to say about that. (It's tempting to feel that the information is outdated, but in my experience it still seems true.)
  - cyberpunk 5y ago
    My point was, that they've not done that, is because it wouldn't help. This is an architectural problem, which even if they had the massive expensive brains behind something like mysql on their team they couldn't fix it. (at least, I'm guessing, I think this kinda architecture doesn't scale even if they could kick the can down the road a few times..)
    - sillysaurusx 5y ago
      Oh, my apologies! Yes, that makes sense.
- deleted 5y ago
  [deleted]
- throwusawayus 5y ago
  it's profoundly strange that github has not properly sharded yet. essentially all large social networks are or have used sharded mysql successfully, this is not rocket science livejournal, facebook, twitter, linkedin, tumblr, pinterest all use (or formerly used) sharded mysql and most of these are at larger db size than github i will also repeat my comment from another recent thread: i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. wtf is going on in the tech world these days
  - drewbug01 5y ago
    > i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. I believe you have the chain of causality backwards here. In fact, I think it suggests that talent that went to planet scale is perhaps not the issue.
    - throwusawayus 5y ago
      i'm not suggesting that these departures are the cause of github's issue. rather, i'm saying i don't understand why such a large group from github was hired by planetscale if they did not have experience successfully sharding or successfully leveraging vitess this is like if you were building a high-rise condo, would you hire the architects or management company from the building that collapsed in surfside florida? sure, they know what NOT to do next time, but that doesn't mean they do know what TO do
      - drewbug01 5y ago
        I understand what you’re getting at… but why is it you presume that the employees now at planetscale are the reason GH couldn’t shard out their databases? Like, there’s another angle here: management, yeah? Another way of reframing it is “maybe the folks hiring at planetscale know the inside baseball about GH infrastructure”. For example: https://www.linkedin.com/in/isamlambert https://www.linkedin.com/in/isamlambert
- pcj-github 5y ago
  Lots of people commenting here about how easy it must be to shard mysql at GitHub scale...
- otterley 5y ago
  Just because you have money doesn’t mean you can hire anyone you want to. Talent is scarce, especially in highly-specialized roles, and people have free will. I think the complexity of GitHub’s data management requirements would surprise you. Better to withhold judgment until you possess all the facts.
prepend 5y ago
My fear is that this seems like a cover excuse for moving off MySQL. The bug will be too hard and they’ll move off. They will choose SQLServer and take a lot of time to convert and then have even more outages.
- speedgoose 5y ago
  I bet on Azure Cosmos DB.
  - deleted 5y ago
    [deleted]
- nimbius 5y ago
  100% agreed. a lift-and-shift migration to Galera and modern MariaDB wouldnt be hard, but knowing MS there are mid managers waiting in the wings to swoop in and drive this into the ground with azure/sqlserver, the former of which posted 8 outages in the past 90 days alone. this is classic Microsoft. spend a ton of money for something very valuable -- in this case virtually all developer marketshare -- and then casually pedal it into the ground while you lie about the KPI's to C levels (IIS marketshare on netcraft as a function of parked websites at GoDaddy to dominate over Apache) and keep it on life support with other revenue streams (XBox) for the next 16 quarters until it becomes a repulsive enough carbuncle to shareholders that it gets the axe (Microsoft phone.) then in a year, limp into the barn with another product nobody else but you could afford to buy (minecraft) and slowly turn it into a KPI farm for Microsoft account metrics to drive some other failing product (Azure) and keep the C level happy while you alienate virtually every player with mechanics or requirements they hate.
  - throwusawayus 5y ago
    galera is not a solution for scaling out writes, full stop galera has lower max writes/sec than a traditional async single master because it's a cluster. the other members of the cluster need to ack the writes, and all members are doing all the writes, so adding machines does not increase your max writes
wincent 5y ago
Linked in the article is this other one, "Partitioning GitHub’s relational databases to handle scale" (https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale/ https://github.blog/2021-09-27-partitioning-githubs-relation...). That describes how there isn't just one "main primary" node; there are multiple clusters, of which `mysql1` is just one (the original one — since then, many others have been partitioned off).
- throwusawayus 5y ago
  from that article it sounds like they are mostly doing "functional partitioning" (moving tables off to other db primary/replica clusters) rather than true sharding (splitting up tables by ranges of data) functional partitioning is a band-aid. you do it when your main cluster is exploding but you need to buy time. it ultimately is a very bad thing, because generally your whole site is depenedent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure!
  - wincent 5y ago
    Towards the end: > In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth.
    - throwusawayus 5y ago
      yes i know, the fact that this is a small blurb at the bottom of the article (which is largely about functional partitioning) exactly proves my point
  - dikei 5y ago
    >> functional partitioning is a band-aid. I disagree, functional partitioning is not a band-aid, but an architectural changes that in the end can reap much more benefit than simple data sharding. >> your whole site is dependent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure! Not necessarily, it can also be that only some parts of your site are dead while others work perfectly fine.
zamalek 5y ago
This is what happens when you handwave CAP theorem. I am shocked at the number of people who are shocked that Github couldn't easily resolve this. Tech debt has to be paid in full, whether it's person-hours, or downtime, or both.
KMag 5y ago
Somehow, I just assumed that because git is based on content-addressed storage DAG that GitHub internally heavily leveraged a distributed hash table and content-addressed storage DAG. With content-addressed storage, all of the stored data is immutable, so caching is significantly simplified, and a DHT can scale very well horizontally. Care still needs to be taken around the transactions that change DAG roots (heads of branches), but the rest is just making sure immutable blobs are sufficiently replicated in the DHT. A DHT isn't a great fit for really rapidly changing data like metrics and such, but I figured that everything required to keep the basic git clone, commit, and web serving functionality is a pretty natural fit for a DHT. I figured even code search had a relatively small amount of storage for indexing recent commits, with compaction of that data resulting in per-token skiplists stored into the DHT. That way, failure outside of the DHT serving path still allows code search for all commits older than the latest (incremental) compaction. Distributed refcounting or Bloom filters could be used for garbage collecting the immutable blobs in the DHT. The probability of a reference cycle in SHA-256 hashes of even quadrillions of immutable blobs is vanishingly small.
- crdrost 5y ago
  Permissioning/ACLs is something that you really want to get synchronous whenever possible, you don't want caching and pipeline delays if you can help it. Some maintainer is acting in bad faith, someone else quickly locks them out of all the repositories that they have not yet thought to corrupt, then it looks really really bad if they find out that they still have permission to run the CI/CD scripts, maybe with malicious substitutions, even though you blocked them from being able to push commits up. The other part of that is, it's not too hard to get right. See what GitHub was doing. People don't change ACLs that often, nowhere near as often as you read them. If they did, you could rate-limit them. And the rest of the problem is, you have one writer and many read copies in one place and you can enforce boundaries on how stale the data gets on the read copies. Two hard problems, first is cache invalidation. Of course, they have found out that they are big enough to shard, one would have hoped that they would have found that out in a gentler way, much sooner. But let's not pretend that their architecture makes no sense, it's a fine architecture, they just happened to outgrow it in a bad way.

eatonphil 5y ago

I can imagine what they're in a rush to refactor.

cyberpunk 5y ago

I mean, they have $MEGABUCKS, they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted, and they still have a giant single db node doing writes and struggle to fail it over. We're doomed >_< You would think it wouldn't be THAT hard to shard something like GitHub effectively. I mean, all user accounts/repos starting with the letter 'a' go to the 'a' cluster and so on seems not exactly science-fiction levels of technology.

sillysaurusx 5y ago
> they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted The Mythical Man Month has a few things to say about that. (It's tempting to feel that the information is outdated, but in my experience it still seems true.)
- cyberpunk 5y ago
  My point was, that they've not done that, is because it wouldn't help. This is an architectural problem, which even if they had the massive expensive brains behind something like mysql on their team they couldn't fix it. (at least, I'm guessing, I think this kinda architecture doesn't scale even if they could kick the can down the road a few times..)
  - sillysaurusx 5y ago
    Oh, my apologies! Yes, that makes sense.
deleted 5y ago
[deleted]
throwusawayus 5y ago
it's profoundly strange that github has not properly sharded yet. essentially all large social networks are or have used sharded mysql successfully, this is not rocket science livejournal, facebook, twitter, linkedin, tumblr, pinterest all use (or formerly used) sharded mysql and most of these are at larger db size than github i will also repeat my comment from another recent thread: i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. wtf is going on in the tech world these days
- drewbug01 5y ago
  > i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. I believe you have the chain of causality backwards here. In fact, I think it suggests that talent that went to planet scale is perhaps not the issue.
  - throwusawayus 5y ago
    i'm not suggesting that these departures are the cause of github's issue. rather, i'm saying i don't understand why such a large group from github was hired by planetscale if they did not have experience successfully sharding or successfully leveraging vitess this is like if you were building a high-rise condo, would you hire the architects or management company from the building that collapsed in surfside florida? sure, they know what NOT to do next time, but that doesn't mean they do know what TO do
    - drewbug01 5y ago
      I understand what you’re getting at… but why is it you presume that the employees now at planetscale are the reason GH couldn’t shard out their databases? Like, there’s another angle here: management, yeah? Another way of reframing it is “maybe the folks hiring at planetscale know the inside baseball about GH infrastructure”. For example: https://www.linkedin.com/in/isamlambert https://www.linkedin.com/in/isamlambert
pcj-github 5y ago
Lots of people commenting here about how easy it must be to shard mysql at GitHub scale...
otterley 5y ago
Just because you have money doesn’t mean you can hire anyone you want to. Talent is scarce, especially in highly-specialized roles, and people have free will. I think the complexity of GitHub’s data management requirements would surprise you. Better to withhold judgment until you possess all the facts.

prepend 5y ago

My fear is that this seems like a cover excuse for moving off MySQL. The bug will be too hard and they’ll move off. They will choose SQLServer and take a lot of time to convert and then have even more outages.

speedgoose 5y ago
I bet on Azure Cosmos DB.
- deleted 5y ago
  [deleted]
nimbius 5y ago
100% agreed. a lift-and-shift migration to Galera and modern MariaDB wouldnt be hard, but knowing MS there are mid managers waiting in the wings to swoop in and drive this into the ground with azure/sqlserver, the former of which posted 8 outages in the past 90 days alone. this is classic Microsoft. spend a ton of money for something very valuable -- in this case virtually all developer marketshare -- and then casually pedal it into the ground while you lie about the KPI's to C levels (IIS marketshare on netcraft as a function of parked websites at GoDaddy to dominate over Apache) and keep it on life support with other revenue streams (XBox) for the next 16 quarters until it becomes a repulsive enough carbuncle to shareholders that it gets the axe (Microsoft phone.) then in a year, limp into the barn with another product nobody else but you could afford to buy (minecraft) and slowly turn it into a KPI farm for Microsoft account metrics to drive some other failing product (Azure) and keep the C level happy while you alienate virtually every player with mechanics or requirements they hate.
- throwusawayus 5y ago
  galera is not a solution for scaling out writes, full stop galera has lower max writes/sec than a traditional async single master because it's a cluster. the other members of the cluster need to ack the writes, and all members are doing all the writes, so adding machines does not increase your max writes

wincent 5y ago

Linked in the article is this other one, "Partitioning GitHub’s relational databases to handle scale" (https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale/ https://github.blog/2021-09-27-partitioning-githubs-relation...). That describes how there isn't just one "main primary" node; there are multiple clusters, of which `mysql1` is just one (the original one — since then, many others have been partitioned off).

throwusawayus 5y ago
from that article it sounds like they are mostly doing "functional partitioning" (moving tables off to other db primary/replica clusters) rather than true sharding (splitting up tables by ranges of data) functional partitioning is a band-aid. you do it when your main cluster is exploding but you need to buy time. it ultimately is a very bad thing, because generally your whole site is depenedent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure!
- wincent 5y ago
  Towards the end: > In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth.
  - throwusawayus 5y ago
    yes i know, the fact that this is a small blurb at the bottom of the article (which is largely about functional partitioning) exactly proves my point
- dikei 5y ago
  >> functional partitioning is a band-aid. I disagree, functional partitioning is not a band-aid, but an architectural changes that in the end can reap much more benefit than simple data sharding. >> your whole site is dependent on every single functional partition being up. it moves you from 1 single point of failure to N single points of failure! Not necessarily, it can also be that only some parts of your site are dead while others work perfectly fine.

zamalek 5y ago

This is what happens when you handwave CAP theorem. I am shocked at the number of people who are shocked that Github couldn't easily resolve this. Tech debt has to be paid in full, whether it's person-hours, or downtime, or both.

KMag 5y ago

Somehow, I just assumed that because git is based on content-addressed storage DAG that GitHub internally heavily leveraged a distributed hash table and content-addressed storage DAG. With content-addressed storage, all of the stored data is immutable, so caching is significantly simplified, and a DHT can scale very well horizontally. Care still needs to be taken around the transactions that change DAG roots (heads of branches), but the rest is just making sure immutable blobs are sufficiently replicated in the DHT. A DHT isn't a great fit for really rapidly changing data like metrics and such, but I figured that everything required to keep the basic git clone, commit, and web serving functionality is a pretty natural fit for a DHT. I figured even code search had a relatively small amount of storage for indexing recent commits, with compaction of that data resulting in per-token skiplists stored into the DHT. That way, failure outside of the DHT serving path still allows code search for all commits older than the latest (incremental) compaction. Distributed refcounting or Bloom filters could be used for garbage collecting the immutable blobs in the DHT. The probability of a reference cycle in SHA-256 hashes of even quadrillions of immutable blobs is vanishingly small.

crdrost 5y ago
Permissioning/ACLs is something that you really want to get synchronous whenever possible, you don't want caching and pipeline delays if you can help it. Some maintainer is acting in bad faith, someone else quickly locks them out of all the repositories that they have not yet thought to corrupt, then it looks really really bad if they find out that they still have permission to run the CI/CD scripts, maybe with malicious substitutions, even though you blocked them from being able to push commits up. The other part of that is, it's not too hard to get right. See what GitHub was doing. People don't change ACLs that often, nowhere near as often as you read them. If they did, you could rate-limit them. And the rest of the problem is, you have one writer and many read copies in one place and you can enforce boundaries on how stale the data gets on the read copies. Two hard problems, first is cache invalidation. Of course, they have found out that they are big enough to shard, one would have hoped that they would have found that out in a gentler way, much sooner. But let's not pretend that their architecture makes no sense, it's a fine architecture, they just happened to outgrow it in a bad way.