6 ms·

I mean, they have $MEGABUCKS, they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted, and they still have a gian

I mean, they have $MEGABUCKS, they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted, and they still have a giant single db node doing writes and struggle to fail it over. We're doomed >_< You would think it wouldn't be THAT hard to shard something like GitHub effectively. I mean, all user accounts/repos starting with the letter 'a' go to the 'a' cluster and so on seems not exactly science-fiction levels of technology.

sillysaurusx 5y ago
> they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted The Mythical Man Month has a few things to say about that. (It's tempting to feel that the information is outdated, but in my experience it still seems true.)
- cyberpunk 5y ago
  My point was, that they've not done that, is because it wouldn't help. This is an architectural problem, which even if they had the massive expensive brains behind something like mysql on their team they couldn't fix it. (at least, I'm guessing, I think this kinda architecture doesn't scale even if they could kick the can down the road a few times..)
  - sillysaurusx 5y ago
    Oh, my apologies! Yes, that makes sense.
deleted 5y ago
[deleted]
throwusawayus 5y ago
it's profoundly strange that github has not properly sharded yet. essentially all large social networks are or have used sharded mysql successfully, this is not rocket science livejournal, facebook, twitter, linkedin, tumblr, pinterest all use (or formerly used) sharded mysql and most of these are at larger db size than github i will also repeat my comment from another recent thread: i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. wtf is going on in the tech world these days
- drewbug01 5y ago
  > i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. I believe you have the chain of causality backwards here. In fact, I think it suggests that talent that went to planet scale is perhaps not the issue.
  - throwusawayus 5y ago
    i'm not suggesting that these departures are the cause of github's issue. rather, i'm saying i don't understand why such a large group from github was hired by planetscale if they did not have experience successfully sharding or successfully leveraging vitess this is like if you were building a high-rise condo, would you hire the architects or management company from the building that collapsed in surfside florida? sure, they know what NOT to do next time, but that doesn't mean they do know what TO do
    - drewbug01 5y ago
      I understand what you’re getting at… but why is it you presume that the employees now at planetscale are the reason GH couldn’t shard out their databases? Like, there’s another angle here: management, yeah? Another way of reframing it is “maybe the folks hiring at planetscale know the inside baseball about GH infrastructure”. For example: https://www.linkedin.com/in/isamlambert https://www.linkedin.com/in/isamlambert
      - throwusawayus 5y ago
        this is my exact point: they hired the former head of GH infrastructure - literally the person directly responsible for all this at github for years - and made him their ceo github should have sharded years ago, every other large mysql user did so much earlier in their growth trajectory
        drewbug01 5y ago
pcj-github 5y ago
Lots of people commenting here about how easy it must be to shard mysql at GitHub scale...
otterley 5y ago
Just because you have money doesn’t mean you can hire anyone you want to. Talent is scarce, especially in highly-specialized roles, and people have free will. I think the complexity of GitHub’s data management requirements would surprise you. Better to withhold judgment until you possess all the facts.

sillysaurusx 5y ago

> they could probably get 1/2 the team who maintains mariadb to come in and work for them if they wanted The Mythical Man Month has a few things to say about that. (It's tempting to feel that the information is outdated, but in my experience it still seems true.)

cyberpunk 5y ago
My point was, that they've not done that, is because it wouldn't help. This is an architectural problem, which even if they had the massive expensive brains behind something like mysql on their team they couldn't fix it. (at least, I'm guessing, I think this kinda architecture doesn't scale even if they could kick the can down the road a few times..)
- sillysaurusx 5y ago
  Oh, my apologies! Yes, that makes sense.

deleted 5y ago

[deleted]

throwusawayus 5y ago

it's profoundly strange that github has not properly sharded yet. essentially all large social networks are or have used sharded mysql successfully, this is not rocket science livejournal, facebook, twitter, linkedin, tumblr, pinterest all use (or formerly used) sharded mysql and most of these are at larger db size than github i will also repeat my comment from another recent thread: i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. wtf is going on in the tech world these days

drewbug01 5y ago
> i just cannot understand how 20+ former github db and infra people recently left to join a db sharding company. this makes no sense whatsoever in light of github's lack of successful sharding. I believe you have the chain of causality backwards here. In fact, I think it suggests that talent that went to planet scale is perhaps not the issue.
- throwusawayus 5y ago
  i'm not suggesting that these departures are the cause of github's issue. rather, i'm saying i don't understand why such a large group from github was hired by planetscale if they did not have experience successfully sharding or successfully leveraging vitess this is like if you were building a high-rise condo, would you hire the architects or management company from the building that collapsed in surfside florida? sure, they know what NOT to do next time, but that doesn't mean they do know what TO do
  - drewbug01 5y ago
    I understand what you’re getting at… but why is it you presume that the employees now at planetscale are the reason GH couldn’t shard out their databases? Like, there’s another angle here: management, yeah? Another way of reframing it is “maybe the folks hiring at planetscale know the inside baseball about GH infrastructure”. For example: https://www.linkedin.com/in/isamlambert https://www.linkedin.com/in/isamlambert
    - throwusawayus 5y ago
      this is my exact point: they hired the former head of GH infrastructure - literally the person directly responsible for all this at github for years - and made him their ceo github should have sharded years ago, every other large mysql user did so much earlier in their growth trajectory
      - drewbug01 5y ago

pcj-github 5y ago

Lots of people commenting here about how easy it must be to shard mysql at GitHub scale...

otterley 5y ago

Just because you have money doesn’t mean you can hire anyone you want to. Talent is scarce, especially in highly-specialized roles, and people have free will. I think the complexity of GitHub’s data management requirements would surprise you. Better to withhold judgment until you possess all the facts.