4 ms·

Blaming software/database complexity as an excuse why GitHub hasn't done so is extremely pathetic. Systems almost always gets more complex over time, not less.

otterley 5y ago
Until you've done it yourself successfully at GitHub's scale, I think you should withhold judgment.
- throwusawayus 5y ago
  why do you blindly assume the person you are replying to has not worked at scale? hundreds of engineers have worked deeply on sharded mysql at massive scale, many of us comment on HN!
  - otterley 5y ago
    My experience is that people who know how challenging big, complex changes can be know better than to trivialize their difficulty when discussing them.
    - throwusawayus 5y ago
      so i would agree when talking about comments like "i could code twitter in a weekend! why so many engineers there!" but this is different. github is a 14 year old company, with annual revenue in the hundreds of millions USD I literally cannot think of any other comparable size and age mysql user who has not successfully sharded long ago and avoided outages of this magnitude. and in return we get hand-wavy excuses of "complexity!!!" from their former vice president of engineering who was previously also their first DBA. criticizing these excuses is not trivializing the complexity, it's more of a "we all did this at our respective companies, who also had a lot of complexity! why can't you? we are github users, we rely on you and are very unhappy, we want to know how this happened" and just getting excuses as an answer.
      - otterley 5y ago
        > I literally cannot think of any other comparable size and age mysql user who has not successfully sharded long ago and avoided outages of this magnitude. How about a little humility? You have no idea who else has similar problems out there. I'm sure Percona and other DB consulting companies would tell you otherwise, as would PlanetScale. (If this weren't true, they wouldn't be viable businesses.) > github is a 14 year old company, with annual revenue in the hundreds of millions USD You're making the classic mistake of thinking that having money means you can just wave your hand and hire whoever you want with whatever talent you need to solve your problems. The world just doesn't work that way. As for the "hand-wavy" excuses, well, the people who know the issues don't need to disclose the dirty details in a public forum, nor are they often able to because of NDAs and other legal encumbrances. And it can be a career-limiting move to throw your colleagues under the bus. You can choose to assume people are incompetent, or instead choose to assume that people are working as best they can under the constraints placed on them. I think it's better to assume the latter.
        throwusawayus 5y ago
        I do in fact know who else would have had similar problems, directly from personal experience and experience of coworkers and friends. large-scale mysql community is fairly insular, lots of talent rotation between the large scale companies and between percona and pythian, everyone knows everyone else, everyone talks war stories at the conference bar and elsewhere. but thanks for making assumptions once again! i am NOT saying that github can hand wave this away with money and hiring. you misunderstand. what i am saying is company of their business size and scale should have already handled this long ago, because that's what literally every other large comparable company has done. there are front-page-of-HN threads about multi-hour github outages every single day this week. show me an example of another similar-size company having an equivalent meltdown please. only real equivalent is twitter during fail-whale days when they were only a few years old. they solved it early on, as should have been done. i am not saying the entirety or even majority of github is incompetent, but i am saying there is clearly something extremely wrong and extremely unusual that led to this, and citing "complexity!" is just a pile of BS.
- infamouscow 5y ago
  There's a lot of people on HN, including myself, that have scaled MySQL beyond what we thought was possible. The hard part isn't solving the scaling problem, but the uncomfortable conversations with leadership about how chunks of roadmap will be drastically delayed. Inevitably, leadership makes a decision to kick the can down the road, further exacerbating the problem.
  - deleted 5y ago
    [deleted]
- krageon 5y ago
  There have to be thousands (if not more) people that have. Why assume that they have not?

otterley 5y ago

Until you've done it yourself successfully at GitHub's scale, I think you should withhold judgment.

throwusawayus 5y ago
why do you blindly assume the person you are replying to has not worked at scale? hundreds of engineers have worked deeply on sharded mysql at massive scale, many of us comment on HN!
- otterley 5y ago
  My experience is that people who know how challenging big, complex changes can be know better than to trivialize their difficulty when discussing them.
  - throwusawayus 5y ago
    so i would agree when talking about comments like "i could code twitter in a weekend! why so many engineers there!" but this is different. github is a 14 year old company, with annual revenue in the hundreds of millions USD I literally cannot think of any other comparable size and age mysql user who has not successfully sharded long ago and avoided outages of this magnitude. and in return we get hand-wavy excuses of "complexity!!!" from their former vice president of engineering who was previously also their first DBA. criticizing these excuses is not trivializing the complexity, it's more of a "we all did this at our respective companies, who also had a lot of complexity! why can't you? we are github users, we rely on you and are very unhappy, we want to know how this happened" and just getting excuses as an answer.
    - otterley 5y ago
      > I literally cannot think of any other comparable size and age mysql user who has not successfully sharded long ago and avoided outages of this magnitude. How about a little humility? You have no idea who else has similar problems out there. I'm sure Percona and other DB consulting companies would tell you otherwise, as would PlanetScale. (If this weren't true, they wouldn't be viable businesses.) > github is a 14 year old company, with annual revenue in the hundreds of millions USD You're making the classic mistake of thinking that having money means you can just wave your hand and hire whoever you want with whatever talent you need to solve your problems. The world just doesn't work that way. As for the "hand-wavy" excuses, well, the people who know the issues don't need to disclose the dirty details in a public forum, nor are they often able to because of NDAs and other legal encumbrances. And it can be a career-limiting move to throw your colleagues under the bus. You can choose to assume people are incompetent, or instead choose to assume that people are working as best they can under the constraints placed on them. I think it's better to assume the latter.
      - throwusawayus 5y ago
        I do in fact know who else would have had similar problems, directly from personal experience and experience of coworkers and friends. large-scale mysql community is fairly insular, lots of talent rotation between the large scale companies and between percona and pythian, everyone knows everyone else, everyone talks war stories at the conference bar and elsewhere. but thanks for making assumptions once again! i am NOT saying that github can hand wave this away with money and hiring. you misunderstand. what i am saying is company of their business size and scale should have already handled this long ago, because that's what literally every other large comparable company has done. there are front-page-of-HN threads about multi-hour github outages every single day this week. show me an example of another similar-size company having an equivalent meltdown please. only real equivalent is twitter during fail-whale days when they were only a few years old. they solved it early on, as should have been done. i am not saying the entirety or even majority of github is incompetent, but i am saying there is clearly something extremely wrong and extremely unusual that led to this, and citing "complexity!" is just a pile of BS.
infamouscow 5y ago
There's a lot of people on HN, including myself, that have scaled MySQL beyond what we thought was possible. The hard part isn't solving the scaling problem, but the uncomfortable conversations with leadership about how chunks of roadmap will be drastically delayed. Inevitably, leadership makes a decision to kick the can down the road, further exacerbating the problem.
- deleted 5y ago
  [deleted]
krageon 5y ago
There have to be thousands (if not more) people that have. Why assume that they have not?