5 ms·

It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help. I feel for them --

by jedberg 6mo ago

It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help. I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR. But they need to make some changes quickly.

giwook 6mo ago
I wouldn't feel too bad for them with their top-of-market comp and valuable RSU packages.
- batshit_beaver 6mo ago
  GitHub doesn't pay top of market.
  - giwook 6mo ago
    You're right. That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years so that 300k package from a few years ago is maybe 350k or more by now. My point is that they are compensated well. They should be feeling pressure to get this stuff right when their product is core infrastructure for a majority of the digital products that exist today.
- jedberg 6mo ago
  I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
  - giwook 6mo ago
    You missed my point.
    - jedberg 6mo ago
      Apparently so did everyone else. What was your point?
      - giwook 6mo ago
        The comment I responded to said they felt bad for GH employees. I was saying I don't feel all that bad given they are well-compensated white collar workers (like many of us here). Life is pretty good if one's biggest concern is work stuff and you're not personally in danger or actively being harmed. That's all I'm saying.
        deleted 6mo ago
        [deleted]
JamesSwift 6mo ago
I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene. Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.
jcgrillo 6mo ago
The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.
- jedberg 6mo ago
  Any sort of trust requirement would break the entire model and cause some serious inequality. How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle, for example?
  - jcgrillo 6mo ago
    That's a hard problem! I don't know. But when we select colleagues we build trust before we let them in the building by interviewing them, looking at their work, checking their references. So maybe there's some sort of analogous process that isn't just "here's a big PR, look at it" that would be useful? If there was such a process, maybe that kid could go through it and become trusted. EDIT: from Github's selfish perspective, this would gatekeep their CI load. I assume (I have no idea, it's just a guess) that mostly serving source code and handling commits is not primarily the scale problem. Instead (again just guessing) probably the vast majority of the compute load due to PRs is running all the CI checks. Nontrivial projects can spawn a hell of a lot of compute per PR, and on every subsequent commit pushed while the PR is open.
  - roadbuster 6mo ago
    > would break the entire model The "model" - GH effectively allowing an overload of their infra - is already broken > How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle By submitting a quality change with a clear description, preferably with unit tests? Is that no longer considered an acceptable hurdle?
    - jedberg 6mo ago
      > By submitting a quality change with a clear description, preferably with unit tests? Is that no longer considered an acceptable hurdle? But the proposal is to specifically disallow that unless the person is already known. That is the model today, the one that people want to get rid of.
zipy124 6mo ago
But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.
- mghackerlady 6mo ago
  they also aren't using azure. IDK what youtube is on, but netflix has actually faced their problems and found solutions (freebsd, mostly)
- the_sleaze_ 6mo ago
  They should migrate to AWS. Its webscale
- jedberg 6mo ago
  Netflix and YouTube both built custom CDNs. Netflix uses AWS for control plane only. Also, respectfully, you have no idea what you're talking about. "Just text" doesn't make it easy to solve. GitHub Actions aren't just text and take a lot of compute.
  - zipy124 5mo ago
    You're right, github actions do indeed take a lot of compute, but the status incidents do not seem to be limited to just actions. I never said "just text" makes it easy to solve, just that I felt Netflix and YouTube solved harder (in terms of serving the load) problems, as demonstrated by their custom CDN's and other engineering feats. Youtube gets a similar number of videos uploaded to it a day as github gets commits now (20 vs 39 million, from the 275 million a week number listed elsewhere in this thread), and I can't believe that those are equivalently hard to serve in terms of compute and bandwidth. I agree that it is not an easy problem to solve when load scales the way it has for them and I feel for the technical guys there, but I don't disagree with the level of dis-satisfaction directed their way when customers who pay GitHub large sums of money don't receive an adequate service.
Scubabear68 6mo ago
"AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that....". What "brunt"? These are not large numbers.
- jedberg 6mo ago
  Before AI coding, a GitHub issue might get one or two PRs after six months. AI coding has made this orders of magnitude bigger. The individual numbers are small, but they add up quickly.
  - Scubabear68 6mo ago
    Maybe I am really dense, but a single issue getting 2 vs 25 PRs seems to be no practical difference.
    - jedberg 6mo ago
      Well two in six months vs 25 in one hour. So that's a 54,000x increase. But also, each PR kicks off a bunch of CI work, often in GitHub Actions.

by jedberg 6mo ago

giwook 6mo ago

I wouldn't feel too bad for them with their top-of-market comp and valuable RSU packages.

batshit_beaver 6mo ago
GitHub doesn't pay top of market.
- giwook 6mo ago
  You're right. That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years so that 300k package from a few years ago is maybe 350k or more by now. My point is that they are compensated well. They should be feeling pressure to get this stuff right when their product is core infrastructure for a majority of the digital products that exist today.
jedberg 6mo ago
I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
- giwook 6mo ago
  You missed my point.
  - jedberg 6mo ago
    Apparently so did everyone else. What was your point?
    - giwook 6mo ago
      The comment I responded to said they felt bad for GH employees. I was saying I don't feel all that bad given they are well-compensated white collar workers (like many of us here). Life is pretty good if one's biggest concern is work stuff and you're not personally in danger or actively being harmed. That's all I'm saying.
      - deleted 6mo ago
        [deleted]

JamesSwift 6mo ago

I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene. Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.

jcgrillo 6mo ago

The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.

jedberg 6mo ago
Any sort of trust requirement would break the entire model and cause some serious inequality. How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle, for example?
- jcgrillo 6mo ago
  That's a hard problem! I don't know. But when we select colleagues we build trust before we let them in the building by interviewing them, looking at their work, checking their references. So maybe there's some sort of analogous process that isn't just "here's a big PR, look at it" that would be useful? If there was such a process, maybe that kid could go through it and become trusted. EDIT: from Github's selfish perspective, this would gatekeep their CI load. I assume (I have no idea, it's just a guess) that mostly serving source code and handling commits is not primarily the scale problem. Instead (again just guessing) probably the vast majority of the compute load due to PRs is running all the CI checks. Nontrivial projects can spawn a hell of a lot of compute per PR, and on every subsequent commit pushed while the PR is open.
- roadbuster 6mo ago
  > would break the entire model The "model" - GH effectively allowing an overload of their infra - is already broken > How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle By submitting a quality change with a clear description, preferably with unit tests? Is that no longer considered an acceptable hurdle?
  - jedberg 6mo ago
    > By submitting a quality change with a clear description, preferably with unit tests? Is that no longer considered an acceptable hurdle? But the proposal is to specifically disallow that unless the person is already known. That is the model today, the one that people want to get rid of.

zipy124 6mo ago

But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.

mghackerlady 6mo ago
they also aren't using azure. IDK what youtube is on, but netflix has actually faced their problems and found solutions (freebsd, mostly)
the_sleaze_ 6mo ago
They should migrate to AWS. Its webscale
jedberg 6mo ago
Netflix and YouTube both built custom CDNs. Netflix uses AWS for control plane only. Also, respectfully, you have no idea what you're talking about. "Just text" doesn't make it easy to solve. GitHub Actions aren't just text and take a lot of compute.
- zipy124 5mo ago
  You're right, github actions do indeed take a lot of compute, but the status incidents do not seem to be limited to just actions. I never said "just text" makes it easy to solve, just that I felt Netflix and YouTube solved harder (in terms of serving the load) problems, as demonstrated by their custom CDN's and other engineering feats. Youtube gets a similar number of videos uploaded to it a day as github gets commits now (20 vs 39 million, from the 275 million a week number listed elsewhere in this thread), and I can't believe that those are equivalently hard to serve in terms of compute and bandwidth. I agree that it is not an easy problem to solve when load scales the way it has for them and I feel for the technical guys there, but I don't disagree with the level of dis-satisfaction directed their way when customers who pay GitHub large sums of money don't receive an adequate service.

Scubabear68 6mo ago

"AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that....". What "brunt"? These are not large numbers.

jedberg 6mo ago
Before AI coding, a GitHub issue might get one or two PRs after six months. AI coding has made this orders of magnitude bigger. The individual numbers are small, but they add up quickly.
- Scubabear68 6mo ago
  Maybe I am really dense, but a single issue getting 2 vs 25 PRs seems to be no practical difference.
  - jedberg 6mo ago
    Well two in six months vs 25 in one hour. So that's a 54,000x increase. But also, each PR kicks off a bunch of CI work, often in GitHub Actions.