10 ms·

Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable

Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? I moved my company over to GH enterprise last year (from AzDO) and I'm considering moving us away to another vendor altogether as a result of the constant partial outages. Things that used to "just work" now are slow in the UI, and GH actions fail to schedule in a reasonable timeframe way more than they ever used to. I enjoy GH copilot as much as the next person, but ultimately I came to GH because I needed a git forge, and I will leave GH if the git forge doesn't work.

kasey_junk 8mo ago
“ I enjoy GH copilot as much as the next person” So not at all?
- 1f60c 8mo ago
  That does seem to be the implication, yes. :D
- nfg 8mo ago
  Really? I’d be interested to hear more. Disclaimer: I work in Microsoft (albeit in a quite disconnected part of it, nothing to do with GitHub or Copilot).
  - macintux 8mo ago
    I’ve only started using it, so maybe I’m holding it wrong, but the other day I asked the IntelliJ plugin to explained two lines of code by referencing the line numbers. It printed & explained two entirely different lines in a different part of the file. I asked again. It picked two lines somewhere else. After using ChatGPT for the last 6 months or so, Copilot feels like a significant downgrade. On the other hand, it did easily diagnose a build failure I was having, so it’s not useless, just not as helpful.
  - 0xy 8mo ago
    Not even Microsoft employees like Copilot. Maybe start why not even your coworkers can use your own slop. https://www.theverge.com/tech/865689/microsoft-claude-code-anthropic-partnership-notepad https://www.theverge.com/tech/865689/microsoft-claude-code-a...
    - nfg 8mo ago
      Sure I love Claude Code too - I use it plenty outside of work. But funnily enough I’ve been asking myself about whether to get my org on board with internal Claude Code trials and was struggling to truly articulate what we were losing versus the Copilot cli. There are some feature gaps - but the pace of work is super and experience is pretty good for me. No one should hit Microsoft over the head for giving people access to Claude code - choice and competition is good!
  - kasey_junk 8mo ago
    In testing for my workflows copilot significantly underperforms the SOTA agents, even when using the exact same models. It's not particularly close either. This has lead to 2 classes of devs at my company a) AI hesitant, who for many copilot is their only interaction, having their worst fears confirmed about how bad AI is. b) AI enthusiasts who are irritated by dealing with management that don't know the difference pushing back on their asks for access to SOTA agents. If I were the frontier labs, and wasn't billions of dollars beholden to Microsoft, I'd cut Copilot off. It poisons the well for adoption of their other systems. I don't deal with the other copilots besides the coding agent variants but I hear similar things about the business application variants. Microsofts AI reputation is in the toilet right now, I'm not sure if its understood how bad it really is within the org.
    - nfg 8mo ago
      Interesting - these head to head comparisons you’re doing with the same model - what harnesses are you comparing, say Claude code / codex versus copilot cli? > I'm not sure if its understood how bad it really is within the org. I can’t speak to that, but there’s a lively culture of people using internal tooling who also extensively use 3p products on projects outside work and are in a reasonable position to assess how well GH copilot works.
      - kasey_junk 8mo ago
        Yeah, I’m only interested in cli and non-interactive agent usage. I don’t compare say the vs code plugins, but do regularly compare say GitHub code reviews. Those comparisons for instance have made us turn _off_ copilot pull requests entirely. All of the agents have false positives (as do humans) but copilot was having negative value in that context.
deleted 8mo ago
[deleted]
sobjornstad 8mo ago
I second this. GitHub used to be a fantastic product. Now it barely even works. Even basic functionality like the timeline updating when I push commits is unreliable. The other day I opened a PR diff (not even a particularly large one) and it took fully 15 seconds after the page visually finished loading -- on a $2,000 dev machine -- before any UI elements became clickable. This happened repeatedly. It is fairly stunning to me that we've come to accept this level of non-functional software as normal.
- kimixa 8mo ago
  We loved Github as a product when it needed to return or profit beyond "getting more users". I feel this is just the natural trajectory for any VC-funded "service" that isn't actually profitable at the time you adopt it. Of course it's going to change for the worse to become profitable.
  - tibbar 8mo ago
    GitHub isn't VC funded at the moment, though. It's owned by Microsoft. Not that this necessarily changes your point.
    - danudey 8mo ago
      > Of course it's going to change for the worse > It's owned by Microsoft. I see no contradictions here.
  - notpushkin 8mo ago
    I don’t get it. Why making the UI shittier would possibly lead to more profit?
    - kimixa 8mo ago
      It seems most of the complaints are about the reliability and infrastructure - which is very much often a direct result of lack of investment and development resources. And then many UI changes people have been complaining about are related to things like copilot being forcibly integrated - which is very much in the "Microsoft expect to gain a profit by encouraging it's use" camp. It's pretty rare companies make a UI because they want a bad UI, it's normally a second order thing from other priorities - such as promoting other services or encouraging more ad impressions or similar.
deleted 8mo ago
[deleted]
zero_shift 8mo ago
As an aside, God, Azure DevOps, what a total pile of crap that product is My "favourite" restriction that an Azure DevOps PR description is limited to a pathetic 4000 characters.
- noodlesUK 8mo ago
  It shows you the level of quality to expect from a Microsoft flagship cloud product...
  - zero_shift 8mo ago
    So I work for a devtools vendor (Snyk) and 6 months ago I signed into Azure DevOps for the first time in my life I couldn't believe it. I actually thought the product was broken. Just from a visual perspective it looked like a student project. And then I got to _using_ the damn thing
    - noodlesUK 8mo ago
      It's also completely unloved. Even MSFT Azure's own documentation regularly treats it as a second class citizen to GitHub. I have no idea why they don't just deprecate the service and officially feature freeze it. Honestly that's the case with a lot of Azure services though.
      - easton 8mo ago
        It's the boards. GitHub issues doesn't let you do all the arcane nonsense Azure DevOps' boards let you do.
        bigfudge 8mo ago
        Isn’t that a feature?
        easton 8mo ago
wnevets 8mo ago
> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? They claim that is what they are doing right now. [1] [1] https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a...
- skywhopper 8mo ago
  "Migrating to Azure" is, unfortunately, often the opposite of "delivering a reliable product".
- semiquaver 8mo ago
  Zero indication that migrating to azure will improve stability over the colos they are in now. The outages aren’t caused by the datacenter, whatever MS execs say.
  - Andrex 8mo ago
    Wasn't the last one even caused by Azure? https://news.ycombinator.com/item?id=46861842 https://news.ycombinator.com/item?id=46861842
- amluto 8mo ago
  The problem with the GH front end being an unbelievably bloated mess will not be even slightly improved by moving to Azure.
co_king_3 8mo ago
> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? It's Microsoft. A reliable product is not a reasonable expectation.
rvz 8mo ago
You might as well self-host at this point as that is far more reliable than depending on GitHub. Additionally, there is no CEO of GitHub this time that is going to save us here. So as I said many years ago [0] in the long term, a better way is to self host or use alternatives such as Codeberg or GitLab which at least you can self host your own. [0] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- none2585 8mo ago
  You can self host GitHub as well
tibbar 8mo ago
Github used to publish some pretty interesting postmortems. Maybe they still do. IIRC that they were struggling with scaling their SQL db and were starting to hit the limits. It's a tough position to be in because you have to either to a massive migration to a data layer with much different semantics, or you have to keep desperately squeezing performance and skirting on the edge of outages with a DB that wasn't really meant to handle what you're doing with it now. The OpenAI blog post on "scaling" Postgres to their current scale has much the same flavor, although I think they're doing it better than Github appears to be doing.
- akshayshah 8mo ago
  I’d be surprised by this: GitHub pretty famously used Vitess, and I’d be surprised if each shard were too big for modern hardware. Based on previous reporting [0], they’re running out of space in the main data center and new management is determined to move to Azure in a hurry. I’d bet that these outages are a combination of a worsening capacity crunch in the old data center and…well, Azure. [0]: https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a...
  - tibbar 8mo ago
    Here’s a post from 2021 about the migration! [0] I guess 2021 is a long time ago now. How did that happen… [0] https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ https://github.blog/engineering/infrastructure/partitioning-...
deleted 8mo ago
[deleted]
bigbuppo 8mo ago
Not going to happen. This is terminal decline. Next step is to kill off free repos, and then they'll start ratcheting up the price to the point that they have one small dedicated engineering team supporting each customer they have. They will have exactly one customer. At some point they'll end up owned by Broadcom, OpenText, Rocket, or Progress.
- tazjin 8mo ago
  Killing off free repos is not going to happen. That would be a suicide move on the level of the Digg redesign, or Tumblr's porn ban. It kind of would be good for everyone if they did do it though. Need to get rid of this monopoly, and maybe people will discover that there are alternatives with actually good workflows out there.
  - bigbuppo 8mo ago
    They are owned by Microsoft. When has Microsoft ever had a good idea?
    - danudey 8mo ago
      Buying Github seems like a good idea? But fucking it up wasn't, so maybe it comes out even.
  - rcakebread 8mo ago
    Microsoft has suicide-ed bigger than Digg.
  - agos 8mo ago
    it would indeed be a suicide move, but note that you are comparing that to two moves that have happened, so it's not unprecedented.
markus_zhang 8mo ago
Maybe take the initiative and move your own first? It definitely would have a bigger effect than begging here.
gerdesj 8mo ago
The ultimate irony is that Linus Thorvalds designed git with the Linux kernel codebase in mind to work without any form of infrastructure centralisation. No repo trumps any other. Surely some of your crazy kids can rummage up a CI pipeline on their laptop? 8) Anyway, I only use GH as something to sync interesting stuff from, so it doesn't get lost.
- lovich 8mo ago
  I wonder how many engineers have even worked on a git repo with multiple remotes. I’ve only worked on a team once where we all were set up as remotes to each other and that was over a decade ago.
  - flowardnut 8mo ago
    hg really spoiled us with these features, though I also haven't used them in ages
    - lovich 8mo ago
      We actually did it with raw git in the cli, but I doubt I could set that up correctly nowadays without pouring over the man pages again.
- yoyohello13 8mo ago
  Setting up a git server for yourself is actually really easy. I use it at home for personal stuff. https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protoco...
Wojtkie 8mo ago
My org just moved to Gitlab because of the GH actions problems.
- bethekidyouwant 8mo ago
  8 hour outage last week
philipallstar 8mo ago
Honestly, Gitlab is pretty decent.

kasey_junk 8mo ago

“ I enjoy GH copilot as much as the next person” So not at all?

1f60c 8mo ago
That does seem to be the implication, yes. :D
nfg 8mo ago
Really? I’d be interested to hear more. Disclaimer: I work in Microsoft (albeit in a quite disconnected part of it, nothing to do with GitHub or Copilot).
- macintux 8mo ago
  I’ve only started using it, so maybe I’m holding it wrong, but the other day I asked the IntelliJ plugin to explained two lines of code by referencing the line numbers. It printed & explained two entirely different lines in a different part of the file. I asked again. It picked two lines somewhere else. After using ChatGPT for the last 6 months or so, Copilot feels like a significant downgrade. On the other hand, it did easily diagnose a build failure I was having, so it’s not useless, just not as helpful.
- 0xy 8mo ago
  Not even Microsoft employees like Copilot. Maybe start why not even your coworkers can use your own slop. https://www.theverge.com/tech/865689/microsoft-claude-code-anthropic-partnership-notepad https://www.theverge.com/tech/865689/microsoft-claude-code-a...
  - nfg 8mo ago
    Sure I love Claude Code too - I use it plenty outside of work. But funnily enough I’ve been asking myself about whether to get my org on board with internal Claude Code trials and was struggling to truly articulate what we were losing versus the Copilot cli. There are some feature gaps - but the pace of work is super and experience is pretty good for me. No one should hit Microsoft over the head for giving people access to Claude code - choice and competition is good!
- kasey_junk 8mo ago
  In testing for my workflows copilot significantly underperforms the SOTA agents, even when using the exact same models. It's not particularly close either. This has lead to 2 classes of devs at my company a) AI hesitant, who for many copilot is their only interaction, having their worst fears confirmed about how bad AI is. b) AI enthusiasts who are irritated by dealing with management that don't know the difference pushing back on their asks for access to SOTA agents. If I were the frontier labs, and wasn't billions of dollars beholden to Microsoft, I'd cut Copilot off. It poisons the well for adoption of their other systems. I don't deal with the other copilots besides the coding agent variants but I hear similar things about the business application variants. Microsofts AI reputation is in the toilet right now, I'm not sure if its understood how bad it really is within the org.
  - nfg 8mo ago
    Interesting - these head to head comparisons you’re doing with the same model - what harnesses are you comparing, say Claude code / codex versus copilot cli? > I'm not sure if its understood how bad it really is within the org. I can’t speak to that, but there’s a lively culture of people using internal tooling who also extensively use 3p products on projects outside work and are in a reasonable position to assess how well GH copilot works.
    - kasey_junk 8mo ago
      Yeah, I’m only interested in cli and non-interactive agent usage. I don’t compare say the vs code plugins, but do regularly compare say GitHub code reviews. Those comparisons for instance have made us turn _off_ copilot pull requests entirely. All of the agents have false positives (as do humans) but copilot was having negative value in that context.

deleted 8mo ago

[deleted]

sobjornstad 8mo ago

I second this. GitHub used to be a fantastic product. Now it barely even works. Even basic functionality like the timeline updating when I push commits is unreliable. The other day I opened a PR diff (not even a particularly large one) and it took fully 15 seconds after the page visually finished loading -- on a $2,000 dev machine -- before any UI elements became clickable. This happened repeatedly. It is fairly stunning to me that we've come to accept this level of non-functional software as normal.

kimixa 8mo ago
We loved Github as a product when it needed to return or profit beyond "getting more users". I feel this is just the natural trajectory for any VC-funded "service" that isn't actually profitable at the time you adopt it. Of course it's going to change for the worse to become profitable.
- tibbar 8mo ago
  GitHub isn't VC funded at the moment, though. It's owned by Microsoft. Not that this necessarily changes your point.
  - danudey 8mo ago
    > Of course it's going to change for the worse > It's owned by Microsoft. I see no contradictions here.
- notpushkin 8mo ago
  I don’t get it. Why making the UI shittier would possibly lead to more profit?
  - kimixa 8mo ago
    It seems most of the complaints are about the reliability and infrastructure - which is very much often a direct result of lack of investment and development resources. And then many UI changes people have been complaining about are related to things like copilot being forcibly integrated - which is very much in the "Microsoft expect to gain a profit by encouraging it's use" camp. It's pretty rare companies make a UI because they want a bad UI, it's normally a second order thing from other priorities - such as promoting other services or encouraging more ad impressions or similar.

deleted 8mo ago

[deleted]

zero_shift 8mo ago

As an aside, God, Azure DevOps, what a total pile of crap that product is My "favourite" restriction that an Azure DevOps PR description is limited to a pathetic 4000 characters.

noodlesUK 8mo ago
It shows you the level of quality to expect from a Microsoft flagship cloud product...
- zero_shift 8mo ago
  So I work for a devtools vendor (Snyk) and 6 months ago I signed into Azure DevOps for the first time in my life I couldn't believe it. I actually thought the product was broken. Just from a visual perspective it looked like a student project. And then I got to _using_ the damn thing
  - noodlesUK 8mo ago
    It's also completely unloved. Even MSFT Azure's own documentation regularly treats it as a second class citizen to GitHub. I have no idea why they don't just deprecate the service and officially feature freeze it. Honestly that's the case with a lot of Azure services though.
    - easton 8mo ago
      It's the boards. GitHub issues doesn't let you do all the arcane nonsense Azure DevOps' boards let you do.
      - bigfudge 8mo ago
        Isn’t that a feature?
        easton 8mo ago

wnevets 8mo ago

> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? They claim that is what they are doing right now. [1] [1] https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a...

skywhopper 8mo ago
"Migrating to Azure" is, unfortunately, often the opposite of "delivering a reliable product".
semiquaver 8mo ago
Zero indication that migrating to azure will improve stability over the colos they are in now. The outages aren’t caused by the datacenter, whatever MS execs say.
- Andrex 8mo ago
  Wasn't the last one even caused by Azure? https://news.ycombinator.com/item?id=46861842 https://news.ycombinator.com/item?id=46861842
amluto 8mo ago
The problem with the GH front end being an unbelievably bloated mess will not be even slightly improved by moving to Azure.

co_king_3 8mo ago

> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? It's Microsoft. A reliable product is not a reasonable expectation.

rvz 8mo ago

You might as well self-host at this point as that is far more reliable than depending on GitHub. Additionally, there is no CEO of GitHub this time that is going to save us here. So as I said many years ago [0] in the long term, a better way is to self host or use alternatives such as Codeberg or GitLab which at least you can self host your own. [0] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803

none2585 8mo ago
You can self host GitHub as well

tibbar 8mo ago

Github used to publish some pretty interesting postmortems. Maybe they still do. IIRC that they were struggling with scaling their SQL db and were starting to hit the limits. It's a tough position to be in because you have to either to a massive migration to a data layer with much different semantics, or you have to keep desperately squeezing performance and skirting on the edge of outages with a DB that wasn't really meant to handle what you're doing with it now. The OpenAI blog post on "scaling" Postgres to their current scale has much the same flavor, although I think they're doing it better than Github appears to be doing.

akshayshah 8mo ago
I’d be surprised by this: GitHub pretty famously used Vitess, and I’d be surprised if each shard were too big for modern hardware. Based on previous reporting [0], they’re running out of space in the main data center and new management is determined to move to Azure in a hurry. I’d bet that these outages are a combination of a worsening capacity crunch in the old data center and…well, Azure. [0]: https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a...
- tibbar 8mo ago
  Here’s a post from 2021 about the migration! [0] I guess 2021 is a long time ago now. How did that happen… [0] https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ https://github.blog/engineering/infrastructure/partitioning-...

deleted 8mo ago

[deleted]

bigbuppo 8mo ago

Not going to happen. This is terminal decline. Next step is to kill off free repos, and then they'll start ratcheting up the price to the point that they have one small dedicated engineering team supporting each customer they have. They will have exactly one customer. At some point they'll end up owned by Broadcom, OpenText, Rocket, or Progress.

tazjin 8mo ago
Killing off free repos is not going to happen. That would be a suicide move on the level of the Digg redesign, or Tumblr's porn ban. It kind of would be good for everyone if they did do it though. Need to get rid of this monopoly, and maybe people will discover that there are alternatives with actually good workflows out there.
- bigbuppo 8mo ago
  They are owned by Microsoft. When has Microsoft ever had a good idea?
  - danudey 8mo ago
    Buying Github seems like a good idea? But fucking it up wasn't, so maybe it comes out even.
- rcakebread 8mo ago
  Microsoft has suicide-ed bigger than Digg.
- agos 8mo ago
  it would indeed be a suicide move, but note that you are comparing that to two moves that have happened, so it's not unprecedented.

markus_zhang 8mo ago

Maybe take the initiative and move your own first? It definitely would have a bigger effect than begging here.

gerdesj 8mo ago

The ultimate irony is that Linus Thorvalds designed git with the Linux kernel codebase in mind to work without any form of infrastructure centralisation. No repo trumps any other. Surely some of your crazy kids can rummage up a CI pipeline on their laptop? 8) Anyway, I only use GH as something to sync interesting stuff from, so it doesn't get lost.

lovich 8mo ago
I wonder how many engineers have even worked on a git repo with multiple remotes. I’ve only worked on a team once where we all were set up as remotes to each other and that was over a decade ago.
- flowardnut 8mo ago
  hg really spoiled us with these features, though I also haven't used them in ages
  - lovich 8mo ago
    We actually did it with raw git in the cli, but I doubt I could set that up correctly nowadays without pouring over the man pages again.
yoyohello13 8mo ago
Setting up a git server for yourself is actually really easy. I use it at home for personal stuff. https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protoco...

Wojtkie 8mo ago

My org just moved to Gitlab because of the GH actions problems.

bethekidyouwant 8mo ago
8 hour outage last week

philipallstar 8mo ago

Honestly, Gitlab is pretty decent.