4 ms·

GitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.

by nr378 1mo ago

weli 1mo ago
That's what I don't understand. They could mitigate their name so much if they just split free/paid/enterprise. It's already shown that enterprise is much more estable and is largely unaffected from service disruptions. Why don't they go one more layer? For sure it's worth the extra complexity.
- gaigalas 1mo ago
  Depending on the cause of the current issues, that move would likely cause more harm to paid services than good. Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up. Doubling the operation doubles the operational challenges.
- lbriner 1mo ago
  There is no such thing as "just split" there is 20+ years of legacy decisions and even if the split is relatively clean it is still probably 1 years work for 200 people for maybe a marginal improvement. The real money is going to go towards, "make this all more reliable".
sdetheridge 1mo ago
According to their status pages (e.g. https://eu.githubstatus.com/ https://eu.githubstatus.com/, https://us.githubstatus.com/ https://us.githubstatus.com/), their Enterprise Cloud uptime for Actions is significantly higher.
- everfrustrated 1mo ago
  That is a different and later product with a confusingly similar name.
- roastedfunction 1mo ago
  “GitHub Enterprise Cloud with data residency” is hosted on separate infrastructure and dedicated subdomains under *.ghe.com. It’s been around since November 2024z It’s not the same thing as GitHub Enterprise Cloud hosted on the shared global network on github.com. https://docs.github.com/en/enterprise-cloud@latest/admin/data-residency/about-github-enterprise-cloud-with-data-residency https://docs.github.com/en/enterprise-cloud@latest/admin/dat...
  - Melatonic 1mo ago
    So confusing and so Microsoft. They love to have licensing so complicated their own sales people aren't up to date and have to rely on third party spreadsheets. Edit: Read that document - why do more people not do the self hosted option with GitHub Enterprise Server ?
    - NewJazz 1mo ago
      At the point you are self hosting, you have many more options ranging from a simple ssh git server with pick your favorite cicd, to gitlab ce, to forgejo, and more.
- vinnymac 1mo ago
  Just to be clear, I am on Github Enterprise, and am also experiencing this disruption both privately and publicly on every org and project I have access to.
john_strinlai 1mo ago
enterprise is mostly separate, is it not? uptimes are significantly more reasonable on the enterprise status pages
- saxonww 1mo ago
  We are in GHEC right now and GitHub Actions is not working. It's been down every time githubstatus.com says it's down.
  - mh- 1mo ago
    Same for us, I'm not even sure what product that other "Enterprise" status page refers to..
  - Melatonic 1mo ago
    What country did you choose to host your data in ? Could be region based
rethab 1mo ago
surely if they did that everybody would complain how github "lost its touch with open source since they now prioritize paid services"
saxonww 1mo ago
They have that-ish as an option: https://docs.github.com/en/enterprise-cloud@latest/admin/data-residency/about-github-enterprise-cloud-with-data-residency https://docs.github.com/en/enterprise-cloud@latest/admin/dat... I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.
- nr378 1mo ago
  Thanks, this option is good to know. We're currently on "GitHub Enterprise Cloud" on github.com and are affected by this outage (even though we use self-hosted runners!), but we're not on "GitHub Enterprise Cloud with data residency" on *.ghe.com, which I understand is/may not be affected by this outage?
  - saxonww 1mo ago
    This is what they've told us. It's represented as basically a separate deployment of the entire GHEC stack, so you're not exposed to the load/scaling issues they believe are the underlying cause of all the github.com outages today.
- herpdyderp 1mo ago
  Do you have any meaningful level of faith in GitHub's ability to deliver on stability? At this point, I have none.
  - creshal 1mo ago
    A lot of the stability problems come from trying to scale a free product on a still WIP cloud solution without costing too much; the same software running on separate, paid for infra has a lot better odds.
  - saxonww 1mo ago
    Meaningful is subjective, but yes I do. It was very stable for many years, and I do believe the recent issues are mostly or all because they were caught flat-footed by the rapid AI-driven load increases. This has been a bad time, though. I'm ready to move back to self-hosting if they can't get it together or we move to GHDR and it's still bad.
VCFundedGenYer 1mo ago
That's what Azure DevOps is supposed to do, but for some reason GitHub has a redundant enterprise division.
flohofwoe 1mo ago
It would probably be better to run projects with extremely high commit/merge frequency on a separate "slop infrastructure", basically like MMOs move cheaters to their own servers ;)

GitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.

by nr378 1mo ago

GitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.

weli 1mo ago

That's what I don't understand. They could mitigate their name so much if they just split free/paid/enterprise. It's already shown that enterprise is much more estable and is largely unaffected from service disruptions. Why don't they go one more layer? For sure it's worth the extra complexity.

gaigalas 1mo ago
Depending on the cause of the current issues, that move would likely cause more harm to paid services than good. Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up. Doubling the operation doubles the operational challenges.
lbriner 1mo ago
There is no such thing as "just split" there is 20+ years of legacy decisions and even if the split is relatively clean it is still probably 1 years work for 200 people for maybe a marginal improvement. The real money is going to go towards, "make this all more reliable".

sdetheridge 1mo ago

According to their status pages (e.g. https://eu.githubstatus.com/ https://eu.githubstatus.com/, https://us.githubstatus.com/ https://us.githubstatus.com/), their Enterprise Cloud uptime for Actions is significantly higher.

everfrustrated 1mo ago
That is a different and later product with a confusingly similar name.
roastedfunction 1mo ago
“GitHub Enterprise Cloud with data residency” is hosted on separate infrastructure and dedicated subdomains under *.ghe.com. It’s been around since November 2024z It’s not the same thing as GitHub Enterprise Cloud hosted on the shared global network on github.com. https://docs.github.com/en/enterprise-cloud@latest/admin/data-residency/about-github-enterprise-cloud-with-data-residency https://docs.github.com/en/enterprise-cloud@latest/admin/dat...
- Melatonic 1mo ago
  So confusing and so Microsoft. They love to have licensing so complicated their own sales people aren't up to date and have to rely on third party spreadsheets. Edit: Read that document - why do more people not do the self hosted option with GitHub Enterprise Server ?
  - NewJazz 1mo ago
    At the point you are self hosting, you have many more options ranging from a simple ssh git server with pick your favorite cicd, to gitlab ce, to forgejo, and more.
vinnymac 1mo ago
Just to be clear, I am on Github Enterprise, and am also experiencing this disruption both privately and publicly on every org and project I have access to.

john_strinlai 1mo ago

enterprise is mostly separate, is it not? uptimes are significantly more reasonable on the enterprise status pages

saxonww 1mo ago
We are in GHEC right now and GitHub Actions is not working. It's been down every time githubstatus.com says it's down.
- mh- 1mo ago
  Same for us, I'm not even sure what product that other "Enterprise" status page refers to..
- Melatonic 1mo ago
  What country did you choose to host your data in ? Could be region based

rethab 1mo ago

surely if they did that everybody would complain how github "lost its touch with open source since they now prioritize paid services"

saxonww 1mo ago

They have that-ish as an option: https://docs.github.com/en/enterprise-cloud@latest/admin/data-residency/about-github-enterprise-cloud-with-data-residency https://docs.github.com/en/enterprise-cloud@latest/admin/dat... I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.

nr378 1mo ago
Thanks, this option is good to know. We're currently on "GitHub Enterprise Cloud" on github.com and are affected by this outage (even though we use self-hosted runners!), but we're not on "GitHub Enterprise Cloud with data residency" on *.ghe.com, which I understand is/may not be affected by this outage?
- saxonww 1mo ago
  This is what they've told us. It's represented as basically a separate deployment of the entire GHEC stack, so you're not exposed to the load/scaling issues they believe are the underlying cause of all the github.com outages today.
herpdyderp 1mo ago
Do you have any meaningful level of faith in GitHub's ability to deliver on stability? At this point, I have none.
- creshal 1mo ago
  A lot of the stability problems come from trying to scale a free product on a still WIP cloud solution without costing too much; the same software running on separate, paid for infra has a lot better odds.
- saxonww 1mo ago
  Meaningful is subjective, but yes I do. It was very stable for many years, and I do believe the recent issues are mostly or all because they were caught flat-footed by the rapid AI-driven load increases. This has been a bad time, though. I'm ready to move back to self-hosting if they can't get it together or we move to GHDR and it's still bad.

VCFundedGenYer 1mo ago

That's what Azure DevOps is supposed to do, but for some reason GitHub has a redundant enterprise division.

flohofwoe 1mo ago

It would probably be better to run projects with extremely high commit/merge frequency on a separate "slop infrastructure", basically like MMOs move cheaters to their own servers ;)