4 ms·

I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to

I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.

x3n0ph3n3 2mo ago
Microsoft is not a serious company anymore.
- hinkley 2mo ago
  They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see. Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can. They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again. I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
  - markus_zhang 2mo ago
    They definitely managed to create a lot of reliable products. Windows NT was one of them. Up to Windows XP the products were pretty solid and each had visible improvement against the previous one. Their language products were/are still solid IMO. Maybe Visual Studio is sluggish, but we can still use an older version if we want. Plus they put a lot of effort optimizing VSCode, too. Even back in the MS-DOS/16-bit Windows days, when things broke down quite easily, I think they still provide the best bang for individual users and developers. There was no competitors who could provide so much value back then.
    - betaby 2mo ago
      > They definitely managed to create a lot of reliable products. > Windows NT was one of them. Disagree. I was admining NT4.0 boxes back then and it was 1. very slow 2. constantly leaking that required weekly scheduled reboots 3. security wise it was nightmare even by that times standard
      - markus_zhang 2mo ago
        Thanks for sharing. It was that bad back then? As a consumer user the stability brought by the NT kernel was great back then, but maybe enterprise wise it was a different story? I never worked as a sys admin so can't say. For the last point what was the golden standard back in the mid-late 90s, if we don't include mainframe/minicomputers? Was it Solaris or BSD? Now that I think about it, I feel sad that I do not have the technical prowess to compare operating systems :/
        betaby 2mo ago
        Beside the NT4 servers we had SunOS 5.x and Digital UNIX (later upgraded to Tru64 UNIX). Yes, from sysadmin point of view they were way more consistent performance wise and less problematic to operate in general. BSDs were not 'approved' but nonetheless we were running DNS and some other internet facing stuff on OpenBSD 2.x if I remember correctly.
    - sunaookami 2mo ago
      reliable? Windows is the only operating system you have to constantly reinstall because it gets slower after a few months. To the point people think it's totally normal to do that!
      - scandals 2mo ago
        Wait. That's not normal? My WSL instances end up getting overwritten when I reinstall Windows so I thought that was just a general OS thing...
  - cyberax 2mo ago
    Microsoft in the 90-s and early 2000-s was a serious company. Windows NT and Office were engineering marvels. Windows 95 was a brilliant hack that allowed 32-bit GUI programs to run on hardware with 2Mb of RAM (Windows had 4Mb of RAM as the official minimum, but you could run it on 2Mb) while preserving compatibility with the majority of DOS software.
- sharts 2mo ago
  One might question your perspective if you ever thought they were serious.
vaneri2007 2mo ago
Maybe switching to https://github.com/gitlabhq https://github.com/gitlabhq :)
duped 2mo ago
I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. ' If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.
- hinkley 2mo ago
  The entire complexity and value of GHA is rent-seeking to keep the lights on for other things. They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
- packetlost 2mo ago
  That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
  - jacobgold 2mo ago
    It's just silly to guess at how a system works, or should work, without digging into the details. Because you simply don't know what you don't know.
    - packetlost 2mo ago
      The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
  - duped 2mo ago
    The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated). I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function. As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates? A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
inigyou 2mo ago
GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner. This is no surprise given standard Microsoft operating procedure - https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242
- exochrono 2mo ago
  they walked that back for now at least - self-hosted gh runners are still free: https://docs.github.com/en/billing/concepts/product-billing/github-actions https://docs.github.com/en/billing/concepts/product-billing/...
  - inigyou 2mo ago
    Well, they're still implemented that way.
    - arandomhuman 2mo ago
      Can you share a source for this? it's not that I don't believe you but I've had difficulties finding one.
    - js2 2mo ago
      Do you have a source for this?
    - judge2020 2mo ago
      More likely is that the log streamer takes up some amount of resources and thus running it requires a baseline level of compute. But I doubt they spin up a small runner per-self-hosted runner, most likely some ruby workers (on distributed azure instances) are dedicated purely to streaming self-hosted runner logs to log consumers. I'm certainly not surprised that they tried to sneak in billing self-hosted runner minutes at some ratio to potentially recoup the costs here. 0: https://docs.github.com/en/enterprise-server@3.21/admin/managing-github-actions-for-your-enterprise/getting-started-with-github-actions-for-your-enterprise/getting-started-with-github-actions-for-github-enterprise-server https://docs.github.com/en/enterprise-server@3.21/admin/mana...
cyanydeez 2mo ago
gitlab-ce can run all the integration tests in docker et al. Sitting on Github these days is the same as sticking to twitter a decade ago, expect next mecha hitler, I suppose.

x3n0ph3n3 2mo ago

Microsoft is not a serious company anymore.

hinkley 2mo ago
They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see. Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can. They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again. I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
- markus_zhang 2mo ago
  They definitely managed to create a lot of reliable products. Windows NT was one of them. Up to Windows XP the products were pretty solid and each had visible improvement against the previous one. Their language products were/are still solid IMO. Maybe Visual Studio is sluggish, but we can still use an older version if we want. Plus they put a lot of effort optimizing VSCode, too. Even back in the MS-DOS/16-bit Windows days, when things broke down quite easily, I think they still provide the best bang for individual users and developers. There was no competitors who could provide so much value back then.
  - betaby 2mo ago
    > They definitely managed to create a lot of reliable products. > Windows NT was one of them. Disagree. I was admining NT4.0 boxes back then and it was 1. very slow 2. constantly leaking that required weekly scheduled reboots 3. security wise it was nightmare even by that times standard
    - markus_zhang 2mo ago
      Thanks for sharing. It was that bad back then? As a consumer user the stability brought by the NT kernel was great back then, but maybe enterprise wise it was a different story? I never worked as a sys admin so can't say. For the last point what was the golden standard back in the mid-late 90s, if we don't include mainframe/minicomputers? Was it Solaris or BSD? Now that I think about it, I feel sad that I do not have the technical prowess to compare operating systems :/
      - betaby 2mo ago
        Beside the NT4 servers we had SunOS 5.x and Digital UNIX (later upgraded to Tru64 UNIX). Yes, from sysadmin point of view they were way more consistent performance wise and less problematic to operate in general. BSDs were not 'approved' but nonetheless we were running DNS and some other internet facing stuff on OpenBSD 2.x if I remember correctly.
  - sunaookami 2mo ago
    reliable? Windows is the only operating system you have to constantly reinstall because it gets slower after a few months. To the point people think it's totally normal to do that!
    - scandals 2mo ago
      Wait. That's not normal? My WSL instances end up getting overwritten when I reinstall Windows so I thought that was just a general OS thing...
- cyberax 2mo ago
  Microsoft in the 90-s and early 2000-s was a serious company. Windows NT and Office were engineering marvels. Windows 95 was a brilliant hack that allowed 32-bit GUI programs to run on hardware with 2Mb of RAM (Windows had 4Mb of RAM as the official minimum, but you could run it on 2Mb) while preserving compatibility with the majority of DOS software.
sharts 2mo ago
One might question your perspective if you ever thought they were serious.

vaneri2007 2mo ago

Maybe switching to https://github.com/gitlabhq https://github.com/gitlabhq :)

duped 2mo ago

I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. ' If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.

hinkley 2mo ago
The entire complexity and value of GHA is rent-seeking to keep the lights on for other things. They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
packetlost 2mo ago
That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
- jacobgold 2mo ago
  It's just silly to guess at how a system works, or should work, without digging into the details. Because you simply don't know what you don't know.
  - packetlost 2mo ago
    The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
- duped 2mo ago
  The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated). I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function. As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates? A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.

inigyou 2mo ago

GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner. This is no surprise given standard Microsoft operating procedure - https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242

exochrono 2mo ago
they walked that back for now at least - self-hosted gh runners are still free: https://docs.github.com/en/billing/concepts/product-billing/github-actions https://docs.github.com/en/billing/concepts/product-billing/...
- inigyou 2mo ago
  Well, they're still implemented that way.
  - arandomhuman 2mo ago
    Can you share a source for this? it's not that I don't believe you but I've had difficulties finding one.
  - js2 2mo ago
    Do you have a source for this?
  - judge2020 2mo ago
    More likely is that the log streamer takes up some amount of resources and thus running it requires a baseline level of compute. But I doubt they spin up a small runner per-self-hosted runner, most likely some ruby workers (on distributed azure instances) are dedicated purely to streaming self-hosted runner logs to log consumers. I'm certainly not surprised that they tried to sneak in billing self-hosted runner minutes at some ratio to potentially recoup the costs here. 0: https://docs.github.com/en/enterprise-server@3.21/admin/managing-github-actions-for-your-enterprise/getting-started-with-github-actions-for-your-enterprise/getting-started-with-github-actions-for-github-enterprise-server https://docs.github.com/en/enterprise-server@3.21/admin/mana...

cyanydeez 2mo ago

gitlab-ce can run all the integration tests in docker et al. Sitting on Github these days is the same as sticking to twitter a decade ago, expect next mecha hitler, I suppose.