4 ms·

Introducing a separate charge specifically targeting those of your customers who choose to self-host your hilariously fragile infrastructure is certainly a choi

Introducing a separate charge specifically targeting those of your customers who choose to self-host your hilariously fragile infrastructure is certainly a choice.. And one I assume is in no way tied to adoption/usage-based KPIs. Of course, if you can just fence in your competition and charge admission, it'd be silly to invest time in building a superior product.

kjuulh 10mo ago
We've self-hosted github actions in the past, and self-hosting it doesn't help all that much with the fragile part. For github it is just as much triggering the actions as it is running them. ;) I hope the product gets some investment, because it has been unstable for such a long time, that on the inside it must be just the usual right now. GitHub has by far the worst uptime of any SaaS tools we use at the moment, and it isn't even close. > Actions is down again, call Brent so he can fix it again...
- btown 10mo ago
  > call Brent so he can fix it again Not sure if a Phoenix Project reference, but if it is, it's certainly in keeping with Github being as fragile as the company in the book!
  - kjuulh 10mo ago
    It is xD On the outside it feels like a product held together with duct tape, wood glue and prayers.
    - cweagans 10mo ago
      Hey, don't insult wood glue like that.
      - chickensong 10mo ago
        Indeed, wood glue is amazing. Such slander is totally uncalled for.
        steve_adams_86 10mo ago
        I don't know, maybe it's a compliment. Wood glue can form bonds stronger than the material it's bonding. So, the wood glue in this case is better than the service it's holding together :)
      - bdangubic 10mo ago
        or prayers
- tracker1 10mo ago
  I tend to just rely on the platform installers, then write my own process scripts to handle the work beyond the runners. Lets me exercise most of the process without having to (re)run the ci/cd processes over and over, which can be cumbersome, and a pain when they do break. The only self-hosted runners I've used have been for internalized deployments separate from the build or (pre)test processes. Aside: I've come to rely on Deno heavily for a lot of my scripting needs since it lets me reference repository modules directly and not require a build/install step head of time... just write TypeScript and run.
  - kjuulh 10mo ago
    We choose github actions because it was tied directly to github providing the best pull-request experience etc. We actually didn't really use github actions templating as we'd got our own stuff for that, so the only thing github actions actually had to do was start, run a few light jobs as the CI was technically run elsewhere and then report the final status. When you've got many 100s of services managing these in actions yaml itself is no bueno. As you mentioned having the option to actually be able to run the CI/CD yourself is a must. Having to wait 5 minutes plus many commits just to test an action drains you very fast. Granted we did end up making the CI so fast (~ 1 minute with dependency cache, ~4 minutes without), that we saw devs running their setup less and less on their personal workstations for development. Except when github actions went down... ;) We used Jenkins self-hosted before and it was far more stable, but a pain to maintain and understand.
- Fabricio20 10mo ago
  We self host the runners in our infrastructure and the builds are over 10x faster than relying on their cloud runners. It's crazy the performance you get from running runners on your own hardware instead of their shared CPU. Literally from ~8m build time on Gradle + Docker down to mere 15s of Gradle + Docker on self hosted CPUs.
  - matsimitsu 10mo ago
    This! We went from 20!! minutes and 1.2k monthly spend on very, very brittle action runs to a full CI run in 4 minutes, always passing, by just by going to Hetzner's server auction page and bid on a 100 euro Ryzen machine.
    - kjuulh 10mo ago
      After self hosting our builds ended up so fast, that we were actually waiting for was GitHub scheduling our agents, rather than it being the job running. It sucked a bit, because we'd optimized it so much, but we on 90th percentile saw that it took 20-30 seconds for github to schedule the jobs as they should. Measured from when the commit hit the branch, to the webhook begin sent.
      - pxc 10mo ago
        My company uses GitHub, GitLab, and Jenkins. We'll soon™ be migrating off of GitLab in favor of GitHub because it's a Microsoft shop and we get some kind of discount on GitHub for spending so much other money with Microsoft. Scheduling jobs, actually getting them running, is virtually instant with GitLab but it's slow AF for GitHub for no discernable reason.
        pxc 10mo ago
        lmao I just realized on this forum writing this way might sound like I own something. To be clear, I don't own shit. I typically write "my employer", and should have here.
- MathiasPius 10mo ago
  In 2023 I quoted a customer some 30 hours to deploy a Kubernetes cluster to Hetzner specifically to run self-hosted GitHub Actions Runners. After 10-ish hours the cluster was operational. The remaining 18 (plus 30-something unbillable to satisfy my conscience) were spent trying and failing to diagnose an issue which is still unsolved to this day[1]. [1]: https://github.com/actions/runner-container-hooks/issues/113 https://github.com/actions/runner-container-hooks/issues/113
featherless 10mo ago
This is absolutely bananas; for my own CI workflow I'll have to pay $140+/month now just to run my own hardware.
- hedgehog 10mo ago
  I'm curious, what are you doing that has over 1000 hours a month of action runtime?
  - featherless 10mo ago
    I run a local Valhalla build cluster to power the https://sidecar.clutch.engineering https://sidecar.clutch.engineering routing engine. The cluster runs daily and takes a significant amount of wall-clock time to build the entire planet. That's about 50% of my CI time; the other 50% is presubmits + App Store builds for Sidecar + CANStudio / ELMCheck. Using GitHub actions to coordinate the Valhalla builds was a nice-to-have, but this is a deal-breaker for my pull request workflows.
    - hedgehog 10mo ago
      Cool, that looks a lot nicer than the OBD scanner app I've been using.
  - Eikon 10mo ago
    On ZeroFS [0] I am doing around 80 000 minutes a month. A lot of it is wasted in build time though, due to a lack of appropriate caching facilities with GitHub actions. [0] https://github.com/Barre/ZeroFS/tree/main/.github/workflows https://github.com/Barre/ZeroFS/tree/main/.github/workflows
    - featherless 10mo ago
      I found that implementing a local cache on the runners has been helpful. Ingress/egress on local network is hella slow, especially when each build has ~10-20GB of artifacts to manage.
zahlman 10mo ago
Meanwhile I'm just running `pytest`, `pyproject-build`, `twine` etc. at the command line.... (People seem to object to this comment. I genuinely do not understand why.)
- misnome 10mo ago
  Because you appear completely oblivious and deliberately naive about the entire purpose of CI.
  - zahlman 10mo ago
    Based on my experience I really do think most people are using it for things that they could perfectly well do locally with far less complication. Perhaps that isn't most use of it; the big projects are really big.
    - wiether 10mo ago
      Care to provide examples? Fundamentally, yes, what you run in a CI pipeline can run locally. That's doesn't mean it should. Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally.
      - yjftsjthsd-h 10mo ago
        > Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally. There are a rather lot of people who do argue that? Like, I actually agree that non-local CI is useful, but this is a poor argument for it.
        wiether 10mo ago
        I'm aware of people arguing for self-hosting some services for personal use. I'm not aware of people arguing for self-hosting team or enterprise services.
awestroke 10mo ago
They still host all artefacts and logs for these self-hosted runs. Probably costs them a fair bit
- featherless 10mo ago
  There's absolutely no way that the cost scales with the usage of my own hardware. I cannot fathom this change in any way or form. Livid.
- deleted 10mo ago
  [deleted]
- gz09 10mo ago
  They already charge for this separately (at least storage). Some compute cost may be justified but you'd wish that this change would come with some commitment of fixing bugs (many open for years) in their CI platform -- as opposed to investing all their resources in a (mostly inferior) LLM agent (copilot).
  - naikrovek 10mo ago
    Copilot uses other models, not (necessarily?) its own, so I’m not sure what you mean.
    - gz09 10mo ago
      It does leverage various models, but - github copilot PR reviews are subpar compared to what I've seen from other services: at least for our PRs they tend to be mostly an (expensive) grammar/spell-check - given that it's github native you'd wish for a good integration with the platform but then when your org is behind a (github) IP whitelist things seem to break often - network firewall for the agent doesn't seem to work properly raised tickets for all these but given how well it works when it does, I might as well just migrate to another service
- newsoftheday 10mo ago
  [flagged]
deleted 10mo ago
[deleted]
naikrovek 10mo ago
Runners aren’t fragile, workflows are. The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub actions runners for 4 years. 100s of thousands of runners have taken jobs, done the work, destroyed themselves, and been replaced with clean runners, all without a single issue with the runners themselves. Workflows on the other hand, they have problems. The whole design is a bit silly
- falsedan 10mo ago
  it's not the runners, it's the orchestration service that's the problem been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates
  - naikrovek 10mo ago
    > before the status page updates That’s not hard, the status page is updated manually, and they wait for support tickets to confirm an issue before they update the status page. (Users are a far better monitoring service than any automated product.) Webhook deliveries do suffer sometimes, which sucks, but that’s not the fault of the Actions orchestration.
    - falsedan 10mo ago
      I'm seeing wonky webhook deliveries for Actions service events, like dropping them completely, while other webhooks work just fine. I struggle to see what else could be responsible for that behaviour. it has to be the case that the Actions service emits events that trigger webhook deliveries & sometimes it messes them up.
  - gheltlkckfn 10mo ago
    The orchestration service has been rewritten from scratch multiple times, in different languages even. How anyone can get it this wrong is beyond me. The one for azure devops is even worse though, pathetic.
nyrikki 10mo ago
I resorted to a local forgejo + woodpecker-ci. Every time I am forced back to GitHub for some reason it confirms I made the right choice. In my experience gitlab always felt clunky and overly complicated on the back end, but for my needs local forgejo is better than the cloud options.

kjuulh 10mo ago

We've self-hosted github actions in the past, and self-hosting it doesn't help all that much with the fragile part. For github it is just as much triggering the actions as it is running them. ;) I hope the product gets some investment, because it has been unstable for such a long time, that on the inside it must be just the usual right now. GitHub has by far the worst uptime of any SaaS tools we use at the moment, and it isn't even close. > Actions is down again, call Brent so he can fix it again...

btown 10mo ago
> call Brent so he can fix it again Not sure if a Phoenix Project reference, but if it is, it's certainly in keeping with Github being as fragile as the company in the book!
- kjuulh 10mo ago
  It is xD On the outside it feels like a product held together with duct tape, wood glue and prayers.
  - cweagans 10mo ago
    Hey, don't insult wood glue like that.
    - chickensong 10mo ago
      Indeed, wood glue is amazing. Such slander is totally uncalled for.
      - steve_adams_86 10mo ago
        I don't know, maybe it's a compliment. Wood glue can form bonds stronger than the material it's bonding. So, the wood glue in this case is better than the service it's holding together :)
    - bdangubic 10mo ago
      or prayers
tracker1 10mo ago
I tend to just rely on the platform installers, then write my own process scripts to handle the work beyond the runners. Lets me exercise most of the process without having to (re)run the ci/cd processes over and over, which can be cumbersome, and a pain when they do break. The only self-hosted runners I've used have been for internalized deployments separate from the build or (pre)test processes. Aside: I've come to rely on Deno heavily for a lot of my scripting needs since it lets me reference repository modules directly and not require a build/install step head of time... just write TypeScript and run.
- kjuulh 10mo ago
  We choose github actions because it was tied directly to github providing the best pull-request experience etc. We actually didn't really use github actions templating as we'd got our own stuff for that, so the only thing github actions actually had to do was start, run a few light jobs as the CI was technically run elsewhere and then report the final status. When you've got many 100s of services managing these in actions yaml itself is no bueno. As you mentioned having the option to actually be able to run the CI/CD yourself is a must. Having to wait 5 minutes plus many commits just to test an action drains you very fast. Granted we did end up making the CI so fast (~ 1 minute with dependency cache, ~4 minutes without), that we saw devs running their setup less and less on their personal workstations for development. Except when github actions went down... ;) We used Jenkins self-hosted before and it was far more stable, but a pain to maintain and understand.
Fabricio20 10mo ago
We self host the runners in our infrastructure and the builds are over 10x faster than relying on their cloud runners. It's crazy the performance you get from running runners on your own hardware instead of their shared CPU. Literally from ~8m build time on Gradle + Docker down to mere 15s of Gradle + Docker on self hosted CPUs.
- matsimitsu 10mo ago
  This! We went from 20!! minutes and 1.2k monthly spend on very, very brittle action runs to a full CI run in 4 minutes, always passing, by just by going to Hetzner's server auction page and bid on a 100 euro Ryzen machine.
  - kjuulh 10mo ago
    After self hosting our builds ended up so fast, that we were actually waiting for was GitHub scheduling our agents, rather than it being the job running. It sucked a bit, because we'd optimized it so much, but we on 90th percentile saw that it took 20-30 seconds for github to schedule the jobs as they should. Measured from when the commit hit the branch, to the webhook begin sent.
    - pxc 10mo ago
      My company uses GitHub, GitLab, and Jenkins. We'll soon™ be migrating off of GitLab in favor of GitHub because it's a Microsoft shop and we get some kind of discount on GitHub for spending so much other money with Microsoft. Scheduling jobs, actually getting them running, is virtually instant with GitLab but it's slow AF for GitHub for no discernable reason.
      - pxc 10mo ago
        lmao I just realized on this forum writing this way might sound like I own something. To be clear, I don't own shit. I typically write "my employer", and should have here.
MathiasPius 10mo ago
In 2023 I quoted a customer some 30 hours to deploy a Kubernetes cluster to Hetzner specifically to run self-hosted GitHub Actions Runners. After 10-ish hours the cluster was operational. The remaining 18 (plus 30-something unbillable to satisfy my conscience) were spent trying and failing to diagnose an issue which is still unsolved to this day[1]. [1]: https://github.com/actions/runner-container-hooks/issues/113 https://github.com/actions/runner-container-hooks/issues/113

featherless 10mo ago

This is absolutely bananas; for my own CI workflow I'll have to pay $140+/month now just to run my own hardware.

hedgehog 10mo ago
I'm curious, what are you doing that has over 1000 hours a month of action runtime?
- featherless 10mo ago
  I run a local Valhalla build cluster to power the https://sidecar.clutch.engineering https://sidecar.clutch.engineering routing engine. The cluster runs daily and takes a significant amount of wall-clock time to build the entire planet. That's about 50% of my CI time; the other 50% is presubmits + App Store builds for Sidecar + CANStudio / ELMCheck. Using GitHub actions to coordinate the Valhalla builds was a nice-to-have, but this is a deal-breaker for my pull request workflows.
  - hedgehog 10mo ago
    Cool, that looks a lot nicer than the OBD scanner app I've been using.
- Eikon 10mo ago
  On ZeroFS [0] I am doing around 80 000 minutes a month. A lot of it is wasted in build time though, due to a lack of appropriate caching facilities with GitHub actions. [0] https://github.com/Barre/ZeroFS/tree/main/.github/workflows https://github.com/Barre/ZeroFS/tree/main/.github/workflows
  - featherless 10mo ago
    I found that implementing a local cache on the runners has been helpful. Ingress/egress on local network is hella slow, especially when each build has ~10-20GB of artifacts to manage.

zahlman 10mo ago

Meanwhile I'm just running `pytest`, `pyproject-build`, `twine` etc. at the command line.... (People seem to object to this comment. I genuinely do not understand why.)

misnome 10mo ago
Because you appear completely oblivious and deliberately naive about the entire purpose of CI.
- zahlman 10mo ago
  Based on my experience I really do think most people are using it for things that they could perfectly well do locally with far less complication. Perhaps that isn't most use of it; the big projects are really big.
  - wiether 10mo ago
    Care to provide examples? Fundamentally, yes, what you run in a CI pipeline can run locally. That's doesn't mean it should. Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally.
    - yjftsjthsd-h 10mo ago
      > Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally. There are a rather lot of people who do argue that? Like, I actually agree that non-local CI is useful, but this is a poor argument for it.
      - wiether 10mo ago
        I'm aware of people arguing for self-hosting some services for personal use. I'm not aware of people arguing for self-hosting team or enterprise services.

awestroke 10mo ago

They still host all artefacts and logs for these self-hosted runs. Probably costs them a fair bit

featherless 10mo ago
There's absolutely no way that the cost scales with the usage of my own hardware. I cannot fathom this change in any way or form. Livid.
deleted 10mo ago
[deleted]
gz09 10mo ago
They already charge for this separately (at least storage). Some compute cost may be justified but you'd wish that this change would come with some commitment of fixing bugs (many open for years) in their CI platform -- as opposed to investing all their resources in a (mostly inferior) LLM agent (copilot).
- naikrovek 10mo ago
  Copilot uses other models, not (necessarily?) its own, so I’m not sure what you mean.
  - gz09 10mo ago
    It does leverage various models, but - github copilot PR reviews are subpar compared to what I've seen from other services: at least for our PRs they tend to be mostly an (expensive) grammar/spell-check - given that it's github native you'd wish for a good integration with the platform but then when your org is behind a (github) IP whitelist things seem to break often - network firewall for the agent doesn't seem to work properly raised tickets for all these but given how well it works when it does, I might as well just migrate to another service
newsoftheday 10mo ago
[flagged]

deleted 10mo ago

[deleted]

naikrovek 10mo ago

Runners aren’t fragile, workflows are. The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub actions runners for 4 years. 100s of thousands of runners have taken jobs, done the work, destroyed themselves, and been replaced with clean runners, all without a single issue with the runners themselves. Workflows on the other hand, they have problems. The whole design is a bit silly

falsedan 10mo ago
it's not the runners, it's the orchestration service that's the problem been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates
- naikrovek 10mo ago
  > before the status page updates That’s not hard, the status page is updated manually, and they wait for support tickets to confirm an issue before they update the status page. (Users are a far better monitoring service than any automated product.) Webhook deliveries do suffer sometimes, which sucks, but that’s not the fault of the Actions orchestration.
  - falsedan 10mo ago
    I'm seeing wonky webhook deliveries for Actions service events, like dropping them completely, while other webhooks work just fine. I struggle to see what else could be responsible for that behaviour. it has to be the case that the Actions service emits events that trigger webhook deliveries & sometimes it messes them up.
- gheltlkckfn 10mo ago
  The orchestration service has been rewritten from scratch multiple times, in different languages even. How anyone can get it this wrong is beyond me. The one for azure devops is even worse though, pathetic.

nyrikki 10mo ago

I resorted to a local forgejo + woodpecker-ci. Every time I am forced back to GitHub for some reason it confirms I made the right choice. In my experience gitlab always felt clunky and overly complicated on the back end, but for my needs local forgejo is better than the cloud options.