4 ms·

Runners aren’t fragile, workflows are. The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub act

Runners aren’t fragile, workflows are. The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub actions runners for 4 years. 100s of thousands of runners have taken jobs, done the work, destroyed themselves, and been replaced with clean runners, all without a single issue with the runners themselves. Workflows on the other hand, they have problems. The whole design is a bit silly

falsedan 10mo ago
it's not the runners, it's the orchestration service that's the problem been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates
- naikrovek 10mo ago
  > before the status page updates That’s not hard, the status page is updated manually, and they wait for support tickets to confirm an issue before they update the status page. (Users are a far better monitoring service than any automated product.) Webhook deliveries do suffer sometimes, which sucks, but that’s not the fault of the Actions orchestration.
  - falsedan 10mo ago
    I'm seeing wonky webhook deliveries for Actions service events, like dropping them completely, while other webhooks work just fine. I struggle to see what else could be responsible for that behaviour. it has to be the case that the Actions service emits events that trigger webhook deliveries & sometimes it messes them up.
- gheltlkckfn 10mo ago
  The orchestration service has been rewritten from scratch multiple times, in different languages even. How anyone can get it this wrong is beyond me. The one for azure devops is even worse though, pathetic.

falsedan 10mo ago

it's not the runners, it's the orchestration service that's the problem been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates

naikrovek 10mo ago
> before the status page updates That’s not hard, the status page is updated manually, and they wait for support tickets to confirm an issue before they update the status page. (Users are a far better monitoring service than any automated product.) Webhook deliveries do suffer sometimes, which sucks, but that’s not the fault of the Actions orchestration.
- falsedan 10mo ago
  I'm seeing wonky webhook deliveries for Actions service events, like dropping them completely, while other webhooks work just fine. I struggle to see what else could be responsible for that behaviour. it has to be the case that the Actions service emits events that trigger webhook deliveries & sometimes it messes them up.
gheltlkckfn 10mo ago
The orchestration service has been rewritten from scratch multiple times, in different languages even. How anyone can get it this wrong is beyond me. The one for azure devops is even worse though, pathetic.