3 ms·
It's not too hard to switch over from GitHub actions to other runners. For example, I wrote up the steps needed in order to use Hugging Face Jobs instead, which
by abidlabs 2mo ago
It's not too hard to switch over from GitHub actions to other runners. For example, I wrote up the steps needed in order to use Hugging Face Jobs instead, which also enables GPU runners and other flavors. I use this for several of the repos I manage (such as Trackio):
https://huggingface.co/blog/github-ci-hf-jobs https://huggingface.co/blog/github-ci-hf-jobs
- timwis 2mo agoIs that not still down though because it still relies on GH actions to orchestrate?
- tom1337 2mo agoIt most likely is. We are using Ubicloud (which seems to work the same way hf jobs does) and it's also down. GitHub is not sending out Webhooks and is also not able to queue external jobs. If we had to emergency deploy anything now I'd have no idea how to really do that...
- Kinrany 2mo agoWait, were Webhooks also down, not notifying about commit pushes?
- tom1337 2mo agoI am not sure if all Webhooks were affected or only those regarding actions. Theres not much information given besides "Webhook deliveries may be delayed." https://www.githubstatus.com/incidents/qcvjkzcs7j74 https://www.githubstatus.com/incidents/qcvjkzcs7j74
- yashap 2mo agoWe use other runners (Blacksmith), and it hasn't helped, still hard down all day long. Brutal outage. As a side note, I started getting these symptoms (actions staying queued forever or not running at all) at 5pm PDT yesterday, intermittently. And definitely full outage by 8:30pm PDT. So for sure full outage for 7 hours and counting, even if you use other runners, and I strongly believe partial outage for ~15.5+ hrs before that, even if it hasn't been acknowledged by GitHub yet.
- idkasam 2mo agoYeah, that’s brutal. And really interesting that you were seeing the symptoms intermittently for 15+ hours before the full outage. You should check out Avrea — they’re building around exactly this problem, with a lot more control and reliability than relying entirely on GitHub Actions. New player, but insanely fast. You can even move your existing runners over with one line, so pretty easy to try.
- nodesocket 2mo agoAlso use Blacksmith for arm64 runners but this outage apparently affected webhooks so their runners are not picking up jobs either.