7 ms·

This feature was delayed every month after May. And yet it is still half baked. We prepared for this with internally shared docs and the branch built in privat

This feature was delayed every month after May. And yet it is still half baked. We prepared for this with internally shared docs and the branch built in private for a while, but still had to roll back yesterday because the scheduler reverted to putting jobs wherever it pleased (including on ephemeral runners that already have a job) and randomly cancels large sets of jobs too. I have been of the opinion that investing into GH Actions at this stage is purely sunken cost (at my org), and I'm not moving until the team behind this thing ships something that doesn't break half the time. These have been seriously frustrating months, because no amount of working around this messy code[1] made of 5 layers of MS style .NET (seriously, deleting a directory goes 5 layers deep in the call stack) will ever produce a stable product. They don't even know their own code base that well, when they first attempted ephemeral runners with `--once` it turned out the thing they produced could never work (because the server-side scheduler loves pipelining jobs to machines and failing miserably when these disappear, job times out after 20 minute of waiting type) [1]: https://github.com/actions/runner https://github.com/actions/runner

mrclean8586 5y ago
Product manager on the GitHub Actions team reporting in, we're sorry to hear about this issue with the rollout of ephemeral runners. Our engineering team is aware of this issue and is heavily prioritizing the investigation and fix. We'd love to look into your specific case if you want shoot me an email: thejoebourneidentity@github.com
- sascha_sl 5y ago
  One, our githubcustomers/ contact is already forwarding anything / setting up a call with the team as needed, and two, that is not a twitter profile I'd ever send a DM to in a professional context, considering you retweet a lot of people diametrically opposed to my existence.
  - mrclean8586 5y ago
    I will check in with our githubcustomers group to help accelerate. I can also directly inform our engineering team if you're open to sending me your information at thejoebourneidentity@github.com Either way, we're looking into this issue and I'll post an update here once we've learned more.
    - p8952 5y ago
      Asking end users of your product to report issues via DM'ing your personal Twitter account, an account which is full of retweets of homophobic garbage, is really REALLY bad. Quietly editing your comment after being called out to hide it is even worse.
      - encryptluks2 5y ago
        Agreed. Thanks for pointing this out. I'm genuinely curious now what their Twitter profile is, but my guess is they'll delete the tweets or remove the Twitter account. Microsoft notoriously hires a lot of people from the Federal Sector who unfortunately appear to be mostly right-wing religious zealots.
        encryptluks2 5y ago
        Nm, found it: https://twitter.com/JoeB_in_NC https://twitter.com/JoeB_in_NC First thing is a retweet from Ben Shapiro.
        geofft 5y ago
        This is the same company whose sales team celebrated their ICE contract with little American flag and eagle emoji (https://www.latimes.com/business/technology/story/2019-10-31/github-ice-contract-defense https://www.latimes.com/business/technology/story/2019-10-31...) and eventually decided to donate all their proceeds from the ICE contract (i.e., their motivation was helping ICE, not making money). Everyone here lives under capitalism. Making money is fine. Saying "If you pay us, we'll take your money" is totally expected. But that's not what they're doing here. But GitHub is now a MAGA-converged organization pushing a social agenda, and you can see it in the quality of their engineering.
- newman314 5y ago
  One issue that I've been dealing with over the last 48hrs is that pushing Docker images to GHCR has been randomly failing with 403 errors. AFAIK, there has been no communication/acknowledgement of this as an issue. It makes it hard to decide to pick GHCR as a registry of choice.
  - thomasmcfarlane 5y ago
    We recently faced this; if you are using the docker/login action I'd give that a check as it turned out it was logging us out by default at the end of each job; resulting in some race conditions when running multiple runners on the same machine (sharing the same docker daemon). Simple fix was to add `logout: false` to the action options.
silasb 5y ago
Has your team considered looking into buildkite? We love the flexibility it gives us. Being able to dynamically build pipelines is a very nice feature that not many others have (at least that I could tell when I was researching).
- sascha_sl 5y ago
  We would like to run other things (probably concourse or argo), but this decision was made way further up to justify picking GitHub as provider. There might also be a Microsoft volume discount involved. If we hard reject actions, we’ll probably end up with the prior status quo: Jenkins.

mrclean8586 5y ago

Product manager on the GitHub Actions team reporting in, we're sorry to hear about this issue with the rollout of ephemeral runners. Our engineering team is aware of this issue and is heavily prioritizing the investigation and fix. We'd love to look into your specific case if you want shoot me an email: thejoebourneidentity@github.com

sascha_sl 5y ago
One, our githubcustomers/ contact is already forwarding anything / setting up a call with the team as needed, and two, that is not a twitter profile I'd ever send a DM to in a professional context, considering you retweet a lot of people diametrically opposed to my existence.
- mrclean8586 5y ago
  I will check in with our githubcustomers group to help accelerate. I can also directly inform our engineering team if you're open to sending me your information at thejoebourneidentity@github.com Either way, we're looking into this issue and I'll post an update here once we've learned more.
  - p8952 5y ago
    Asking end users of your product to report issues via DM'ing your personal Twitter account, an account which is full of retweets of homophobic garbage, is really REALLY bad. Quietly editing your comment after being called out to hide it is even worse.
    - encryptluks2 5y ago
      Agreed. Thanks for pointing this out. I'm genuinely curious now what their Twitter profile is, but my guess is they'll delete the tweets or remove the Twitter account. Microsoft notoriously hires a lot of people from the Federal Sector who unfortunately appear to be mostly right-wing religious zealots.
      - encryptluks2 5y ago
        Nm, found it: https://twitter.com/JoeB_in_NC https://twitter.com/JoeB_in_NC First thing is a retweet from Ben Shapiro.
        geofft 5y ago
        This is the same company whose sales team celebrated their ICE contract with little American flag and eagle emoji (https://www.latimes.com/business/technology/story/2019-10-31/github-ice-contract-defense https://www.latimes.com/business/technology/story/2019-10-31...) and eventually decided to donate all their proceeds from the ICE contract (i.e., their motivation was helping ICE, not making money). Everyone here lives under capitalism. Making money is fine. Saying "If you pay us, we'll take your money" is totally expected. But that's not what they're doing here. But GitHub is now a MAGA-converged organization pushing a social agenda, and you can see it in the quality of their engineering.
newman314 5y ago
One issue that I've been dealing with over the last 48hrs is that pushing Docker images to GHCR has been randomly failing with 403 errors. AFAIK, there has been no communication/acknowledgement of this as an issue. It makes it hard to decide to pick GHCR as a registry of choice.
- thomasmcfarlane 5y ago
  We recently faced this; if you are using the docker/login action I'd give that a check as it turned out it was logging us out by default at the end of each job; resulting in some race conditions when running multiple runners on the same machine (sharing the same docker daemon). Simple fix was to add `logout: false` to the action options.

silasb 5y ago

Has your team considered looking into buildkite? We love the flexibility it gives us. Being able to dynamically build pipelines is a very nice feature that not many others have (at least that I could tell when I was researching).

sascha_sl 5y ago
We would like to run other things (probably concourse or argo), but this decision was made way further up to justify picking GitHub as provider. There might also be a Microsoft volume discount involved. If we hard reject actions, we’ll probably end up with the prior status quo: Jenkins.