13 ms·
GitHub Actions Incident 29.3
- deltaci 4y agothis is already the third time github actions is down this week at wednesday morning
- mdaniel 4y agomaybe they have a "no deploy on Friday" rule :-D
- capableweb 4y agoI sure hope not, GitHub is supposed to be matured infrastructure at this point, where most if not all changes going into production should be very well tested and nothing that multiple people haven't verified as being correct should end up being deployed and released. Besides, Microsoft surely has 24/7 watch of their infrastructure, even on weekends, it's a huge company.
- andrewxdiamond 4y agoBugs are a function of change, not a function of maturity. Just because they can have people come in on weekends to fix things doesn’t mean they like doing that. I know many “mature” software platforms that do not deploy on Fridays or off hours at all
- bastardoperator 4y agoA true global company doesn't have off hours.
- andrewxdiamond 4y agoAs someone who runs one of the most global APIs in the world, I promise you, I do in fact sleep
- bastardoperator 4y agoThat's why a global team is important, when you sleep they work, when they sleep you work. Work is constant when you're servicing the globe.
- andrewxdiamond 4y agoAnd work is constantly slow as well, since it’s impossible to get people in the room at the same time
- bastardoperator 4y agoWhy does everyone need to be in the room? I have a groomed backlog and can talk to people async as needed. We also record meetings if you missed them and depending on the context of the meeting or importance, we'll hold timezone friendly meetings for everyone as required.
- andrewxdiamond 4y ago¯\_(ツ)_/¯ We have a different work philosophy. I do work with teams in India and England, and it’s painful to accomplish anything cross team
- capableweb 4y agoEveryone employed by your company, who work in the same industry stops and starts working at the same time, all around the world?
- andrewxdiamond 4y agoNo, but my team owns our own APIs. We are all located in the same area and we go oncall for our service.
- zamnos 4y agoWhat's your pager rotation like? I want to say you have follows-the-sun, and so your on-call shifts are 12-hours long and you swap with a team on the other side of the world from you so you can get said sleep, but I don't want to just assume that.
- zamnos 4y agoLess "on" hours then? Even Google has diurnal patterns when there's a lower amount of traffic simply due to the fact that humans are unevenly distributed across the Earth's surface. And Google does code freezes for the holidays where they don't deploy at all.
- outworlder 4y ago> Besides, Microsoft surely has 24/7 watch of their infrastructure, even on weekends "watching" with a dedicated team vs "waking up everyone in engg because things are on fire" are two very different things. Besides, size doesn't work that way. The larger the organization and the more complex the product is, the higher the chance some unexpected interaction will occur. There are processes and automation that can mitigate this, but one can never be completely certain. Not even the aviation industry has mastered that.
- deleted 4y ago[deleted]
- rvz 4y agoOnce again, just two days ago [0], the whole of GitHub went down, after the RSA key leakage and the certificate key expiry on its user facing site. It is also apparent that GitHub Actions has chronically been struggling to operate normally for at least once a month for years. There is no question that GitHub has been more unreliable than if you were to use a self-hosted GitLab or Gittea instance yourself as I said before [1]. [0] https://news.ycombinator.com/item?id=35325850 https://news.ycombinator.com/item?id=35325850 [1] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- carlmr 4y agoGittea is great but it doesn't replace GHA, only the rest of GitHub.
- dunno7456 4y agoGitea + Woodpecker CI works pretty nicely.
- aojdwhsd 4y agoThey've actually just rolled out an early version of Gitea Actions in the most recent release, which are mostly compatible with GitHub actions. https://blog.gitea.io/2023/03/gitea-1.19.0-is-released/#-gitea-actions-21937httpsgithubcomgo-giteagiteaissues21937 https://blog.gitea.io/2023/03/gitea-1.19.0-is-released/#-git...
- temp_account_32 4y agoFunnily enough, GitLab is also melting down at the moment, with pipelines not running and pull requests not functioning: https://status.gitlab.com/ https://status.gitlab.com/
- kingds 4y agowhat's a pull request?
- tmpz22 4y agoA pull request is a process which can merge new code into existing code. "Software Engineer John was tasked to add a new logo to the website, when he was done he submitted a pull request of his feature branch into his organization's github repository for the website so that his team members could approve the changes before automation (like Github Actions) deployed live as a new version of the website."
- nightfly 4y agoGitlab has "merge requests" the comment you're replying was trying to point out an error in an annoying way
- kingds 4y agoi tried to add a lil laughing emoji to indicate that my comment was in jest but apparently those get stripped out of comments? my bad.
- c-hendricks 4y agoYeah HN doesn't want people to be _too_ expressive.
- midasuni 4y ago:D doesn’t get stripped
- chatmasta 4y agoAt least we don't need to scroll too far back in our comment history to copy/paste our arguments from the thread two days ago.
- sithlord 4y agoWonder if these were managed by a defunct team from India?
- racl101 4y agoOh I get it. [touches nose]
- gxt 4y agoAt what point are organisations going to ask themselves wether this is intentional or not. Your velocity is disrupted by a likely competitor. I'd move out.
- esotericimpl 4y ago[dead]
- nimbius 4y agohow is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages. https://www.githubstatus.com/history https://www.githubstatus.com/history github has had 55 outages in 3 months. thats nearly an outage every two days. the last six months of 2022 had 74 outages. In many shops thats tangibly worse than what their local greybeard Linux admin maintains. arguments against spinning up my own gitlab/gitea/jenkins/whatever in podman under systemd are starting to ring pretty hollow lately.
- _gabe_ 4y ago> how is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages. I mean Github Actions was released 5 years ago[0]. I imagine the infrastructure for actions is more susceptible to outages than the fairly simple features Github offered previously. It makes sense that the number of outages would increase with the additional complexity in the infrastructure. [0]: https://resources.github.com/devops/tools/automation/actions/ https://resources.github.com/devops/tools/automation/actions...
- xxpor 4y agocomments like these are what end up incentivizing companies to hide outages and report all green all the time
- web3-is-a-scam 4y agoGithub is so crappy now, it feels like something is always wrong with it. Thanks Microsoft.
- jacobsenscott 4y agoNobody's stock goes down when actions go down. Nobody's stock goes up when actions are working. But everyone's stock goes up when you have mass layoffs. Working as designed.
- riffic 4y agonot sure what the SLA is on Actions but outages are a regular occurrence with these kinds of systems and are incredibly expensive to move to the next 9 of availability. it's certainly a risk you'll need to evaluate when planning your desired build process.
- riffic 4y agowhat is the "29.3" in the title supposed to represent? is that supposed to indicate a date of March 29th? I do not see a reference to this on the incident page itself.
- nurgasemetey 4y agoIt is because titles will be duplicated on search if 29.3 is not added
- riffic 4y agois it a date? is it a version? I'm just jumping up and down how awful this representation is. iso8601 (YYYY-MM-DD) is a settled debate at this point.
- usrme 4y agoAzure is also having somewhat widespread issues at the moment, so I'd venture to guess that these two are also linked.