5 ms·
Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousa
by bouk 4mo ago
Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...
- sebmellen 4mo agoSame here. You’d think they could at least separate out the GitHub-hosted and self-hosted runners, so you’re still able to dispatch jobs if the self-hosted runners are down.
- ketzu 4mo agoIf the job queue is down, that wouldn't help, would it? On my repo the jobs do not get scheduled on the PRs at all, so I assume that separation wouldn't help for todays issue.
- voxic11 4mo agoThey have the github enterprise domain separated out and its working fine right now https://us.githubstatus.com/posts/dashboard https://us.githubstatus.com/posts/dashboard
- anon7000 4mo agoI’m not convinced they actually do, because GHE on the cloud tends to have the same problems as the main outages. Probably costs extra to be “single tenant” or whatever
- sofixa 4mo agoDepending on how many thousands of $ per year, it would probably be cheaper and more reliable to self-host GitLab. It's better in terms of organisational structure (you can have one, including access and secret inheritance), and (personal view) Gitlab-CI is better than GitHub Actions because it doesn't push you towards a JavaScript/NPM style dependency hell. And it's actually fairly easy to self-hosted, with options from a single machine with an omnibus package that handles everything to a full blown autoscaling Kubernetes deployment.
- hsbauauvhabzb 4mo agoSounds good until you see their cvedetails page
- sofixa 4mo agoI mean, the GitHub Actions supply chain risks and attacks definitely compensate for any GitLab security vulnerabilities you can think of.
- user43928 4mo ago[dead]
- lazystone 4mo agoHide it behind VPN, so it's not accessible from outside.
- hsbauauvhabzb 4mo agoNow patching becomes a responsibility, unless your organisation is willing to run knowingly vulnerable software.
- PunchyHamster 4mo agoWhen you own it you can just limit it into vpn-ed company users, that significantly cuts down on the area that can be hit
- decodebytes 4mo agoSame thoughts - we use an action to ship to production, its builds an image, pushes it to ECS which triggers a deployment. We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.
- mystifyingpoi 4mo agoSounds like a very easy process to rewrite in bash/python and have it on hand if needed.
- dnnddidiej 4mo agoIt is a control pain
- re-thc 4mo ago> For a service we're paying thousands of $ per year for even though we host runners ourselves... Wait until you charge you for self-hosting runners. Oh wait. They already tried.
- the8472 4mo ago./deploy.sh
- pluc 4mo agoSure. Don't use GitHub. You can now hire me as an overpriced consultant instead of paying Microsoft.
- cryo32 4mo agoYou should never entirely depend on a third party service for deployments. Been burned too many times on that one.
- 999900000999 4mo agoOk. Move to EC2. Darn AWS is down. Alright, run it on a Mac Mini in your basement. Ahh dawn, your ISP is having issues. Good thing you have a backup 5G hotspot. Ohh no, the power is out. Eventually you have to trust someone else. GitHub is a tragedy of the Commons. Too many people are using it, and Microsoft isn't willing to handle it correctly. Feels like a very good business opportunity. Minimum 50k yearly contracts, GitHub with actual uptime. GitPro ?
- bee_rider 4mo agoMaybe we need a split between source management and distribution? The former looks like git[hub] to me, the latter maybe more like a Linux distro repo?
- sleight42 4mo agoIt's almost as though GitHub should never have let itself be sold to Microsoft...
- 999900000999 4mo agoI'm sure the VCs who invested in GitHub disagree. This is supposed to be Hacker News! Who is coming up with a startup to fill the gap !
- cryo32 4mo agoWe’re actually moving back to redundant data centres due to all of those problems. Aggregate risk is too high.
- bouk 4mo agoWe could still deploy manually but it's suboptimal! And we're 'flying blind' without CI runs
- Cthulhu_ 4mo agoIt's always best to be portable - always be able to do builds and releases locally (at least, once you get the keys - it shouldn't be possible by default), then add things like github actions on top as convenience.
- yoyohello13 4mo agoSelf host gitlab. If you already host runners it’s not a big lift.
- xaerise 4mo agoEven if there is features that are similar, most of gitlabs features are for paying customers only.
- yoyohello13 4mo agoOP said they already pay for GitHub. We pay for the premium tier of Gitlab at my work and it’s definitely worth it.
- Salgat 4mo agoIt's funny, when we were acquired they started moving us to Github actions but it seems that maybe we should stay on our old crusty self-hosted Jenkins setup...