7 ms·
Why don't we all collectively mirror our repos to gitlab and switch development to there during the monthly outage?
by Kelteseth 3y ago
Why don't we all collectively mirror our repos to gitlab and switch development to there during the monthly outage?
- geraldwhen 3y agoEnterprise says I have to use this unreliable garbage.
- bombolo 3y agoYep. We could self host. But it's forbidden.
- geraldwhen 3y agoWe migrated recently. On prem was never down, but since moving to GitHub were more down than up.
- lol768 3y agoIt's definitely no less reliable than GitLab, where a good 300GB of database data got deleted in production by accident...
- ishanjain28 3y agoAnd how often that has happened? Seems a little harsh
- subtra3t 3y agoJust like calling GitHub "unreliable garbage"...
- ishanjain28 3y agoGithub has been down hundreds of times this year alone. They have reported outages 72 times this year and there are multiple times when services are unavailable and they don't report it on the status page. I don't see how the two situations are comparable
- lol768 3y ago> there are multiple times when services are unavailable and they don't report it on the status page. There's no evidence that the exact same doesn't happen with GitLab. I've had it (consistently) 500 on me in the past when there's nothing on their status page to indicate any issues.
- ishanjain28 3y agoOkay Agreed, Except That's not the point of discussion. I didn't say Gitlab doesn't lie about it or heck, That it doesn't have worse uptime than Github. My argument is that a company erasing 300GB production database once is not a stain on their competency and that it can not be compared to a company which has very frequent outages which also happens to lie when they have outages.
- 8organicbits 3y agogitlab.com or self-hosted?
- hnarn 3y agogitlab.com is implied since it happening on a self-hosted instance would have nothing to do with gitlab as a service (they can't be responsible for your on-site backups). > Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab...
- 8organicbits 3y agoAh I see the link. I'd caution that many people choose between github.com, gitlab.com, and gitlab self-hosted. The reliability of self-hosted gitlab is meaningful, especially when operated competently. People need to know if there are safeguards or foot guns. Backups alone can't prevent data loss.
- namaria 3y agoSubstitute capital expenditure for operating expenses? With interest rates going up? It was already a tough sell with negative real rates...
- jonas-w 3y agowhy particularly GitLab?
- misnome 3y agohttps://status.gitlab.com/ https://status.gitlab.com/ lists 27 incidents this year, so far.
- alpaca128 3y agohttps://www.githubstatus.com/history https://www.githubstatus.com/history lists 72 incidents since January
- misnome 3y agoThe point isn't that GitLab has more, the point is that running these things at global scale is pretty complicated, and everyone has problems. "Just switch to GitLab" is pithy but isn't in itself an actual solution.
- tpxl 3y agoYou can self-host GitLab and have few, if any, incidents that get resolved very quickly. Worked for a company that had no incidents that I observed in ~3 years, now work at a company that had ~2 incidents in 1.5 years.
- misnome 3y agoWe have a self-hosted Premium instance and have 30min of downtime _every day_ while the database is frozen and backed up. We've been told that it's a known issue being discussed with GitLab but that could just be CYA. But in any case, it's the "at scale, while changing" that tends to cause problems. Perhaps this is a continuing argument for self-hosting, especially if you don't have to expose the instance publicly. But then, if that's an option, you can also self-host GitHub (though I have heard less anecdotes about the stability of that).
- grumple 3y ago> We have a self-hosted Premium instance and have 30min of downtime _every day_ while the database is frozen and backed up. I'm confused. You can do zero-downtime backups and replication of databases. I don't know what your company / Gitlab are doing but it seems wrong.
- Hamuko 3y agoGitLab is quite a bit more expensive. If you have GitHub Enterprise with the security features, it's $70/month/user whereas you'll need to get GitLab Ultimate for the security features, which is $99/month/user.
- progval 3y agoFeature mismatch of anything outside Git. And no one wrote the tooling needed to synchronize issues, pull/merge requests, ... back and forth.
- Lutger 3y agoEven so, for a lot of devs its still easier as a temporary collaboration point than sending patches via mail.
- electroly 3y agoGitHub outages aren't nearly long or often enough to consider this. Git is distributed, just keep working locally until GitHub is back up. GitHub outages are nowhere near the threshold of pain I'd require to introduce a second Git hosting provider to the mix. Really, GitHub outages barely hurt at all. It's not like an AWS or Cloudflare outage which is more likely to be a production disaster. Every outage a bunch of people on HN start screaming about owning their own on-prem destiny or wondering why we're still on GitHub. Nothing changes because it's not nearly as bad as those people are making it out to be. Life is all about tradeoffs.
- bombolo 3y agoDepends how your company is set up. Some people can't run tests locally and just push commits to have some magic run the tests online.