19 ms·

https://www.githubstatus.com/history https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own

https://www.githubstatus.com/history https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

edgyquant 5y ago
I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold. I have real problems to solve I don’t want to deal with a git repo manager on top of that.
- jeltz 5y ago
  Maybe you are in a different time zone because our organization certainly noticed and was disrupted by this.
  - edgyquant 5y ago
    I’m on PST time, some of our other devs are on the east coast and one is in India. I think we’re spread out enough it should be an issue but maybe we prioritize different things.
    - jeltz 5y ago
      We are in CET and maybe we use Github differently than you.
- gjulianm 5y ago
  At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.
  - xondono 5y ago
    I’d say it depends, I run my own on prem server and gitlab was a PITA. Too many moving parts, updating took too much of my time, and I never felt “safe”. Moving to gitea solved all of those issues for me (thus far), now I’m looking into adding other stuff like CI through Drone.
    - edgyquant 5y ago
      It definitely depends. We’re pretty early stage and I’m the senior engineer+infrastructure guy so running our own gitea instance or whatever is just more time that I’m almost out of.
    - Inversechi 5y ago
      Did you consider woodpecker instead of drone? It's basically an evolved fork of the OSS version. https://woodpecker-ci.org/ https://woodpecker-ci.org/
      - xondono 5y ago
        Didn’t even know about it. I’ll check it out. Thanks!
    - KronisLV 5y ago
      Curiously, this was also my own experience! I actually wrote a bit about the migration process, as well as the reasons for migrating over to Gitea, Nexus and Drone CI as opposed to using GitLab, GitLab Registry and GitLab CI: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-nexus-and-drone https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-... With containers, it's actually a pretty good experience that's not too hard to setup or manage.
  - nightpool 5y ago
    Self-hosted GitLab got a good callout yesterday from Microsoft, it appears to be a favorite of LAPSUS$: https://www.microsoft.com/security/blog/2022/03/22/dev-0537-criminal-actor-targeting-organizations-for-data-exfiltration-and-destruction/ https://www.microsoft.com/security/blog/2022/03/22/dev-0537-... Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much more secure to let the software's developers host it as well.
- thaeli 5y ago
  Also, looking at this it seems like GitHub isn't doing the common SaaS thing of just lying on their status page. Many providers, both internal and external, would look a lot worse if they had honest status pages.
  - mirekrusin 5y ago
    They are green for good 15 minutes from first moment i see problems, not the first time, it happens actually quite often. Maybe that's the time they need to confirm/cross check/write status update, don't know.
    - georgemcbay 5y ago
      While quicker reporting would be better, 15 minutes is anecdotally a lot better than I see from most other services where their status pages will report all-clear hours into full outages.
      - thaeli 5y ago
        Yeah, I'm legit impressed with a 15 minute time here.
    - judge2020 5y ago
      They probably allow regular SREs to trigger an incident on the status page on their own, when the likes of AWS and other bigger cloud providers are rumored to need approval from a VP[0] to update the status page. 0: https://news.ycombinator.com/item?id=29475756 https://news.ycombinator.com/item?id=29475756
  - ishanjain28 5y ago
    They do intentionally or not lie about this on their status page. From December 25th to December 31st 2021, Github actions had network problems almost every single day for hours and the status page was green out through out that period. Same thing also happened few months back. It feels like they do this manually and it's only done when enough people are effected.
  - 5y ago
- jmartens 5y ago
  My company monitors the functionality, performance and availability of apps like Github, and we have certainly noticed the increase in issues lately.
  - edgyquant 5y ago
    We were actually talking about implementing this last week. Not for GitHub but for slack as it seems to have issues once a month or so.
- SkyPuncher 5y ago
  > I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge These have been minor inconveniences for us - at worst. Most of the time it simply means people jump to something else then come back later in the day. Failing tests and PR feedback cycles are more of a blocker to our team than these outages.
- ManWith2Plans 5y ago
  I will say that for us this is a huge deal. We're a devops services company, and our customers expect their deployment pipelines to work. This is becoming a huge pain-point for a few of our customers and we recommended Github Actions to them. A couple of our customers want us to move away from GitHub actions because of how disruptive outages have been.
- wnevets 5y ago
  > I’ve hardly noticed the three month surge. This has been my experience as well. I don't know if that means GitHub is being overly transparent about issues or I've just been lucky but I would hate if people punished services for being transparent and informative on their status pages.
  - zenexer 5y ago
    GitHub's outages have hit me hard over the past week or so. I don't think it's a matter of them being transparent--if anything, I was hitting errors well before their status page updated. Yesterday it was completely unusable for much of my workday, and today tasks that normally take me a few minutes have been taking hours.
- jhugo 5y ago
  I think the impact was for some reason not consistent between users (maybe due to geographical factors or maybe sharding of accounts?). We're in Asia and I think we've had three different days recently where we couldn't actually get much work done due to GitHub being flakey or down for the entire day and our CI/CD and development processes being built around it. We ended up moving off GitHub onto a self-hosted system, which took about a day of work for one engineer (CI/CD itself was already self-hosted, so just Git, issues and PRs), and there have already been two more GitHub outages since then.
- jacobr 5y ago
  20 PRs waiting in line for half a day to be merged is pretty annoying. We’ve had that on multiple occasions the last few weeks due to GitHub incidents.
rvz 5y ago
Well, I think I have said that since 2020 [0] and it is self-evident that you are better off self-hosting your own Git repo. If you can host a website you can do it. If GNOME, ReactOS, Wireguard, Linux Kernel Project, Mozilla, etc can do it, so can you. Or even use it as a backup / failsafe just in case. But going 'all in' on GitHub just doesn't make any sense anymore. [0] https://hn.algolia.com/?dateRange=all&page=1&prefix=true&query=author%3Arvz%20Github%20self%20host&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=1&prefix=true&que...
- rglullis 5y ago
  It's almost like people forget that git is a Distributed Version Control System, after all...
  - djbusby 5y ago
    GitHub/Lab are for more than just code repo
    - aaaaaaaaata 5y ago
      Can project management features not be made part of a dumb repo on the db side? (Spoiler: yes, and many projects have explored this — setup unfortunately has never been as easy as "we'll invite u to the gh, check ur email". Perhaps with decentralization push of web3/QR etc, we'll get over the hump.
      - rglullis 5y ago
        I think that parent means also things like CI, release repository, PR review, etc. These are not easily portable, but honestly is because of this lock-in that I prefer to use separate/independent tools. For my open source project [0], I am putting things on github and it is the link that give to most people, but in reality is just a mirror to the gitlab repository[1], which I use for CI and static page hosting, and the "project management" is done on Taiga [2] [0]: https://github.com/mushroomlabs/hub20 [1]: https://gitlab.com/mushroomlabs/hub20/hub20 [2]: https://tree.taiga.io/project/lullis-mushroomlabshub20
Jiejeing 5y ago
If you are a closed org, that is. Running your own gitea or gitlab with registration enabled and having to deal with spam is a real hurdle.
- julianlam 5y ago
  Is it not possible to restrict access to the git server from a VPN server only? Just off the top of my head, that's one thing you can do.
  - mlyle 5y ago
    Yah, that's a "closed org". When you need to deal with the public at large, you need to deal with user registration issues and spam.
  - Dobbs 5y ago
    So now every person who wants to contribute to your open source project has to setup a VPN client? The parent comment was explicitly about non-closed (e.g. private) orgs.
chockchocschoir 5y ago
> At this rate the benefits of running your own gitea or gitlab are starting to become competitive No need, just use Codeberg.org instead. They run Gitea and is a free collaboration platform (+ git hosting) for free projects. FOSS/OSS should really consider alternatives to GitHub and GitLab, especially when there are much more FOSS/OSS friendly platforms around.
Chris2048 5y ago
> running your own assuming that would be flawless, which it wouldn't
belter 5y ago
Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.
- ejb999 5y ago
  thats not the kind of progression you like to see - that is, error rates increasing over time instead of decreasing.
  - TheRealPomax 5y ago
    Only if you believe those numbers mean anything. What are the errors for? Github has been adding lots of features and subproducts over the years, becoming a bigger and bigger platform as a result. What you want is the error-per-component, which may very well have actually gone down, with error spikes coming from "when github adds a completely new feature and it goes through a slew of incidents in its first year". The bigger the feature, the more incidents. Without more detailed numbers, there's literally no conclusion to draw here.
    - ejb999 5y ago
      Every place I have ever worked reported incidents going down would be good, not up.
      - TheRealPomax 5y ago
        Every place I ever worked at understood that if you x3 the codebase/infra/interaction surface/etc, you can expect x3 errors. If the total number of errors don't go up as you grow you're doing amazing, and if they go down even though you're landing more and more code for more and more features and subproducts, you have a genuine miracle.
        AlexandrB 5y ago
adamsmith143 5y ago
Is it really though? Are engineers committing so frequently that they can't make it through a few hours without Github?
- imiric 5y ago
  GitHub doesn't just host Git repositories. It's the central location for discussions, issues, code reviews, milestone planning, and any CI process like testing or releases. If it's unavailable whole teams can be interrupted. Git is distributed. GitHub is very much not.
- jeffwask 5y ago
  Yes
- queuebert 5y ago
  Maybe they measure performance in git commits.
- deleted 5y ago
  [deleted]
- lallysingh 5y ago
  It depends on how many engineers you have! But also, there are plenty of other functions in GH besides raw git, like Wiki/PR/Issues/test/deploy pipelines, etc. It can become pretty critical.
- duped 5y ago
  An outage of a few hours can tank a release deadline for me, so yes.
dbrgn 5y ago
Does Gitea support some kind of federation / cross-instance PRs? That's the main thing I'd miss from a self-hosted instance, the ease of getting contributions. After all, you don't even need Gitea for pure Git hosting. If you have a server with SSH access, just init a bare repo in a directory, push to that, and you're ready to go. No web UI needed. The reason I'm still using GitHub is not code hosting. It's collaboration.
- dbrgn 5y ago
  It seems there's a tracking issue here, but it seems stalled: https://github.com/go-gitea/gitea/issues/1612 https://github.com/go-gitea/gitea/issues/1612
- tokumei 5y ago
  > If you have a server with SSH access, just init a bare repo in a directory, push to that, and you're ready to go. No web UI needed. Used to do that years ago for my personal projects. Honestly does the trick.
- brimble 5y ago
  Gitea gets you: a nice GitHub-like web GUI, including for stuff like managing users; 2FA; some integrations; web hooks without having to add git-hooks to all your repos; and extremely-useful-to-some-projects features like git-lfs support. If you don't want or need those things, bare git repos are fine and certainly easier to support (not that Gitea's that hard, though a few issues/PRs I've noticed have caused me more than a little concern about the overall quality of the project).
- encryptluks2 5y ago
  But by using GitHub for "collaboration" you are sacrificing decentralization.
  - mst 5y ago
    Absolutely, and I genuinely hate that. But for new open source projects of mine the ease of contribution and user expectation of a github repository are a trade-off worth making even so (I also maintain a self hosted master git repo that I consider the source of truth to -me- but these days it syncs from, rather than to, github, just because of the trade-offs involved).
mhh__ 5y ago
The company I work for has a bunch of non-programmers using and working in gitlab (or "the git"), I can't really see it happening with GitHub regardless of where it was hosted. Gitlab just seems better for actually running a software project.
mrkurt 5y ago
If you want companies to be honest on their status pages (I do!), you can't just count incidents like that. Status pages can be an amazing place to communicate all kinds of problems. Most issues have a relatively narrow impact, but the impacted people _still_ benefit from seeing them listed.
- jmartens 5y ago
  How can we solve this as customers, or push the vendor to do better?
  - mrkurt 5y ago
    Use vendors who do a good job communicating status, basically. I don't think you can change AWS behavior. But if you find a hosting company who does an amazing job with their status updates, put some apps there (_my_ company does an ok job with status page updates, we're getting better, it's not amazing yet).
  - encryptluks2 5y ago
    Stop being a customer of crappy vendors
    - copperx 5y ago
      What cloud provider does better status pages than AWS?
      - drusepth 5y ago
        The snarky answer is "literally all of them", but one real answer is that I've been pretty happy with GCP's status reporting for the past year-ish I've used them. I've only noticed a few incidents, but every time I've checked the status it was already updated. They also occasionally provide workarounds on the live incident pages if you need to be back up before the issue is fixed on their end.
      - mst 5y ago
gwbas1c 5y ago
> At this rate the benefits of running your own gitea or gitlab are starting to become competitive When you host things yourself, you still have downtime. And, having worked with Github for over a decade, the actual disruption to my work is from downtime is much less than if I had to host my own. That being said: I briefly worked for a company that hosted its own source code control system. For us, as a small team, it wasn't worth it. The system was outdated and hosted in an insecure manner. No one ever did any "admin" work except the founder. He ran it because he had irrational fears of switching, not because of any tangible advantages over Github (and competitors.) Keep in mind that Github (and competitors) are often cheaper than the time needed to invest in hosting your own. (Estimate 10-20 hours a year of invested time. Calculate your hourly rate. Github and competitors are cheaper.) In order to come ahead, you need tangible benefits other than "I think I can have less downtime."
- megous 5y ago
  Dunno, I got blocked from my work SaaS hosted gitlab for about a month by cloudflare. Nobody at gitlab or cf helped. I only figured it myself after about 4 hours of research, that it was caused by some disabled (by me years ago) web tracking APIs no-one should have hard dependence on. I certainly would not have this problem on self hosted instance, because it would not be behind CF. I'm sure I'd have other problems though. :) All software is crap. You can be either spending time fixing it yourself, or spending time begging online for fixes/help from some SaaS company/community with resolution time in months, somtimes, all that while you may not be able to use it fully. Also with SaaS it will be constantly shifting under you. Things will be moved around, restyled, iconized, popupized, etc. This doesn't help productivity either. With self-hosting, you can at least avoid upgrading, if you dislike this kind of thing. Or choose FOSS software that values UX permanency/stability, which seems to be really hard ask from SaaS business.
ironmagma 5y ago
We run Gitea at my company. In fact, we forked it. It could reeeaaaalllly use a rewrite. If anyone is even mildly ambitious about creating a new alternative to Github/Gitea, it's a great time to do that.
- KronisLV 5y ago
  Another self-hosted project in the space that i've seen was GitBucket, although it runs on the JVM (not necessarily a bad thing, just different from Go): https://gitbucket.github.io/ https://gitbucket.github.io/
- mynameismon 5y ago
  You might be interested in sourcehut: https://sr.ht https://sr.ht
Melatonic 5y ago
The marketing for building your own new "private cloud" will begin soon I am sure :-D
- pid-1 5y ago
  You just made a few Openstack consultants raise from their graves.
TheRealPomax 5y ago
But how many of those actually affected you? For example, no amount of issues around codespaces or github packages would impact my professional use of github, so whether there are 21 or 5000 or those parts get permanently taken offline makes no difference in what I need out of the platform. How many core incidents? The part that affects whether you can even push to and pull from a repo, and access issues and PRs? Because everything else is nice to have, but you can do work perfectly fine without them if they go down for a few hours.
- CanSpice 5y ago
  Yesterday's affected me, I couldn't pull or push and when I tried to look at the repo to do PRs I got 500 errors. That only lasted maybe 30 minutes though.
- sitzkrieg 5y ago
  i could not even sso login so it was a bit more impactful than it sounds on the paper
  - TheRealPomax 5y ago
    Absolutely, but could you not log in every single time, or just "once this year so far"?
- jeltz 5y ago
  I was affected by the one last week, the one yesterday and the one today. The one today was harmless but the other two disrupted out work. All three were "core incidents", but the one today felt shorter.
sonicggg 5y ago
Come on, don't be so dramatic. This is not a 911 call center, people will survive these minor outages.
- mdellavo 5y ago
  sure but three days in a row?
devwastaken 5y ago
And whom pays for fixing it? Downtimes of self hosted systems using external software can be far longer. GitHub, unlike Amazon and friends, doesn't lie about their downtime. Every saas has hundreds of downtime instances across the board every month. Some are small enough you don't see them. Yet the services still work exceptionally well - and when they don't they get fixed in a quick manner. What takes them an hour would take most private orgs a day.
- everfrustrated 5y ago
  GitHub is notorious for only noticing outages once the USA morning starts. If you're using GitHub in Europe or Asia it's not uncommon for GitHub to be offline for many hours before they acknowledge anying.
- AlexandrB 5y ago
  > GitHub, unlike Amazon and friends, doesn't lie about their downtime. Are you kidding? The last 2 incidents were called "degraded performance". Where "degraded" meant I would get nothing but 500 errors accessing GitHub.com either via browser or git itself for the duration of the outage. How is this not lying?
ransom1538 5y ago
"21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive." Oh stop the drama. Fine. Setup your gitlab.
dijonman2 5y ago
GitHub enterprise is amazing, but I agree that a centrally hosted Git instance of any variety is a liability. With the advent of the Okta breach I think we will see a reverse in the centralization trend.

edgyquant 5y ago

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold. I have real problems to solve I don’t want to deal with a git repo manager on top of that.

jeltz 5y ago
Maybe you are in a different time zone because our organization certainly noticed and was disrupted by this.
- edgyquant 5y ago
  I’m on PST time, some of our other devs are on the east coast and one is in India. I think we’re spread out enough it should be an issue but maybe we prioritize different things.
  - jeltz 5y ago
    We are in CET and maybe we use Github differently than you.
gjulianm 5y ago
At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.
- xondono 5y ago
  I’d say it depends, I run my own on prem server and gitlab was a PITA. Too many moving parts, updating took too much of my time, and I never felt “safe”. Moving to gitea solved all of those issues for me (thus far), now I’m looking into adding other stuff like CI through Drone.
  - edgyquant 5y ago
    It definitely depends. We’re pretty early stage and I’m the senior engineer+infrastructure guy so running our own gitea instance or whatever is just more time that I’m almost out of.
  - Inversechi 5y ago
    Did you consider woodpecker instead of drone? It's basically an evolved fork of the OSS version. https://woodpecker-ci.org/ https://woodpecker-ci.org/
    - xondono 5y ago
      Didn’t even know about it. I’ll check it out. Thanks!
  - KronisLV 5y ago
    Curiously, this was also my own experience! I actually wrote a bit about the migration process, as well as the reasons for migrating over to Gitea, Nexus and Drone CI as opposed to using GitLab, GitLab Registry and GitLab CI: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-nexus-and-drone https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-... With containers, it's actually a pretty good experience that's not too hard to setup or manage.
- nightpool 5y ago
  Self-hosted GitLab got a good callout yesterday from Microsoft, it appears to be a favorite of LAPSUS$: https://www.microsoft.com/security/blog/2022/03/22/dev-0537-criminal-actor-targeting-organizations-for-data-exfiltration-and-destruction/ https://www.microsoft.com/security/blog/2022/03/22/dev-0537-... Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much more secure to let the software's developers host it as well.
thaeli 5y ago
Also, looking at this it seems like GitHub isn't doing the common SaaS thing of just lying on their status page. Many providers, both internal and external, would look a lot worse if they had honest status pages.
- mirekrusin 5y ago
  They are green for good 15 minutes from first moment i see problems, not the first time, it happens actually quite often. Maybe that's the time they need to confirm/cross check/write status update, don't know.
  - georgemcbay 5y ago
    While quicker reporting would be better, 15 minutes is anecdotally a lot better than I see from most other services where their status pages will report all-clear hours into full outages.
    - thaeli 5y ago
      Yeah, I'm legit impressed with a 15 minute time here.
  - judge2020 5y ago
    They probably allow regular SREs to trigger an incident on the status page on their own, when the likes of AWS and other bigger cloud providers are rumored to need approval from a VP[0] to update the status page. 0: https://news.ycombinator.com/item?id=29475756 https://news.ycombinator.com/item?id=29475756
- ishanjain28 5y ago
  They do intentionally or not lie about this on their status page. From December 25th to December 31st 2021, Github actions had network problems almost every single day for hours and the status page was green out through out that period. Same thing also happened few months back. It feels like they do this manually and it's only done when enough people are effected.
- 5y ago
jmartens 5y ago
My company monitors the functionality, performance and availability of apps like Github, and we have certainly noticed the increase in issues lately.
- edgyquant 5y ago
  We were actually talking about implementing this last week. Not for GitHub but for slack as it seems to have issues once a month or so.
SkyPuncher 5y ago
> I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge These have been minor inconveniences for us - at worst. Most of the time it simply means people jump to something else then come back later in the day. Failing tests and PR feedback cycles are more of a blocker to our team than these outages.
ManWith2Plans 5y ago
I will say that for us this is a huge deal. We're a devops services company, and our customers expect their deployment pipelines to work. This is becoming a huge pain-point for a few of our customers and we recommended Github Actions to them. A couple of our customers want us to move away from GitHub actions because of how disruptive outages have been.
wnevets 5y ago
> I’ve hardly noticed the three month surge. This has been my experience as well. I don't know if that means GitHub is being overly transparent about issues or I've just been lucky but I would hate if people punished services for being transparent and informative on their status pages.
- zenexer 5y ago
  GitHub's outages have hit me hard over the past week or so. I don't think it's a matter of them being transparent--if anything, I was hitting errors well before their status page updated. Yesterday it was completely unusable for much of my workday, and today tasks that normally take me a few minutes have been taking hours.
jhugo 5y ago
I think the impact was for some reason not consistent between users (maybe due to geographical factors or maybe sharding of accounts?). We're in Asia and I think we've had three different days recently where we couldn't actually get much work done due to GitHub being flakey or down for the entire day and our CI/CD and development processes being built around it. We ended up moving off GitHub onto a self-hosted system, which took about a day of work for one engineer (CI/CD itself was already self-hosted, so just Git, issues and PRs), and there have already been two more GitHub outages since then.
jacobr 5y ago
20 PRs waiting in line for half a day to be merged is pretty annoying. We’ve had that on multiple occasions the last few weeks due to GitHub incidents.

rvz 5y ago

Well, I think I have said that since 2020 [0] and it is self-evident that you are better off self-hosting your own Git repo. If you can host a website you can do it. If GNOME, ReactOS, Wireguard, Linux Kernel Project, Mozilla, etc can do it, so can you. Or even use it as a backup / failsafe just in case. But going 'all in' on GitHub just doesn't make any sense anymore. [0] https://hn.algolia.com/?dateRange=all&page=1&prefix=true&query=author%3Arvz%20Github%20self%20host&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=1&prefix=true&que...

rglullis 5y ago
It's almost like people forget that git is a Distributed Version Control System, after all...
- djbusby 5y ago
  GitHub/Lab are for more than just code repo
  - aaaaaaaaata 5y ago
    Can project management features not be made part of a dumb repo on the db side? (Spoiler: yes, and many projects have explored this — setup unfortunately has never been as easy as "we'll invite u to the gh, check ur email". Perhaps with decentralization push of web3/QR etc, we'll get over the hump.
    - rglullis 5y ago
      I think that parent means also things like CI, release repository, PR review, etc. These are not easily portable, but honestly is because of this lock-in that I prefer to use separate/independent tools. For my open source project [0], I am putting things on github and it is the link that give to most people, but in reality is just a mirror to the gitlab repository[1], which I use for CI and static page hosting, and the "project management" is done on Taiga [2] [0]: https://github.com/mushroomlabs/hub20 [1]: https://gitlab.com/mushroomlabs/hub20/hub20 [2]: https://tree.taiga.io/project/lullis-mushroomlabshub20

Jiejeing 5y ago

If you are a closed org, that is. Running your own gitea or gitlab with registration enabled and having to deal with spam is a real hurdle.

julianlam 5y ago
Is it not possible to restrict access to the git server from a VPN server only? Just off the top of my head, that's one thing you can do.
- mlyle 5y ago
  Yah, that's a "closed org". When you need to deal with the public at large, you need to deal with user registration issues and spam.
- Dobbs 5y ago
  So now every person who wants to contribute to your open source project has to setup a VPN client? The parent comment was explicitly about non-closed (e.g. private) orgs.

chockchocschoir 5y ago

> At this rate the benefits of running your own gitea or gitlab are starting to become competitive No need, just use Codeberg.org instead. They run Gitea and is a free collaboration platform (+ git hosting) for free projects. FOSS/OSS should really consider alternatives to GitHub and GitLab, especially when there are much more FOSS/OSS friendly platforms around.

Chris2048 5y ago

> running your own assuming that would be flawless, which it wouldn't

belter 5y ago

Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.

ejb999 5y ago
thats not the kind of progression you like to see - that is, error rates increasing over time instead of decreasing.
- TheRealPomax 5y ago
  Only if you believe those numbers mean anything. What are the errors for? Github has been adding lots of features and subproducts over the years, becoming a bigger and bigger platform as a result. What you want is the error-per-component, which may very well have actually gone down, with error spikes coming from "when github adds a completely new feature and it goes through a slew of incidents in its first year". The bigger the feature, the more incidents. Without more detailed numbers, there's literally no conclusion to draw here.
  - ejb999 5y ago
    Every place I have ever worked reported incidents going down would be good, not up.
    - TheRealPomax 5y ago
      Every place I ever worked at understood that if you x3 the codebase/infra/interaction surface/etc, you can expect x3 errors. If the total number of errors don't go up as you grow you're doing amazing, and if they go down even though you're landing more and more code for more and more features and subproducts, you have a genuine miracle.
      - AlexandrB 5y ago

adamsmith143 5y ago

Is it really though? Are engineers committing so frequently that they can't make it through a few hours without Github?

imiric 5y ago
GitHub doesn't just host Git repositories. It's the central location for discussions, issues, code reviews, milestone planning, and any CI process like testing or releases. If it's unavailable whole teams can be interrupted. Git is distributed. GitHub is very much not.
jeffwask 5y ago
Yes
queuebert 5y ago
Maybe they measure performance in git commits.
deleted 5y ago
[deleted]
lallysingh 5y ago
It depends on how many engineers you have! But also, there are plenty of other functions in GH besides raw git, like Wiki/PR/Issues/test/deploy pipelines, etc. It can become pretty critical.
duped 5y ago
An outage of a few hours can tank a release deadline for me, so yes.

dbrgn 5y ago

Does Gitea support some kind of federation / cross-instance PRs? That's the main thing I'd miss from a self-hosted instance, the ease of getting contributions. After all, you don't even need Gitea for pure Git hosting. If you have a server with SSH access, just init a bare repo in a directory, push to that, and you're ready to go. No web UI needed. The reason I'm still using GitHub is not code hosting. It's collaboration.

dbrgn 5y ago
It seems there's a tracking issue here, but it seems stalled: https://github.com/go-gitea/gitea/issues/1612 https://github.com/go-gitea/gitea/issues/1612
tokumei 5y ago
> If you have a server with SSH access, just init a bare repo in a directory, push to that, and you're ready to go. No web UI needed. Used to do that years ago for my personal projects. Honestly does the trick.
brimble 5y ago
Gitea gets you: a nice GitHub-like web GUI, including for stuff like managing users; 2FA; some integrations; web hooks without having to add git-hooks to all your repos; and extremely-useful-to-some-projects features like git-lfs support. If you don't want or need those things, bare git repos are fine and certainly easier to support (not that Gitea's that hard, though a few issues/PRs I've noticed have caused me more than a little concern about the overall quality of the project).
encryptluks2 5y ago
But by using GitHub for "collaboration" you are sacrificing decentralization.
- mst 5y ago
  Absolutely, and I genuinely hate that. But for new open source projects of mine the ease of contribution and user expectation of a github repository are a trade-off worth making even so (I also maintain a self hosted master git repo that I consider the source of truth to -me- but these days it syncs from, rather than to, github, just because of the trade-offs involved).

mhh__ 5y ago

The company I work for has a bunch of non-programmers using and working in gitlab (or "the git"), I can't really see it happening with GitHub regardless of where it was hosted. Gitlab just seems better for actually running a software project.

mrkurt 5y ago

If you want companies to be honest on their status pages (I do!), you can't just count incidents like that. Status pages can be an amazing place to communicate all kinds of problems. Most issues have a relatively narrow impact, but the impacted people _still_ benefit from seeing them listed.

jmartens 5y ago
How can we solve this as customers, or push the vendor to do better?
- mrkurt 5y ago
  Use vendors who do a good job communicating status, basically. I don't think you can change AWS behavior. But if you find a hosting company who does an amazing job with their status updates, put some apps there (_my_ company does an ok job with status page updates, we're getting better, it's not amazing yet).
- encryptluks2 5y ago
  Stop being a customer of crappy vendors
  - copperx 5y ago
    What cloud provider does better status pages than AWS?
    - drusepth 5y ago
      The snarky answer is "literally all of them", but one real answer is that I've been pretty happy with GCP's status reporting for the past year-ish I've used them. I've only noticed a few incidents, but every time I've checked the status it was already updated. They also occasionally provide workarounds on the live incident pages if you need to be back up before the issue is fixed on their end.
    - mst 5y ago

gwbas1c 5y ago

> At this rate the benefits of running your own gitea or gitlab are starting to become competitive When you host things yourself, you still have downtime. And, having worked with Github for over a decade, the actual disruption to my work is from downtime is much less than if I had to host my own. That being said: I briefly worked for a company that hosted its own source code control system. For us, as a small team, it wasn't worth it. The system was outdated and hosted in an insecure manner. No one ever did any "admin" work except the founder. He ran it because he had irrational fears of switching, not because of any tangible advantages over Github (and competitors.) Keep in mind that Github (and competitors) are often cheaper than the time needed to invest in hosting your own. (Estimate 10-20 hours a year of invested time. Calculate your hourly rate. Github and competitors are cheaper.) In order to come ahead, you need tangible benefits other than "I think I can have less downtime."

megous 5y ago
Dunno, I got blocked from my work SaaS hosted gitlab for about a month by cloudflare. Nobody at gitlab or cf helped. I only figured it myself after about 4 hours of research, that it was caused by some disabled (by me years ago) web tracking APIs no-one should have hard dependence on. I certainly would not have this problem on self hosted instance, because it would not be behind CF. I'm sure I'd have other problems though. :) All software is crap. You can be either spending time fixing it yourself, or spending time begging online for fixes/help from some SaaS company/community with resolution time in months, somtimes, all that while you may not be able to use it fully. Also with SaaS it will be constantly shifting under you. Things will be moved around, restyled, iconized, popupized, etc. This doesn't help productivity either. With self-hosting, you can at least avoid upgrading, if you dislike this kind of thing. Or choose FOSS software that values UX permanency/stability, which seems to be really hard ask from SaaS business.

ironmagma 5y ago

We run Gitea at my company. In fact, we forked it. It could reeeaaaalllly use a rewrite. If anyone is even mildly ambitious about creating a new alternative to Github/Gitea, it's a great time to do that.

KronisLV 5y ago
Another self-hosted project in the space that i've seen was GitBucket, although it runs on the JVM (not necessarily a bad thing, just different from Go): https://gitbucket.github.io/ https://gitbucket.github.io/
mynameismon 5y ago
You might be interested in sourcehut: https://sr.ht https://sr.ht

Melatonic 5y ago

The marketing for building your own new "private cloud" will begin soon I am sure :-D

pid-1 5y ago
You just made a few Openstack consultants raise from their graves.

TheRealPomax 5y ago

But how many of those actually affected you? For example, no amount of issues around codespaces or github packages would impact my professional use of github, so whether there are 21 or 5000 or those parts get permanently taken offline makes no difference in what I need out of the platform. How many core incidents? The part that affects whether you can even push to and pull from a repo, and access issues and PRs? Because everything else is nice to have, but you can do work perfectly fine without them if they go down for a few hours.

CanSpice 5y ago
Yesterday's affected me, I couldn't pull or push and when I tried to look at the repo to do PRs I got 500 errors. That only lasted maybe 30 minutes though.
sitzkrieg 5y ago
i could not even sso login so it was a bit more impactful than it sounds on the paper
- TheRealPomax 5y ago
  Absolutely, but could you not log in every single time, or just "once this year so far"?
jeltz 5y ago
I was affected by the one last week, the one yesterday and the one today. The one today was harmless but the other two disrupted out work. All three were "core incidents", but the one today felt shorter.

sonicggg 5y ago

Come on, don't be so dramatic. This is not a 911 call center, people will survive these minor outages.

mdellavo 5y ago
sure but three days in a row?

devwastaken 5y ago

And whom pays for fixing it? Downtimes of self hosted systems using external software can be far longer. GitHub, unlike Amazon and friends, doesn't lie about their downtime. Every saas has hundreds of downtime instances across the board every month. Some are small enough you don't see them. Yet the services still work exceptionally well - and when they don't they get fixed in a quick manner. What takes them an hour would take most private orgs a day.

everfrustrated 5y ago
GitHub is notorious for only noticing outages once the USA morning starts. If you're using GitHub in Europe or Asia it's not uncommon for GitHub to be offline for many hours before they acknowledge anying.
AlexandrB 5y ago
> GitHub, unlike Amazon and friends, doesn't lie about their downtime. Are you kidding? The last 2 incidents were called "degraded performance". Where "degraded" meant I would get nothing but 500 errors accessing GitHub.com either via browser or git itself for the duration of the outage. How is this not lying?

ransom1538 5y ago

"21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive." Oh stop the drama. Fine. Setup your gitlab.

dijonman2 5y ago

GitHub enterprise is amazing, but I agree that a centrally hosted Git instance of any variety is a liability. With the advent of the Okta breach I think we will see a reverse in the centralization trend.