8 ms·

This has been happening pretty much every day for a long time. Maybe this will push more people towards some decentralization. A self-hosted Gitea instance my o

This has been happening pretty much every day for a long time. Maybe this will push more people towards some decentralization. A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. None. Same with our CI system. Yes, I know, GitHub is 10⁹⁹⁹ times larger than our puny Gitea instance, and that's why they're having issues, but why should I care when one is working, and the other one isn't?

chockchocschoir 5y ago
It's not about being larger, it's about doing changes. GitHub used to be pretty stable before, when they didn't really do any (user facing) changes. It stood still in terms of features, but was also pretty stable overall. Compared to GitHub, how often does your Gitea instance go through code changes? How many major features have been switched on in that instance during the past year without any hiccups?
- freedomben 5y ago
  I tried to look at the Gitea repo to see how fast they get commits, but github is 500ing :-( https://github.com/go-gitea/gitea https://github.com/go-gitea/gitea
  - chockchocschoir 5y ago
    Rest assure, a open source project with ~100 contributors (whereas maybe 10% regularly contributes) has less activity and changes than a multi-international company focusing a SaaS with "1000 to 5000" employees worldwide.
    - judge2020 5y ago
      For perspective, GitHub has 4,000 employees who list them as their employer on LinkedIn and GitLab has 1,700.
- Deukhoofd 5y ago
  I run Gitea myself as well. I generally throw in a new release once a month or so, never had any real hiccups.
  - rolph 5y ago
    it sounds like you dont make code changes unless they are required for function.
    - Deukhoofd 5y ago
      Also for security reasons, and because some features are nice to have. But you're right in that I'm definitely not randomly updating sometimes. When I do update I basically just back up the database, throw in a new Gitea binary, and restart the service. As the releases have been fairly well tested in general, this hasn't given me issues so far.
  - chockchocschoir 5y ago
    Again, how much it actually change across those releases? GitHub most likely are going through more changes on a weekly basis (if not daily) than what Gitea does on a monthly basis.
    - Deukhoofd 5y ago
      The major releases generally change quite a lot in one go, for example: https://blog.gitea.io/2022/02/gitea-1.16.0-and-1.16.1-released/ https://blog.gitea.io/2022/02/gitea-1.16.0-and-1.16.1-releas... https://blog.gitea.io/2021/08/gitea-1.15.0-is-released/ https://blog.gitea.io/2021/08/gitea-1.15.0-is-released/ But I'm sure Github changes a lot more. However, the question is if those changes are worth the instability. For my workflow I just want my repos to be accessible, have my source, issues, and wiki accessible, and of course my basic git operations working. If that core functionality is actually not working at times, I'd definitely consider changing the git provider.
- dogleash 5y ago
  GitHub's feature churn is irrelevant to the comparison unless those features are used in parent poster's workflow.
  - chockchocschoir 5y ago
    Really? No, comparing the uptime of two services (actually, one service and one open source self-hosted service) it's relevant to consider _why_ one would have more downtime than another, even if they don't use those features. Even if the parent posters workflow doesn't use the new/deprecated features, they will still be affected by the downtime GitHub have monthly.
    - Nextgrid 5y ago
      I don't think anyone is overlooking the reasons why one might have more downtime than the other. The problem is that ultimately the downtime itself matters and not the reason, and if you don't need any of the features that GitHub offers, then the self-hosted route is a better option.
      - falcolas 5y ago
        > if you don't need any of the features that GitHub offers In particular, all of the new features whose addition causes instability.
        Nextgrid 5y ago
        I think it's the development process (as in moving things around and potentially introducing bugs) than the new features per-se.
        falcolas 5y ago
        With all due respect to the developers, it would not speak well for GitHub if they can't maintain stability while refactoring. There's a lot of testing processes and CI processes explicitly around making those actions safe. New features breaking is a lot more understandable - even expected - than regressions and refactoring failures.
- newswasboring 5y ago
  An infrastructure company focusing on feature churn just sounds wrong.
  - woah 5y ago
    What? Github is a social networking company.
    - newswasboring 5y ago
      I'm a bit dumb about these things on text, but did you drop an "/s" somewhere? I understand it has sort of a social networking layer on top, but that is not its main function right? This would be like calling a calculator app a social networking app because it allows me to share my results :D.
      - chockchocschoir 5y ago
        GitHub has always branded itself as a "social collaboration platform" rather than anything else (well, first "git hosting" but secondly the social stuff). Here is a early version of their landing page from 2008 (the year they launched): https://web.archive.org/web/20081111061111/http://github.com/ https://web.archive.org/web/20081111061111/http://github.com... Notice the logo says "Social code hosting" and the messaging of the page is mostly around popular repositories, collaborative features and other social elements.
skeeter2020 5y ago
>> A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. None. Same with our CI system. And when they do go down, how long will they be unavailable? When this happens (and it will happen) please post here so we can say "why are you running infrastructure yourself?"
- scandox 5y ago
  5 years of a Gogs instance. Zero downtime. I'll come back to you when it goes down.
  - ramesh31 5y ago
    does gogs plan to support any kind of CI integration like gh actions?
    - scandox 5y ago
      No it doesn't. I fully confess the feature set is extremely minimal. But the point here is that running your own infrastructure can make a lot of sense if you know what your precise requirements are (or are not).
    - sreevisakh 5y ago
      Gogs and gitea are designed to be a single-binary minimal source code hosting service. Additional services like CI within that binary is likely to bloat it up and make it less reliable. However, it's easy to integrate a CI service to it - like Woodpecker CI [1]. Woodpecker also has a minimalist approach like gogs. If you instead prefer a full package, then gitlab is the best solution. [1] https://woodpecker-ci.org/docs/administration/vcs/gogs https://woodpecker-ci.org/docs/administration/vcs/gogs
- Nextgrid 5y ago
  Very likely to be lower than GitHub/etc, because it's a much simpler system that can be reinstalled from scratch in 30 minutes by an experienced admin on brand new hardware if needed.
contravariant 5y ago
10^999, really? Rule of thumb is that 10^100 of anything doesn't fit in the observable universe.
rvz 5y ago
> A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. > Yes, I know, GitHub is 10⁹⁹⁹ times larger than our puny Gitea instance, and that's why they're having issues Exactly. That is also my overall point. [0] It makes no sense to go 'all in' on GitHub and something goes down and everyone is stuck once again. [0] https://news.ycombinator.com/item?id=30711442 https://news.ycombinator.com/item?id=30711442
freedomben 5y ago
Mind sharing some details on your install? Which database are you using (especially wondering about sqlite with gitea)? Are you using the docker install, or from package, or in k8s? No pressure, but I'm seriously considering standing one up but it would need to be production ready.
- raxxorrax 5y ago
  Gitea is absolutely production ready, we use an older version by now and don't even think about it much because it just works so well. Exactly what you want in production (production = software should work more often). Actually not "vendoring" dependencies from github is very much not production ready in many cases.
- qbasic_forever 5y ago
  IMHO storage is the big issue to figure out with a production ready gitea installation. Gitea only supports the local filesystem for storing bare repo data and not anything like S3 or other distributed or highly available storage systems (it supports S3 only for git LFS data right now, it is a long awaited open issue to get the bare repos there too). So the big issue for any non-trivial or highly available setup is going to be how you get high availability for the local storage volume. There are tons of options with various tradeoffs and levels of complexity here--simple local disk RAID, distributed filesystems like gluster or ceph, etc. I think this is the real crux of getting a good gitea instance going. No matter what figure out and test a good backup and recovery solution!
  - freedomben 5y ago
    Ah interesting! I read somewhere that it uses the database for that, but I think they were mistaken. Thanks that's quite helpful. If we do it in prod we'll have to accept single-instance.
    - qbasic_forever 5y ago
      Yeah I forgot to mention too a lot of complexity goes away if you host it on a good IaaS platform. On AWS for example you could just mount an EBS volume as the bare repo root and use RDS for a database. Then you just have to deploy and manage an EC2 instance to host gitea's main process. It's more expensive to use those managed platforms but might be well worth the cost vs. doing it all yourself with a bare metal deployment of everything.
corobo 5y ago
Opposite anecdata my GitLab install absolutely self destructed itself at one point and I had to rebuild it myself, waste of a weekend
- ashildr 5y ago
  > my GitLab install absolutely self destructed itself That’s frightening, we’ve been using it for multiple years now. It’s running fine and the only short downtime is every few weeks for the update. What happened to your instance?
  - corobo 5y ago
    More fairly to GitLab it was the underlying server that broke initially. The issue from there was that I couldn't restore from the backup as the newly installed server was also a newer version of GitLab Tried restoring it to the older version to run through the upgrade path and the process didn't want to work. Wish I could be more specific here but I gave up on it and decided it'd just be quicker to whip up a script to recreate all the groups/projects/CICD in a new install and push repos from backups Don't let your GitLab server get outdated and don't consider it a valid backup unless it was taken with the latest version to avoid the same Also it wasn't relevant here but make sure you're backing up the etc files too, while that wasn't an issue for me it could easily trip you over in a worst case scenario ( https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets https://docs.gitlab.com/charts/backup-restore/backup.html#ba... ) Not a complaint, it's just one of the hats that needs wearing sometimes, would have been easier on me if someone else was handling it though :P
    - ashildr 5y ago
      Thank you. From what I know direct updates to the next major releases of gitlab are only supported from relatively recent releases. I guess that’s what caught you. We’ve been updating about as soon as an update is released for a while now, with very good results. (I’d rather have to restore a backup or a snapshot than to be running without the latest security fixes. If gitlab is down for an hour (didn’t happen in ~3 years) at least all developers have their local repo.)
qbasic_forever 5y ago
Git is distributed, you should be able to completely detach from your remote and still work effectively. I think a lot of big projects get too lazy about good, reproducible build environments and just rely on centralized CI to do everything.
- gmueckl 5y ago
  If you haven't noticed, the whole point of GitHub is to infest git-using environments with so many centralized features that user's get trapped using that service. Seems like issues and actions are doing that job nicely!
- MattGaiser 5y ago
  GitHub isn't just a repo in many cases. It is project management software.
candiddevmike 5y ago
Gitea is nice, what CI/CD system do you use with it? I used it in the past with Drone, and Drone left a lot to be desired (coming from/going back to GitHub actions, Drone's job syntax is a bit more complicated and focused on containers).
- ncrmro 5y ago
  I've used Drone CI in the past with Gitea. I still prefer gitlabs CI tho
- 5e92cb50239222b 5y ago
  Drone CI has been working fine for our purposes. I agree that it may not be the most feature-packed thing in the world (for example, caching support is missing completely), but there are lots of plugins and it's easy to write your own. We've been using exec and ssh runners when containers are not enough (for example, when you need to juggle a few VMs or build some complicated Docker image, and using docker-in-docker is not fun). https://docs.drone.io/pipeline/exec/overview/ https://docs.drone.io/pipeline/exec/overview/ https://docs.drone.io/pipeline/ssh/overview/ https://docs.drone.io/pipeline/ssh/overview/ GitHub Actions are nice… when they're actually working.

chockchocschoir 5y ago

It's not about being larger, it's about doing changes. GitHub used to be pretty stable before, when they didn't really do any (user facing) changes. It stood still in terms of features, but was also pretty stable overall. Compared to GitHub, how often does your Gitea instance go through code changes? How many major features have been switched on in that instance during the past year without any hiccups?

freedomben 5y ago
I tried to look at the Gitea repo to see how fast they get commits, but github is 500ing :-( https://github.com/go-gitea/gitea https://github.com/go-gitea/gitea
- chockchocschoir 5y ago
  Rest assure, a open source project with ~100 contributors (whereas maybe 10% regularly contributes) has less activity and changes than a multi-international company focusing a SaaS with "1000 to 5000" employees worldwide.
  - judge2020 5y ago
    For perspective, GitHub has 4,000 employees who list them as their employer on LinkedIn and GitLab has 1,700.
Deukhoofd 5y ago
I run Gitea myself as well. I generally throw in a new release once a month or so, never had any real hiccups.
- rolph 5y ago
  it sounds like you dont make code changes unless they are required for function.
  - Deukhoofd 5y ago
    Also for security reasons, and because some features are nice to have. But you're right in that I'm definitely not randomly updating sometimes. When I do update I basically just back up the database, throw in a new Gitea binary, and restart the service. As the releases have been fairly well tested in general, this hasn't given me issues so far.
- chockchocschoir 5y ago
  Again, how much it actually change across those releases? GitHub most likely are going through more changes on a weekly basis (if not daily) than what Gitea does on a monthly basis.
  - Deukhoofd 5y ago
    The major releases generally change quite a lot in one go, for example: https://blog.gitea.io/2022/02/gitea-1.16.0-and-1.16.1-released/ https://blog.gitea.io/2022/02/gitea-1.16.0-and-1.16.1-releas... https://blog.gitea.io/2021/08/gitea-1.15.0-is-released/ https://blog.gitea.io/2021/08/gitea-1.15.0-is-released/ But I'm sure Github changes a lot more. However, the question is if those changes are worth the instability. For my workflow I just want my repos to be accessible, have my source, issues, and wiki accessible, and of course my basic git operations working. If that core functionality is actually not working at times, I'd definitely consider changing the git provider.
dogleash 5y ago
GitHub's feature churn is irrelevant to the comparison unless those features are used in parent poster's workflow.
- chockchocschoir 5y ago
  Really? No, comparing the uptime of two services (actually, one service and one open source self-hosted service) it's relevant to consider _why_ one would have more downtime than another, even if they don't use those features. Even if the parent posters workflow doesn't use the new/deprecated features, they will still be affected by the downtime GitHub have monthly.
  - Nextgrid 5y ago
    I don't think anyone is overlooking the reasons why one might have more downtime than the other. The problem is that ultimately the downtime itself matters and not the reason, and if you don't need any of the features that GitHub offers, then the self-hosted route is a better option.
    - falcolas 5y ago
      > if you don't need any of the features that GitHub offers In particular, all of the new features whose addition causes instability.
      - Nextgrid 5y ago
        I think it's the development process (as in moving things around and potentially introducing bugs) than the new features per-se.
        falcolas 5y ago
        With all due respect to the developers, it would not speak well for GitHub if they can't maintain stability while refactoring. There's a lot of testing processes and CI processes explicitly around making those actions safe. New features breaking is a lot more understandable - even expected - than regressions and refactoring failures.
newswasboring 5y ago
An infrastructure company focusing on feature churn just sounds wrong.
- woah 5y ago
  What? Github is a social networking company.
  - newswasboring 5y ago
    I'm a bit dumb about these things on text, but did you drop an "/s" somewhere? I understand it has sort of a social networking layer on top, but that is not its main function right? This would be like calling a calculator app a social networking app because it allows me to share my results :D.
    - chockchocschoir 5y ago
      GitHub has always branded itself as a "social collaboration platform" rather than anything else (well, first "git hosting" but secondly the social stuff). Here is a early version of their landing page from 2008 (the year they launched): https://web.archive.org/web/20081111061111/http://github.com/ https://web.archive.org/web/20081111061111/http://github.com... Notice the logo says "Social code hosting" and the messaging of the page is mostly around popular repositories, collaborative features and other social elements.

skeeter2020 5y ago

>> A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. None. Same with our CI system. And when they do go down, how long will they be unavailable? When this happens (and it will happen) please post here so we can say "why are you running infrastructure yourself?"

scandox 5y ago
5 years of a Gogs instance. Zero downtime. I'll come back to you when it goes down.
- ramesh31 5y ago
  does gogs plan to support any kind of CI integration like gh actions?
  - scandox 5y ago
    No it doesn't. I fully confess the feature set is extremely minimal. But the point here is that running your own infrastructure can make a lot of sense if you know what your precise requirements are (or are not).
  - sreevisakh 5y ago
    Gogs and gitea are designed to be a single-binary minimal source code hosting service. Additional services like CI within that binary is likely to bloat it up and make it less reliable. However, it's easy to integrate a CI service to it - like Woodpecker CI [1]. Woodpecker also has a minimalist approach like gogs. If you instead prefer a full package, then gitlab is the best solution. [1] https://woodpecker-ci.org/docs/administration/vcs/gogs https://woodpecker-ci.org/docs/administration/vcs/gogs
Nextgrid 5y ago
Very likely to be lower than GitHub/etc, because it's a much simpler system that can be reinstalled from scratch in 30 minutes by an experienced admin on brand new hardware if needed.

contravariant 5y ago

10^999, really? Rule of thumb is that 10^100 of anything doesn't fit in the observable universe.

rvz 5y ago

> A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. > Yes, I know, GitHub is 10⁹⁹⁹ times larger than our puny Gitea instance, and that's why they're having issues Exactly. That is also my overall point. [0] It makes no sense to go 'all in' on GitHub and something goes down and everyone is stuck once again. [0] https://news.ycombinator.com/item?id=30711442 https://news.ycombinator.com/item?id=30711442

freedomben 5y ago

Mind sharing some details on your install? Which database are you using (especially wondering about sqlite with gitea)? Are you using the docker install, or from package, or in k8s? No pressure, but I'm seriously considering standing one up but it would need to be production ready.

raxxorrax 5y ago
Gitea is absolutely production ready, we use an older version by now and don't even think about it much because it just works so well. Exactly what you want in production (production = software should work more often). Actually not "vendoring" dependencies from github is very much not production ready in many cases.
qbasic_forever 5y ago
IMHO storage is the big issue to figure out with a production ready gitea installation. Gitea only supports the local filesystem for storing bare repo data and not anything like S3 or other distributed or highly available storage systems (it supports S3 only for git LFS data right now, it is a long awaited open issue to get the bare repos there too). So the big issue for any non-trivial or highly available setup is going to be how you get high availability for the local storage volume. There are tons of options with various tradeoffs and levels of complexity here--simple local disk RAID, distributed filesystems like gluster or ceph, etc. I think this is the real crux of getting a good gitea instance going. No matter what figure out and test a good backup and recovery solution!
- freedomben 5y ago
  Ah interesting! I read somewhere that it uses the database for that, but I think they were mistaken. Thanks that's quite helpful. If we do it in prod we'll have to accept single-instance.
  - qbasic_forever 5y ago
    Yeah I forgot to mention too a lot of complexity goes away if you host it on a good IaaS platform. On AWS for example you could just mount an EBS volume as the bare repo root and use RDS for a database. Then you just have to deploy and manage an EC2 instance to host gitea's main process. It's more expensive to use those managed platforms but might be well worth the cost vs. doing it all yourself with a bare metal deployment of everything.

corobo 5y ago

Opposite anecdata my GitLab install absolutely self destructed itself at one point and I had to rebuild it myself, waste of a weekend

ashildr 5y ago
> my GitLab install absolutely self destructed itself That’s frightening, we’ve been using it for multiple years now. It’s running fine and the only short downtime is every few weeks for the update. What happened to your instance?
- corobo 5y ago
  More fairly to GitLab it was the underlying server that broke initially. The issue from there was that I couldn't restore from the backup as the newly installed server was also a newer version of GitLab Tried restoring it to the older version to run through the upgrade path and the process didn't want to work. Wish I could be more specific here but I gave up on it and decided it'd just be quicker to whip up a script to recreate all the groups/projects/CICD in a new install and push repos from backups Don't let your GitLab server get outdated and don't consider it a valid backup unless it was taken with the latest version to avoid the same Also it wasn't relevant here but make sure you're backing up the etc files too, while that wasn't an issue for me it could easily trip you over in a worst case scenario ( https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets https://docs.gitlab.com/charts/backup-restore/backup.html#ba... ) Not a complaint, it's just one of the hats that needs wearing sometimes, would have been easier on me if someone else was handling it though :P
  - ashildr 5y ago
    Thank you. From what I know direct updates to the next major releases of gitlab are only supported from relatively recent releases. I guess that’s what caught you. We’ve been updating about as soon as an update is released for a while now, with very good results. (I’d rather have to restore a backup or a snapshot than to be running without the latest security fixes. If gitlab is down for an hour (didn’t happen in ~3 years) at least all developers have their local repo.)

qbasic_forever 5y ago

Git is distributed, you should be able to completely detach from your remote and still work effectively. I think a lot of big projects get too lazy about good, reproducible build environments and just rely on centralized CI to do everything.

gmueckl 5y ago
If you haven't noticed, the whole point of GitHub is to infest git-using environments with so many centralized features that user's get trapped using that service. Seems like issues and actions are doing that job nicely!
MattGaiser 5y ago
GitHub isn't just a repo in many cases. It is project management software.

candiddevmike 5y ago

Gitea is nice, what CI/CD system do you use with it? I used it in the past with Drone, and Drone left a lot to be desired (coming from/going back to GitHub actions, Drone's job syntax is a bit more complicated and focused on containers).

ncrmro 5y ago
I've used Drone CI in the past with Gitea. I still prefer gitlabs CI tho
5e92cb50239222b 5y ago
Drone CI has been working fine for our purposes. I agree that it may not be the most feature-packed thing in the world (for example, caching support is missing completely), but there are lots of plugins and it's easy to write your own. We've been using exec and ssh runners when containers are not enough (for example, when you need to juggle a few VMs or build some complicated Docker image, and using docker-in-docker is not fun). https://docs.drone.io/pipeline/exec/overview/ https://docs.drone.io/pipeline/exec/overview/ https://docs.drone.io/pipeline/ssh/overview/ https://docs.drone.io/pipeline/ssh/overview/ GitHub Actions are nice… when they're actually working.