4 ms·

>> A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. None. Same with our CI system. And when they do go d

by skeeter2020 5y ago

>> A self-hosted Gitea instance my organization has been using had exactly zero downtime in the past year. None. Same with our CI system. And when they do go down, how long will they be unavailable? When this happens (and it will happen) please post here so we can say "why are you running infrastructure yourself?"

scandox 5y ago
5 years of a Gogs instance. Zero downtime. I'll come back to you when it goes down.
- ramesh31 5y ago
  does gogs plan to support any kind of CI integration like gh actions?
  - scandox 5y ago
    No it doesn't. I fully confess the feature set is extremely minimal. But the point here is that running your own infrastructure can make a lot of sense if you know what your precise requirements are (or are not).
  - sreevisakh 5y ago
    Gogs and gitea are designed to be a single-binary minimal source code hosting service. Additional services like CI within that binary is likely to bloat it up and make it less reliable. However, it's easy to integrate a CI service to it - like Woodpecker CI [1]. Woodpecker also has a minimalist approach like gogs. If you instead prefer a full package, then gitlab is the best solution. [1] https://woodpecker-ci.org/docs/administration/vcs/gogs https://woodpecker-ci.org/docs/administration/vcs/gogs
Nextgrid 5y ago
Very likely to be lower than GitHub/etc, because it's a much simpler system that can be reinstalled from scratch in 30 minutes by an experienced admin on brand new hardware if needed.
5e92cb50239222b 5y ago
Less than you've already wasted waiting for GitHub to come up from the dead in the last year alone. If you're insinuating that we'll have to restore everything from scratch, don't worry, the machine is being continuously replicated and sysadmins are trained to restore everything chop-chop.
jchw 5y ago
I ran a GitLab instance for a company pretty much solo and it never wound up going down outside of when I was upgrading it, that I can recall. The thing about that is, it’s a small scale instance. If an upgrade or change failed, I could literally delete everything and restore from backup in under an hour, and never really needed to do that (although to be clear, I did actually test that it was possible.) Now I’ll tell you the real problem with it: it worked great for people who were located in the U.S., but it was terrible for certain remote employees due to increased latency, and the fix for that would be to set up a complicated replication setup that puts data closer to people not near the main instance. There’s obviously other issues, but that is one that is probably commonly overlooked that most SaaS solutions can do better on.
- Jarwain 5y ago
  What gitlab features do you utilize where latency actually is a problem? I currently manage an internal gitlab instance with some remote coworkers, but I haven't heard any complaints about latency. Although tbf, I feel like we've barely scratched the surface of what gitlab supports
  - jchw 5y ago
    The most impacted user had ~500ms latency to the server, which is enough that even pageloads are bound to feel slow (especially because Gitlab itself is not necessarily the fastest thing in the world sometimes.) And if I am remembering correctly, I think that was basically it: the UI pageloads just had enough chained round trips that they were way too slow. In retrospect, there may have been some ways to work around this that we didn’t end up exploring. When I moved to a new company, the GitLab instance was still up, and last I was aware, many years down the road, it is still in use. Feature-wise, we were very much satisfied, and the GitLab CI model with Docker executors worked very well for us.
    - sytse 5y ago
      If you have high latency to a GitLab server consider setting up a GitLab Geo server locally https://docs.gitlab.com/ee/administration/geo/ https://docs.gitlab.com/ee/administration/geo/ to improve things (this is a paid feature).
ghusto 5y ago
With something as simple as serving git for your own organisation, when you do get downtime it's going to be a case of restoring from backup and changing DNS. No need for complicated infrastructure, so bringing it back really is that simple. Worst case is losing commits between going down and backups, but even then it'll be minimal if you set your backup to simply push to a another origin. I pay for someone else to worry about most services, but git doesn't have to be one of them.
- munificent 5y ago
  The worst case is discovering the hard way that your backup system isn't working and you haven't been diligent about spending the time to periodically test restores.
  - ghusto 5y ago
    Pushing to multiple origins is not a hard thing.
    - munificent 5y ago
      The risk is less about losing commits, since those tend to live on every devs machines too. It's about losing all of the other data like the issue tracker, comments, etc.
  - dspillett 5y ago
    That's why testing of backups should be at least partially automated. Though you will always find a point where the automation, and automated tests to make sure that is working, and so on, get to the point where making a human responsible for sanity checking is most resource effective.
- thefreeman 5y ago
  If you think "serving git" is the only role github is serving for organizations you are probably not the target audience.
chousuke 5y ago
Depends on your setup. For a simple git server on AWS I might have it on a single-node ASG so that when something terminates the instance, the ASG notices and launches a new one. The initialization scripts would then attach and mount a pre-existing EBS volume and start services. Very simple to set up, and as long as your data volume is not lost (which can happen, but is very rare), such a system would recover from a host failure by itself faster than most people could react to it going down. In the unlikely case that you lose the EBS volume, you'd just recover from snapshots manually.
thrashh 5y ago
I once worked at a place that self hosted a lot and they had fail-over, backups and restores dialed in If something like Git went down — which it never — it was a few clicks away from coming back Working with competent people is a blast
- x86_64Ubuntu 5y ago
  You also need an organization that will spend the money to make that happen. Many of us would like to have such systems where we work, and are capable enough to implement them, but the Finance people say 'NO'.
johnisgood 5y ago
Infra I have been running myself only goes down when I reboot INTENTIONALLY.
aprdm 5y ago
I've been running gitlab for ~5y in companies with ~200 devs and haven't had a single downtime, it also takes very little work/effort.
dijit 5y ago
I'm honestly sick of this conversation. Have we as an industry forgot how to make systems? that's pretty telling if so. Why are people so utterly afraid of running their own systems. If a SaaS platform goes down it's followed by "#hugops; running large systems is hard!", when someone says "Well, if it's hard, why not break it into smaller pieces and run it myself" it's met directly with "Do you think you can do better!?!". Honestly, probably, maybe not. Why does it matter? Those two things cannot be true simultaneously. You cannot say "running big systems is hard" and "they can run a big system better than you can run a small system". I'm not going to fall into the trap of arguing points like the fact that if you own the data you can have your own D/R & backup strategies and the fact you can run maintenance's on your own time (hell: you can run maintenance's to migrate things around, which is more than github can do, which is not an indictment of them, just the nature of being a hosted service). Honestly, I miss sysadmins. People who were not afraid of servers, even if they said "no" sometimes. Because, you know, being afraid of servers and hosting things is just pathetic.
- PradeetPatel 5y ago
  From an organisational perspective, outsourcing infrastructure is the safest and cost effective way to meet Regulatory & Compliance requirements. When you run your own servers, not only do you need to maintain them, you are also responsible for their certification and getting them audited. As a manager in a medium sized company, I think it is essential to identify what "needs building" and what could be bought to ensure the needs of stakeholders are met in a timely manner.
  - dijit 5y ago
    Heh, time was "compliance" was the reason to host your own services. It is a lot easier to be compliant with many things if you're hosting your own services. Back when I was working in eCommerce (PCI-DSS Tier 1, Cardholder data on premises) it was basically impossible to use hosted services. As then, it is now: "Compliance" is just enough nebulous of a word to prevent critical thinking.
    - 5y ago
manicdee 5y ago
It only takes one month long outage of an international undersea cable to negate all the benefit of SaaS when that SaaS is hosted in or heavily depends on other services hosted in the USA or Europe. DNS goes down for a particular dot com? BGP hijinks? Backhoe through the major fibre serving your workplace? All of these are going to happen and will lead to downtime out of your control or the control of the SaaS provider. Thankfully git is designed to be decentralised and developers can continue to work even if they have to use paper books as reference material and not Stack Overflow.

scandox 5y ago

5 years of a Gogs instance. Zero downtime. I'll come back to you when it goes down.

ramesh31 5y ago
does gogs plan to support any kind of CI integration like gh actions?
- scandox 5y ago
  No it doesn't. I fully confess the feature set is extremely minimal. But the point here is that running your own infrastructure can make a lot of sense if you know what your precise requirements are (or are not).
- sreevisakh 5y ago
  Gogs and gitea are designed to be a single-binary minimal source code hosting service. Additional services like CI within that binary is likely to bloat it up and make it less reliable. However, it's easy to integrate a CI service to it - like Woodpecker CI [1]. Woodpecker also has a minimalist approach like gogs. If you instead prefer a full package, then gitlab is the best solution. [1] https://woodpecker-ci.org/docs/administration/vcs/gogs https://woodpecker-ci.org/docs/administration/vcs/gogs

Nextgrid 5y ago

Very likely to be lower than GitHub/etc, because it's a much simpler system that can be reinstalled from scratch in 30 minutes by an experienced admin on brand new hardware if needed.

5e92cb50239222b 5y ago

Less than you've already wasted waiting for GitHub to come up from the dead in the last year alone. If you're insinuating that we'll have to restore everything from scratch, don't worry, the machine is being continuously replicated and sysadmins are trained to restore everything chop-chop.

jchw 5y ago

I ran a GitLab instance for a company pretty much solo and it never wound up going down outside of when I was upgrading it, that I can recall. The thing about that is, it’s a small scale instance. If an upgrade or change failed, I could literally delete everything and restore from backup in under an hour, and never really needed to do that (although to be clear, I did actually test that it was possible.) Now I’ll tell you the real problem with it: it worked great for people who were located in the U.S., but it was terrible for certain remote employees due to increased latency, and the fix for that would be to set up a complicated replication setup that puts data closer to people not near the main instance. There’s obviously other issues, but that is one that is probably commonly overlooked that most SaaS solutions can do better on.

Jarwain 5y ago
What gitlab features do you utilize where latency actually is a problem? I currently manage an internal gitlab instance with some remote coworkers, but I haven't heard any complaints about latency. Although tbf, I feel like we've barely scratched the surface of what gitlab supports
- jchw 5y ago
  The most impacted user had ~500ms latency to the server, which is enough that even pageloads are bound to feel slow (especially because Gitlab itself is not necessarily the fastest thing in the world sometimes.) And if I am remembering correctly, I think that was basically it: the UI pageloads just had enough chained round trips that they were way too slow. In retrospect, there may have been some ways to work around this that we didn’t end up exploring. When I moved to a new company, the GitLab instance was still up, and last I was aware, many years down the road, it is still in use. Feature-wise, we were very much satisfied, and the GitLab CI model with Docker executors worked very well for us.
  - sytse 5y ago
    If you have high latency to a GitLab server consider setting up a GitLab Geo server locally https://docs.gitlab.com/ee/administration/geo/ https://docs.gitlab.com/ee/administration/geo/ to improve things (this is a paid feature).

ghusto 5y ago

With something as simple as serving git for your own organisation, when you do get downtime it's going to be a case of restoring from backup and changing DNS. No need for complicated infrastructure, so bringing it back really is that simple. Worst case is losing commits between going down and backups, but even then it'll be minimal if you set your backup to simply push to a another origin. I pay for someone else to worry about most services, but git doesn't have to be one of them.

munificent 5y ago
The worst case is discovering the hard way that your backup system isn't working and you haven't been diligent about spending the time to periodically test restores.
- ghusto 5y ago
  Pushing to multiple origins is not a hard thing.
  - munificent 5y ago
    The risk is less about losing commits, since those tend to live on every devs machines too. It's about losing all of the other data like the issue tracker, comments, etc.
- dspillett 5y ago
  That's why testing of backups should be at least partially automated. Though you will always find a point where the automation, and automated tests to make sure that is working, and so on, get to the point where making a human responsible for sanity checking is most resource effective.
thefreeman 5y ago
If you think "serving git" is the only role github is serving for organizations you are probably not the target audience.

chousuke 5y ago

Depends on your setup. For a simple git server on AWS I might have it on a single-node ASG so that when something terminates the instance, the ASG notices and launches a new one. The initialization scripts would then attach and mount a pre-existing EBS volume and start services. Very simple to set up, and as long as your data volume is not lost (which can happen, but is very rare), such a system would recover from a host failure by itself faster than most people could react to it going down. In the unlikely case that you lose the EBS volume, you'd just recover from snapshots manually.

thrashh 5y ago

I once worked at a place that self hosted a lot and they had fail-over, backups and restores dialed in If something like Git went down — which it never — it was a few clicks away from coming back Working with competent people is a blast

x86_64Ubuntu 5y ago
You also need an organization that will spend the money to make that happen. Many of us would like to have such systems where we work, and are capable enough to implement them, but the Finance people say 'NO'.

johnisgood 5y ago

Infra I have been running myself only goes down when I reboot INTENTIONALLY.

aprdm 5y ago

I've been running gitlab for ~5y in companies with ~200 devs and haven't had a single downtime, it also takes very little work/effort.

dijit 5y ago

I'm honestly sick of this conversation. Have we as an industry forgot how to make systems? that's pretty telling if so. Why are people so utterly afraid of running their own systems. If a SaaS platform goes down it's followed by "#hugops; running large systems is hard!", when someone says "Well, if it's hard, why not break it into smaller pieces and run it myself" it's met directly with "Do you think you can do better!?!". Honestly, probably, maybe not. Why does it matter? Those two things cannot be true simultaneously. You cannot say "running big systems is hard" and "they can run a big system better than you can run a small system". I'm not going to fall into the trap of arguing points like the fact that if you own the data you can have your own D/R & backup strategies and the fact you can run maintenance's on your own time (hell: you can run maintenance's to migrate things around, which is more than github can do, which is not an indictment of them, just the nature of being a hosted service). Honestly, I miss sysadmins. People who were not afraid of servers, even if they said "no" sometimes. Because, you know, being afraid of servers and hosting things is just pathetic.

PradeetPatel 5y ago
From an organisational perspective, outsourcing infrastructure is the safest and cost effective way to meet Regulatory & Compliance requirements. When you run your own servers, not only do you need to maintain them, you are also responsible for their certification and getting them audited. As a manager in a medium sized company, I think it is essential to identify what "needs building" and what could be bought to ensure the needs of stakeholders are met in a timely manner.
- dijit 5y ago
  Heh, time was "compliance" was the reason to host your own services. It is a lot easier to be compliant with many things if you're hosting your own services. Back when I was working in eCommerce (PCI-DSS Tier 1, Cardholder data on premises) it was basically impossible to use hosted services. As then, it is now: "Compliance" is just enough nebulous of a word to prevent critical thinking.
  - 5y ago

manicdee 5y ago

It only takes one month long outage of an international undersea cable to negate all the benefit of SaaS when that SaaS is hosted in or heavily depends on other services hosted in the USA or Europe. DNS goes down for a particular dot com? BGP hijinks? Backhoe through the major fibre serving your workplace? All of these are going to happen and will lead to downtime out of your control or the control of the SaaS provider. Thankfully git is designed to be decentralised and developers can continue to work even if they have to use paper books as reference material and not Stack Overflow.