4 ms·

That isn't everyone's experience. Our onprem Gitlab CI/CD sometimes breaks, usually because the Docker daemon went down for unclear reasons. The fix is easy, ju

That isn't everyone's experience. Our onprem Gitlab CI/CD sometimes breaks, usually because the Docker daemon went down for unclear reasons. The fix is easy, just restart the daemon, but still. Even managing a server may not be a trivial task depending on the available competency in the organization. Upgrades and backups are way more problematic. A while ago I discovered that backups broke and I had to fix it. I setup automatic APT upgrades for Gitlab, but every few major versions or so a manual upgrade process is required. In an organization where people are less vigilant, and/or haven't setup monitoring of backups, this can easily lead to a disaster down the road. None of this is particularly hard — for me. But for some others this may as well be black magic. Furthermore all this work is a nuisance, a distraction from what I usually do.

jacquesm 5y ago
It sounds as though you could probably learn a thing or two from how the GP has set things up if their experience is so much different from yours.
- dx034 5y ago
  I actually never encountered the problem with the crashing daemon, but we also have periodic restarts (they don't take too long and can be scheduled to happen at night). Upgrades aren't automated but as the server is only exposed to the internal network (and also not fully exposed there), the attack surface is much lower. Keeping servers up to date isn't hard, and we don't always insist on running the newest Gitlab version. We make sure to upgrade if there's a very critical bug or if we lag too far behind. It's true, upgrades for Gitlab could be smoother, but we've never had significant down time from it. And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding.
  - hutzlibu 5y ago
    "And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding" Taking a actual break works too, I think. I mean, it is a virtue to see good in everything ..
Nextgrid 5y ago
I bet the GitHub/Lab/Atlassian/etc customers affected by the recent outages would be delighted to be able to SSH in and fix it or at least manually extract any required data until the outage is properly resolved.
qeternity 5y ago
> Furthermore all this work is a nuisance, a distraction from what I usually do. We said the same. Now we lose twice as much time due to Gitlab.com outages
cyberpunk 5y ago
> automatic APT upgrades Please, please don't ever do this. Or if you do, make sure you're taking a snapshot of the box every hour or something..
- dgb23 5y ago
  Or just before the upgrade?
  - mikepurvis 5y ago
    That's how we do it— snapshot the box, apt dist-upgrade, snapshot again, gitlab upgrade, snapshot again. A few days later, drop the snapshots.
- heurisko 5y ago
  What about automatic security APT updates?
- FooBarWidget 5y ago
  If you don't have a dedicated sysadmin who's always on top of new Gitlab versions, then the alternative to automatic upgrades is letting insecure versions run for an arbitrary time until someone takes the time to upgrade. Gitlab is merely 1 of the 200 things we have to keep track of. Besides, we already make weekly automatic backups, stored remotely and encrypted. And Gitlab automatically makes a locally-stored backup before each upgrade, which is good enough unless the disk fails right after we discover that the upgrade is broken.
  - vorpalhex 5y ago
    You can configure just security patches and not full upgrades.
jhugo 5y ago
Your mention of APT suggests to me that you're trying to maintain a stable deployment on a pet server, whereas GP's mention of Terraform suggests they've built some sort of pipeline for repeatable deployment. GP's approach does have a bigger learning curve for those that have never worked that way, but after that the ongoing maintenance is simpler, less time consuming and much lower risk. That probably explains most of the difference in your experiences.
- dx034 5y ago
  We don't have an overly complex setup, just some automations along the way to make things easier the long way. Sure, it's a bit more complicated than manually installing a server, but learning how to use terraform and automating patching/backups etc isn't rocket science. Anyone with some knowledge of Linux/Programming shouldn't struggle setting this up within a few days. And it adds knowledge to the company that's very valuable. With Gitlab, as with most software, restarting it regularly outside of office hours is the best way to ensure a stable service. Also something SaaS can't do, as they never know when their customers need the software.
  - thih9 5y ago
    Many companies with on prem setup also don’t know when their devs need the software. Some have people in multiple timezones, some do async work, etc.
    - vorpalhex 5y ago
      Yeah but how bad is a 5 min restart for that one dude in Australia at odd hours?
      - thih9 5y ago
        How bad is a github downtime? I guess it all depends.
- FooBarWidget 5y ago
  Our deployment is repeatable. Pet server but managed with Ansible. If it's on fire, provision new server, rerun Ansible, restore backup. Server manually provisioned instead of Terraform, but provisioning the server isn't the hard part anyway.
tapoxi 5y ago
This seems like a Docker issue more than a GitLab issue. We run ours in GKE, so the infrastructure is managed and auto scales.

jacquesm 5y ago

It sounds as though you could probably learn a thing or two from how the GP has set things up if their experience is so much different from yours.

dx034 5y ago
I actually never encountered the problem with the crashing daemon, but we also have periodic restarts (they don't take too long and can be scheduled to happen at night). Upgrades aren't automated but as the server is only exposed to the internal network (and also not fully exposed there), the attack surface is much lower. Keeping servers up to date isn't hard, and we don't always insist on running the newest Gitlab version. We make sure to upgrade if there's a very critical bug or if we lag too far behind. It's true, upgrades for Gitlab could be smoother, but we've never had significant down time from it. And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding.
- hutzlibu 5y ago
  "And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding" Taking a actual break works too, I think. I mean, it is a virtue to see good in everything ..

Nextgrid 5y ago

I bet the GitHub/Lab/Atlassian/etc customers affected by the recent outages would be delighted to be able to SSH in and fix it or at least manually extract any required data until the outage is properly resolved.

qeternity 5y ago

> Furthermore all this work is a nuisance, a distraction from what I usually do. We said the same. Now we lose twice as much time due to Gitlab.com outages

cyberpunk 5y ago

> automatic APT upgrades Please, please don't ever do this. Or if you do, make sure you're taking a snapshot of the box every hour or something..

dgb23 5y ago
Or just before the upgrade?
- mikepurvis 5y ago
  That's how we do it— snapshot the box, apt dist-upgrade, snapshot again, gitlab upgrade, snapshot again. A few days later, drop the snapshots.
heurisko 5y ago
What about automatic security APT updates?
FooBarWidget 5y ago
If you don't have a dedicated sysadmin who's always on top of new Gitlab versions, then the alternative to automatic upgrades is letting insecure versions run for an arbitrary time until someone takes the time to upgrade. Gitlab is merely 1 of the 200 things we have to keep track of. Besides, we already make weekly automatic backups, stored remotely and encrypted. And Gitlab automatically makes a locally-stored backup before each upgrade, which is good enough unless the disk fails right after we discover that the upgrade is broken.
- vorpalhex 5y ago
  You can configure just security patches and not full upgrades.

jhugo 5y ago

Your mention of APT suggests to me that you're trying to maintain a stable deployment on a pet server, whereas GP's mention of Terraform suggests they've built some sort of pipeline for repeatable deployment. GP's approach does have a bigger learning curve for those that have never worked that way, but after that the ongoing maintenance is simpler, less time consuming and much lower risk. That probably explains most of the difference in your experiences.

dx034 5y ago
We don't have an overly complex setup, just some automations along the way to make things easier the long way. Sure, it's a bit more complicated than manually installing a server, but learning how to use terraform and automating patching/backups etc isn't rocket science. Anyone with some knowledge of Linux/Programming shouldn't struggle setting this up within a few days. And it adds knowledge to the company that's very valuable. With Gitlab, as with most software, restarting it regularly outside of office hours is the best way to ensure a stable service. Also something SaaS can't do, as they never know when their customers need the software.
- thih9 5y ago
  Many companies with on prem setup also don’t know when their devs need the software. Some have people in multiple timezones, some do async work, etc.
  - vorpalhex 5y ago
    Yeah but how bad is a 5 min restart for that one dude in Australia at odd hours?
    - thih9 5y ago
      How bad is a github downtime? I guess it all depends.
FooBarWidget 5y ago
Our deployment is repeatable. Pet server but managed with Ansible. If it's on fire, provision new server, rerun Ansible, restore backup. Server manually provisioned instead of Terraform, but provisioning the server isn't the hard part anyway.

tapoxi 5y ago

This seems like a Docker issue more than a GitLab issue. We run ours in GKE, so the infrastructure is managed and auto scales.