7 ms·

And here we are with our on prem Gitlab with CI/CD that hasn't been down for 2+ years (except for very short, planned maintenance) and has 100% predictable cost

by dx034 5y ago

And here we are with our on prem Gitlab with CI/CD that hasn't been down for 2+ years (except for very short, planned maintenance) and has 100% predictable cost at much lower rates than cloud services. We can still scale within minutes if needed (thanks to Terraform). Dedicated servers are just so cheap, esp with sth like CI/CD that's perfect to host at cheaper providers. Seeing constant issues with Gitlab and Github products just reinforces that decision.

qeternity 5y ago
We switched from self-hosted Gitlab to Gitlab.com in the name of reducing infra burden...my oh my do I regret that choice. Same experience to you - no downtime that wasn't planned when we self-hosted. Load was very predictable, and ran on k8s anyway. Much, much faster than the hosted version.
- dx034 5y ago
  And I don't get what infra burden there is. We run Gitlab, Mattermost, Databases etc and running/maintaining those is still just a very minor part in the work of some developers (who like and are good at infra). Put everything behind Tailscale/Cloudflare Tunnel and your attack surface is very low to start with. Most deployment and patching can be automated anyway, so there's not much left. In the end we save a lot of cost, avoid lock-in and keep much more knowledge in-house.
  - qeternity 5y ago
    It’s just one more thing to backup, test, monitor, etc. Upgrade migrations didn’t always succeed so then you need to go back and remember what you did last time to fix it. It’s a cognitive burden and distraction. In theory it’s cheaper to pay Gitlab to handle for us. Except that now we lose more time due to Gitlab.com outages than we spent maintaining on prem.
    - Cthulhu_ 5y ago
      That's exactly it, it adds a list of nodes to the "required knowledge" graph of your company, adding job positions, expenses, people, and (operational?) risk to your company that I'm sure already has its own list of things to worry about. There's so many companies out there whose infrastructure is run by this One Guy. Any company needs to assess whether they can risk losing their One Guy (see also: Bus Factor), or if they'll just have to swallow the cost and (some) risk of (some) downtime by using a service provider.
      - dx034 5y ago
        Any decent developer should be able to install and maintain on-prem Gitlab. It's not harder than running most managed services on AWS (I'd say it's simpler). So you already have the people, you just need a few minutes (literaly minutes, not hours) of their time per week/month. There's operational risk, but that doesn't go away if you shift to Gitlab.com infra. Just in this case, you can control all of it.
        Aicy 5y ago
        In theory, yes, but in practice most of the difficulty in maintaining an on-prem Gitlab will be that individual company's bespoke installation of it.
        qeternity 5y ago
        It's really not this simple. Even if it were just minutes (it isn't, but agree it's not huge overhead) you still have the operational risk when things inevitably go wrong. Now you're impacting your orgs ability to be productive (super expensive) and you incur the direct cost of fixing. Who do you think is going to be better at triaging Gitlab: someone looking after on-prem once a month, or Gitlab's team that do this full time every day. That's why people pay for SaaS. The problem with Github and Gitlab is that it's actually turning out that those companies aren't that good at running their own product, which is tilting the economics of the above.
        5y ago
- Nextgrid 5y ago
  An issue everyone seems to overlook is that an SaaS typically needs to be scalable to the entire world while your on-prem service pretty much never does. The SaaS will have more moving parts & operator activity which in turn makes outages more likely, and that "operator activity" can't be planned for unlike on-prem where you can schedule maintenance out-of-hours to make sure any accidents don't affect the business.
rvz 5y ago
This. That's how you do it (and also how many open-source organizations have done it) as I have said and predicted years ago here going against 'centralizing everything to GitHub' [0] Fast forward precisely 2 years ago since that prediction was made in (April 14th, 2020), GitHub has become (unsurprisingly) unreliable, especially for those that went 'all in'. Self-hosting for your own organization seems to have been more reliable in the long run. > Seeing constant issues with Gitlab and Github products just reinforces that decision. This also. Especially the SaSS ones. [0] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- oars 5y ago
  At my start-up we are all in on SaaS because we don't have enough engineers to self-host our things. The previous F500 company I worked at had enough engineers to self-host their things but thought SaaS was also the future and migrated a lot of their key tools and services onto these cloud platforms. I wonder if both my start-up and the F500 company will suffer in the long run because of these decisions...
  - dx034 5y ago
    If you have enough engineers to develop a product, you have enough to host your own Gitlab. Most engineers are expected to understand some ops anyway for CI/CD, setting up Gitlab really isn't harder than that (from experience). Especially with low usage, these instances basically never go down. Most of the overhead is managing users, permissions etc, that won't go away with SaaS.
  - mindwok 5y ago
    Gitlab in particular is unbelievably easy to self host. At my last gig I ran it on a t4g Debian instance with unattended upgrades and then just rebooted once a week at like 3am, literally never touched it again and it costs nothing to run.
    - Cthulhu_ 5y ago
      But how many people would know what to do if something did happen?
FooBarWidget 5y ago
That isn't everyone's experience. Our onprem Gitlab CI/CD sometimes breaks, usually because the Docker daemon went down for unclear reasons. The fix is easy, just restart the daemon, but still. Even managing a server may not be a trivial task depending on the available competency in the organization. Upgrades and backups are way more problematic. A while ago I discovered that backups broke and I had to fix it. I setup automatic APT upgrades for Gitlab, but every few major versions or so a manual upgrade process is required. In an organization where people are less vigilant, and/or haven't setup monitoring of backups, this can easily lead to a disaster down the road. None of this is particularly hard — for me. But for some others this may as well be black magic. Furthermore all this work is a nuisance, a distraction from what I usually do.
- jacquesm 5y ago
  It sounds as though you could probably learn a thing or two from how the GP has set things up if their experience is so much different from yours.
  - dx034 5y ago
    I actually never encountered the problem with the crashing daemon, but we also have periodic restarts (they don't take too long and can be scheduled to happen at night). Upgrades aren't automated but as the server is only exposed to the internal network (and also not fully exposed there), the attack surface is much lower. Keeping servers up to date isn't hard, and we don't always insist on running the newest Gitlab version. We make sure to upgrade if there's a very critical bug or if we lag too far behind. It's true, upgrades for Gitlab could be smoother, but we've never had significant down time from it. And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding.
    - hutzlibu 5y ago
      "And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding" Taking a actual break works too, I think. I mean, it is a virtue to see good in everything ..
Cthulhu_ 5y ago
Congratulations, you got lucky; your comment is an example of survivorship bias. You happen to have the technical know-how and people to run your own infrastructure, but this isn't the default.
- jacquesm 5y ago
  If you have the technical knowhow to develop complex applications and do application and systems level administration for your own application then you can run a gitlab on prem installation. You already have most or all of the required knowledge.
  - deleted 5y ago
    [deleted]
- dx034 5y ago
  I'm not sure if it's a good thing if no software developer in a company has the skill to install and maintain an application on a linux server. To me that's no additional knowledge for a developer, but one of the basics they should know anyway.
  - jhugo 5y ago
    It probably depends what sort of software your company builds. A frontend web design shop might well have developers who have little or no relevant experience for self-hosting something like Gitlab.
  - Aicy 5y ago
    Except it's more complex than that. Ideally you want to have multiple runners, and for them to scale so that developers aren't endlessly waiting in job queues. On top of that you might want proper network segregation, and for it to be on a separate virtual network to your workloads to reduce your blast radius. You might have problems with connecting your server to third party SaaS outside your network, etc etc etc
    - 5y ago
jhugo 5y ago
Yup, after the first series of outages (which were much worse for us in Asia than for companies in the US, because it seemed that nobody at GitHub noticed everything was down until the US woke up) we moved all our code hosting from GitHub to self-hosted Gitea (our CI/CD was already self-hosted with Tekton), and set up mirror repos from GitHub for all our dependencies too. Before we were losing productivity for an entire day due to GitHub outages, now we only notice because there's a post on HN :) As for the cost of self-hosting, we already host our own services, Gitea is no more complex than any of those. I wouldn't recommend self-hosting to a company with no backend/ops capability in-house, but for any other company it's fairly obviously a better choice IMO.
danielvaughn 5y ago
I thought gitlab onprem was an enterprise feature? You’re saying it’s cheaper than cloud?
- Karunamon 5y ago
  Gitlab has enterprise and community versions that are self hostable. You may be thinking of GitHub.
dncornholio 5y ago
Gitlab is great, the downside is managing updates and frequent vulnerabilities found

by dx034 5y ago

qeternity 5y ago

We switched from self-hosted Gitlab to Gitlab.com in the name of reducing infra burden...my oh my do I regret that choice. Same experience to you - no downtime that wasn't planned when we self-hosted. Load was very predictable, and ran on k8s anyway. Much, much faster than the hosted version.

dx034 5y ago
And I don't get what infra burden there is. We run Gitlab, Mattermost, Databases etc and running/maintaining those is still just a very minor part in the work of some developers (who like and are good at infra). Put everything behind Tailscale/Cloudflare Tunnel and your attack surface is very low to start with. Most deployment and patching can be automated anyway, so there's not much left. In the end we save a lot of cost, avoid lock-in and keep much more knowledge in-house.
- qeternity 5y ago
  It’s just one more thing to backup, test, monitor, etc. Upgrade migrations didn’t always succeed so then you need to go back and remember what you did last time to fix it. It’s a cognitive burden and distraction. In theory it’s cheaper to pay Gitlab to handle for us. Except that now we lose more time due to Gitlab.com outages than we spent maintaining on prem.
  - Cthulhu_ 5y ago
    That's exactly it, it adds a list of nodes to the "required knowledge" graph of your company, adding job positions, expenses, people, and (operational?) risk to your company that I'm sure already has its own list of things to worry about. There's so many companies out there whose infrastructure is run by this One Guy. Any company needs to assess whether they can risk losing their One Guy (see also: Bus Factor), or if they'll just have to swallow the cost and (some) risk of (some) downtime by using a service provider.
    - dx034 5y ago
      Any decent developer should be able to install and maintain on-prem Gitlab. It's not harder than running most managed services on AWS (I'd say it's simpler). So you already have the people, you just need a few minutes (literaly minutes, not hours) of their time per week/month. There's operational risk, but that doesn't go away if you shift to Gitlab.com infra. Just in this case, you can control all of it.
      - Aicy 5y ago
        In theory, yes, but in practice most of the difficulty in maintaining an on-prem Gitlab will be that individual company's bespoke installation of it.
      - qeternity 5y ago
        It's really not this simple. Even if it were just minutes (it isn't, but agree it's not huge overhead) you still have the operational risk when things inevitably go wrong. Now you're impacting your orgs ability to be productive (super expensive) and you incur the direct cost of fixing. Who do you think is going to be better at triaging Gitlab: someone looking after on-prem once a month, or Gitlab's team that do this full time every day. That's why people pay for SaaS. The problem with Github and Gitlab is that it's actually turning out that those companies aren't that good at running their own product, which is tilting the economics of the above.
        5y ago
Nextgrid 5y ago
An issue everyone seems to overlook is that an SaaS typically needs to be scalable to the entire world while your on-prem service pretty much never does. The SaaS will have more moving parts & operator activity which in turn makes outages more likely, and that "operator activity" can't be planned for unlike on-prem where you can schedule maintenance out-of-hours to make sure any accidents don't affect the business.

rvz 5y ago

This. That's how you do it (and also how many open-source organizations have done it) as I have said and predicted years ago here going against 'centralizing everything to GitHub' [0] Fast forward precisely 2 years ago since that prediction was made in (April 14th, 2020), GitHub has become (unsurprisingly) unreliable, especially for those that went 'all in'. Self-hosting for your own organization seems to have been more reliable in the long run. > Seeing constant issues with Gitlab and Github products just reinforces that decision. This also. Especially the SaSS ones. [0] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803

oars 5y ago
At my start-up we are all in on SaaS because we don't have enough engineers to self-host our things. The previous F500 company I worked at had enough engineers to self-host their things but thought SaaS was also the future and migrated a lot of their key tools and services onto these cloud platforms. I wonder if both my start-up and the F500 company will suffer in the long run because of these decisions...
- dx034 5y ago
  If you have enough engineers to develop a product, you have enough to host your own Gitlab. Most engineers are expected to understand some ops anyway for CI/CD, setting up Gitlab really isn't harder than that (from experience). Especially with low usage, these instances basically never go down. Most of the overhead is managing users, permissions etc, that won't go away with SaaS.
- mindwok 5y ago
  Gitlab in particular is unbelievably easy to self host. At my last gig I ran it on a t4g Debian instance with unattended upgrades and then just rebooted once a week at like 3am, literally never touched it again and it costs nothing to run.
  - Cthulhu_ 5y ago
    But how many people would know what to do if something did happen?

FooBarWidget 5y ago

That isn't everyone's experience. Our onprem Gitlab CI/CD sometimes breaks, usually because the Docker daemon went down for unclear reasons. The fix is easy, just restart the daemon, but still. Even managing a server may not be a trivial task depending on the available competency in the organization. Upgrades and backups are way more problematic. A while ago I discovered that backups broke and I had to fix it. I setup automatic APT upgrades for Gitlab, but every few major versions or so a manual upgrade process is required. In an organization where people are less vigilant, and/or haven't setup monitoring of backups, this can easily lead to a disaster down the road. None of this is particularly hard — for me. But for some others this may as well be black magic. Furthermore all this work is a nuisance, a distraction from what I usually do.

jacquesm 5y ago
It sounds as though you could probably learn a thing or two from how the GP has set things up if their experience is so much different from yours.
- dx034 5y ago
  I actually never encountered the problem with the crashing daemon, but we also have periodic restarts (they don't take too long and can be scheduled to happen at night). Upgrades aren't automated but as the server is only exposed to the internal network (and also not fully exposed there), the attack surface is much lower. Keeping servers up to date isn't hard, and we don't always insist on running the newest Gitlab version. We make sure to upgrade if there's a very critical bug or if we lag too far behind. It's true, upgrades for Gitlab could be smoother, but we've never had significant down time from it. And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding.
  - hutzlibu 5y ago
    "And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding" Taking a actual break works too, I think. I mean, it is a virtue to see good in everything ..

Cthulhu_ 5y ago

Congratulations, you got lucky; your comment is an example of survivorship bias. You happen to have the technical know-how and people to run your own infrastructure, but this isn't the default.

jacquesm 5y ago
If you have the technical knowhow to develop complex applications and do application and systems level administration for your own application then you can run a gitlab on prem installation. You already have most or all of the required knowledge.
- deleted 5y ago
  [deleted]
dx034 5y ago
I'm not sure if it's a good thing if no software developer in a company has the skill to install and maintain an application on a linux server. To me that's no additional knowledge for a developer, but one of the basics they should know anyway.
- jhugo 5y ago
  It probably depends what sort of software your company builds. A frontend web design shop might well have developers who have little or no relevant experience for self-hosting something like Gitlab.
- Aicy 5y ago
  Except it's more complex than that. Ideally you want to have multiple runners, and for them to scale so that developers aren't endlessly waiting in job queues. On top of that you might want proper network segregation, and for it to be on a separate virtual network to your workloads to reduce your blast radius. You might have problems with connecting your server to third party SaaS outside your network, etc etc etc
  - 5y ago

jhugo 5y ago

Yup, after the first series of outages (which were much worse for us in Asia than for companies in the US, because it seemed that nobody at GitHub noticed everything was down until the US woke up) we moved all our code hosting from GitHub to self-hosted Gitea (our CI/CD was already self-hosted with Tekton), and set up mirror repos from GitHub for all our dependencies too. Before we were losing productivity for an entire day due to GitHub outages, now we only notice because there's a post on HN :) As for the cost of self-hosting, we already host our own services, Gitea is no more complex than any of those. I wouldn't recommend self-hosting to a company with no backend/ops capability in-house, but for any other company it's fairly obviously a better choice IMO.

danielvaughn 5y ago

I thought gitlab onprem was an enterprise feature? You’re saying it’s cheaper than cloud?

Karunamon 5y ago
Gitlab has enterprise and community versions that are self hostable. You may be thinking of GitHub.

dncornholio 5y ago

Gitlab is great, the downside is managing updates and frequent vulnerabilities found