3 ms·

We switched from self-hosted Gitlab to Gitlab.com in the name of reducing infra burden...my oh my do I regret that choice. Same experience to you - no downtime

We switched from self-hosted Gitlab to Gitlab.com in the name of reducing infra burden...my oh my do I regret that choice. Same experience to you - no downtime that wasn't planned when we self-hosted. Load was very predictable, and ran on k8s anyway. Much, much faster than the hosted version.

dx034 5y ago
And I don't get what infra burden there is. We run Gitlab, Mattermost, Databases etc and running/maintaining those is still just a very minor part in the work of some developers (who like and are good at infra). Put everything behind Tailscale/Cloudflare Tunnel and your attack surface is very low to start with. Most deployment and patching can be automated anyway, so there's not much left. In the end we save a lot of cost, avoid lock-in and keep much more knowledge in-house.
- qeternity 5y ago
  It’s just one more thing to backup, test, monitor, etc. Upgrade migrations didn’t always succeed so then you need to go back and remember what you did last time to fix it. It’s a cognitive burden and distraction. In theory it’s cheaper to pay Gitlab to handle for us. Except that now we lose more time due to Gitlab.com outages than we spent maintaining on prem.
  - Cthulhu_ 5y ago
    That's exactly it, it adds a list of nodes to the "required knowledge" graph of your company, adding job positions, expenses, people, and (operational?) risk to your company that I'm sure already has its own list of things to worry about. There's so many companies out there whose infrastructure is run by this One Guy. Any company needs to assess whether they can risk losing their One Guy (see also: Bus Factor), or if they'll just have to swallow the cost and (some) risk of (some) downtime by using a service provider.
    - dx034 5y ago
      Any decent developer should be able to install and maintain on-prem Gitlab. It's not harder than running most managed services on AWS (I'd say it's simpler). So you already have the people, you just need a few minutes (literaly minutes, not hours) of their time per week/month. There's operational risk, but that doesn't go away if you shift to Gitlab.com infra. Just in this case, you can control all of it.
      - Aicy 5y ago
        In theory, yes, but in practice most of the difficulty in maintaining an on-prem Gitlab will be that individual company's bespoke installation of it.
      - qeternity 5y ago
        It's really not this simple. Even if it were just minutes (it isn't, but agree it's not huge overhead) you still have the operational risk when things inevitably go wrong. Now you're impacting your orgs ability to be productive (super expensive) and you incur the direct cost of fixing. Who do you think is going to be better at triaging Gitlab: someone looking after on-prem once a month, or Gitlab's team that do this full time every day. That's why people pay for SaaS. The problem with Github and Gitlab is that it's actually turning out that those companies aren't that good at running their own product, which is tilting the economics of the above.
        Nextgrid 5y ago
        > someone looking after on-prem once a month, or Gitlab's team that do this full time every day Keep in mind that Gitlab.com sees much more load & activity than your on-prem equivalent, and their maintenance schedules don't line up with yours. Ideally, you don't want someone messing with your VCS/CI "every day". Your on-prem is not only very unlikely to need any of the moving parts that a world-scalable system needs, but any maintenance can also be planned out-of-hours to make sure eventual operator errors don't cause downtime during business hours.
      - Aeolun 5y ago
  - dx034 5y ago
    Backups tend to take the most work, especially if you verify them periodically. But at least that means you have backups. I'm not sure I would want Gitlab to have all my data without a local copy for such important information.
Nextgrid 5y ago
An issue everyone seems to overlook is that an SaaS typically needs to be scalable to the entire world while your on-prem service pretty much never does. The SaaS will have more moving parts & operator activity which in turn makes outages more likely, and that "operator activity" can't be planned for unlike on-prem where you can schedule maintenance out-of-hours to make sure any accidents don't affect the business.

dx034 5y ago

And I don't get what infra burden there is. We run Gitlab, Mattermost, Databases etc and running/maintaining those is still just a very minor part in the work of some developers (who like and are good at infra). Put everything behind Tailscale/Cloudflare Tunnel and your attack surface is very low to start with. Most deployment and patching can be automated anyway, so there's not much left. In the end we save a lot of cost, avoid lock-in and keep much more knowledge in-house.

qeternity 5y ago
It’s just one more thing to backup, test, monitor, etc. Upgrade migrations didn’t always succeed so then you need to go back and remember what you did last time to fix it. It’s a cognitive burden and distraction. In theory it’s cheaper to pay Gitlab to handle for us. Except that now we lose more time due to Gitlab.com outages than we spent maintaining on prem.
- Cthulhu_ 5y ago
  That's exactly it, it adds a list of nodes to the "required knowledge" graph of your company, adding job positions, expenses, people, and (operational?) risk to your company that I'm sure already has its own list of things to worry about. There's so many companies out there whose infrastructure is run by this One Guy. Any company needs to assess whether they can risk losing their One Guy (see also: Bus Factor), or if they'll just have to swallow the cost and (some) risk of (some) downtime by using a service provider.
  - dx034 5y ago
    Any decent developer should be able to install and maintain on-prem Gitlab. It's not harder than running most managed services on AWS (I'd say it's simpler). So you already have the people, you just need a few minutes (literaly minutes, not hours) of their time per week/month. There's operational risk, but that doesn't go away if you shift to Gitlab.com infra. Just in this case, you can control all of it.
    - Aicy 5y ago
      In theory, yes, but in practice most of the difficulty in maintaining an on-prem Gitlab will be that individual company's bespoke installation of it.
    - qeternity 5y ago
      It's really not this simple. Even if it were just minutes (it isn't, but agree it's not huge overhead) you still have the operational risk when things inevitably go wrong. Now you're impacting your orgs ability to be productive (super expensive) and you incur the direct cost of fixing. Who do you think is going to be better at triaging Gitlab: someone looking after on-prem once a month, or Gitlab's team that do this full time every day. That's why people pay for SaaS. The problem with Github and Gitlab is that it's actually turning out that those companies aren't that good at running their own product, which is tilting the economics of the above.
      - Nextgrid 5y ago
        > someone looking after on-prem once a month, or Gitlab's team that do this full time every day Keep in mind that Gitlab.com sees much more load & activity than your on-prem equivalent, and their maintenance schedules don't line up with yours. Ideally, you don't want someone messing with your VCS/CI "every day". Your on-prem is not only very unlikely to need any of the moving parts that a world-scalable system needs, but any maintenance can also be planned out-of-hours to make sure eventual operator errors don't cause downtime during business hours.
    - Aeolun 5y ago
- dx034 5y ago
  Backups tend to take the most work, especially if you verify them periodically. But at least that means you have backups. I'm not sure I would want Gitlab to have all my data without a local copy for such important information.

Nextgrid 5y ago

An issue everyone seems to overlook is that an SaaS typically needs to be scalable to the entire world while your on-prem service pretty much never does. The SaaS will have more moving parts & operator activity which in turn makes outages more likely, and that "operator activity" can't be planned for unlike on-prem where you can schedule maintenance out-of-hours to make sure any accidents don't affect the business.