6 ms·

devops sysadmin here. I started with gitlab about a year ago and can honestly say I wish I had taken gogs/gitea instead. The main problem for me is gitlabs ut

devops sysadmin here. I started with gitlab about a year ago and can honestly say I wish I had taken gogs/gitea instead. The main problem for me is gitlabs utter dearth of somewhat counterproductive features. Git LFS support is almost a cruel joke in gitlab as git operations under the hood now take inexorably more ram to complete. In turn im rewarded with more traditional RCS programmers asking why git has problems with their newly requested LFS and a 40gb video file they decided to store. gitlab is also so large it requires its own chef deployment to competently install it from omnibus, and has no HA roadmap in sight unless you want to break apart its rube-goldberg structure and attempt to HA the individual components of it. Other things like Wiki, local runner CI, pages and integration with a corporate Jira ticket system seem like the excessive demands of programmers without enough skill to understand that there is tangible merit in ablating these systems into separate applications entirely and growing them as needed. gogs and gittea allow you to put the brakes on developers that just want to get the code out the door in order to study and implement competent things like competent CI, blue/green deploys, immutable infrastructure and reproduceable builds that target scalable platforms like Kubernetes instead of one-offs that just reward coders with a merciful standup and the chance to say "its done." if you use gittea for no other reason, it is also massively less resource hungry than a gitlab installation. im currently throwing 4 cores and 8 gigs of ram at my gitlab install with another 2.3tb of disk and frankly no end in sight. The gitlab rake task for backups has sent oomkiller on a rampage 3 times so far and takes forever. Large installs of 1000 users or more and you'll begin to see why gitlab might not be the best choice. with programmers, the thirst is real...nobody cares about features if the repo wont clone.

vanderreeah 9y ago
Pedantic notice: an utter dearth of counterproductive features would seem to be a good thing rather than a problem.
- crankylinuxuser 9y ago
  As another systems engineer, I'd much rather have clearly delineated features and how to interconnect them, rather than a cornucopia of disparate and badly documented "features". I'm thinking of the class of "Does:X, Y, Z" but Z requires unstated dependency of A and C, and C requires you to have a specific database organization type that was asked at the initialization of the system. And there would be no hint that you had to choose the DB type, and how that would have ramifications 10 steps down the line. Going back, what I'd rather have is more in line with the Unix philosophy. Do something and do it well. Let me connect somethings to other somethings. But Gitlab is a cathedral, and disorganized at that.
  - kevinherron 9y ago
    I think you're missing the point as well; "dearth" is a lack of. A lack of counter-productive features is a good thing.
    - Izkata 9y ago
      I think you've misunderstood who is responding to who. A "dearth of somewhat counterproductive features" is being described as a bad thing in the OP, using an incorrect/nonexistent definition of "dearth" as "there are too many somewhat counterproductive features".
- IncRnd 9y ago
  As they say - less is more.
- code_duck 9y ago
  I was confused by that, too. I have a problem with the word ‘dearth’ in that I can often not remember whether it means an abundance or scarcity. So I looked it up, and found that it means scarcity. Perhaps GP makes the same mistake I do and misused dearth to mean abundance?
  - adrianratnapala 9y ago
    Yes, it means scarcity and the GP used it to mean abundance. My mnemonic is that "dearth" looks like "death".
- Terr_ 9y ago
  I suspect they meant more like "profusion."
jancsika 9y ago
> [...] and implement competent things like competent CI [...] I've got an OSX build box with a bunch of virtualbox VMs so I can build for Gnu/Linux (i386, x86_64, and arm), OSX, and Windows. Single slow box, so one runner at a time. It has to be running OSX because I haven't found any way to run an OSX VM on any other platform. I want to trigger CI on Gitlab merge requests. What does $competent_CI expand to?
- mschuster91 9y ago
  > It has to be running OSX because I haven't found any way to run an OSX VM on any other platform. It is fairly easy to get decent OS X virtualization using Debian as base OS and QEMU-KVM for virtualization. You won't have niceties such as GPU acceleration but it will work. Search for "OSX KVM" on Google, it will take you where you want - unfortunately a direct link would expose me to legal liabilities. Also, be warned that using this method may violate Apple's EULA.
  - freedomben 9y ago
    Have you figured out a way to bootstrap it without a Mac? I successfully got an image running under Vbox on Fedora, but I had to build the image first on a Mac and then export it as an appliance. Also, performance was atrocious. It was so bad as to basically make it unusable. 5 second delay before processing mouse and key events.
    - kakwa_ 9y ago
      It's possible to create an OSX VM under KVM. It's a little clunky, you get very bad graphics performances, and you can run in really weird issues (for example, the NIC not going-up properly), but it works, and if you are using it as a CI platform and/or only sshing into it, it's more than usable. It's really handy when you are testing/adding support for projects (like C/C++ with cmake) on OSX.
  - justrobert 9y ago
    Hackintosh systems are easy as long as you're not looking to update them after everything is working correctly. They are fragile systems that one doesn't really want to risk production environments on in my opinion.
the_duke 9y ago
I concur with you on the resource usage of Gitlab. It's a complex system, and takes up a lot of resources. I haven't been able to get it to run in a stable manner with less than 8GB of RAM on a very moderately used setup. BUT I very much disagree with your general rejection of it's worth. Yes, the feature set is ballooning, and some of them are very half baked. But I find tremendous value in a integrated solution that offers code hosting, wikis, issue tracking and a tightly integrated CI without having to jump between 4 different systems. There are a lot of use cases where Gitlab is overkill, but for others it is a really great solution that even comes for free. Also, as long as the system has enough resources, I've found it to run very reliably and upgrades are painless. (I also agree though that the backup task is a PITA. Really slow and resource hungry).
- KaoruAoiShiho 9y ago
  Well gitea/gogs also have code hosting, wikis, issue tracking, and while CI is not tightly integrated it is fairly easy to integrate. So the value has to come from something else :) GitLab has integrated chat and kanban boards so there is other cool stuff in Gitlab but it's not really any of your listed reasons. Edit: I would have to say if I were to choose GitLab the biggest reason would be the UI. The front-end just looks a lot better. If Gogs/gitea could get a sponsor to help clean it up it would be a lot more popular.
  - naranha 9y ago
    What would you recommend as CI solution for gitea? I'm looking for something that's self hosted, foss and does not require an admin to set it up for every project. GitLab offers this, but I haven't found anything that competes really.
    - cobbzilla 9y ago
      We use a Gitea for source control and Jenkins for CI. There's an existing jenkins-git plugin that makes connecting them very easy. We switched from GitHub to Gitea last year and love it. We looked at Gitlab but it reminded me too much of phabricator - trying to be all things to all people, bloating all over. I prefer specialized tools that focus on a smaller core of functionality and interoperate easily with the other pieces.
nfrankel 9y ago
I was about to ask "why not Gitlab?". Thanks for answering it in advance.
mschuster91 9y ago
> gitlab is also so large it requires its own chef deployment to competently install it from omnibus, and has no HA roadmap in sight unless you want to break apart its rube-goldberg structure and attempt to HA the individual components of it. Use the gitlab-ce docker container and save yourself the hassle. Literally takes less than 10min to set up and a version upgrade is nothing more than docker stop, docker rm, docker run <new version> away. Only thing I haven't solved with this is HA but you can use a central NFS mountpoint with an HA'd setup there, and simply start a new container on another machine with the NFS mount... will require manual intervention, yes, but it's more efficient than trying to break up the dozens of moving parts.
- jedberg 9y ago
  NFS will give you redundancy but not high availability. In fact it will probably lower your availability the first time NFS gets hung. But if manual intervention works for your setup, then you probably don’t really need HA anyway, just redundancy.
  - sytse 9y ago
    We're working towards Gitaly 1.0 that would allow you to run GitLab without NFS. This isn't HA but it would prevent one NFS server from taking down the whole cluster.
- sytse 9y ago
  Thanks for posting. HA is a paid feature of GitLab and will likely stay a paid feature. We're working hard on a cloud native alternative to the docker container that works well on Kubernetes. There is more information on https://gitlab.com/charts/gitlab/ https://gitlab.com/charts/gitlab/ and you can see the activity on https://gitlab.com/charts/gitlab/commits/master https://gitlab.com/charts/gitlab/commits/master Since it is cloud native I assume it will allow autoscaling and it neatly orders every component in its own container. Today GitLab is using a lot of RAM and the best way to reduce that is to make the application server multithreaded https://gitlab.com/gitlab-org/gitlab-ce/issues/3592 https://gitlab.com/gitlab-org/gitlab-ce/issues/3592 The first priority is to add rubocop rules for https://github.com/covermymeds/rubocop-thread_safety https://github.com/covermymeds/rubocop-thread_safety There is a WIP merge request in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1899 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1899 Any help is appreciated.
avar 9y ago
> gitlab [...] also [...] has no HA roadmap in sight unless you > want to break apart its rube-goldberg structure and attempt to > HA the individual components of it. We run a very large in-house GitLab installation and the HA is easy because: 1) GitLab stores data on the FS, and you can outsource this problem to e.g. a NetApp filer. 2) The data it doesn't store on NFS it stores in psql or MySQL, which has established HA solutions. Once you have that set up you can load-balance the GitLab nodes and provision / destroy them at will.
- sytse 9y ago
  Thanks, you can indeed do HA this way. When Gitaly 1.0 lands you won't have to mount the NFS volumes on the application servers anymore as everything will go over gRPC.
  - avar 9y ago
    That just splits up the task of application server & git executor into two, it doesn't get rid of the dependency on a monolithic storage backend. I understand that this is something gitlab.com is working towards and excited about, presumably because it'll allow for provisioning web workers unrelated to the CPU you're spending on running git, but for us I don't see it being useful. I hope GitLab eventually gets something like GitHub Spokes[1] which'll allow for provisioning the entire setup on a bunch of dumb boxes with RAID 0. 1. https://githubengineering.com/introducing-dgit/ https://githubengineering.com/introducing-dgit/
    - sytse 9y ago
      The advantage is that if one storage nose is offline the rest of the site can still function. Spokes is more elaborate can can deal with node failure. We plan to rely on cloud storage so we can quickly attach a new VM with Gitaly.
- qaq 9y ago
  It's hard to imagine that TCO for NetApp + Gitlab is less than Github or some other hosted solution.
erikb 9y ago
Exactly my feeling, and I just tried it once on an AWS instance or similar. It was the only serious program running on the machine and it dragged down all the resources, worse than Jenkins ever could. All that doesn't say gittea is much better, though.
dozzie 9y ago
> [...] programmers asking why git has problems with their newly requested LFS and a 40gb video file they decided to store. The thing is, git, as our industry uses it, is not really a good tool for keeping artifacts, and "video" sounds like an artifact, not a source file. I'd like to see more artifact repositories, especially the ones that you can talk to from command line or from a Python/Ruby/whatever script. I haven't found any such thing, so I wrote my own.
- merlincorey 9y ago
  What is your artifact repository called and what are the main features? I'd love to check it out.
  - severine 9y ago
    Maybe https://github.com/dozzie/grailbag https://github.com/dozzie/grailbag
  - dozzie 9y ago
    As severine found out, it's called GrailBag (https://github.com/dozzie/grailbag https://github.com/dozzie/grailbag). Docs and README are lacking, to put it mildly, so I don't deem it ready for being published, but I don't exactly hide it either. From the features, GrailBag stores files along with key:value pairs as metadata (not surprising), has a command line client to list/modify/upload/download/delete the artifacts, and has a Python module for doing the same from a more sophisticated script. Additionally there is an interpreter of a simple language that describes directory tree where artifacts will be deployed (which artifacts to download, how to name the files, what directories and symlinks to create). This way you can write a cron script that deploys whatever you have in the repository, e.g. in a cold-standby server scenario (two servers having the same configuration, but one shut down and only powered on when the first one fails; on the first run of the cron task, the spare server downloads all the missing files, including whatever got stored while the server was shut down).
Vaslo 9y ago
I had a class last semester that submitted everything through Gitlab and when we tried to run unit tests to test for the correct code, it would hang for 7-8 minutes. Very frustrating. Happened on all the assignments so was not just a one time thing.
- matteeyah 9y ago
  Could you provide more info about this? What do you mean by "it would hang"? Any specific component that was unresponsive? Was this on a self-hosted instance or on GitLab.com? We'd love to make sure this doesn't happen again.
- JoeMalt 9y ago
  I had a similar class that used Gigs instead and never had any performance issues. It didn't feel as polished as GitLab though.
lttlrck 9y ago
Yeah I didn’t have much luck with LFS but we didn’t really need it so it wasn’t a problem to just disable. I love gitlab. I have it running for a small team on a Digital Ocean Droplet it was trivial to get running. If we had gone for self-hosting we’d have gone the docker route. Having CI built in was a massive boon when we switched over. My only complaint really is the resource usage but it isn’t causing problems (8GB). Aside from that I love it.
robbiemitchell 9y ago
Could you describe these specs in money terms? This reads like it's painful to throw 4 cores and 8 GB RAM at the problem -- what is that costing you?

vanderreeah 9y ago

Pedantic notice: an utter dearth of counterproductive features would seem to be a good thing rather than a problem.

crankylinuxuser 9y ago
As another systems engineer, I'd much rather have clearly delineated features and how to interconnect them, rather than a cornucopia of disparate and badly documented "features". I'm thinking of the class of "Does:X, Y, Z" but Z requires unstated dependency of A and C, and C requires you to have a specific database organization type that was asked at the initialization of the system. And there would be no hint that you had to choose the DB type, and how that would have ramifications 10 steps down the line. Going back, what I'd rather have is more in line with the Unix philosophy. Do something and do it well. Let me connect somethings to other somethings. But Gitlab is a cathedral, and disorganized at that.
- kevinherron 9y ago
  I think you're missing the point as well; "dearth" is a lack of. A lack of counter-productive features is a good thing.
  - Izkata 9y ago
    I think you've misunderstood who is responding to who. A "dearth of somewhat counterproductive features" is being described as a bad thing in the OP, using an incorrect/nonexistent definition of "dearth" as "there are too many somewhat counterproductive features".
IncRnd 9y ago
As they say - less is more.
code_duck 9y ago
I was confused by that, too. I have a problem with the word ‘dearth’ in that I can often not remember whether it means an abundance or scarcity. So I looked it up, and found that it means scarcity. Perhaps GP makes the same mistake I do and misused dearth to mean abundance?
- adrianratnapala 9y ago
  Yes, it means scarcity and the GP used it to mean abundance. My mnemonic is that "dearth" looks like "death".
Terr_ 9y ago
I suspect they meant more like "profusion."

jancsika 9y ago

> [...] and implement competent things like competent CI [...] I've got an OSX build box with a bunch of virtualbox VMs so I can build for Gnu/Linux (i386, x86_64, and arm), OSX, and Windows. Single slow box, so one runner at a time. It has to be running OSX because I haven't found any way to run an OSX VM on any other platform. I want to trigger CI on Gitlab merge requests. What does $competent_CI expand to?

mschuster91 9y ago
> It has to be running OSX because I haven't found any way to run an OSX VM on any other platform. It is fairly easy to get decent OS X virtualization using Debian as base OS and QEMU-KVM for virtualization. You won't have niceties such as GPU acceleration but it will work. Search for "OSX KVM" on Google, it will take you where you want - unfortunately a direct link would expose me to legal liabilities. Also, be warned that using this method may violate Apple's EULA.
- freedomben 9y ago
  Have you figured out a way to bootstrap it without a Mac? I successfully got an image running under Vbox on Fedora, but I had to build the image first on a Mac and then export it as an appliance. Also, performance was atrocious. It was so bad as to basically make it unusable. 5 second delay before processing mouse and key events.
  - kakwa_ 9y ago
    It's possible to create an OSX VM under KVM. It's a little clunky, you get very bad graphics performances, and you can run in really weird issues (for example, the NIC not going-up properly), but it works, and if you are using it as a CI platform and/or only sshing into it, it's more than usable. It's really handy when you are testing/adding support for projects (like C/C++ with cmake) on OSX.
- justrobert 9y ago
  Hackintosh systems are easy as long as you're not looking to update them after everything is working correctly. They are fragile systems that one doesn't really want to risk production environments on in my opinion.

the_duke 9y ago

I concur with you on the resource usage of Gitlab. It's a complex system, and takes up a lot of resources. I haven't been able to get it to run in a stable manner with less than 8GB of RAM on a very moderately used setup. BUT I very much disagree with your general rejection of it's worth. Yes, the feature set is ballooning, and some of them are very half baked. But I find tremendous value in a integrated solution that offers code hosting, wikis, issue tracking and a tightly integrated CI without having to jump between 4 different systems. There are a lot of use cases where Gitlab is overkill, but for others it is a really great solution that even comes for free. Also, as long as the system has enough resources, I've found it to run very reliably and upgrades are painless. (I also agree though that the backup task is a PITA. Really slow and resource hungry).

KaoruAoiShiho 9y ago
Well gitea/gogs also have code hosting, wikis, issue tracking, and while CI is not tightly integrated it is fairly easy to integrate. So the value has to come from something else :) GitLab has integrated chat and kanban boards so there is other cool stuff in Gitlab but it's not really any of your listed reasons. Edit: I would have to say if I were to choose GitLab the biggest reason would be the UI. The front-end just looks a lot better. If Gogs/gitea could get a sponsor to help clean it up it would be a lot more popular.
- naranha 9y ago
  What would you recommend as CI solution for gitea? I'm looking for something that's self hosted, foss and does not require an admin to set it up for every project. GitLab offers this, but I haven't found anything that competes really.
  - cobbzilla 9y ago
    We use a Gitea for source control and Jenkins for CI. There's an existing jenkins-git plugin that makes connecting them very easy. We switched from GitHub to Gitea last year and love it. We looked at Gitlab but it reminded me too much of phabricator - trying to be all things to all people, bloating all over. I prefer specialized tools that focus on a smaller core of functionality and interoperate easily with the other pieces.

nfrankel 9y ago

I was about to ask "why not Gitlab?". Thanks for answering it in advance.

mschuster91 9y ago

> gitlab is also so large it requires its own chef deployment to competently install it from omnibus, and has no HA roadmap in sight unless you want to break apart its rube-goldberg structure and attempt to HA the individual components of it. Use the gitlab-ce docker container and save yourself the hassle. Literally takes less than 10min to set up and a version upgrade is nothing more than docker stop, docker rm, docker run <new version> away. Only thing I haven't solved with this is HA but you can use a central NFS mountpoint with an HA'd setup there, and simply start a new container on another machine with the NFS mount... will require manual intervention, yes, but it's more efficient than trying to break up the dozens of moving parts.

jedberg 9y ago
NFS will give you redundancy but not high availability. In fact it will probably lower your availability the first time NFS gets hung. But if manual intervention works for your setup, then you probably don’t really need HA anyway, just redundancy.
- sytse 9y ago
  We're working towards Gitaly 1.0 that would allow you to run GitLab without NFS. This isn't HA but it would prevent one NFS server from taking down the whole cluster.
sytse 9y ago
Thanks for posting. HA is a paid feature of GitLab and will likely stay a paid feature. We're working hard on a cloud native alternative to the docker container that works well on Kubernetes. There is more information on https://gitlab.com/charts/gitlab/ https://gitlab.com/charts/gitlab/ and you can see the activity on https://gitlab.com/charts/gitlab/commits/master https://gitlab.com/charts/gitlab/commits/master Since it is cloud native I assume it will allow autoscaling and it neatly orders every component in its own container. Today GitLab is using a lot of RAM and the best way to reduce that is to make the application server multithreaded https://gitlab.com/gitlab-org/gitlab-ce/issues/3592 https://gitlab.com/gitlab-org/gitlab-ce/issues/3592 The first priority is to add rubocop rules for https://github.com/covermymeds/rubocop-thread_safety https://github.com/covermymeds/rubocop-thread_safety There is a WIP merge request in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1899 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1899 Any help is appreciated.

avar 9y ago

> gitlab [...] also [...] has no HA roadmap in sight unless you > want to break apart its rube-goldberg structure and attempt to > HA the individual components of it. We run a very large in-house GitLab installation and the HA is easy because: 1) GitLab stores data on the FS, and you can outsource this problem to e.g. a NetApp filer. 2) The data it doesn't store on NFS it stores in psql or MySQL, which has established HA solutions. Once you have that set up you can load-balance the GitLab nodes and provision / destroy them at will.

sytse 9y ago
Thanks, you can indeed do HA this way. When Gitaly 1.0 lands you won't have to mount the NFS volumes on the application servers anymore as everything will go over gRPC.
- avar 9y ago
  That just splits up the task of application server & git executor into two, it doesn't get rid of the dependency on a monolithic storage backend. I understand that this is something gitlab.com is working towards and excited about, presumably because it'll allow for provisioning web workers unrelated to the CPU you're spending on running git, but for us I don't see it being useful. I hope GitLab eventually gets something like GitHub Spokes[1] which'll allow for provisioning the entire setup on a bunch of dumb boxes with RAID 0. 1. https://githubengineering.com/introducing-dgit/ https://githubengineering.com/introducing-dgit/
  - sytse 9y ago
    The advantage is that if one storage nose is offline the rest of the site can still function. Spokes is more elaborate can can deal with node failure. We plan to rely on cloud storage so we can quickly attach a new VM with Gitaly.
qaq 9y ago
It's hard to imagine that TCO for NetApp + Gitlab is less than Github or some other hosted solution.

erikb 9y ago

Exactly my feeling, and I just tried it once on an AWS instance or similar. It was the only serious program running on the machine and it dragged down all the resources, worse than Jenkins ever could. All that doesn't say gittea is much better, though.

dozzie 9y ago

> [...] programmers asking why git has problems with their newly requested LFS and a 40gb video file they decided to store. The thing is, git, as our industry uses it, is not really a good tool for keeping artifacts, and "video" sounds like an artifact, not a source file. I'd like to see more artifact repositories, especially the ones that you can talk to from command line or from a Python/Ruby/whatever script. I haven't found any such thing, so I wrote my own.

merlincorey 9y ago
What is your artifact repository called and what are the main features? I'd love to check it out.
- severine 9y ago
  Maybe https://github.com/dozzie/grailbag https://github.com/dozzie/grailbag
- dozzie 9y ago
  As severine found out, it's called GrailBag (https://github.com/dozzie/grailbag https://github.com/dozzie/grailbag). Docs and README are lacking, to put it mildly, so I don't deem it ready for being published, but I don't exactly hide it either. From the features, GrailBag stores files along with key:value pairs as metadata (not surprising), has a command line client to list/modify/upload/download/delete the artifacts, and has a Python module for doing the same from a more sophisticated script. Additionally there is an interpreter of a simple language that describes directory tree where artifacts will be deployed (which artifacts to download, how to name the files, what directories and symlinks to create). This way you can write a cron script that deploys whatever you have in the repository, e.g. in a cold-standby server scenario (two servers having the same configuration, but one shut down and only powered on when the first one fails; on the first run of the cron task, the spare server downloads all the missing files, including whatever got stored while the server was shut down).

Vaslo 9y ago

I had a class last semester that submitted everything through Gitlab and when we tried to run unit tests to test for the correct code, it would hang for 7-8 minutes. Very frustrating. Happened on all the assignments so was not just a one time thing.

matteeyah 9y ago
Could you provide more info about this? What do you mean by "it would hang"? Any specific component that was unresponsive? Was this on a self-hosted instance or on GitLab.com? We'd love to make sure this doesn't happen again.
JoeMalt 9y ago
I had a similar class that used Gigs instead and never had any performance issues. It didn't feel as polished as GitLab though.

lttlrck 9y ago

Yeah I didn’t have much luck with LFS but we didn’t really need it so it wasn’t a problem to just disable. I love gitlab. I have it running for a small team on a Digital Ocean Droplet it was trivial to get running. If we had gone for self-hosting we’d have gone the docker route. Having CI built in was a massive boon when we switched over. My only complaint really is the resource usage but it isn’t causing problems (8GB). Aside from that I love it.

robbiemitchell 9y ago

Could you describe these specs in money terms? This reads like it's painful to throw 4 cores and 8 GB RAM at the problem -- what is that costing you?