3 ms·

Our internal gitlab spends an hour every night intermittently responding and at least 20 minutes literally unresponsive because of running the “backup” and “upg

Our internal gitlab spends an hour every night intermittently responding and at least 20 minutes literally unresponsive because of running the “backup” and “upgrade” steps. I’m told that we’re talking to the Gitlab premium people and it’s currently unavoidable. We don’t have a large instance. “Maintenance mode” makes the whole process slower, so we don’t turn it on. And I think premium now costs the same as self-hosted GitHub? Has anyone had similar bad experiences with medium company (~500 employees, probably 1/4-1/3 developers) enterprise self-hosting?

jeff_carr 4y ago
Have you tried gitea? It's super hard core. Also, it's written in golang. The only thing that will cause gitea to fail, as far as I can tell, is the hardware. It's basically a daemon on top of the linux kernel. Stand on the shoulders of giants. When you can.
- kenniskrag 4y ago
  even the gitea people are using github for development. The reason they mention on the migration issue is, that not all features needed are ready. They e.g. implemented issue deletation a few months ago. https://github.com/go-gitea/gitea/issues/1029 https://github.com/go-gitea/gitea/issues/1029
  - vanous 4y ago
    >even the gitea people are using github for development. The reason they mention on the migration issue is, that not all features needed are ready. They e.g. implemented issue deletation a few months ago. Actually,data gravity is the issue, not features. https://github.com/go-gitea/gitea/pull/18165 https://github.com/go-gitea/gitea/pull/18165 needs to be merged to be able to leave Github.
    - kenniskrag 4y ago
      They are writing an import function im this pull request. I think the code an pull requests are easy migrated manually. The 2k open issues would have to be cleaned up first. Then migrate the remaining issues via api. One issue could be the non existing user on the import side.
    - knewter 4y ago
      "data gravity" is itself a useful term but I maintain irritation at the person I believe coined it because they presided over the death of the basho era. Is there another pithy term that means this that I can use?
- bogwog 4y ago
  Gitea has had reliability issues whenever I've tried to use it. And by "reliability issues" I mean the thing would spit out an "internal server error" message if I clicked around too fast in the UI while it's running locally on the same PC. It is a lot faster and lighter than Gitlab, but I don't really care about that for something like this. Gitlab is fast enough, and since I run it on a dedicated server, I don't really care how much memory and resources it's using as long as it doesn't exceed the limits of the server. EDIT: to be fair though, I haven't used gitea in at least a year. Things might have improved in that time.
mgbmtl 4y ago
Presumably this is only the SQL backup? (you can backup almost everything else using a regular backup system) 20 minutes seems really long for running an SQL backup. How big is the database? Is the disk/storage fast? Have you tried replicating the database and running the backup off the secondary server? Restarts are annoying though, and fairly slow (2-3 minutes?), but less than once per week?
fuzzy2 4y ago
Maybe using filesystem snapshots (either directly on supported filesystems or using LVM or the like) could remove the bottleneck here. Are you monitoring the system status? How’s the CPU and I/O load when the backup is running?
lbotos 4y ago
My email is in my profile if you have an open ticket number for this issue that I can take a look at when I'm at my desk Monday.

jeff_carr 4y ago

Have you tried gitea? It's super hard core. Also, it's written in golang. The only thing that will cause gitea to fail, as far as I can tell, is the hardware. It's basically a daemon on top of the linux kernel. Stand on the shoulders of giants. When you can.

kenniskrag 4y ago
even the gitea people are using github for development. The reason they mention on the migration issue is, that not all features needed are ready. They e.g. implemented issue deletation a few months ago. https://github.com/go-gitea/gitea/issues/1029 https://github.com/go-gitea/gitea/issues/1029
- vanous 4y ago
  >even the gitea people are using github for development. The reason they mention on the migration issue is, that not all features needed are ready. They e.g. implemented issue deletation a few months ago. Actually,data gravity is the issue, not features. https://github.com/go-gitea/gitea/pull/18165 https://github.com/go-gitea/gitea/pull/18165 needs to be merged to be able to leave Github.
  - kenniskrag 4y ago
    They are writing an import function im this pull request. I think the code an pull requests are easy migrated manually. The 2k open issues would have to be cleaned up first. Then migrate the remaining issues via api. One issue could be the non existing user on the import side.
  - knewter 4y ago
    "data gravity" is itself a useful term but I maintain irritation at the person I believe coined it because they presided over the death of the basho era. Is there another pithy term that means this that I can use?
bogwog 4y ago
Gitea has had reliability issues whenever I've tried to use it. And by "reliability issues" I mean the thing would spit out an "internal server error" message if I clicked around too fast in the UI while it's running locally on the same PC. It is a lot faster and lighter than Gitlab, but I don't really care about that for something like this. Gitlab is fast enough, and since I run it on a dedicated server, I don't really care how much memory and resources it's using as long as it doesn't exceed the limits of the server. EDIT: to be fair though, I haven't used gitea in at least a year. Things might have improved in that time.

mgbmtl 4y ago

Presumably this is only the SQL backup? (you can backup almost everything else using a regular backup system) 20 minutes seems really long for running an SQL backup. How big is the database? Is the disk/storage fast? Have you tried replicating the database and running the backup off the secondary server? Restarts are annoying though, and fairly slow (2-3 minutes?), but less than once per week?

fuzzy2 4y ago

Maybe using filesystem snapshots (either directly on supported filesystems or using LVM or the like) could remove the bottleneck here. Are you monitoring the system status? How’s the CPU and I/O load when the backup is running?

lbotos 4y ago

My email is in my profile if you have an open ticket number for this issue that I can take a look at when I'm at my desk Monday.