5 ms·

Gitlab's servers have been slow in my experience. Sometimes taking 10-15 seconds to push up a new commit. I got tired of waiting and moved everything back to bi

by dglass 10y ago

Gitlab's servers have been slow in my experience. Sometimes taking 10-15 seconds to push up a new commit. I got tired of waiting and moved everything back to bitbucket. Both gitlab and bitbuckets source code browsing is slow and a little clunky. GitHub's source code browsing is definitely the best.

sytse 10y ago
You're right, we have to do better on performance. We solved most of the time waiting when pushing a new commit, see the API timings slide on https://www.scribd.com/doc/316471059/GitLab-Infrastructure-20160621 https://www.scribd.com/doc/316471059/GitLab-Infrastructure-2... The web interface is still slower than we like. We've doubled the team of performance engineers and we're making progress, see https://gitlab.com/gitlab-com/infrastructure/issues/59 https://gitlab.com/gitlab-com/infrastructure/issues/59 and all issues labeled with performance https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&sort=created_desc&state=opened&utf8=%E2%9C%93&label-name=&issue_search=performanc https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&sor...
- dineshp2 10y ago
  At a high level, what are the major areas where you face performance issues? Does the choice of RoR play a part in performance problems? Can performance be improved by deploying more servers? PS: I really enjoy using Gitlab, and would be ready to replace Github with Gitlab in my workflow if the performance improves.
  - connorshea 10y ago
    My understanding (as a frontend engineer, so I'm not as intimately familiar with the problems as some others are) is that the biggest bottleneck at this point is with the file system. We're working on switching to CephFS which looks very promising. See the relevant issue for more info: https://gitlab.com/gitlab-com/operations/issues/1 https://gitlab.com/gitlab-com/operations/issues/1 I can get someone from the backend performance team to comment too if you'd like :)
    - deleted 10y ago
      [deleted]
    - dineshp2 10y ago
      > I can get someone from the backend performance team to comment too if you'd like :) Sure! > the biggest bottleneck at this point is with the file system. Would something like AWS EFS [1] solve the problem? It might not be a practical solution from a cost perspective though. Eg: for 50TB of data at $0.3/GB/month 0.3*1024*50 = $15360/month [1] https://aws.amazon.com/efs/ https://aws.amazon.com/efs/
      - pcarranza 10y ago
        The main issue with this is that this would be a vendor specific solution that would lock everybody. We build a product that you can host yourself. If we only solve the scaling issue by pushing it down to a specific vendor, then there is no actual solution. The way we are facing the problem is first by enabling a really easy and simple form of sharding (that would solve most of the issues that a lot of big customers may face), and then by using an open source underlaying filesystem that can scale reasonably well.
  - pcarranza 10y ago
    Hi dineshp, I'm Pablo Carranza, and I am currently leading the effort on infrastructure at GitLab, this includes performance. > what are the major areas where you face performance issues? > Does the choice of RoR play a part in performance problems? RoR does not play a part in the performance problems as much as any other language choice. Our performance problems come from at least 3 different fronts: lack of caching in some specific points making us call the same complex/slow operations many times, NFS (filesystem) performance as a whole, and algorithms that worked really well at small scale, but not anymore, both at app level and at DB level. I think that RoR is a really good option for building a product fact, and eventually it is necessary to start specializing specific parts that do not perform anymore, and just replacing what cannot be specialized. The key element here is that we need to measure first to see where the problem is. > Can performance be improved by deploying more servers? Not really, more front end servers means more load on the NFS backend, so there is no easy solution here. The first step into fixing this issue right now is this one: https://gitlab.com/gitlab-com/infrastructure/issues/139 https://gitlab.com/gitlab-com/infrastructure/issues/139 and then we will start playing with a distributed FS as a longer term solution: https://gitlab.com/gitlab-com/operations/issues/1 https://gitlab.com/gitlab-com/operations/issues/1 As a final note, we are using some specific issues to measure performance as a blackbox, and those issues are seeing some really good progress lately. So stay tuned :) https://www.youtube.com/watch?v=czes-oa0yik https://www.youtube.com/watch?v=czes-oa0yik
    - dineshp2 10y ago
      Thank you for the helpful answer! Could you also comment on this https://news.ycombinator.com/item?id=12056991 https://news.ycombinator.com/item?id=12056991
- pcarranza 10y ago
  The original document is here, it is publicly visible: https://docs.google.com/presentation/d/1qxezL1lSn9SyLGMZRXFUjPOcDIEzlT-LsrSYh38M7ZI/edit?pref=2&pli=1#slide=id.g147aad4f31_0_25 https://docs.google.com/presentation/d/1qxezL1lSn9SyLGMZRXFU... Regarding web performance, these last days we had really good progress: https://gitlab.com/gitlab-com/infrastructure/issues/193#note_12955525 https://gitlab.com/gitlab-com/infrastructure/issues/193#note...
gcb0 10y ago
github cheats for performance. our github enterprise will not bother try show diffs on some pull requests if you changed more than a hundred lines or so. very worthless. my workflow now is to always check the diffs locally because i don't trust theirs. never used gitlab or bitbucket a lot so i don't know if this treachery is there too.
- sytse 10y ago
  I assume you're referring to GitLab.com specifically. We also don't show large diffs, but I think the limit is thousands of lines. In the upcoming release (July 22) we're working on a collapsable diff feature https://gitlab.com/gitlab-org/gitlab-ce/issues/14103 https://gitlab.com/gitlab-org/gitlab-ce/issues/14103 to make working with large diffs easier.
- kannonboy 10y ago
  Bitbucket engineer here! In Bitbucket Cloud - i.e. bitbucket.org - we cap the diff size for pull requests at 10000 lines. In Bitbucket Server we also cap the number of lines in a particular diff at 10000 by default, but you can override this number[0], along with all other timeouts and thresholds with our various configuration properties. One of the engineering values of the Server team is "no hardcoded constants" - so you can configure basically any property that you like to suit your particular deployment. Both Bitbucket Server and Cloud also generate a subtly different - and in our opinion more correct - diff than you'll find in GitLab and GitHub. Bitbucket actually creates a hypothetical merge commit between your two branches and shows the diff between it and the tip of the target branch. This means we can nicely render merge conflicts in the UI, and show how your target branch will actually be affected by the merge (rather than just the changes on the source branch). I wrote an article that discusses our merge algorithm in more depth[1] a little while ago. [0]: https://confluence.atlassian.com/bitbucketserver/bitbucket-server-config-properties-776640155.html#BitbucketServerconfigproperties-Paging https://confluence.atlassian.com/bitbucketserver/bitbucket-s... [1]: https://developer.atlassian.com/blog/2015/01/a-better-pull-request/ https://developer.atlassian.com/blog/2015/01/a-better-pull-r... edit: correcting my previous statement on diff sizing in Bitbucket Cloud
- dmitry-k 10y ago

by dglass 10y ago

sytse 10y ago

You're right, we have to do better on performance. We solved most of the time waiting when pushing a new commit, see the API timings slide on https://www.scribd.com/doc/316471059/GitLab-Infrastructure-20160621 https://www.scribd.com/doc/316471059/GitLab-Infrastructure-2... The web interface is still slower than we like. We've doubled the team of performance engineers and we're making progress, see https://gitlab.com/gitlab-com/infrastructure/issues/59 https://gitlab.com/gitlab-com/infrastructure/issues/59 and all issues labeled with performance https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&sort=created_desc&state=opened&utf8=%E2%9C%93&label-name=&issue_search=performanc https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&sor...

dineshp2 10y ago
At a high level, what are the major areas where you face performance issues? Does the choice of RoR play a part in performance problems? Can performance be improved by deploying more servers? PS: I really enjoy using Gitlab, and would be ready to replace Github with Gitlab in my workflow if the performance improves.
- connorshea 10y ago
  My understanding (as a frontend engineer, so I'm not as intimately familiar with the problems as some others are) is that the biggest bottleneck at this point is with the file system. We're working on switching to CephFS which looks very promising. See the relevant issue for more info: https://gitlab.com/gitlab-com/operations/issues/1 https://gitlab.com/gitlab-com/operations/issues/1 I can get someone from the backend performance team to comment too if you'd like :)
  - deleted 10y ago
    [deleted]
  - dineshp2 10y ago
    > I can get someone from the backend performance team to comment too if you'd like :) Sure! > the biggest bottleneck at this point is with the file system. Would something like AWS EFS [1] solve the problem? It might not be a practical solution from a cost perspective though. Eg: for 50TB of data at $0.3/GB/month 0.3*1024*50 = $15360/month [1] https://aws.amazon.com/efs/ https://aws.amazon.com/efs/
    - pcarranza 10y ago
      The main issue with this is that this would be a vendor specific solution that would lock everybody. We build a product that you can host yourself. If we only solve the scaling issue by pushing it down to a specific vendor, then there is no actual solution. The way we are facing the problem is first by enabling a really easy and simple form of sharding (that would solve most of the issues that a lot of big customers may face), and then by using an open source underlaying filesystem that can scale reasonably well.
- pcarranza 10y ago
  Hi dineshp, I'm Pablo Carranza, and I am currently leading the effort on infrastructure at GitLab, this includes performance. > what are the major areas where you face performance issues? > Does the choice of RoR play a part in performance problems? RoR does not play a part in the performance problems as much as any other language choice. Our performance problems come from at least 3 different fronts: lack of caching in some specific points making us call the same complex/slow operations many times, NFS (filesystem) performance as a whole, and algorithms that worked really well at small scale, but not anymore, both at app level and at DB level. I think that RoR is a really good option for building a product fact, and eventually it is necessary to start specializing specific parts that do not perform anymore, and just replacing what cannot be specialized. The key element here is that we need to measure first to see where the problem is. > Can performance be improved by deploying more servers? Not really, more front end servers means more load on the NFS backend, so there is no easy solution here. The first step into fixing this issue right now is this one: https://gitlab.com/gitlab-com/infrastructure/issues/139 https://gitlab.com/gitlab-com/infrastructure/issues/139 and then we will start playing with a distributed FS as a longer term solution: https://gitlab.com/gitlab-com/operations/issues/1 https://gitlab.com/gitlab-com/operations/issues/1 As a final note, we are using some specific issues to measure performance as a blackbox, and those issues are seeing some really good progress lately. So stay tuned :) https://www.youtube.com/watch?v=czes-oa0yik https://www.youtube.com/watch?v=czes-oa0yik
  - dineshp2 10y ago
    Thank you for the helpful answer! Could you also comment on this https://news.ycombinator.com/item?id=12056991 https://news.ycombinator.com/item?id=12056991
pcarranza 10y ago
The original document is here, it is publicly visible: https://docs.google.com/presentation/d/1qxezL1lSn9SyLGMZRXFUjPOcDIEzlT-LsrSYh38M7ZI/edit?pref=2&pli=1#slide=id.g147aad4f31_0_25 https://docs.google.com/presentation/d/1qxezL1lSn9SyLGMZRXFU... Regarding web performance, these last days we had really good progress: https://gitlab.com/gitlab-com/infrastructure/issues/193#note_12955525 https://gitlab.com/gitlab-com/infrastructure/issues/193#note...

gcb0 10y ago

github cheats for performance. our github enterprise will not bother try show diffs on some pull requests if you changed more than a hundred lines or so. very worthless. my workflow now is to always check the diffs locally because i don't trust theirs. never used gitlab or bitbucket a lot so i don't know if this treachery is there too.

sytse 10y ago
I assume you're referring to GitLab.com specifically. We also don't show large diffs, but I think the limit is thousands of lines. In the upcoming release (July 22) we're working on a collapsable diff feature https://gitlab.com/gitlab-org/gitlab-ce/issues/14103 https://gitlab.com/gitlab-org/gitlab-ce/issues/14103 to make working with large diffs easier.
kannonboy 10y ago
Bitbucket engineer here! In Bitbucket Cloud - i.e. bitbucket.org - we cap the diff size for pull requests at 10000 lines. In Bitbucket Server we also cap the number of lines in a particular diff at 10000 by default, but you can override this number[0], along with all other timeouts and thresholds with our various configuration properties. One of the engineering values of the Server team is "no hardcoded constants" - so you can configure basically any property that you like to suit your particular deployment. Both Bitbucket Server and Cloud also generate a subtly different - and in our opinion more correct - diff than you'll find in GitLab and GitHub. Bitbucket actually creates a hypothetical merge commit between your two branches and shows the diff between it and the tip of the target branch. This means we can nicely render merge conflicts in the UI, and show how your target branch will actually be affected by the merge (rather than just the changes on the source branch). I wrote an article that discusses our merge algorithm in more depth[1] a little while ago. [0]: https://confluence.atlassian.com/bitbucketserver/bitbucket-server-config-properties-776640155.html#BitbucketServerconfigproperties-Paging https://confluence.atlassian.com/bitbucketserver/bitbucket-s... [1]: https://developer.atlassian.com/blog/2015/01/a-better-pull-request/ https://developer.atlassian.com/blog/2015/01/a-better-pull-r... edit: correcting my previous statement on diff sizing in Bitbucket Cloud
dmitry-k 10y ago