5 ms·

Great to hear you notice the performance improvements. We're very happy with the improvements but we still have long way to go. We want the 99% latency under 1s

by sytse 10y ago

Great to hear you notice the performance improvements. We're very happy with the improvements but we still have long way to go. We want the 99% latency under 1s and right now it is 2s: https://drive.google.com/file/d/0BzQDcBnEfNRZSTZYVXlyc1FGdEk/view https://drive.google.com/file/d/0BzQDcBnEfNRZSTZYVXlyc1FGdEk... There are some controller timings that are red https://drive.google.com/a/gitlab.com/file/d/0BzQDcBnEfNRZZVZiRXFLOWNVQTA/view?usp=sharing https://drive.google.com/a/gitlab.com/file/d/0BzQDcBnEfNRZZV... And the git access timings are a sea of red https://drive.google.com/a/gitlab.com/file/d/0BzQDcBnEfNRZTzg3NFN4UXU3Qmc/view?usp=sharing https://drive.google.com/a/gitlab.com/file/d/0BzQDcBnEfNRZTz... Git access will be improved now that we have Gitaly as part of this release. In GitLab 9.1 we want to move git access from the application server to the file server. That should greatly reduce the latency.

tokenizerrr 10y ago
Can't access any of those links, just get a login prompt.
- zegerjan 10y ago
  That is the internal one, Sid meant to link to the public monitoring, located at http://monitor.gitlab.net/ http://monitor.gitlab.net/. It doesn't have the daily overview dashboard, but the fleet overview might be an indicator too (http://monitor.gitlab.net/dashboard/db/fleet-overview http://monitor.gitlab.net/dashboard/db/fleet-overview).
- sytse 10y ago
  Sorry about that, we've replaced the links with screenshots. The problem is that we can't measure Unicorn timings with Prometheus since it is multi-process. Our Prometheus team is working to fix this for everyone.
superplussed 10y ago
I'm glad this is a priority for you guys Sid, the one thing keeping me from using Gitlab is that it doesn't feel as snappy as Github. But as a point of reference, I'm also a guy that won't use Atom because it doesn't feel as snappy as Sublime. For some of us, speed comes first.
- sytse 10y ago
  Speed is essential for a tool like GitLab that people use intensely. We've made great progress in having better monitoring to see what the cause is. For 9.1 we've scheduled many more performance improvements https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=performance&milestone_title=9.1 https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf... than we were able to complete in 9.0 https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=performance&milestone_title=9.0 https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf...
  - Bombthecat 10y ago
    What about memory usage?
- sdflkd 10y ago
  Is Sublime really an editor of choice if speed comes first?
  - tracker1 10y ago
    It's all relative... what kind of computer are you using, the cpu speed, memory available, even OS and drive/ssd can make differences. Even on an i7 with 32gb and an SSD atom was too slow for a really long time... in fact after trying earlier releases of atom and brackets, I almost didn't even try VS Code. I'm really glad I did though. In any case, it's just a matter of relative performance and perception in relation to what one runs on, and their needs.
  - jfktrey 10y ago
    Well, compared to Atom, yes. Two somewhat similar tools with trade offs between them, one of them being speed.
    - dotancohen 10y ago
      Note that Atom is written in Javascript, so the run time is extraordinarily CPU heavy. Even though many Electron (Javascript desktop framework) apps may have snappy UIs, they're typically even worse than Java for CPU (battery life, etc).
jamesmiller5 10y ago
Taking a look at Gitaly & the conversations around it, I don't see much in the way of consensus. Are you using Redis for the distributed lock or is this still a centralized system? My query being you can effectively use the native pre-receive hooks & wrap the git shell to allow consensus without needing to replace git with an RPC system like Gitaly.
- sytse 10y ago
  Gitaly is still a centralized system. It shards the git repositories over multiple git servers in one datacenter. We looked into Git Ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ke... for a distributed system based on hooks. But this requires a sync per repository (very slow with 100k+ repos per server) and it is no longer actively being worked on by Google.
  - eclark 10y ago
    Next time you think about distributing git, you should take a look at the combination of jgit and a distributed key value store. A little while ago I spent a night and I was able to get a fully distributed git http server up and running. I used HBase and was able to get it most things working. That allows the only contention to be around refs. Everything else is well distributed and spread across as many machines as you need. Caching is handled on the key value store side, and the http server side. I didn't get to git gc work though.
    - jamesmiller5 10y ago
      > That allows the only contention to be around refs. This is solved without modifying the git client using a distributed/shared lock wrapping the git-shell & git-hooks (pre-receive) . If you were to modify the git server behavior to say use epaxos, one could provide even better latency guaranties https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf . I'm not sure about the GC operation though, did you use the DfsGarbageCollector that is part of Jgit? I thought that was provided as part of the DFS interface. http://download.eclipse.org/jgit/docs/jgit-2.0.0.201206130900-r/apidocs/org/eclipse/jgit/storage/dfs/DfsGarbageCollector.html http://download.eclipse.org/jgit/docs/jgit-2.0.0.20120613090...
gooeyblob 10y ago
Looks like those last two screenshots are Grafana - do you mind sharing the dashboard JSON for those views? Very cool!
- sytse 10y ago
  Thanks. Most of our monitoring is public on http://monitor.gitlab.net/ http://monitor.gitlab.net/ the things I showed are not because we're still working to make Prometheus work with multi process applications. I'm not sure the Json is public.
- fnord123 10y ago
  There's also http://arewefastyet.gitlab.com http://arewefastyet.gitlab.com

by sytse 10y ago

tokenizerrr 10y ago

Can't access any of those links, just get a login prompt.

zegerjan 10y ago
That is the internal one, Sid meant to link to the public monitoring, located at http://monitor.gitlab.net/ http://monitor.gitlab.net/. It doesn't have the daily overview dashboard, but the fleet overview might be an indicator too (http://monitor.gitlab.net/dashboard/db/fleet-overview http://monitor.gitlab.net/dashboard/db/fleet-overview).
sytse 10y ago
Sorry about that, we've replaced the links with screenshots. The problem is that we can't measure Unicorn timings with Prometheus since it is multi-process. Our Prometheus team is working to fix this for everyone.

superplussed 10y ago

I'm glad this is a priority for you guys Sid, the one thing keeping me from using Gitlab is that it doesn't feel as snappy as Github. But as a point of reference, I'm also a guy that won't use Atom because it doesn't feel as snappy as Sublime. For some of us, speed comes first.

sytse 10y ago
Speed is essential for a tool like GitLab that people use intensely. We've made great progress in having better monitoring to see what the cause is. For 9.1 we've scheduled many more performance improvements https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=performance&milestone_title=9.1 https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf... than we were able to complete in 9.0 https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=performance&milestone_title=9.0 https://gitlab.com/gitlab-org/gitlab-ce/issues?scope=all&utf...
- Bombthecat 10y ago
  What about memory usage?
sdflkd 10y ago
Is Sublime really an editor of choice if speed comes first?
- tracker1 10y ago
  It's all relative... what kind of computer are you using, the cpu speed, memory available, even OS and drive/ssd can make differences. Even on an i7 with 32gb and an SSD atom was too slow for a really long time... in fact after trying earlier releases of atom and brackets, I almost didn't even try VS Code. I'm really glad I did though. In any case, it's just a matter of relative performance and perception in relation to what one runs on, and their needs.
- jfktrey 10y ago
  Well, compared to Atom, yes. Two somewhat similar tools with trade offs between them, one of them being speed.
  - dotancohen 10y ago
    Note that Atom is written in Javascript, so the run time is extraordinarily CPU heavy. Even though many Electron (Javascript desktop framework) apps may have snappy UIs, they're typically even worse than Java for CPU (battery life, etc).

jamesmiller5 10y ago

Taking a look at Gitaly & the conversations around it, I don't see much in the way of consensus. Are you using Redis for the distributed lock or is this still a centralized system? My query being you can effectively use the native pre-receive hooks & wrap the git shell to allow consensus without needing to replace git with an RPC system like Gitaly.

sytse 10y ago
Gitaly is still a centralized system. It shards the git repositories over multiple git servers in one datacenter. We looked into Git Ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ke... for a distributed system based on hooks. But this requires a sync per repository (very slow with 100k+ repos per server) and it is no longer actively being worked on by Google.
- eclark 10y ago
  Next time you think about distributing git, you should take a look at the combination of jgit and a distributed key value store. A little while ago I spent a night and I was able to get a fully distributed git http server up and running. I used HBase and was able to get it most things working. That allows the only contention to be around refs. Everything else is well distributed and spread across as many machines as you need. Caching is handled on the key value store side, and the http server side. I didn't get to git gc work though.
  - jamesmiller5 10y ago
    > That allows the only contention to be around refs. This is solved without modifying the git client using a distributed/shared lock wrapping the git-shell & git-hooks (pre-receive) . If you were to modify the git server behavior to say use epaxos, one could provide even better latency guaranties https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf . I'm not sure about the GC operation though, did you use the DfsGarbageCollector that is part of Jgit? I thought that was provided as part of the DFS interface. http://download.eclipse.org/jgit/docs/jgit-2.0.0.201206130900-r/apidocs/org/eclipse/jgit/storage/dfs/DfsGarbageCollector.html http://download.eclipse.org/jgit/docs/jgit-2.0.0.20120613090...

gooeyblob 10y ago

Looks like those last two screenshots are Grafana - do you mind sharing the dashboard JSON for those views? Very cool!

sytse 10y ago
Thanks. Most of our monitoring is public on http://monitor.gitlab.net/ http://monitor.gitlab.net/ the things I showed are not because we're still working to make Prometheus work with multi process applications. I'm not sure the Json is public.
fnord123 10y ago
There's also http://arewefastyet.gitlab.com http://arewefastyet.gitlab.com