4 ms·

Yep, it's still among the slowest. In terms of speed, popular source forges rank as follows: 1. SourceHut 2. Codeberg 3. GitHub 4. Pagure 5. GitLab 6. Bit

Yep, it's still among the slowest. In terms of speed, popular source forges rank as follows: 1. SourceHut 2. Codeberg 3. GitHub 4. Pagure 5. GitLab 6. Bitbucket Source: https://forgeperf.org/ https://forgeperf.org/

odensc 6y ago
I see that site is maintained by SourceHut though - are the test run from within their own DC? That could contribute to the performance score.
- battery_cowboy 6y ago
  No, they do it in a fair way, I believe if you read the page it will tell you how they fairly measure performance between the forges. Edit: see "Expand for details on methodology" right at the top of the page.
  - odensc 6y ago
    Yes, that explains their methodology but not the location of the hardware. If I go and run a Lighthouse audit on localhost vs. a server in Philadelphia there's going to be an inherent difference.
    - jraph 6y ago
      I remember reading that they also ran the benchmark in Germany and got similar results when first reading about this here on hn. Anyway, it suffice to use gitlab, github, bitbucket and sourcehub for a few seconds to notice the slowness of the three first sites compared to the last one. We didn't really need the numbers to be convinced but I hope they will produce some pressure to be faster.
    - battery_cowboy 6y ago
      Sorry, I seemed to remember they did something to make it fair, I don't recall the location. I use sr.ht and I've used the others and, for me, it feels like 'turbo mode' compared to them. Especially git pushing: it's faster than pushing to my dokku server that only I use!
- cosmojg 6y ago
  Very true. I'm in a different part of the country (I'm in Boston, they're in Philadelphia) and tried running the benchmarks myself about a week ago. Unfortunately, I didn't save the results, but if you take my word for it, I remember getting similar results with only slightly different numbers. Your best bet, of course, is to run the benchmarks yourself: https://git.sr.ht/~sircmpwn/forgeperf https://git.sr.ht/~sircmpwn/forgeperf Some other people ran the benchmarks from different places in Europe and got similar results with "nothing significantly different" from those published: https://news.ycombinator.com/item?id=22900649 https://news.ycombinator.com/item?id=22900649
  - odensc 6y ago
    I'll try it out later today. Would be interesting to see actually because, completely anecdotally, GitLab seems faster than sr.ht in my daily usage.
    - cosmojg 6y ago
      Don't make the same mistake that I did! Save your results and share them with the class. :)
      - odensc 6y ago
        Here: https://report-delta.now.sh https://report-delta.now.sh SourceHut is still the winner of most, but GitHub is a lot faster than in the report on forgeperf.org. I think the perceived speediness of GitHub for me can be mostly attributed to the fact they don't trigger a full page reload for every navigation. Whereas SourceHut is rendered completely server-side AFAIK. (I meant GitHub in my previous comment, fyi.)
dsissitka 6y ago
I was curious so I ran the summary page test. Their results: 100 SourceHut 87 GitHub 53 GitLab My results: 100 SourceHut 87 GitHub 20 GitLab - 1 20 GitLab - 2 19 GitLab - 3 I reran GitLab twice because the results seemed a little low.
gygitlab 6y ago
Thanks for sharing this. I'm part of the team within GitLab's Quality Engineering group that is tasked with performance testing GitLab. We've been working on vastly increasing our performance testing and visibility for about just under a year now. We know there's work to be done with GitLab's performance and it's a top priority for us moving forward. We've blogged recently on our (Quality) efforts here - https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/ https://about.gitlab.com/blog/2020/02/18/how-were-building-u... and you can also see all the issues we've got here - https://gitlab.com/gitlab-org/gitlab/-/issues?label_name[]=Quality:performance-issues https://gitlab.com/gitlab-org/gitlab/-/issues?label_name[]=Q.... I've been digging into these results today as a priority. While we're using different tooling (k6 and SiteSpeed respectively) and a different methodology (SourceHut is running tests with simulated low bandwidth) our tests are indeed generally showing the same pattern and I've confirmed we have issues raised to cover the problem areas. Alongside this we identified a few gaps in our performance testing that we wanted to address immediately: * Our File Blame page (and API) in particular was sobering to witness and we've added that into our tool's tests today (that you should be able to see in our results from tomorrow - https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k https://gitlab.com/gitlab-org/quality/performance/-/wikis/Be...). We've raised 2 top priority issues for that area that will hopefully get tackled soon. * Our file view page, with larger files, is also not as good as it should be. We've pivoted our tooling to test the worse case scenario today with a much larger file and raised an issue for it also for our dev teams. As part of the company wide effort to improve our performance I'm always keen to see new tests being done or highlights of areas we haven't been able to cover yet so thanks again for highlighting this and hopefully you'll see our numbers continuing to improve soon!

odensc 6y ago

I see that site is maintained by SourceHut though - are the test run from within their own DC? That could contribute to the performance score.

battery_cowboy 6y ago
No, they do it in a fair way, I believe if you read the page it will tell you how they fairly measure performance between the forges. Edit: see "Expand for details on methodology" right at the top of the page.
- odensc 6y ago
  Yes, that explains their methodology but not the location of the hardware. If I go and run a Lighthouse audit on localhost vs. a server in Philadelphia there's going to be an inherent difference.
  - jraph 6y ago
    I remember reading that they also ran the benchmark in Germany and got similar results when first reading about this here on hn. Anyway, it suffice to use gitlab, github, bitbucket and sourcehub for a few seconds to notice the slowness of the three first sites compared to the last one. We didn't really need the numbers to be convinced but I hope they will produce some pressure to be faster.
  - battery_cowboy 6y ago
    Sorry, I seemed to remember they did something to make it fair, I don't recall the location. I use sr.ht and I've used the others and, for me, it feels like 'turbo mode' compared to them. Especially git pushing: it's faster than pushing to my dokku server that only I use!
cosmojg 6y ago
Very true. I'm in a different part of the country (I'm in Boston, they're in Philadelphia) and tried running the benchmarks myself about a week ago. Unfortunately, I didn't save the results, but if you take my word for it, I remember getting similar results with only slightly different numbers. Your best bet, of course, is to run the benchmarks yourself: https://git.sr.ht/~sircmpwn/forgeperf https://git.sr.ht/~sircmpwn/forgeperf Some other people ran the benchmarks from different places in Europe and got similar results with "nothing significantly different" from those published: https://news.ycombinator.com/item?id=22900649 https://news.ycombinator.com/item?id=22900649
- odensc 6y ago
  I'll try it out later today. Would be interesting to see actually because, completely anecdotally, GitLab seems faster than sr.ht in my daily usage.
  - cosmojg 6y ago
    Don't make the same mistake that I did! Save your results and share them with the class. :)
    - odensc 6y ago
      Here: https://report-delta.now.sh https://report-delta.now.sh SourceHut is still the winner of most, but GitHub is a lot faster than in the report on forgeperf.org. I think the perceived speediness of GitHub for me can be mostly attributed to the fact they don't trigger a full page reload for every navigation. Whereas SourceHut is rendered completely server-side AFAIK. (I meant GitHub in my previous comment, fyi.)

dsissitka 6y ago

I was curious so I ran the summary page test. Their results: 100 SourceHut 87 GitHub 53 GitLab My results: 100 SourceHut 87 GitHub 20 GitLab - 1 20 GitLab - 2 19 GitLab - 3 I reran GitLab twice because the results seemed a little low.

gygitlab 6y ago

Thanks for sharing this. I'm part of the team within GitLab's Quality Engineering group that is tasked with performance testing GitLab. We've been working on vastly increasing our performance testing and visibility for about just under a year now. We know there's work to be done with GitLab's performance and it's a top priority for us moving forward. We've blogged recently on our (Quality) efforts here - https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/ https://about.gitlab.com/blog/2020/02/18/how-were-building-u... and you can also see all the issues we've got here - https://gitlab.com/gitlab-org/gitlab/-/issues?label_name[]=Quality:performance-issues https://gitlab.com/gitlab-org/gitlab/-/issues?label_name[]=Q.... I've been digging into these results today as a priority. While we're using different tooling (k6 and SiteSpeed respectively) and a different methodology (SourceHut is running tests with simulated low bandwidth) our tests are indeed generally showing the same pattern and I've confirmed we have issues raised to cover the problem areas. Alongside this we identified a few gaps in our performance testing that we wanted to address immediately: * Our File Blame page (and API) in particular was sobering to witness and we've added that into our tool's tests today (that you should be able to see in our results from tomorrow - https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k https://gitlab.com/gitlab-org/quality/performance/-/wikis/Be...). We've raised 2 top priority issues for that area that will hopefully get tackled soon. * Our file view page, with larger files, is also not as good as it should be. We've pivoted our tooling to test the worse case scenario today with a much larger file and raised an issue for it also for our dev teams. As part of the company wide effort to improve our performance I'm always keen to see new tests being done or highlights of areas we haven't been able to cover yet so thanks again for highlighting this and hopefully you'll see our numbers continuing to improve soon!