5 ms·

I wonder how Gitea will hold up to all the traffic? Even Gitea hosts their stuff on Github.

by cowmix 4y ago

codefined 4y ago
Gitea is self-hosted, so Blender will be running their own instance.
- qwertox 4y ago
  That's not what parent means. Gitea is an alternative to GitHub, yet they use GitHub for hosting their code.
  - SV_BubbleTime 4y ago
    That kind of makes sense for disaster planning though. If Gitea discovered some massive CVE, and volentarily went dark to fix it... They would want their code and fix hosted somewhere else.
    - bawolff 4y ago
      Mirroring the code to github (or somewhere else) would solve that problem just as well. Not dogfooding your own product seems like a huge red flag to me.
      - throwaway894345 4y ago
        There's clearly the intention to self-host, they seem to be waiting for support to ingest data from a GitHub data export which seems pretty reasonable.
        techknowlogick 4y ago
        Yes, this is exactly the case. Lots of data, and low ratelimits don't mix. All except for the main repo have already migrated to gitea.com Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.
stryan 4y ago
I don't believe Gitea is hosting on Github for traffic reasons but for feature completeness. Relevant issue is here[0]. [0]https://github.com/go-gitea/gitea/issues/1029 https://github.com/go-gitea/gitea/issues/1029
- lucideer 4y ago
  That's a great ticket, & also links to a really nice & insightful comment from one of the Fog Creek folk https://lobste.rs/s/gokjbo/gitea_1_1_0_released#c_dg9pwe https://lobste.rs/s/gokjbo/gitea_1_1_0_released#c_dg9pwe
  - vidanay 4y ago
    I still dream of a world where FogBugz & Kiln gained the respect they deserved. Mercurial was pretty damn nice.
    - slondr 4y ago
      Was? It still is very nice.
- aidenn0 4y ago
  Looks like https://github.com/go-gitea/gitea/pull/18165 https://github.com/go-gitea/gitea/pull/18165 is the last missing checkbox...
tylersmith 4y ago
Gitea does not use Github because the software can't handle their load requirements, but because they started using GH before Gitea had all the features they needed. They are actively working on hosting Gitea on Gitea, with the last piece being an importer to move over all their GH data. Blender seems to have been ok manually migrating some things and abandoning other things, since they weren't likely to ever get a Phab->Gitea importer anyways. https://github.com/go-gitea/gitea/issues/1029 https://github.com/go-gitea/gitea/issues/1029
- bawolff 4y ago
  How much traffic could they possibly have? Quite frankly either gitea has absolutely terrible scalability or something else is off here. Edit: i misread the parent comment. My bad.
  - yamtaddle 4y ago
    Huh? It isn't because of inability to handle load, that they host on GH. Gitea performs alright. It can serve surprisingly-heavy traffic from a potato-tier VM with a SQLite backend (assuming the host network can keep up with it, anyway). Not Github-scale (but Github has... a lot of traffic, including a whole lot of write traffic) but quite a bit, and that's with basically the worst-possible hosting situation for handling high load.
    - techknowlogick 4y ago
      That's correct. It is not due to load, gitea.com already has most of the projects repos migrated already, just not the main one due to ratelimits and exporting all the issues. Gitea.com has thousands of users, and multi-TB traffic monthly on a fairly modest setup. Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.
  - tylersmith 4y ago
    Tbf my comment was phrased a bit ambiguously, but I couldn't think of anything less ambiguous.
yamtaddle 4y ago
Doubt it'll be a problem. Github's scaling challenges have more to do with the write volume than the read volume. Even a project as big and active as Blender shouldn't make Gitea sweat too much on relatively modest hardware. Anonymous clones and such will surely be https, so, quite cache-friendly. I wanna say Gitea has some config options for hosting static files (like releases) on s3-compatible storage then serving directly from there, so even that shouldn't be a problem. But I may be mis-remembering that part. And, again, a CDN or Varnish or whatever would be able to serve those files just as well from Gitea as from anywhere else.
- techknowlogick 4y ago
  Yup, that's right. Non-repo data (releases, avatars, packages, etc..) can all be hosted in s3-compat storage. If you are interested in some benchmarks that Blender did you can find them here: https://code.blender.org/2023/02/gitea-test-drive/ https://code.blender.org/2023/02/gitea-test-drive/ Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.
techknowlogick 4y ago
Hi, I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea. The reason why the project itself is still on GitHub is due to the massive amount of metadata (20K+ issues and PRs, and all the comments, reactions, and other metadata attached to them) around the project, and the ratelimiting that is slowing the export. All the non-primary repos have long since moved to gitea.com.
- vfclists 4y ago
  I start getting worried about projects when people say stuff like "I'm one of the members of the technical oversight committee of the Gitea project" How many committees are there and how many people on each committee?
  - jolheiser 4y ago
    The TOC is our owners team, but three are from the company and three are from the community. The reason to say it as a disclaimer is because we are going to have some bias.
- bastardoperator 4y ago
  You can use the API to export all of the metadata in a single request: https://docs.github.com/en/rest/migrations/orgs#start-an-organization-migration https://docs.github.com/en/rest/migrations/orgs#start-an-org... https://docs.github.com/en/rest/migrations/orgs#download-an-organization-migration-archive https://docs.github.com/en/rest/migrations/orgs#download-an-... Using a Github App you get 15K requests per hour. Even with pagination(100 records per request) you should be able to get at all of this data within a couple of hours. I suspect you lack reliable transformation tools to take github metadata and convert it to gitea versus the limitations of GitHub.
  - techknowlogick 4y ago
    Thanks, yeah. We have started to look at that alternative endpoint, but it doesn't have the exact data as the api, nor is it always in the exact same format. Although as I have run the migration myself several times to attempt to debug issues, it takes over 24hours due to ratelimiting. Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.

I wonder how Gitea will hold up to all the traffic? Even Gitea hosts their stuff on Github.

by cowmix 4y ago

I wonder how Gitea will hold up to all the traffic? Even Gitea hosts their stuff on Github.

codefined 4y ago

Gitea is self-hosted, so Blender will be running their own instance.

qwertox 4y ago
That's not what parent means. Gitea is an alternative to GitHub, yet they use GitHub for hosting their code.
- SV_BubbleTime 4y ago
  That kind of makes sense for disaster planning though. If Gitea discovered some massive CVE, and volentarily went dark to fix it... They would want their code and fix hosted somewhere else.
  - bawolff 4y ago
    Mirroring the code to github (or somewhere else) would solve that problem just as well. Not dogfooding your own product seems like a huge red flag to me.
    - throwaway894345 4y ago
      There's clearly the intention to self-host, they seem to be waiting for support to ingest data from a GitHub data export which seems pretty reasonable.
      - techknowlogick 4y ago
        Yes, this is exactly the case. Lots of data, and low ratelimits don't mix. All except for the main repo have already migrated to gitea.com Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.

stryan 4y ago

I don't believe Gitea is hosting on Github for traffic reasons but for feature completeness. Relevant issue is here[0]. [0]https://github.com/go-gitea/gitea/issues/1029 https://github.com/go-gitea/gitea/issues/1029

lucideer 4y ago
That's a great ticket, & also links to a really nice & insightful comment from one of the Fog Creek folk https://lobste.rs/s/gokjbo/gitea_1_1_0_released#c_dg9pwe https://lobste.rs/s/gokjbo/gitea_1_1_0_released#c_dg9pwe
- vidanay 4y ago
  I still dream of a world where FogBugz & Kiln gained the respect they deserved. Mercurial was pretty damn nice.
  - slondr 4y ago
    Was? It still is very nice.
aidenn0 4y ago
Looks like https://github.com/go-gitea/gitea/pull/18165 https://github.com/go-gitea/gitea/pull/18165 is the last missing checkbox...

tylersmith 4y ago

Gitea does not use Github because the software can't handle their load requirements, but because they started using GH before Gitea had all the features they needed. They are actively working on hosting Gitea on Gitea, with the last piece being an importer to move over all their GH data. Blender seems to have been ok manually migrating some things and abandoning other things, since they weren't likely to ever get a Phab->Gitea importer anyways. https://github.com/go-gitea/gitea/issues/1029 https://github.com/go-gitea/gitea/issues/1029

bawolff 4y ago
How much traffic could they possibly have? Quite frankly either gitea has absolutely terrible scalability or something else is off here. Edit: i misread the parent comment. My bad.
- yamtaddle 4y ago
  Huh? It isn't because of inability to handle load, that they host on GH. Gitea performs alright. It can serve surprisingly-heavy traffic from a potato-tier VM with a SQLite backend (assuming the host network can keep up with it, anyway). Not Github-scale (but Github has... a lot of traffic, including a whole lot of write traffic) but quite a bit, and that's with basically the worst-possible hosting situation for handling high load.
  - techknowlogick 4y ago
    That's correct. It is not due to load, gitea.com already has most of the projects repos migrated already, just not the main one due to ratelimits and exporting all the issues. Gitea.com has thousands of users, and multi-TB traffic monthly on a fairly modest setup. Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.
- tylersmith 4y ago
  Tbf my comment was phrased a bit ambiguously, but I couldn't think of anything less ambiguous.

yamtaddle 4y ago

Doubt it'll be a problem. Github's scaling challenges have more to do with the write volume than the read volume. Even a project as big and active as Blender shouldn't make Gitea sweat too much on relatively modest hardware. Anonymous clones and such will surely be https, so, quite cache-friendly. I wanna say Gitea has some config options for hosting static files (like releases) on s3-compatible storage then serving directly from there, so even that shouldn't be a problem. But I may be mis-remembering that part. And, again, a CDN or Varnish or whatever would be able to serve those files just as well from Gitea as from anywhere else.

techknowlogick 4y ago
Yup, that's right. Non-repo data (releases, avatars, packages, etc..) can all be hosted in s3-compat storage. If you are interested in some benchmarks that Blender did you can find them here: https://code.blender.org/2023/02/gitea-test-drive/ https://code.blender.org/2023/02/gitea-test-drive/ Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.

techknowlogick 4y ago

Hi, I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea. The reason why the project itself is still on GitHub is due to the massive amount of metadata (20K+ issues and PRs, and all the comments, reactions, and other metadata attached to them) around the project, and the ratelimiting that is slowing the export. All the non-primary repos have long since moved to gitea.com.

vfclists 4y ago
I start getting worried about projects when people say stuff like "I'm one of the members of the technical oversight committee of the Gitea project" How many committees are there and how many people on each committee?
- jolheiser 4y ago
  The TOC is our owners team, but three are from the company and three are from the community. The reason to say it as a disclaimer is because we are going to have some bias.
bastardoperator 4y ago
You can use the API to export all of the metadata in a single request: https://docs.github.com/en/rest/migrations/orgs#start-an-organization-migration https://docs.github.com/en/rest/migrations/orgs#start-an-org... https://docs.github.com/en/rest/migrations/orgs#download-an-organization-migration-archive https://docs.github.com/en/rest/migrations/orgs#download-an-... Using a Github App you get 15K requests per hour. Even with pagination(100 records per request) you should be able to get at all of this data within a couple of hours. I suspect you lack reliable transformation tools to take github metadata and convert it to gitea versus the limitations of GitHub.
- techknowlogick 4y ago
  Thanks, yeah. We have started to look at that alternative endpoint, but it doesn't have the exact data as the api, nor is it always in the exact same format. Although as I have run the migration myself several times to attempt to debug issues, it takes over 24hours due to ratelimiting. Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.