3 ms·

Hi, I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea. The reason why the project itself is sti

Hi, I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea. The reason why the project itself is still on GitHub is due to the massive amount of metadata (20K+ issues and PRs, and all the comments, reactions, and other metadata attached to them) around the project, and the ratelimiting that is slowing the export. All the non-primary repos have long since moved to gitea.com.

vfclists 4y ago
I start getting worried about projects when people say stuff like "I'm one of the members of the technical oversight committee of the Gitea project" How many committees are there and how many people on each committee?
- jolheiser 4y ago
  The TOC is our owners team, but three are from the company and three are from the community. The reason to say it as a disclaimer is because we are going to have some bias.
bastardoperator 4y ago
You can use the API to export all of the metadata in a single request: https://docs.github.com/en/rest/migrations/orgs#start-an-organization-migration https://docs.github.com/en/rest/migrations/orgs#start-an-org... https://docs.github.com/en/rest/migrations/orgs#download-an-organization-migration-archive https://docs.github.com/en/rest/migrations/orgs#download-an-... Using a Github App you get 15K requests per hour. Even with pagination(100 records per request) you should be able to get at all of this data within a couple of hours. I suspect you lack reliable transformation tools to take github metadata and convert it to gitea versus the limitations of GitHub.
- techknowlogick 4y ago
  Thanks, yeah. We have started to look at that alternative endpoint, but it doesn't have the exact data as the api, nor is it always in the exact same format. Although as I have run the migration myself several times to attempt to debug issues, it takes over 24hours due to ratelimiting. Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.
  - bastardoperator 4y ago
    Create more GitHub Apps. If you're hitting rate limits you can double your capacity by adding a new app and using round robin to cycle through requests/tokens. If you're using a PAT, you only have 5k requests and will bottleneck quickly.
    - techknowlogick 4y ago
      I.... uhh... can't say I tried that... as it's against the TOS... cough All joking aside, they have banned accounts from other projects attempting to do the same, so if I were to do that (and I'm not saying that I would), I'd want to make sure that the exporter is solid prior to risking any account.
      - bastardoperator 4y ago
        It's not against the TOS, the documentation is very clear: "Keep these ideas in mind when creating GitHub Apps: - A user or organization can own up to 100 GitHub Apps." https://docs.github.com/en/developers/apps/getting-started-with-apps/about-apps https://docs.github.com/en/developers/apps/getting-started-w... Which is 1.5 million API requests per org per hour.

vfclists 4y ago

I start getting worried about projects when people say stuff like "I'm one of the members of the technical oversight committee of the Gitea project" How many committees are there and how many people on each committee?

jolheiser 4y ago
The TOC is our owners team, but three are from the company and three are from the community. The reason to say it as a disclaimer is because we are going to have some bias.

bastardoperator 4y ago

You can use the API to export all of the metadata in a single request: https://docs.github.com/en/rest/migrations/orgs#start-an-organization-migration https://docs.github.com/en/rest/migrations/orgs#start-an-org... https://docs.github.com/en/rest/migrations/orgs#download-an-organization-migration-archive https://docs.github.com/en/rest/migrations/orgs#download-an-... Using a Github App you get 15K requests per hour. Even with pagination(100 records per request) you should be able to get at all of this data within a couple of hours. I suspect you lack reliable transformation tools to take github metadata and convert it to gitea versus the limitations of GitHub.

techknowlogick 4y ago
Thanks, yeah. We have started to look at that alternative endpoint, but it doesn't have the exact data as the api, nor is it always in the exact same format. Although as I have run the migration myself several times to attempt to debug issues, it takes over 24hours due to ratelimiting. Disclaimer: I'm one of the members of the technical oversight committee of the Gitea project, and am employed to work on Gitea.
- bastardoperator 4y ago
  Create more GitHub Apps. If you're hitting rate limits you can double your capacity by adding a new app and using round robin to cycle through requests/tokens. If you're using a PAT, you only have 5k requests and will bottleneck quickly.
  - techknowlogick 4y ago
    I.... uhh... can't say I tried that... as it's against the TOS... cough All joking aside, they have banned accounts from other projects attempting to do the same, so if I were to do that (and I'm not saying that I would), I'd want to make sure that the exporter is solid prior to risking any account.
    - bastardoperator 4y ago
      It's not against the TOS, the documentation is very clear: "Keep these ideas in mind when creating GitHub Apps: - A user or organization can own up to 100 GitHub Apps." https://docs.github.com/en/developers/apps/getting-started-with-apps/about-apps https://docs.github.com/en/developers/apps/getting-started-w... Which is 1.5 million API requests per org per hour.