3 ms·

> thousounds of requests an hour from bots That's not much for any modern server so I genuinely don't understand the frustration. I'm pretty certain gitea shou

> thousounds of requests an hour from bots That's not much for any modern server so I genuinely don't understand the frustration. I'm pretty certain gitea should be able to handle thousands of read requests per minute (not per hour) without even breaking a sweat.

q3k 1y ago
Serving file content/diff requests from gitea/forgejo is quite expensive computationally. And these bots tend to tarpit themselves when they come across eg. a Linux repo mirror. https://social.hackerspace.pl/@q3k/114358881508370524 https://social.hackerspace.pl/@q3k/114358881508370524
- rollcat 1y ago
  I think at this point every self-hosted forge should block diffs from anonymous users. Also: Anubis and go-away, but also: some people are on old browsers or underpowered computers.
- diggan 1y ago
  > Serving file content/diff requests from gitea/forgejo is quite expensive computationally One time, sure. But unauthenticated requests would surely be cached, authenticated ones skip the cache (just like HN works :) ), as most internet-facing websites end up using this pattern.
  - q3k 1y ago
    You can't feasibly cache large reposotories' diffs/content-at-version without reimplementing a significant part of git - this stuff is extremely high cardinality and you'd just constantly thrash the cache the moment someone does a BFS/DFS through available links (as these bots tend to do).
  - Sesse__ 1y ago
    There are _lots_ of objects in a large git repository. E.g., I happen to have a fork of VLC lying around. VLC has 70k+ commits (on that version). Each commit has about 10k files. The typical AI crawler wants, for every commit, to download every file (so 700M objects), every tarball (70k+ .tar.gz files), and the blame layer of every file (700M objects, where blame has to look back on average 35k commits). Plus some more. Saying “just cache this” is not sustainable. And this is only one repository; the only reasonable way to deal with this is some sort of traffic mitigation, you cannot just deal with the traffic as the happy path.
p3rls 1y ago
and this is how the entire web was turned into wordpress slop and cryptoscams
hinkley 1y ago
We were seeing over a million hits per hour from bots and I agree with GP. It’s fucking out of control. And it’s 100x worse at least if you sell vanity URLs, because the good bots cannot tell that they’re sending you 100 simultaneous requests by throttling on one domain and hitting five others instead.

q3k 1y ago

Serving file content/diff requests from gitea/forgejo is quite expensive computationally. And these bots tend to tarpit themselves when they come across eg. a Linux repo mirror. https://social.hackerspace.pl/@q3k/114358881508370524 https://social.hackerspace.pl/@q3k/114358881508370524

rollcat 1y ago
I think at this point every self-hosted forge should block diffs from anonymous users. Also: Anubis and go-away, but also: some people are on old browsers or underpowered computers.
diggan 1y ago
> Serving file content/diff requests from gitea/forgejo is quite expensive computationally One time, sure. But unauthenticated requests would surely be cached, authenticated ones skip the cache (just like HN works :) ), as most internet-facing websites end up using this pattern.
- q3k 1y ago
  You can't feasibly cache large reposotories' diffs/content-at-version without reimplementing a significant part of git - this stuff is extremely high cardinality and you'd just constantly thrash the cache the moment someone does a BFS/DFS through available links (as these bots tend to do).
- Sesse__ 1y ago
  There are _lots_ of objects in a large git repository. E.g., I happen to have a fork of VLC lying around. VLC has 70k+ commits (on that version). Each commit has about 10k files. The typical AI crawler wants, for every commit, to download every file (so 700M objects), every tarball (70k+ .tar.gz files), and the blame layer of every file (700M objects, where blame has to look back on average 35k commits). Plus some more. Saying “just cache this” is not sustainable. And this is only one repository; the only reasonable way to deal with this is some sort of traffic mitigation, you cannot just deal with the traffic as the happy path.

p3rls 1y ago

and this is how the entire web was turned into wordpress slop and cryptoscams

hinkley 1y ago

We were seeing over a million hits per hour from bots and I agree with GP. It’s fucking out of control. And it’s 100x worse at least if you sell vanity URLs, because the good bots cannot tell that they’re sending you 100 simultaneous requests by throttling on one domain and hitting five others instead.