4 ms·

I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many d

by m4lvin 4y ago

I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many different places. So they would have to count requests from the specific user agent and by that point it is again special purpose work and load on sr.ht (whereas general rate limiting per IP might be a good idea anyway).

brnt 4y ago
Per blogpost: > git clone requests with a GoModuleMirror User-Agent will receive a 429 So only one counter seems to be needed. Sure, it's doing work for an adversarial party, but it's not much extra work either.
- m4lvin 4y ago
  Ah, not sure I understand then. Would it still need one counter, or rather a flag "has been cloned by google in last 24h" for each go module?
- landhar 4y ago
  But since it looks like the proxy fleet is made of a large set of nodes and each one of them only keeps track of their own state, if sourcehut were to only keep one counter for the whole user-agent it would lead to a large number of nodes not being able to refresh their cache for potentially long periods of time (there's nothing that would guarantee for example that it's always a new node that gets to make the one request per hour). That's precisely why it's so unfair to ask the sourcehut team to try to come up with a solution to this problem: it's the very design of the proxy that Google put together that is causing this issue in the first place. And the sourcehut team has no control over their design. And to add insult to injury: when the sourcehut team offers recommendations on how to improve their implementation, Google responds with "it's too much work".
  - TeMPOraL 4y ago
    And to add a cherry on top, the only reason a single user-agent counter is even a possibility to discuss is because Drew made the Go team put a unique user-agent on the proxy in the early stage of this issue report.
chlorion 4y ago
The Linux firewall is designed specifically to be able to track and count packets by IP address and other metedata, and can do so extremely efficiently. Setting up an IP based rate limiting rule in nftables takes a few minutes at max, and I am not a professional sysadmin.

by m4lvin 4y ago

brnt 4y ago

Per blogpost: > git clone requests with a GoModuleMirror User-Agent will receive a 429 So only one counter seems to be needed. Sure, it's doing work for an adversarial party, but it's not much extra work either.

m4lvin 4y ago
Ah, not sure I understand then. Would it still need one counter, or rather a flag "has been cloned by google in last 24h" for each go module?
landhar 4y ago
But since it looks like the proxy fleet is made of a large set of nodes and each one of them only keeps track of their own state, if sourcehut were to only keep one counter for the whole user-agent it would lead to a large number of nodes not being able to refresh their cache for potentially long periods of time (there's nothing that would guarantee for example that it's always a new node that gets to make the one request per hour). That's precisely why it's so unfair to ask the sourcehut team to try to come up with a solution to this problem: it's the very design of the proxy that Google put together that is causing this issue in the first place. And the sourcehut team has no control over their design. And to add insult to injury: when the sourcehut team offers recommendations on how to improve their implementation, Google responds with "it's too much work".
- TeMPOraL 4y ago
  And to add a cherry on top, the only reason a single user-agent counter is even a possibility to discuss is because Drew made the Go team put a unique user-agent on the proxy in the early stage of this issue report.

chlorion 4y ago

The Linux firewall is designed specifically to be able to track and count packets by IP address and other metedata, and can do so extremely efficiently. Setting up an IP based rate limiting rule in nftables takes a few minutes at max, and I am not a professional sysadmin.