5 ms·

Seems like they could also let one request per hour through, say, and then serve up the rest a 429. Users trying to clone their project would hit an almost cer

by brnt 4y ago

Seems like they could also let one request per hour through, say, and then serve up the rest a 429. Users trying to clone their project would hit an almost certainly up to date Google cache and thus be happy and sr.ht save on pretty much all that traffic and thus be also happy.

Pet_Ant 4y ago
I like this because it feels like malicious compliance and it would allow Google to seemlessly continue working should Google figure out their caching situation. Plus it uses a rarely used HTTP status. Love it.
- lanstin 4y ago
  I have seen more 429, almost all from large cloud providers, a fair amount in the last 5 years. Never before then, I was used to making rate limiting an in-band part of the API explicitly. AWS S3 will give you 429s before giving you 503s when you hit it with a go program touching every object or every version in a giant bucket from a 56 core machine and a few hundred Go routine workers. It is a good signal: if you do exponential back off per worker on the 429s, you won’t see the 503s.
nouveaux 4y ago
I think the actions here are meant to generate attention. Google is acting in bad faith and needs to be called out.
m4lvin 4y ago
I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many different places. So they would have to count requests from the specific user agent and by that point it is again special purpose work and load on sr.ht (whereas general rate limiting per IP might be a good idea anyway).
- brnt 4y ago
  Per blogpost: > git clone requests with a GoModuleMirror User-Agent will receive a 429 So only one counter seems to be needed. Sure, it's doing work for an adversarial party, but it's not much extra work either.
  - m4lvin 4y ago
    Ah, not sure I understand then. Would it still need one counter, or rather a flag "has been cloned by google in last 24h" for each go module?
  - landhar 4y ago
    But since it looks like the proxy fleet is made of a large set of nodes and each one of them only keeps track of their own state, if sourcehut were to only keep one counter for the whole user-agent it would lead to a large number of nodes not being able to refresh their cache for potentially long periods of time (there's nothing that would guarantee for example that it's always a new node that gets to make the one request per hour). That's precisely why it's so unfair to ask the sourcehut team to try to come up with a solution to this problem: it's the very design of the proxy that Google put together that is causing this issue in the first place. And the sourcehut team has no control over their design. And to add insult to injury: when the sourcehut team offers recommendations on how to improve their implementation, Google responds with "it's too much work".
    - TeMPOraL 4y ago
      And to add a cherry on top, the only reason a single user-agent counter is even a possibility to discuss is because Drew made the Go team put a unique user-agent on the proxy in the early stage of this issue report.
- chlorion 4y ago
  The Linux firewall is designed specifically to be able to track and count packets by IP address and other metedata, and can do so extremely efficiently. Setting up an IP based rate limiting rule in nftables takes a few minutes at max, and I am not a professional sysadmin.
wut42 4y ago
Why the burden of making this google-specific rate-limiter should fall on Sourcehut ? They are a small team and a user-agent-per-repo specific limiter have rare uses. Google should definitely fix their shit, but when we see that a googler said "it would be a fair bit of extra work for us to read robots.txt", I understand the frustration on Sourcehut side, and banning an user agent is usually just a single line of configuration to add.

by brnt 4y ago

Pet_Ant 4y ago

I like this because it feels like malicious compliance and it would allow Google to seemlessly continue working should Google figure out their caching situation. Plus it uses a rarely used HTTP status. Love it.

lanstin 4y ago
I have seen more 429, almost all from large cloud providers, a fair amount in the last 5 years. Never before then, I was used to making rate limiting an in-band part of the API explicitly. AWS S3 will give you 429s before giving you 503s when you hit it with a go program touching every object or every version in a giant bucket from a 56 core machine and a few hundred Go routine workers. It is a good signal: if you do exponential back off per worker on the 429s, you won’t see the 503s.

nouveaux 4y ago

I think the actions here are meant to generate attention. Google is acting in bad faith and needs to be called out.

m4lvin 4y ago

I thought that too, but then sr.ht needs to keep state which IP made how many requests already, and as the post mentions google makes these requests from many different places. So they would have to count requests from the specific user agent and by that point it is again special purpose work and load on sr.ht (whereas general rate limiting per IP might be a good idea anyway).

brnt 4y ago
Per blogpost: > git clone requests with a GoModuleMirror User-Agent will receive a 429 So only one counter seems to be needed. Sure, it's doing work for an adversarial party, but it's not much extra work either.
- m4lvin 4y ago
  Ah, not sure I understand then. Would it still need one counter, or rather a flag "has been cloned by google in last 24h" for each go module?
- landhar 4y ago
  But since it looks like the proxy fleet is made of a large set of nodes and each one of them only keeps track of their own state, if sourcehut were to only keep one counter for the whole user-agent it would lead to a large number of nodes not being able to refresh their cache for potentially long periods of time (there's nothing that would guarantee for example that it's always a new node that gets to make the one request per hour). That's precisely why it's so unfair to ask the sourcehut team to try to come up with a solution to this problem: it's the very design of the proxy that Google put together that is causing this issue in the first place. And the sourcehut team has no control over their design. And to add insult to injury: when the sourcehut team offers recommendations on how to improve their implementation, Google responds with "it's too much work".
  - TeMPOraL 4y ago
    And to add a cherry on top, the only reason a single user-agent counter is even a possibility to discuss is because Drew made the Go team put a unique user-agent on the proxy in the early stage of this issue report.
chlorion 4y ago
The Linux firewall is designed specifically to be able to track and count packets by IP address and other metedata, and can do so extremely efficiently. Setting up an IP based rate limiting rule in nftables takes a few minutes at max, and I am not a professional sysadmin.

wut42 4y ago

Why the burden of making this google-specific rate-limiter should fall on Sourcehut ? They are a small team and a user-agent-per-repo specific limiter have rare uses. Google should definitely fix their shit, but when we see that a googler said "it would be a fair bit of extra work for us to read robots.txt", I understand the frustration on Sourcehut side, and banning an user agent is usually just a single line of configuration to add.