4 ms·

Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.

by radq 3y ago

p-e-w 3y ago
Indeed. Tens of thousands of requests per second from just 64 hosts? So they allow individual hosts to make hundreds of requests per second, sustained? That sounds crazy. Even for a burst limit hundreds per second would be extremely high.
- akiselev 3y ago
  Archive.org is a core utility for the web to the point where Wikipedia and many other sites would collapse without it in the sense that many if not most of their outbound links would be dead forever. I’m pretty sure it would even impact the US justice system [1]. Obviously judges aren’t going to have to worry about reasonable rate limits but if these DDoSes are rare, I’d much rather they dealt with them on a case by case basis. Without some complex dynamic rate limit that scales based on available compute and demand, rate limiting would be a blunt solution that will necessarily generate false positives. [1] https://www.theregister.com/2018/09/04/wayback_machine_legit/ https://www.theregister.com/2018/09/04/wayback_machine_legit...
  - Defletter 3y ago
    Probably not just the US justice system. I also guess that academia would take a hit too. Not every citation is of a published paper.
  - p-e-w 3y ago
    Rate limiting is (usually) done on a per-host basis. The only users who would be adversely affected are those who perform hundreds of requests per second from a single system. Virtually every major website has per-host inbound request limits. This is completely standard practice and Archive.org is the odd one here. And DDoS isn't the only concern. Legitimate users that run poorly written Python scripts that make insane numbers of requests can hog server resources, and rate limits with appropriate error messages linking to resources documenting efficient access patterns can improve the experience for everyone, and drastically cut costs for the service operators.
    - phkx 3y ago
      It‘s rather per public IP, such that e.g. behind a corporate proxy you may experience rate limiting even for regular use, just because you’re sharing a few IP addresses with a large number of users. Better hope that you get an increased quota for your external IPs in that case.
  - microtherion 3y ago
    > I’m pretty sure it would even impact the US justice system What an old fashioned concern! Don't you know that the US justice system uses ChatGPT as an archive retrieval system nowadays? /s
hackernewds 3y ago
Precisely. What kinds of legitimate processes would require this kind of utility from Archive.org?
deleted 3y ago
[deleted]

Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.

by radq 3y ago

Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.

p-e-w 3y ago

Indeed. Tens of thousands of requests per second from just 64 hosts? So they allow individual hosts to make hundreds of requests per second, sustained? That sounds crazy. Even for a burst limit hundreds per second would be extremely high.

akiselev 3y ago
Archive.org is a core utility for the web to the point where Wikipedia and many other sites would collapse without it in the sense that many if not most of their outbound links would be dead forever. I’m pretty sure it would even impact the US justice system [1]. Obviously judges aren’t going to have to worry about reasonable rate limits but if these DDoSes are rare, I’d much rather they dealt with them on a case by case basis. Without some complex dynamic rate limit that scales based on available compute and demand, rate limiting would be a blunt solution that will necessarily generate false positives. [1] https://www.theregister.com/2018/09/04/wayback_machine_legit/ https://www.theregister.com/2018/09/04/wayback_machine_legit...
- Defletter 3y ago
  Probably not just the US justice system. I also guess that academia would take a hit too. Not every citation is of a published paper.
- p-e-w 3y ago
  Rate limiting is (usually) done on a per-host basis. The only users who would be adversely affected are those who perform hundreds of requests per second from a single system. Virtually every major website has per-host inbound request limits. This is completely standard practice and Archive.org is the odd one here. And DDoS isn't the only concern. Legitimate users that run poorly written Python scripts that make insane numbers of requests can hog server resources, and rate limits with appropriate error messages linking to resources documenting efficient access patterns can improve the experience for everyone, and drastically cut costs for the service operators.
  - phkx 3y ago
    It‘s rather per public IP, such that e.g. behind a corporate proxy you may experience rate limiting even for regular use, just because you’re sharing a few IP addresses with a large number of users. Better hope that you get an increased quota for your external IPs in that case.
- microtherion 3y ago
  > I’m pretty sure it would even impact the US justice system What an old fashioned concern! Don't you know that the US justice system uses ChatGPT as an archive retrieval system nowadays? /s

hackernewds 3y ago

Precisely. What kinds of legitimate processes would require this kind of utility from Archive.org?

deleted 3y ago

[deleted]