3 ms·
One million hits in 24 hours is only 11 TPS, if that's causing issues, then Anthropic isn't the problem, your application or hosting is.
by iLoveOncall 2y ago
One million hits in 24 hours is only 11 TPS, if that's causing issues, then Anthropic isn't the problem, your application or hosting is.
- choppaface 2y agoIf 90% of the content doesn’t get daily attention then the spider can blow out the caches and cause inordinate costs.
- ericholscher 2y agoYea, this happened to us, and you can see the cache profile of a crawler here: https://fosstodon.org/@readthedocs/112877477202118215 https://fosstodon.org/@readthedocs/112877477202118215 A bunch of the traffic hit the origin.
- sitkack 2y agoNot trying to splain you, but if you can turn off caching for one hit wonders, everything would be greatly improved.
- iLoveOncall 2y agoCaches? You could not cache anything and 11 TPS would be just fine to run on a gameboy.
- xboxnolifes 2y agoNot if that content is multiple gigbytes of data per file.
- iLoveOncall 2y agoWhich isn't the case for the extreme majority of all websites? We're talking about ifixit.com here, a random guide page (so quite heavy in pictures) is only 6.8MB. I think they'll survive this massive DDOS of 76MBPS. Same in the case of ReadTheDocs which was linked above, 10TB of traffic over 6 days, an incredible 19.29MBPS sustained. Has humanity even built technology powerful enough to handle this massive assault?!