4 ms·
1. I fear you may be underestimating the volume of bot traffic websites are now receiving. I recommend reading this article to get an idea of the scale of the p
by whoisyc 1y ago
1. I fear you may be underestimating the volume of bot traffic websites are now receiving. I recommend reading this article to get an idea of the scale of the problem: https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ https://thelibre.news/foss-infrastructure-is-under-attack-by...
2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source project with binary download links for all past versions for multiple platforms, each one being a multi megabyte blob, and a scraper starts hitting these links, you will run into bandwidth problems very quickly, unless you live in a utopia where bandwidth is free.
3. You are underestimating the difficulty of caching dynamic pages. Cache invalidation is hard, they say. One notably problematic example is Git blames. So far I am not aware of any existing solution for caching blames, and jury rigging your own will likely not be any easier than the “solution” explored in the TFA.
- busymom0 1y agoShouldn't such big blogs be put on something like CloudFlare R2 or BackBlaze or even S3 with their caching in front? Instead of having your server handle such file downloads.
- phantomathkg 1y agoCache also cost money as well. Nothing is free.
- deleted 1y ago[deleted]
- charcircuit 1y ago>you live in a utopia where bandwidth is free. It's called peering agreements and they are very common. There's a reason social media and sites like YouTube, Twitch, TikTok don't immediately go out of business. The bandwidth is free for most users.
- chii 1y agothere's only a handful of entities in the world that are capable of peering. Most people have to pay for their bandwidth.
- charcircuit 1y agoIt can be done by whoever is providing you internet connectivity. Not everywhere adds extra charges for bandwith.
- hartator 1y ago> 2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source project with binary download links for all past versions for multiple platforms, each one being a multi megabyte blob, and a scraper starts hitting these links, you will run into bandwidth problems very quickly, unless you live in a utopia where bandwidth is free. All of this is (and should) cached on a cdn. You can go 1000 QPS on this in that config.
- tdeck 1y agoI think a person should be able to set up a small website on a niche topic without ever having to set up a CDN. Until very recently this was the case, so it's sad to see that simplicity go away purely to satisfy the data needs of shady scrapers.