3 ms·
> 2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source
by hartator 1y ago
> 2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source project with binary download links for all past versions for multiple platforms, each one being a multi megabyte blob, and a scraper starts hitting these links, you will run into bandwidth problems very quickly, unless you live in a utopia where bandwidth is free.
All of this is (and should) cached on a cdn. You can go 1000 QPS on this in that config.
- tdeck 1y agoI think a person should be able to set up a small website on a niche topic without ever having to set up a CDN. Until very recently this was the case, so it's sad to see that simplicity go away purely to satisfy the data needs of shady scrapers.