3 ms·
True. But it still wastes your server resources, right? And it's sad that you have to accept that as part of the "cost" of hosting a site ...
by thisislife2 5mo ago
True. But it still wastes your server resources, right? And it's sad that you have to accept that as part of the "cost" of hosting a site ...
- ndriscoll 5mo agoWhat resources are you concerned about? An n100 minipc should be capable of serving something like a blog at 20k+ requests/second (or saturating its network).
- deleted 5mo ago[deleted]
- thisislife2 4mo agoSure. But 25,000 (bot hits / day) x 50kB (static webpage) = 1.25 GB / day / page x 30 (days) = 37.5 GB of data transfer / month / page. And that's assuming a static resource - if any of the resource is dynamically generated by the server, there's a CPU and memory cost too. Overall, even if you treat the impact on the server as "negligible", it's still an unnecessary waste of resource was the point I was trying to convey. Here's a more real-world projection of the cost and server impact - The Bandwidth Cost of AI Crawlers: What Scraping Really Costs Publishers - https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost