2 ms·

They are not a problem unless you "believe" it is a problem. I estimate around 20-25K hits to my website from bots per day and I have all cloudflare protections

by csomar 4mo ago

They are not a problem unless you "believe" it is a problem. I estimate around 20-25K hits to my website from bots per day and I have all cloudflare protections disabled. Any decently optimized server should be able to easily handle that. (it's roughly 1 request every 3 seconds).

specialp 4mo ago
Yes and that is just the bot background radiation of the internet. I run a primary source of information site and these botnets are aggressive to a DDOS level. All to do some sort of scraping. Because they have sophisticated enough tactics to DDOS us if they wanted to. However I am not sure their objective as they have wasted enough of our resources to have scraped all our content 1000s of times over. That 25k traffic is a couple of minutes for us. And that adds up. 80-90pct of our traffic is this
thisislife2 4mo ago
True. But it still wastes your server resources, right? And it's sad that you have to accept that as part of the "cost" of hosting a site ...
- ndriscoll 4mo ago
  What resources are you concerned about? An n100 minipc should be capable of serving something like a blog at 20k+ requests/second (or saturating its network).
  - deleted 4mo ago
    [deleted]
  - thisislife2 4mo ago
    Sure. But 25,000 (bot hits / day) x 50kB (static webpage) = 1.25 GB / day / page x 30 (days) = 37.5 GB of data transfer / month / page. And that's assuming a static resource - if any of the resource is dynamically generated by the server, there's a CPU and memory cost too. Overall, even if you treat the impact on the server as "negligible", it's still an unnecessary waste of resource was the point I was trying to convey. Here's a more real-world projection of the cost and server impact - The Bandwidth Cost of AI Crawlers: What Scraping Really Costs Publishers - https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost
HWR_14 4mo ago
Assuming that the bots aren't repackaging your content and preventing users from seeing your blog by serving that content to them first.

by csomar 4mo ago

specialp 4mo ago

Yes and that is just the bot background radiation of the internet. I run a primary source of information site and these botnets are aggressive to a DDOS level. All to do some sort of scraping. Because they have sophisticated enough tactics to DDOS us if they wanted to. However I am not sure their objective as they have wasted enough of our resources to have scraped all our content 1000s of times over. That 25k traffic is a couple of minutes for us. And that adds up. 80-90pct of our traffic is this

thisislife2 4mo ago

True. But it still wastes your server resources, right? And it's sad that you have to accept that as part of the "cost" of hosting a site ...

ndriscoll 4mo ago
What resources are you concerned about? An n100 minipc should be capable of serving something like a blog at 20k+ requests/second (or saturating its network).
- deleted 4mo ago
  [deleted]
- thisislife2 4mo ago
  Sure. But 25,000 (bot hits / day) x 50kB (static webpage) = 1.25 GB / day / page x 30 (days) = 37.5 GB of data transfer / month / page. And that's assuming a static resource - if any of the resource is dynamically generated by the server, there's a CPU and memory cost too. Overall, even if you treat the impact on the server as "negligible", it's still an unnecessary waste of resource was the point I was trying to convey. Here's a more real-world projection of the cost and server impact - The Bandwidth Cost of AI Crawlers: What Scraping Really Costs Publishers - https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost

HWR_14 4mo ago

Assuming that the bots aren't repackaging your content and preventing users from seeing your blog by serving that content to them first.