4 ms·
I've had a number of content sites I've shut down a few sites in the last few days because of the toll these aggressive AI bots. Alexa seems like the worst.
by ipaddr 2y ago
I've had a number of content sites
I've shut down a few sites in the last few days because of the toll these aggressive AI bots. Alexa seems like the worst.
These were created 20 years ago and updated over the years. I use to get traffic but that's been slowed to 1,000 or less legitimate visitors over the last year. But now I have to deal with server down emails caused by these aggressive bots that don't respect the robots file.
- Aurornis 2y ago> Alexa seems like the worst. Many of the bots disguise themselves as coming from Amazon or other big company. Amazon has a page where you can check some details to see if it’s really their crawler or someone imitating it.
- deleted 2y ago[deleted]
- spookie 2y agoyup, actually most that I've seen are impersonating amazon
- svelle 2y agoIt's funny how every time this topic comes up someone says "I've had this happen and x is the worst" With x being any of the big AI providers. Just a couple minutes ago I read the same in another thread and it was Anthropic. A couple weeks back it was Meta. My conclusion is that they're all equally terrible then.
- Aurornis 2y agoAll of the crawlers present themselves as being from one of the major companies, even if they’re not. Setting user-agent headers is easy.
- cyanydeez 2y agoat the same time, all the AI providers have some kind of web based AI agent, so let snot pretend they're crafting their services in care of other peoples websites.
- CaptainFever 2y agoI highly doubt that people are using AI agent features so frequently and so concentrated-ly that it brings down websites.
- ipaddr 2y agoI agree they are all creating negative value for site owners. In my personal experience this week blocking Amazon solved my server overload issue.