3 ms·

This headline sucks. It suggest that AI crawlers' bad behavior isn't intentional. Robots.txt has existed for ages, and crawler operators know exactly how to pla

This headline sucks. It suggest that AI crawlers' bad behavior isn't intentional. Robots.txt has existed for ages, and crawler operators know exactly how to play nice. They simply choose not to. Media outlets need to stop attributing ignorance or incompetence to what is clearly willful action. These megacorps need to be held accountable, even on the little things.

soco 2y ago
Technically it's true - they have not learned, but because nobody taught them to. Because nobody slapped any over-reaching wrist, ever, and if somebody even just thinks about, then it's all "damn EU bureaucrats".
mrweasel 2y ago
> It suggest that AI crawlers' bad behaviour isn't intentional. The robots.txt is only part of the problem, simply overloading sites is another. Maybe you don't really mind that your site is getting scraped, but you do mind that it crashes due to poorly written scrapers. It was always understood that your scrappers should limit their load to something that a site can handle. Google has always excelled at this, I've never experienced the GoogleBot being an issue, nor the BingBot. The developers who work at the AI companies however are less talented and less careering. If their bot crashes a site, meeeh, try again later. I don't think they DDoS sites deliberately, they just don't have the skills to fix their shitty scrapper. If we want the problem to go away, ask AWS, Azure, GCP, Alibaba Cloud and others for a way to report bad behaviour. Make their hosting providers take action.
- danaris 2y ago
  I will go one step further: It doesn't matter whether they have the skills or not, because they don't care. As long as they can siphon up enough data to increase their revenue or stock price by 0.001¢, they're perfectly content to leave small independent websites shattered and unusable under the weight of their aggressive, unrelenting scraper bots. I'm firmly of the opinion that they should all be considered hostile, predatory companies, regardless of what supposed "value" they bring. Train your river-guzzling models with data that's paid for or given voluntarily and then we can talk about your "value".
freeAgent 2y ago
I’m not sure whether you’re familiar with the Register or not, but they definitely get it and aren’t being nice. They’re being sarcastic.

soco 2y ago

Technically it's true - they have not learned, but because nobody taught them to. Because nobody slapped any over-reaching wrist, ever, and if somebody even just thinks about, then it's all "damn EU bureaucrats".

mrweasel 2y ago

> It suggest that AI crawlers' bad behaviour isn't intentional. The robots.txt is only part of the problem, simply overloading sites is another. Maybe you don't really mind that your site is getting scraped, but you do mind that it crashes due to poorly written scrapers. It was always understood that your scrappers should limit their load to something that a site can handle. Google has always excelled at this, I've never experienced the GoogleBot being an issue, nor the BingBot. The developers who work at the AI companies however are less talented and less careering. If their bot crashes a site, meeeh, try again later. I don't think they DDoS sites deliberately, they just don't have the skills to fix their shitty scrapper. If we want the problem to go away, ask AWS, Azure, GCP, Alibaba Cloud and others for a way to report bad behaviour. Make their hosting providers take action.

danaris 2y ago
I will go one step further: It doesn't matter whether they have the skills or not, because they don't care. As long as they can siphon up enough data to increase their revenue or stock price by 0.001¢, they're perfectly content to leave small independent websites shattered and unusable under the weight of their aggressive, unrelenting scraper bots. I'm firmly of the opinion that they should all be considered hostile, predatory companies, regardless of what supposed "value" they bring. Train your river-guzzling models with data that's paid for or given voluntarily and then we can talk about your "value".

freeAgent 2y ago

I’m not sure whether you’re familiar with the Register or not, but they definitely get it and aren’t being nice. They’re being sarcastic.