5 ms·

To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "

To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.

GuinansEyebrows 2y ago
I think there’s room to argue that that’s a matter of perspective. The intent of the attacker is not that important to the people being attacked.
joepie91_ 2y ago
> The crawlers could just as well be search engine startups. And yet they are not. So what does that tell you?
soco 2y ago
Correct, they could - but they are not. This is about the unaccountability, and if I was charitable (which I'm not) I'd add also incompetence, of the techbros leading the AI giants. Are we still expecting anything like "ethics"? I hope the few engineers reading HN will still have some, but the higher you go the foreign the concept gets.
- FeepingCreature 2y ago
  Sure, I just think focusing it on AI companies misses the reason this happens. It's not an "AI company problem", it's a "tech company problem". It just happens that AI companies are the tech companies that externalize their incompetence with crawlers at this point in time.
MrArthegor 2y ago
Yes, any crawler or some traffic peak can do this if your infrastructure is not well engineered
- tecleandor 2y ago
  You're getting it the wrong way. It's: Any crawler that's not well engineered, that doesn't follow robots.txt, that fakes its User Agent, that doesn't allow you to contact hem, that fetches content an indiscriminate number of times, repeatedly, all day long... can do this to your infrastructure unless you're a giant. What these crawlers are doing is akin to DDoS attacks.
- mrweasel 2y ago
  Please do explain how you'd engineer a site to deal with barrage of poorly written scrapers descending upon it. After you've done geo-ip routing, implemented various levels of caching, separated read/write traffic and bought an ever increasing amount of bandwidth, what is there left to do? You could also get CloudFlare, or some other CDN, but depending on your size that might not be within your budget. I don't get why the rest of the internet should subsidize these AI companies. They're not profitable and live of venture capital and increase the operation costs of everyone else.
johnnyanmac 2y ago
In the same way a drunk driver isn't deliberately trying to run over pedestrians, I suppose. I think gross negligence is in many ways worse than malice. A malicious actor can at least be somewhat reasoned with.
mdp2021 2y ago
> has nothing to do with AI Surely has very little to do with "intelligence".
GuinansEyebrows 2y ago
The only way to stop a bad AI with a gun is a good AI with a gun?
insane_dreamer 2y ago
It’s not incompetence. These large AI companies have (or can hire) the competency to engineer proper crawlers. This is deliberate due to lack of accountability and “who’s gonna stop them?”
Vegenoid 2y ago
> "FOSS sites damaged by extreme incompetence and unaccountability." Given this info, the natural next question is “who is doing the harm?” The answer is “AI companies”. Most people would now view the situation as having a lot to do with AI companies.
- FeepingCreature 2y ago
  As a matter of fact, yes. As a matter of cause, no. Being an AI company doesn't make these companies especially incompetent; rather, this is the normal tech company level of incompetence, and being an AI company causes it to externalize via crawling. Bing used to do the same thing. (It might still do it, I just haven't heard about it in a while.)

GuinansEyebrows 2y ago

I think there’s room to argue that that’s a matter of perspective. The intent of the attacker is not that important to the people being attacked.

joepie91_ 2y ago

> The crawlers could just as well be search engine startups. And yet they are not. So what does that tell you?

soco 2y ago

Correct, they could - but they are not. This is about the unaccountability, and if I was charitable (which I'm not) I'd add also incompetence, of the techbros leading the AI giants. Are we still expecting anything like "ethics"? I hope the few engineers reading HN will still have some, but the higher you go the foreign the concept gets.

FeepingCreature 2y ago
Sure, I just think focusing it on AI companies misses the reason this happens. It's not an "AI company problem", it's a "tech company problem". It just happens that AI companies are the tech companies that externalize their incompetence with crawlers at this point in time.

MrArthegor 2y ago

Yes, any crawler or some traffic peak can do this if your infrastructure is not well engineered

tecleandor 2y ago
You're getting it the wrong way. It's: Any crawler that's not well engineered, that doesn't follow robots.txt, that fakes its User Agent, that doesn't allow you to contact hem, that fetches content an indiscriminate number of times, repeatedly, all day long... can do this to your infrastructure unless you're a giant. What these crawlers are doing is akin to DDoS attacks.
mrweasel 2y ago
Please do explain how you'd engineer a site to deal with barrage of poorly written scrapers descending upon it. After you've done geo-ip routing, implemented various levels of caching, separated read/write traffic and bought an ever increasing amount of bandwidth, what is there left to do? You could also get CloudFlare, or some other CDN, but depending on your size that might not be within your budget. I don't get why the rest of the internet should subsidize these AI companies. They're not profitable and live of venture capital and increase the operation costs of everyone else.

johnnyanmac 2y ago

In the same way a drunk driver isn't deliberately trying to run over pedestrians, I suppose. I think gross negligence is in many ways worse than malice. A malicious actor can at least be somewhat reasoned with.

mdp2021 2y ago

> has nothing to do with AI Surely has very little to do with "intelligence".

GuinansEyebrows 2y ago

The only way to stop a bad AI with a gun is a good AI with a gun?

insane_dreamer 2y ago

It’s not incompetence. These large AI companies have (or can hire) the competency to engineer proper crawlers. This is deliberate due to lack of accountability and “who’s gonna stop them?”

Vegenoid 2y ago

> "FOSS sites damaged by extreme incompetence and unaccountability." Given this info, the natural next question is “who is doing the harm?” The answer is “AI companies”. Most people would now view the situation as having a lot to do with AI companies.

FeepingCreature 2y ago
As a matter of fact, yes. As a matter of cause, no. Being an AI company doesn't make these companies especially incompetent; rather, this is the normal tech company level of incompetence, and being an AI company causes it to externalize via crawling. Bing used to do the same thing. (It might still do it, I just haven't heard about it in a while.)