4 ms·

Sadly it hard to tell if this is an actual DDoS attack, or scrappers descending on the site. It all looks very similar. The search engines always seemed happy

Sadly it hard to tell if this is an actual DDoS attack, or scrappers descending on the site. It all looks very similar. The search engines always seemed happy to announce that they are in fact GoogleBot/BingBot/Yahoo/whatever and frequently provided you with their expected IP ranges. The modern companies, mostly AI companies, seems to be more interested in flying under the radar, and have less respect for the internet infrastructure at a whole. So we're now at a point where I can't tell if it's an ill willed DDoS attack or just shitty AI startup number 7 reloading training data.

piokoch 2y ago
Yes, search engines were not hiding, as it was website owner interest involved here as well - without those search bots their sites would not be indexed and searchable in the Internet. So there was kind of win-win situation, in most typical cases at least, as for instance publishers complained about deep links, etc. because their ads revenue was hurt. AI scrapping bots provide zero value for sites owners.
- Valodim 2y ago
  Is this really true? If I have a marketing website for a product, isn't it in my interest to have that marketing incorporated in AI models?
jeroenhd 2y ago
> The modern companies, mostly AI companies, seems to be more interested in flying under the radar, and have less respect for the internet infrastructure at a whole I think that makes a lot of sense. Google's goal is (or perhaps used to be) providing a network of links. The more they scrape you, the more visitors you may end up receiving, and the better your website performs (monetarily, or just in terms of providing information to the world). With AI companies, the goal is to consume and replace. In their best case scenario, your website will never receive a visitor again. You won't get anything in return for providing content to AI companies. That means there's no reason for website administrators to permit the good ones, especially for people who use subscriptions or ads to support their website operating costs.
- eadmund 2y ago
  > With AI companies, the goal is to consume and replace. I don’t think that’s really true. The AI companies’ goal is to consume and create something else. > You won't get anything in return for providing content to AI companies. That was the original problem with websites in general, and the ‘solution’ was ads. It would be really, really cool if the thing which finally makes micropayments happen is AI. And then we humans could use micropayments too. Of course, the worst of both worlds would be micropayments and ads.
  - mrweasel 2y ago
    You can have non-commercial websites. Plenty of people have blogs or personal websites, sites that support a business, sites where you already pay and in this case it was the ScummVM website, an open source project. A lot of those sites are at risk of being made irrelevant by AI companies who really don't give a shit about your motivations for doing something for free. If their crawler kills your site and their LLM steals views by regurgitation answers based on your work, so be it, you served your purpose. If you want to talk payment: Ask the AI companies to pay you when they generate an answer based on your work, a license fee. That will kill their business model pretty quickly.
    - eadmund 2y ago
      > … their crawler kills your site and their LLM steals views by regurgitation answers based on your work How is that different from a human being reading my underwater basket weaving site and starting his own, ‘stealing’ ‘my’ views? Or a thousand human beings out of the billions on Earth doing the same thing?
      - mystified5016 2y ago
        Because that other person doesn't have the unilateral and un-appealable power to put their website in place of yours in search engines.
      - poincaredisk 2y ago
philipwhiuk 2y ago
It's DDoS either way even if it's not an attack.

piokoch 2y ago

Yes, search engines were not hiding, as it was website owner interest involved here as well - without those search bots their sites would not be indexed and searchable in the Internet. So there was kind of win-win situation, in most typical cases at least, as for instance publishers complained about deep links, etc. because their ads revenue was hurt. AI scrapping bots provide zero value for sites owners.

Valodim 2y ago
Is this really true? If I have a marketing website for a product, isn't it in my interest to have that marketing incorporated in AI models?

jeroenhd 2y ago

> The modern companies, mostly AI companies, seems to be more interested in flying under the radar, and have less respect for the internet infrastructure at a whole I think that makes a lot of sense. Google's goal is (or perhaps used to be) providing a network of links. The more they scrape you, the more visitors you may end up receiving, and the better your website performs (monetarily, or just in terms of providing information to the world). With AI companies, the goal is to consume and replace. In their best case scenario, your website will never receive a visitor again. You won't get anything in return for providing content to AI companies. That means there's no reason for website administrators to permit the good ones, especially for people who use subscriptions or ads to support their website operating costs.

eadmund 2y ago
> With AI companies, the goal is to consume and replace. I don’t think that’s really true. The AI companies’ goal is to consume and create something else. > You won't get anything in return for providing content to AI companies. That was the original problem with websites in general, and the ‘solution’ was ads. It would be really, really cool if the thing which finally makes micropayments happen is AI. And then we humans could use micropayments too. Of course, the worst of both worlds would be micropayments and ads.
- mrweasel 2y ago
  You can have non-commercial websites. Plenty of people have blogs or personal websites, sites that support a business, sites where you already pay and in this case it was the ScummVM website, an open source project. A lot of those sites are at risk of being made irrelevant by AI companies who really don't give a shit about your motivations for doing something for free. If their crawler kills your site and their LLM steals views by regurgitation answers based on your work, so be it, you served your purpose. If you want to talk payment: Ask the AI companies to pay you when they generate an answer based on your work, a license fee. That will kill their business model pretty quickly.
  - eadmund 2y ago
    > … their crawler kills your site and their LLM steals views by regurgitation answers based on your work How is that different from a human being reading my underwater basket weaving site and starting his own, ‘stealing’ ‘my’ views? Or a thousand human beings out of the billions on Earth doing the same thing?
    - mystified5016 2y ago
      Because that other person doesn't have the unilateral and un-appealable power to put their website in place of yours in search engines.
    - poincaredisk 2y ago

philipwhiuk 2y ago

It's DDoS either way even if it's not an attack.