6 ms·

The crawlers can simply stop identifying themselves via custom user agent, can't they? Also why are "AI" crawlers are worse than "normal" crawlers? Either way

The crawlers can simply stop identifying themselves via custom user agent, can't they? Also why are "AI" crawlers are worse than "normal" crawlers? Either way, this is an exercise in futility.

karaterobot 3y ago
> Also why are "AI" crawlers are worse than "normal" crawlers? A search engine will index your content to bring people to it through search. An AI crawler will take your content to recapitulate it and sell it to others. Obviously it's more complicated than this, but this is how one might see it who wishes to use this file. > Either way, this is an exercise in futility. Not necessarily disqualifying. Laws against theft are also futile, in the sense that honest people don't need them and dishonest people don't follow them, and history since at least Hammurabi has been replete with examples of such laws not stopping theft. And yet. Seems worth the calories it costs to say "for the record, I do not give my consent for what you're doing".
- cabirum 3y ago
  Search engines are not the beacons of holiness - they sell ads, they sell data on who searched what, they manipulate results. Search engines and AI things are typically owned by the same company. AIs are fed with the data collected by a search engine. The only difference is whether AI gets the data in realtime or waits for the search engine to collect another data dump. Fighting windmills as I see it.
  - nunez 3y ago
    Search engines manipulate results way less aggressively than LLMs do.
vouaobrasil 3y ago
> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the more vocal opponents are against it, the more those downsides will become apparent. Hopefully, it will bring the ruin of all AI companies and research.
- cabirum 3y ago
  I'm kind of out of the loop in regard why do we need to hate on AI? The bubble will burst given time, like all the other bubbles before. What needed is indifference, not hate.
  - starbugs 3y ago
    > What needed is indifference, not hate. Agree that hate is not the solution. But saying "no" decidedly and loudly is absolutely something that people need to relearn and I am happy to see that some do now. Just being indifferent is not sufficient. Finding and taking measures to protect yourself and others is a positive way to approach this IMO.
  - nerdjon 3y ago
    I think we need to be careful to assume that the AI bubble is really going to burst, there is way too much money being put into it by big players for them to just give up on it. I mean Microsoft is even pushing for a new key on Windows keyboards. We also have to look at this bubble differently. With Crypto there was a general technical level that was required to use it that many people just did not have. I would say the same is true for most bubbles we have seen. AI is generally available for anyone that has enough understanding on how to open a browser to use. Too many non technical people have just accepted it without understanding the risks. Edit: You can really see just how bad this is going to get if you look at Apple. How they are deemed to be "behind" because they don't have generative AI right now and how they are so desperate that they are looking at Google. I really don't think this is going to burst and really all that is going to happen is even more consolidation and we will be screwed.
    - starbugs 3y ago
      > Edit: You can really see just how bad this is going to get if you look at Apple. How they are deemed to be "behind" because they don't have generative AI right now and how they are so desperate that they are looking at Google. > I really don't think this is going to burst and really all that is going to happen is even more consolidation and we will be screwed. It's always FOMO until it's FUD.

The crawlers can simply stop identifying themselves via custom user agent, can't they? Also why are "AI" crawlers are worse than "normal" crawlers? Either way, this is an exercise in futility.

karaterobot 3y ago

> Also why are "AI" crawlers are worse than "normal" crawlers? A search engine will index your content to bring people to it through search. An AI crawler will take your content to recapitulate it and sell it to others. Obviously it's more complicated than this, but this is how one might see it who wishes to use this file. > Either way, this is an exercise in futility. Not necessarily disqualifying. Laws against theft are also futile, in the sense that honest people don't need them and dishonest people don't follow them, and history since at least Hammurabi has been replete with examples of such laws not stopping theft. And yet. Seems worth the calories it costs to say "for the record, I do not give my consent for what you're doing".

cabirum 3y ago
Search engines are not the beacons of holiness - they sell ads, they sell data on who searched what, they manipulate results. Search engines and AI things are typically owned by the same company. AIs are fed with the data collected by a search engine. The only difference is whether AI gets the data in realtime or waits for the search engine to collect another data dump. Fighting windmills as I see it.
- nunez 3y ago
  Search engines manipulate results way less aggressively than LLMs do.

vouaobrasil 3y ago

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the more vocal opponents are against it, the more those downsides will become apparent. Hopefully, it will bring the ruin of all AI companies and research.

cabirum 3y ago
I'm kind of out of the loop in regard why do we need to hate on AI? The bubble will burst given time, like all the other bubbles before. What needed is indifference, not hate.
- starbugs 3y ago
  > What needed is indifference, not hate. Agree that hate is not the solution. But saying "no" decidedly and loudly is absolutely something that people need to relearn and I am happy to see that some do now. Just being indifferent is not sufficient. Finding and taking measures to protect yourself and others is a positive way to approach this IMO.
- nerdjon 3y ago
  I think we need to be careful to assume that the AI bubble is really going to burst, there is way too much money being put into it by big players for them to just give up on it. I mean Microsoft is even pushing for a new key on Windows keyboards. We also have to look at this bubble differently. With Crypto there was a general technical level that was required to use it that many people just did not have. I would say the same is true for most bubbles we have seen. AI is generally available for anyone that has enough understanding on how to open a browser to use. Too many non technical people have just accepted it without understanding the risks. Edit: You can really see just how bad this is going to get if you look at Apple. How they are deemed to be "behind" because they don't have generative AI right now and how they are so desperate that they are looking at Google. I really don't think this is going to burst and really all that is going to happen is even more consolidation and we will be screwed.
  - starbugs 3y ago
    > Edit: You can really see just how bad this is going to get if you look at Apple. How they are deemed to be "behind" because they don't have generative AI right now and how they are so desperate that they are looking at Google. > I really don't think this is going to burst and really all that is going to happen is even more consolidation and we will be screwed. It's always FOMO until it's FUD.