9 ms·

Block the Bots That Feed “AI” Models by Scraping Your Website

26 points by sohkamyung 3y ago

nuc1e0n 3y ago
There is a * user agent that can be used to prevent all bots that honor robots.txt from crawling a site. Specific user agents can then be granted additional access afterward.
- JackGreyhat 3y ago
  Do you use this approach? If so, are you willing to share an example robots.txt file? What bots would be allowed? Google, Bing, ...? Those?
  - nuc1e0n 3y ago
    robots.txt is a standard and has been for many years. It seems that https://www.robotstxt.org https://www.robotstxt.org is the main repository for its definition. Any robots.txt I might suggest wouldn't necessarily be appropriate for anyone else. It's a policy that site stakeholders need to decide for themselves I'd say.
maitland 3y ago
I'd be amazed if ai companies like openai respect robots.txt or any other good faith mechanism.
- nuc1e0n 3y ago
  Seems to me that any organisation that doesn't honor robots.txt would have a tougher time justifying to people that they are acting in good faith. Just my two cents of course. I understand OpenAI does honor it, which is to be expected from responsible orgs.
  - deleted 3y ago
    [deleted]
- JohnFen 3y ago
  Yeah, I don't trust that they will all respect it. Just like you can't trust any other scrapers to respect these mechanisms, and just like you couldn't trust websites to honor the DNT signal. if it's voluntary and companies will make more money by not doing it, then at least some are not going to do it. So, from my point of view, this effort is mostly meaningless. It's not enough to get me to open my websites again, anyway.
  - nuc1e0n 3y ago
    It's true that some dishonest folk might not honor it, but it does communicate the wishes of the stakeholders of a website in machine readable terms. It means owners of bots cannot claim permission to use content is granted, as clear intent in robots.txt would show it is not.
    - JohnFen 3y ago
      > as clear intent in robots.txt would show it is not. Would it? The primary purpose of robots.txt is not actually to lock out bots (that's why respecting it is not mandatory). It's to give the bots guidance as to which parts of your site are appropriate for them and which parts are not. This may make the "clear intent" argument weak in court.
      - nuc1e0n 3y ago
        > This may make the "clear intent" argument weak in court. The standard has the keyword "Disallow", not "Avoid". I can't speak for anyone else of course, but that seems a pretty clear indicator of intent to me. By that I mean a site's stakeholders want to indicate that certain bots are disallowed from crawling a portion of their website.
        JohnFen 3y ago
        But you and I aren't judges in a court. They go by different rules, such as the official intent and meaning of the robots.txt system itself. I'm not saying a court wouldn't find intent signaled, I don't know, only that it's not clear-cut that it would.
- brianjking 3y ago
  I mean they literally wrote their own crawler which has docs for it. I'm sure they'll respect it. https://platform.openai.com/docs/gptbot https://platform.openai.com/docs/gptbot What isn't known is if those same sites will be also included in corpuses such as CommonCrawl or ThePile, leading to being included in training as is.
- jesterson 3y ago
  All this "AI" garbage bots are being filtered on IP level for me.
  - JohnFen 3y ago
    IP level filtering is purely a "better than nothing" approach, though. It's very fragile and porous. You have to constantly monitor your logs to find the new IP addresses to filter out as they are adopted, but the only way you'll see them in your logs is after they've already done the scraping.
    - jesterson 3y ago
      There are multiple behaviour-based tools to use for spotting and terminating this sort of activities. I do agree however, it's a very tedious semi-automated task of monitoring logs, which many business wouldn't bother to have.
ris 3y ago
AI hysteria is is going to get worse before it gets better.
- gumballindie 3y ago
  Depends on what drives clicks the nvidia’s stock price.
nabogh 3y ago
I don't really understand the viewpoint that people shouldn't be allowed to do math on data that I openly publish. I guess it's about protecting your livelihood.
- gumballindie 3y ago
  Yea generally speaking protecting what you own from thieves is about protecting your livelihood.
- JohnFen 3y ago
  I'm not worried about protecting my livelihood. I just don't want to contribute to training any AI.
  - nabogh 3y ago
    Is your objection because of who is doing the training or just AI in general? Like do you have qualms with an individual scraping your site? And what about scraping for non-AI purposes?
    - JohnFen 3y ago
      I don't want to contribute to any AI models, regardless of who is making them. I don't have an issue with scraping for most other purposes, but there's no way to differentiate based on purpose.
extraduder_ire 3y ago
Is there anything like gitignore.io for robots.txt files? Like, select a set of known bots and get a list of rules tailored to them. Bytedance's crawler should probably be blocked too, based on a recent mattKC video. I assume google's training bard with their search engine crawling data, so you have an incentive to allow them.
astrange 3y ago
Authors and artists have been spending a lot of time talking about AI exclusively with each other, which is why they keep proposing solutions to made up problems like this.
LinuxBender 3y ago
I suppose there is no harm in adding things to robots.txt but I would not expect anyone to honor it. People that work on bleeding edge technology are also commonly the risk takers that move fast, break things while sometimes acting a bit cavalier. Adding to this I assume perhaps incorrectly that these companies have the funding to write their scrapers using Chrome headless so the same techniques I used to block archive.is and other bots will not be useful. If something should not be scraped I think it should be placed behind strong authentication and the AUP/ToS must state in clear and concise language explicitly what the sites content may be used for. Even then I would expect risk takers to simply ignore it and accept any legal findings to be the cost of doing business. Or perhaps I have become a bit too skeptical and jaded with time and experience. I will go run a lap.
NoZebra120vClip 3y ago
It would seem rather pointless to try and block any sort of crawler from crawling your site, if it's already been crawled by someone like the Internet Archive. Because all any future crawler needs to do is crawl through the various archives and plunder their content, rather than being so concerned about what live sites say that they can or cannot do directly.

26 points by sohkamyung 3y ago

There is a * user agent that can be used to prevent all bots that honor robots.txt from crawling a site. Specific user agents can then be granted additional access afterward.

JackGreyhat 3y ago
Do you use this approach? If so, are you willing to share an example robots.txt file? What bots would be allowed? Google, Bing, ...? Those?
- nuc1e0n 3y ago
  robots.txt is a standard and has been for many years. It seems that https://www.robotstxt.org https://www.robotstxt.org is the main repository for its definition. Any robots.txt I might suggest wouldn't necessarily be appropriate for anyone else. It's a policy that site stakeholders need to decide for themselves I'd say.

maitland 3y ago

I'd be amazed if ai companies like openai respect robots.txt or any other good faith mechanism.

nuc1e0n 3y ago
Seems to me that any organisation that doesn't honor robots.txt would have a tougher time justifying to people that they are acting in good faith. Just my two cents of course. I understand OpenAI does honor it, which is to be expected from responsible orgs.
- deleted 3y ago
  [deleted]
JohnFen 3y ago
Yeah, I don't trust that they will all respect it. Just like you can't trust any other scrapers to respect these mechanisms, and just like you couldn't trust websites to honor the DNT signal. if it's voluntary and companies will make more money by not doing it, then at least some are not going to do it. So, from my point of view, this effort is mostly meaningless. It's not enough to get me to open my websites again, anyway.
- nuc1e0n 3y ago
  It's true that some dishonest folk might not honor it, but it does communicate the wishes of the stakeholders of a website in machine readable terms. It means owners of bots cannot claim permission to use content is granted, as clear intent in robots.txt would show it is not.
  - JohnFen 3y ago
    > as clear intent in robots.txt would show it is not. Would it? The primary purpose of robots.txt is not actually to lock out bots (that's why respecting it is not mandatory). It's to give the bots guidance as to which parts of your site are appropriate for them and which parts are not. This may make the "clear intent" argument weak in court.
    - nuc1e0n 3y ago
      > This may make the "clear intent" argument weak in court. The standard has the keyword "Disallow", not "Avoid". I can't speak for anyone else of course, but that seems a pretty clear indicator of intent to me. By that I mean a site's stakeholders want to indicate that certain bots are disallowed from crawling a portion of their website.
      - JohnFen 3y ago
        But you and I aren't judges in a court. They go by different rules, such as the official intent and meaning of the robots.txt system itself. I'm not saying a court wouldn't find intent signaled, I don't know, only that it's not clear-cut that it would.
brianjking 3y ago
I mean they literally wrote their own crawler which has docs for it. I'm sure they'll respect it. https://platform.openai.com/docs/gptbot https://platform.openai.com/docs/gptbot What isn't known is if those same sites will be also included in corpuses such as CommonCrawl or ThePile, leading to being included in training as is.
jesterson 3y ago
All this "AI" garbage bots are being filtered on IP level for me.
- JohnFen 3y ago
  IP level filtering is purely a "better than nothing" approach, though. It's very fragile and porous. You have to constantly monitor your logs to find the new IP addresses to filter out as they are adopted, but the only way you'll see them in your logs is after they've already done the scraping.
  - jesterson 3y ago
    There are multiple behaviour-based tools to use for spotting and terminating this sort of activities. I do agree however, it's a very tedious semi-automated task of monitoring logs, which many business wouldn't bother to have.

ris 3y ago

AI hysteria is is going to get worse before it gets better.

gumballindie 3y ago
Depends on what drives clicks the nvidia’s stock price.

nabogh 3y ago

I don't really understand the viewpoint that people shouldn't be allowed to do math on data that I openly publish. I guess it's about protecting your livelihood.

gumballindie 3y ago
Yea generally speaking protecting what you own from thieves is about protecting your livelihood.
JohnFen 3y ago
I'm not worried about protecting my livelihood. I just don't want to contribute to training any AI.
- nabogh 3y ago
  Is your objection because of who is doing the training or just AI in general? Like do you have qualms with an individual scraping your site? And what about scraping for non-AI purposes?
  - JohnFen 3y ago
    I don't want to contribute to any AI models, regardless of who is making them. I don't have an issue with scraping for most other purposes, but there's no way to differentiate based on purpose.

extraduder_ire 3y ago

Is there anything like gitignore.io for robots.txt files? Like, select a set of known bots and get a list of rules tailored to them. Bytedance's crawler should probably be blocked too, based on a recent mattKC video. I assume google's training bard with their search engine crawling data, so you have an incentive to allow them.

astrange 3y ago

Authors and artists have been spending a lot of time talking about AI exclusively with each other, which is why they keep proposing solutions to made up problems like this.

LinuxBender 3y ago

I suppose there is no harm in adding things to robots.txt but I would not expect anyone to honor it. People that work on bleeding edge technology are also commonly the risk takers that move fast, break things while sometimes acting a bit cavalier. Adding to this I assume perhaps incorrectly that these companies have the funding to write their scrapers using Chrome headless so the same techniques I used to block archive.is and other bots will not be useful. If something should not be scraped I think it should be placed behind strong authentication and the AUP/ToS must state in clear and concise language explicitly what the sites content may be used for. Even then I would expect risk takers to simply ignore it and accept any legal findings to be the cost of doing business. Or perhaps I have become a bit too skeptical and jaded with time and experience. I will go run a lap.

NoZebra120vClip 3y ago

It would seem rather pointless to try and block any sort of crawler from crawling your site, if it's already been crawled by someone like the Internet Archive. Because all any future crawler needs to do is crawl through the various archives and plunder their content, rather than being so concerned about what live sites say that they can or cannot do directly.