3 ms·

Leave and go where. Seems like a losing battle, eventually all platforms will be co-opted to feed AI. Even if you have a private web site (blog), it can be sc

Leave and go where. Seems like a losing battle, eventually all platforms will be co-opted to feed AI. Even if you have a private web site (blog), it can be scraped, and how are the little guys going to fight it?

erickhill 2y ago
Can we not put lines in our robots.txt files to block being crawled?
- wkat4242 2y ago
  Yeah and even better: share blocklists of known AI crawler IPs so we can just block them. Robots.txt is too voluntary.
- CrazyStat 2y ago
  AI companies are ignoring robots.txt in the race to slurp up the entire internet [1]. [1] https://www.reuters.com/technology/artificial-intelligence/multiple-ai-companies-bypassing-web-standard-scrape-publisher-sites-licensing-2024-06-21/ https://www.reuters.com/technology/artificial-intelligence/m...
  - deleted 2y ago
    [deleted]
- cjf101 2y ago
  There was a bunch of reporting on how AI companies and researchers were using tools that ignored robots.txt. It's a "polite request" that these companies had a strong incentive to ignore, so they did. That incentive is still there, so it is likely that some of them will continue to do so.
  - Ukv 2y ago
    CommonCrawl[0] and the companies training models I'm aware of[1][2][3] all respect robots.txt for their crawling. If we're thinking of the same reporting, it was based on a claim by TollBit (a content licensing startup) which was in turn based the fact that "Perplexity had a feature where a user could prompt a specific URL within the answer engine to summarize it". Actions performed by tools acting as a user agent (like archive.today, or webpage-to-PDF site, or a translation site) aren't crawlers and aren't what robots.txt is designed for, but either way the feature is disabled now. [0]: https://commoncrawl.org/faq https://commoncrawl.org/faq [1]: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots [2]: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler https://support.anthropic.com/en/articles/8896518-does-anthr... [3]: https://blog.google/technology/ai/an-update-on-web-publisher-controls/ https://blog.google/technology/ai/an-update-on-web-publisher...
    - FrustratedMonky 2y ago
      Robots.txt is a suggestions. As is reporting on using it. The companies that are ignoring robots.txt, are also probably the companies not advertising that they are ignoring robots.txt.
      - Ukv 2y ago
        The EU's AI act points to the DSM directive's text and data mining exemption, allowing for commercial data mining so long as machine-readable opt-outs are respected - robots.txt is typically taken as the established standard for this. In the US it is a suggestion (so long as Fair Use holds up) but all I've seen suggests that the major players are respecting it, and minor players tend to just use CommonCrawl which also does. Definitely possible that some slip through the cracks, but I don't think it's as useless as is being suggested.
        FrustratedMonky 2y ago
        Technically, robot.txt isn't enforcing anything, so it is just trust. ""OpenAI CTO doesn't know what data was used to train the company's video generating platform, Sora"" https://www.youtube.com/watch?v=4AYbZG3h14w https://www.youtube.com/watch?v=4AYbZG3h14w Funny. If I can browse to it, it is public right? That is how some people's logic goes. And how OpenAI argued 2 years ago when GPT3.5/ChatGPT first started getting traction.
- rvbissell 2y ago
  Someone who doesn't care about polluting our corpus is not going to care about your robots.txt

erickhill 2y ago

Can we not put lines in our robots.txt files to block being crawled?

wkat4242 2y ago
Yeah and even better: share blocklists of known AI crawler IPs so we can just block them. Robots.txt is too voluntary.
CrazyStat 2y ago
AI companies are ignoring robots.txt in the race to slurp up the entire internet [1]. [1] https://www.reuters.com/technology/artificial-intelligence/multiple-ai-companies-bypassing-web-standard-scrape-publisher-sites-licensing-2024-06-21/ https://www.reuters.com/technology/artificial-intelligence/m...
- deleted 2y ago
  [deleted]
cjf101 2y ago
There was a bunch of reporting on how AI companies and researchers were using tools that ignored robots.txt. It's a "polite request" that these companies had a strong incentive to ignore, so they did. That incentive is still there, so it is likely that some of them will continue to do so.
- Ukv 2y ago
  CommonCrawl[0] and the companies training models I'm aware of[1][2][3] all respect robots.txt for their crawling. If we're thinking of the same reporting, it was based on a claim by TollBit (a content licensing startup) which was in turn based the fact that "Perplexity had a feature where a user could prompt a specific URL within the answer engine to summarize it". Actions performed by tools acting as a user agent (like archive.today, or webpage-to-PDF site, or a translation site) aren't crawlers and aren't what robots.txt is designed for, but either way the feature is disabled now. [0]: https://commoncrawl.org/faq https://commoncrawl.org/faq [1]: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots [2]: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler https://support.anthropic.com/en/articles/8896518-does-anthr... [3]: https://blog.google/technology/ai/an-update-on-web-publisher-controls/ https://blog.google/technology/ai/an-update-on-web-publisher...
  - FrustratedMonky 2y ago
    Robots.txt is a suggestions. As is reporting on using it. The companies that are ignoring robots.txt, are also probably the companies not advertising that they are ignoring robots.txt.
    - Ukv 2y ago
      The EU's AI act points to the DSM directive's text and data mining exemption, allowing for commercial data mining so long as machine-readable opt-outs are respected - robots.txt is typically taken as the established standard for this. In the US it is a suggestion (so long as Fair Use holds up) but all I've seen suggests that the major players are respecting it, and minor players tend to just use CommonCrawl which also does. Definitely possible that some slip through the cracks, but I don't think it's as useless as is being suggested.
      - FrustratedMonky 2y ago
        Technically, robot.txt isn't enforcing anything, so it is just trust. ""OpenAI CTO doesn't know what data was used to train the company's video generating platform, Sora"" https://www.youtube.com/watch?v=4AYbZG3h14w https://www.youtube.com/watch?v=4AYbZG3h14w Funny. If I can browse to it, it is public right? That is how some people's logic goes. And how OpenAI argued 2 years ago when GPT3.5/ChatGPT first started getting traction.
rvbissell 2y ago
Someone who doesn't care about polluting our corpus is not going to care about your robots.txt