3 ms·

curious, will the robots.txt be really honored? maybe legal issue if not?

cdme 3y ago
It should be — it’s an accepted standard and would make their behavior even more dubious should they ignore it.
ghxst 3y ago
I don't see how a robots.txt itself would be legally binding, but I guess you could make an argument for the terms of service? Would love to know more about this as well.
29athrowaway 3y ago
It will not be honored. The ones that honor it will lose to the ones that do.
- __loam 3y ago
  So we should make it illegal not to honor it then, if the incentives are to engage in anti-social behavior.
- noahtallen 3y ago
  Given that Google has honored robots.txt for many years, not sure this is true. It mostly means that blocked sites might not show up on the biggest, most trustworthy platforms. If these AI platforms become a big way people consume content from the internet, authors will have an interest in being there. Today, people make money from web traffic, and therefore want their sites indexed by search indices. If the same happens with AI eventually, authors will probably follow. What people get from having their sites ingested by AI is still unknown today — if the model doesn’t send traffic to your site, you can’t make ad money and can’t support your site (if it’s big). Data licensing that allows AI platforms to pay authors for useful content may be a solution someday
  - 29athrowaway 3y ago
    There are many entities out there crawling and scraping the web in non-compliant ways.
matt_heimer 3y ago
robots.txt being an honor system has always been a bit weird and search engine treatment is odder still. Google will still index your disallowed pages but refuse to show a description since they are couldn't read the page. The premise being that Google could have discovered your URLs from other pages and they don't consider a robots.txt disallow to indicate that you don't want them in search, just that you don't want Google to directly read the page. To actually disallow indexing of a page a page you need to allow it in robots and then add a noindex to the page itself. No one has really explained how these AI agents will treat robots.txt. Does it just mean that Bard/Gemini won't directly read the page but might somehow incorporate it into a dataset? For a test I asked Gemini "What is the URL of the String api documentation for Java 6?" which is disallowed by robots.txt. The response was: Unfortunately, Oracle does not maintain separate API documentation for older versions of Java like Java 6. The official documentation website (https://docs.oracle.com/javase/7/docs/api/ https://docs.oracle.com/javase/7/docs/api/) only provides documentation for the latest versions. Oracle has archieved some of the older Java API documentation but 6 is still available at https://docs.oracle.com/javase/6/docs/api/index.html https://docs.oracle.com/javase/6/docs/api/index.html. Funny enough the Java 7 API docs which Gemini linked too is also disallowed but maybe that happened after the model was trained.
- fragmede 3y ago
  OpenAI documents how their crawler honors robots.txt. https://platform.openai.com/docs/gptbot https://platform.openai.com/docs/gptbot
  - __loam 3y ago
    After they ingested everything
    - fragmede 3y ago
      We don't actually know what they trained on. If we want to trust them, they've always honored robots.txt, because why wouldn't they? On the other hand, if we want to believe they're as capricious as we want to make them out to be, they've got a copy of LibGen, and every single torrent out there to train it in on, on top of using robots.txt to find what you don't want crawled. robots.txt is great - it tells you where your evil scraper should go to get the good stuff.

cdme 3y ago

It should be — it’s an accepted standard and would make their behavior even more dubious should they ignore it.

I don't see how a robots.txt itself would be legally binding, but I guess you could make an argument for the terms of service? Would love to know more about this as well.

29athrowaway 3y ago

It will not be honored. The ones that honor it will lose to the ones that do.

__loam 3y ago
So we should make it illegal not to honor it then, if the incentives are to engage in anti-social behavior.
noahtallen 3y ago
Given that Google has honored robots.txt for many years, not sure this is true. It mostly means that blocked sites might not show up on the biggest, most trustworthy platforms. If these AI platforms become a big way people consume content from the internet, authors will have an interest in being there. Today, people make money from web traffic, and therefore want their sites indexed by search indices. If the same happens with AI eventually, authors will probably follow. What people get from having their sites ingested by AI is still unknown today — if the model doesn’t send traffic to your site, you can’t make ad money and can’t support your site (if it’s big). Data licensing that allows AI platforms to pay authors for useful content may be a solution someday
- 29athrowaway 3y ago
  There are many entities out there crawling and scraping the web in non-compliant ways.

matt_heimer 3y ago

robots.txt being an honor system has always been a bit weird and search engine treatment is odder still. Google will still index your disallowed pages but refuse to show a description since they are couldn't read the page. The premise being that Google could have discovered your URLs from other pages and they don't consider a robots.txt disallow to indicate that you don't want them in search, just that you don't want Google to directly read the page. To actually disallow indexing of a page a page you need to allow it in robots and then add a noindex to the page itself. No one has really explained how these AI agents will treat robots.txt. Does it just mean that Bard/Gemini won't directly read the page but might somehow incorporate it into a dataset? For a test I asked Gemini "What is the URL of the String api documentation for Java 6?" which is disallowed by robots.txt. The response was: Unfortunately, Oracle does not maintain separate API documentation for older versions of Java like Java 6. The official documentation website (https://docs.oracle.com/javase/7/docs/api/ https://docs.oracle.com/javase/7/docs/api/) only provides documentation for the latest versions. Oracle has archieved some of the older Java API documentation but 6 is still available at https://docs.oracle.com/javase/6/docs/api/index.html https://docs.oracle.com/javase/6/docs/api/index.html. Funny enough the Java 7 API docs which Gemini linked too is also disallowed but maybe that happened after the model was trained.

fragmede 3y ago
OpenAI documents how their crawler honors robots.txt. https://platform.openai.com/docs/gptbot https://platform.openai.com/docs/gptbot
- __loam 3y ago
  After they ingested everything
  - fragmede 3y ago
    We don't actually know what they trained on. If we want to trust them, they've always honored robots.txt, because why wouldn't they? On the other hand, if we want to believe they're as capricious as we want to make them out to be, they've got a copy of LibGen, and every single torrent out there to train it in on, on top of using robots.txt to find what you don't want crawled. robots.txt is great - it tells you where your evil scraper should go to get the good stuff.