13 ms·

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement t

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to align legislation with what the public wants. We need test cases. The thing I somewhat struggle with is that after 20-30 years of calls for shorter copyright terms, lesser restrictions on content you access publicly, and what you can do with it, we are now in the situation where the arguments are quickly leaning the other way. "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... In many ways an ai.txt would be worse than doing nothing as it's a meaningless veneer that would be ignored, but pointed to as the answer.

shanebellone 3y ago
"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.
- alwayslikethis 3y ago
  Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts
  - shanebellone 3y ago
    "Why not serve fake garbage indistinguishable from real content by a computer, like LLM output?" Serving more than the minimum wastes resources. Worse yet, a better solution would cost my time. "Sending errors just incentivizes bot owners to fix the identifiable parts" Sure, someone could make or configure their scraper perfectly. "Perfect" is now the table stakes though. Edit: My solution strives to cause an unproportional expense in order to circumvent. I want 10x on my time.
  - twelve40 3y ago
    it'd be cool to be able to fingerprint that garbage, too. Like, sprinkle some hashes here and there (or something like that) so that you can later uniquely look up your own "content" being stolen by chatbots and which ones.
    - shanebellone 3y ago
      You can. I can't think of the appropriate term though. Hopefully someone else chimes in here with a link.
  - ape4 3y ago
    I like this idea. Of course it would have to be only to robots that visit a page disallowed by the robots.txt
  - dspillett 3y ago
    > Sending errors just incentivizes bot owners to fix the identifiable parts Nah. It'll just make them fake their identity so it is harder to tell the traffic is from a bot.
- spc476 3y ago
  It might be a better idea to serve up a 418 ("I'm a tea pot") with a line line text file saying "I'm not an HTTP server". That solved a problem I had with bots making HTTP requests to my gopher server [1]. Serving up a 503 informs the bot that there's a server issue and it may try again later. A 418 informs the bot that it made an erroneous request and such an odd error code might get someone to look into it and stop. [1] https://boston.conman.org/2019/09/30.2 https://boston.conman.org/2019/09/30.2
  - shanebellone 3y ago
    This is very interesting. I've bookmarked the link. Thanks for sharing. I believe minimal is best and this might fit nicely within my larger system. Do you approach other problems with a similar mindset?
George83728 3y ago
robots.txt works for the major search engines who voluntarily abide by it, so it isn't a failed system. Just because it doesn't work on everybody doesn't mean it's useless.
Karunamon 3y ago
On the contrary, it works perfectly well for normal, non-bad actors running services used by most of the public. That includes search engines and stuff like archive.org. A robots.txt set to deny all will result in your site not showing up on any search engine that matters. It doesn't work for bad actors, but then again, nothing really does.
bombolo 3y ago
> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... AI is being used to do copyright laundering, at the same time "we", the people who can't afford to run our own AI, are still subject to absurd rules that AI owners get to ignore, apparently.
- rhn_mk1 3y ago
  The barrier to running an AI model is getting lower every day, so the threshold for ignoring copyright is getting lower with it.
  - gavinhoward 3y ago
    You are mistaken if you think companies will allow common people to ignore copyright on their IP. The only IP that will be allowed to be stolen is that of other common people.
    - alphanullmeric 3y ago
      You can’t steal an idea.
      - gavinhoward 3y ago
        Yes, you can. You can do that by making the monetary value of the idea zero when it used to be non-zero.
        alphanullmeric 3y ago
        Stealing requires physical force, whether that be on the property or person. By your logic, independently discovering the idea is also stealing.
        gavinhoward 3y ago
        Stealing does not require physical force.
brookst 3y ago
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.
- samwillis 3y ago
  Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction. If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static). With an ai.txt there is no potential downside to violating it - unless we get new legislation enforcing its legal standing. The nature of ML models is that it's opaque what content exactly it's trained on, there is no obvious retaliation or retribution.
  - Wowfunhappy 3y ago
    > But my point is that it's not a system that we should copy for preventing the use of content in training AI. I don't think that's what OP is envisioning based on their post!
  - BurningFrog 3y ago
    OP is trying to give helpful info to the AI, not set boundaries for it.
  - capableweb 3y ago
    > But my point is that it's not a system that we should copy for preventing the use of content in training AI The purpose OP is suggesting in the submission is the opposite, help AI crawlers to understand what the page/website is about without actually having to infer the purpose from the content itself.
    - Xelynega 3y ago
      Isn't that the entire point of the semantic web?
prepend 3y ago
> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... While I’m sure others than you share this opinion, I don’t think it’s as uniform as the more common “shorten/rationalize copyright terms and fair use” crowd “we.” I consider myself a knowledge worker and a pretty staunch proponent of floss and am perfectly fine with training AI on everything publicly available. While create stuff, I don’t make a living off selling particular copies of things I make, so my self preservation bias isn’t kicking in as much as someone who does want to sell items of their work. But I also made some pretty explicit choices in the 90s based on where I thought IP would go so I was never in a position where I had to sell copies to survive. My decision was more pragmatic first and philosophical second. I think someone entering the workforce now probably wants to align their livelihood with AI training on everything and not go against that. Even if US/Euro law limits training, there’s no way all other countries are going to, so it’s going to happen. And I don’t think it’s worth locking down the world to try to stop AIs from training on text, images, etc.
- JohnFen 3y ago
  Fair enough. But there should be some mechanism where people who don't want their works to contribute to AI training to be able to prevent that without having to resort to removing their works from the web.
  - prepend 3y ago
    I think people who don’t want their content contributing to AI shouldn’t have it on the public web. There are many ways to restrict access. Use one of them. But if you respond to an anonymous http request with content then it shouldn’t matter if it’s a robot looking at it or a human (or a man or a woman or whatever). I think this both for simplicity and that I foresee a future where human consciousness is simulated and basically an AI. I don’t want to have rules that biological humans can view and digital humans can’t.
    - JohnFen 3y ago
      > I think people who don’t want their content contributing to AI shouldn’t have it on the public web. Practically speaking, that's the only effective solution. I just think that it's a shame that's necessary. It would be better for everyone if there wasn't a disincentive to making works publicly available. > I don’t want to have rules that biological humans can view and digital humans can’t. This is a point we disagree on. And "digital humans"? I would argue that such a thing can't exist, if you mean "human" in any way other than rough analogy.
mschuster91 3y ago
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. That depends what you expect from it. For the purpose of limiting crawlers, at least the major search engines respect it.
majewsky 3y ago
> All a robots.txt is is a polite request to please follow the rules in it At least in my country (Germany), respecting robots.txt is a legal requirement for data mining. See German Copyright Code, section 44b: https://www.gesetze-im-internet.de/urhg/__44b.html https://www.gesetze-im-internet.de/urhg/__44b.html (IANAL)
FinnKuhn 3y ago
if you do data mining in the EU you are legally required to respect robots.txt afaik
- LawTalkingGuy 3y ago
  This is how it is a failed system. It doesn't really do what most people who have one think it does and yet it gets rolled into law and everyone now has to deal with it constantly while still not fixing the original issue. It's like the EU doesn't understand that bad law has a negative value.
IanCal 3y ago
The poster wants the opposite - a way of explicitly helping AI systems/etc to use their site. If people ignore it, they're just giving up a bit of help.
- circuit10 3y ago
  Yes, I feel like this person only read the title and not the text of the post and made an assumption
  - benatkin 3y ago
    Not an assumption. Just a response to the title of the article.
    - circuit10 3y ago
      They said "Using robots.txt as a model for anything doesn't work." but it does work for the case described in the text
      - benatkin 3y ago
        What works? This is an idea, not anything that's been throughly tried.
none_to_remain 3y ago
Robots.txt is meant as an aid to crawlers. "This stuff is not useful to index," rather than a blocking mechanism
safety1st 3y ago
> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... This gross generalization of other people's views on important issues is really offensive. My view is that the Copyright Act of 1976 had it about right when they established the duration of copyright. My view is that members of Congress were handsomely rewarded by a specific corporation to carve out special exceptions to this law because they wanted larger profits. "We" didn't call the Copyright Term Extension Act of 1998 the "Mickey Mouse Act" for nothing. It's also no coincidence that Disney is now the largest media company in the world. Reducing copyright term extension has everything to do with restoring competition and creativity to our economy, and reversing corruption that borders on white collar crime. It has nothing to do with AI. Don't recruit me into some bullshit argument that rewrites history and entrenches Disney's ill-gotten monopoly.
- casey2 3y ago
  Disney has a monopoly on media because they still have the copyright to their IP from the 20s? LOL! Companies that can leverage this new wave of AI will have, in reality, 1000x the advantage that you believe Disney has.
  - safety1st 3y ago
    Yes. There's this little thing called brand value. Disney has one of the most valuable brands in the world. Forbes estimated it at being worth about $60 billion as I recall. That brand was built heavily over many decades on IP that dates back to the 1920s, such as the most recognizable Disney character, Mickey Mouse. They manipulated the law to enhance the value of that IP and thereby gained an edge over their competitors. That's a big part of why they now enjoy such a dominant position. None of this is especially controversial (you will get a very different spin from Disney of course). If you want to comment about how business works you should read history and learn how business works first. AI luminary that you are, if you choose to remain ignorant then I guess this whole cycle will happen again with AI.
  - Spivak 3y ago
    The argument goes that copyright has allowed massive corporations to buy up and exert near total control over all of our shared stories. And when you own the cultural touchstones of whole generations that gives you power that no one else can ever wield. There is a massive amount of amazing stories based on ancient myths because it's one of the few large corpora that isn't copywritten. Once you see it in media you can't unsee it. The only space where that kind of creativity can thrive anymore is fan-fiction which lives in weird limbo where it's illegal but the copyright owners don't care. And when you want to bring any of it to the mainstream you have to hide it, all of Ali Hazelwoods books are reworked fanfics because she can't use the actual characters that inspired her -- her most famous book "The Love Hypothesis" is a Reylo fic. Go check out https://archiveofourown.org/media https://archiveofourown.org/media and see how many works are owned by a few large corporations.
Retric 3y ago
That’s factually incorrect. Germany for example does explicitly give machine readable permissions like robots.txt legal weight. In general without a fair use exemption or permission from robots.txt saving a copy of a website’s content to your own servers is copyright infringement. Purely factual information like Amazon’s prices isn’t protected by copyright, but if you want to save artwork or source files to train AI, that’s a copyright issue even before you get into the possibility of your AI being considered a derivative work.
deleted 3y ago
[deleted]
hoofhearted 3y ago
I don’t really agree with your sentiment. Robots.txt have served the simple purpose of directing bots like Google to the different parts of your website since the beginning of internet time. They still serve the same purpose, they tell bots where to go, and most importantly, they tell bots how to find your site map. Robots.txt is not there to prevent malicious crawlers from accessing pages as you have suggested. The robots.txt file acts simply like a garden gate. The good and honest people will honor the gate, while the more malicious might ignore it and hop the fence or something.
mort96 3y ago
Do you think there's a space between "you will never ever get to do anything at all with popular media until at least a hundred years after you're dead" and "anyone and any company can do anything they with everything I produce as long as it goes through an LLM"? Is it really so hard to think people may be against both of those extremes? There's a phrase I like which describes what you're doing. It's "vaguely gesturing at imagined hypocrisy".
shadowgovt 3y ago
I disagree it has failed as a system. While it does not substitute for authentication / authorization, reputable crawlers respect it, and there'd be a lot more traffic load on sites if they didn't have a way to tell reputable crawlers "please stop." Similarly, extending robots.txt to direct AI would have a similar effect: not sufficient, but useful (if for no other reason than to make it easy to distinguish reputable AI projects from ones that feel like they own the Internet to do with as they please).
FpUser 3y ago
>"All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative" Up until the point when some person / entity with the deep pockets will put a clear license / terms of use on their site that prohibits ignoring of robots.txt and would be willing to sue the ignorant.
hiddencost 3y ago
The point of robots.txt is to inform well behaved scrapers about how to behave. It is not designed nor intended to prevent bad actors. Which is good design: don't pretend to solve problems you can't.
drbawb 3y ago
It's not actually contradictory at all when you consider the root of the issue is about power asymmetry between individual creators and the corporations. Copyright terms were lobbied for, and primarily benefit the large corporations. They're symbolic of corporate overreach, that's why they're unpopular. Meanwhile, now that the laws are inconvenient for them, tech companies are straight up ignoring labeling their training data to respect IP law. Labeling the data would be expensive, thereby eroding profits. The loss of usable data would also harm the efficacy of their models, and the time spent classifying the data will hamper their iteration time. The ideas are only dissonant if you are looking at the trees (copyright term, DMCA, right to repair, etc.) and not the forest: which is a class struggle between a few thousand billionaires versus the rest of humanity.
waffletower 3y ago
I have the belief that models should be allowed to ingest everything, just as a human is allowed. We are not yet at the stage where AI is autonomous, they currently are designed to require human agency for input, human agency for evaluation of output, and finally human agency for the dissemination of select output. This last important stage is well understood in the field of photography, but currently ignored in AI stewardship dialogues. Ultimately, it is the responsibility of the human agent who selects AI information products to determine its legality and appropriateness, just as if they had snapped a photograph and are wrestling with the decision whether or not it should be distributed in a particular medium. It takes a fairly selfish consciousness to become obsessed with the desire to prevent AI models access to information and disregard the collective benefits of rich information availability to training.
- musTY8 3y ago
  I like your view. From my limited knowledge, I speculate that if AI was developed further, and given as much publicly available data across a variety of scholarly topics as reasonably possible, it could potentially use statistics and such to help us find correlations across many different fields that a human would never think of. Whether it would revolutionize analytics or not I don't know as I am not qualified to say, but it's fun to dream of positive change these tools could bring.
zitterbewegung 3y ago
Considering the state of ai right now you probably are better off deleting your robots.txt file
bachmeier 3y ago
> All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. I don't know that this is true for the US. As far back as I can remember, there have been questions about whether a robots.txt file means you don't have permission to engage in those activities. The CFAA is one law that has repeatedly come up. See for example https://www.natlawreview.com/article/doj-revises-policy-cfaa-prosecution-to-reflect-developments-web-scraping-and-other https://www.natlawreview.com/article/doj-revises-policy-cfaa... It might be the case that there is nothing there legally, but I don't think I'd describe the actions of search engines as being driven by a moral imperative.
Macha 3y ago
> The thing I somewhat struggle with is that after 20-30 years of calls for shorter copyright terms, lesser restrictions on content you access publicly, and what you can do with it, we are now in the situation where the arguments are quickly leaning the other way. "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... Are these mutually exclusive? If you couldn't make Avengers movie Thanos memes but all the 90s X-Men and Spiderman content was a free for all, I think a lot of people would take that trade off.
omoikane 3y ago
There is no legal agreement to follow robots.txt, but it appears to have came up a few times (from the first search result for "court cases involving robots.txt"): https://www.robotstxt.org/faq/legal.html https://www.robotstxt.org/faq/legal.html If an "ai.txt" were to exist, I hope it's a signal for opt-in rather than opt-out. Whereas "robots.txt" being an explicit signal for opt-out might be useful because people who build public websites generally want their websites to be discovered, it seemed unlikely that training unknown AI would be a use case that content creators had in mind, considering that most existing content predates current AI systems.
balaji1 3y ago
With search engines and other crawlers, there wasn't easy ways to monetize "copyright theft" at scale. Google, which had the biggest share of eyeballs, was much more equitable in sharing revenue to content producers (who wanted to monetize). And Google was probably more just in taking action against copyright theft. Individual high value IP was always much less accessible (not available as a webpage on the internet). Gen AI/LLMs with the internet scale data is too powerful and maybe easier to monetize.
User23 3y ago
> there is no "legal" agreement to follow those rules Yes there certainly is[1]. The robots.txt clearly specifies authorized use and violating it exceeds that authorization. Now granted good luck getting the FBI to doorkick their friends at Google and other politically connected tech companies, but as the law is written crawlers need to honor the site owner's robots.txt. [1] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
quenix 3y ago
I think you fundamentally misunderstood the OP's point. They're not trying to use their ai.txt as any sort of deterrent, legal or otherwise. They are trying to use it as a form of extended metadata for training AIs. Essentially, "ah I see you're training using my website! Here's some extra info about it: [...]"
yafbum 3y ago
robots.txt is a successful coordination mechanism between website operators and crawlers. It is not in any way a security mechanism meant to address adversarial situations, as you seem to make it out to be.
wwweston 3y ago
> The thing I somewhat struggle with is that after 20-30 years of calls for shorter copyright terms, lesser restrictions on content you access publicly, and what you can do with it, we are now in the situation where the arguments are quickly leaning the other way. There've always been solid human arguments for sustaining copyright legally. The balance is the tricky part. On one hand we had a period where terms got too long, and some of the really aggressive legal enforcement from 20 years ago before stakeholders actually figured out how to get into digital markets were was entitled and useless. The pendulum also swung the other way with things like buffet streaming services essentially offering an economic bargain for creators with a sliver of compensatory difference from piracy but with none of piracy's actual benefits (people who simply pirate know they're not participating in a relationship of economic support with creators and might be persuaded to, someone who uses Spotify is under the illusion there's something fully legit on that front). But the fundamental copyright bargain -- creators can recoup investments of time and effort in proportion to how popular engagement with their work is -- has always made sense. > "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... Both these things can be true: (1) Using a work as training data for AI is a very novel use, it's entirely plausible there should be novel considerations and rights to go with it. (2) The incentive & benefits of copyrights have diminishing returns the longer the horizons are, while the cost in terms of social inaccessibility only increase. Where that's balanced out precisely is a debatable question, but something longer than a human lifespan is probably on the wrong side.
dragonwriter 3y ago
> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... I don’t know who “we” are, but I absolutely don't want “stricter copyright law when it comes to AI”. More clarity? Sure. Narrowing fair use? No fucking way.

shanebellone 3y ago

"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.

alwayslikethis 3y ago
Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts
- shanebellone 3y ago
  "Why not serve fake garbage indistinguishable from real content by a computer, like LLM output?" Serving more than the minimum wastes resources. Worse yet, a better solution would cost my time. "Sending errors just incentivizes bot owners to fix the identifiable parts" Sure, someone could make or configure their scraper perfectly. "Perfect" is now the table stakes though. Edit: My solution strives to cause an unproportional expense in order to circumvent. I want 10x on my time.
- twelve40 3y ago
  it'd be cool to be able to fingerprint that garbage, too. Like, sprinkle some hashes here and there (or something like that) so that you can later uniquely look up your own "content" being stolen by chatbots and which ones.
  - shanebellone 3y ago
    You can. I can't think of the appropriate term though. Hopefully someone else chimes in here with a link.
- ape4 3y ago
  I like this idea. Of course it would have to be only to robots that visit a page disallowed by the robots.txt
- dspillett 3y ago
  > Sending errors just incentivizes bot owners to fix the identifiable parts Nah. It'll just make them fake their identity so it is harder to tell the traffic is from a bot.
spc476 3y ago
It might be a better idea to serve up a 418 ("I'm a tea pot") with a line line text file saying "I'm not an HTTP server". That solved a problem I had with bots making HTTP requests to my gopher server [1]. Serving up a 503 informs the bot that there's a server issue and it may try again later. A 418 informs the bot that it made an erroneous request and such an odd error code might get someone to look into it and stop. [1] https://boston.conman.org/2019/09/30.2 https://boston.conman.org/2019/09/30.2
- shanebellone 3y ago
  This is very interesting. I've bookmarked the link. Thanks for sharing. I believe minimal is best and this might fit nicely within my larger system. Do you approach other problems with a similar mindset?

George83728 3y ago

robots.txt works for the major search engines who voluntarily abide by it, so it isn't a failed system. Just because it doesn't work on everybody doesn't mean it's useless.

Karunamon 3y ago

On the contrary, it works perfectly well for normal, non-bad actors running services used by most of the public. That includes search engines and stuff like archive.org. A robots.txt set to deny all will result in your site not showing up on any search engine that matters. It doesn't work for bad actors, but then again, nothing really does.

bombolo 3y ago

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... AI is being used to do copyright laundering, at the same time "we", the people who can't afford to run our own AI, are still subject to absurd rules that AI owners get to ignore, apparently.

rhn_mk1 3y ago
The barrier to running an AI model is getting lower every day, so the threshold for ignoring copyright is getting lower with it.
- gavinhoward 3y ago
  You are mistaken if you think companies will allow common people to ignore copyright on their IP. The only IP that will be allowed to be stolen is that of other common people.
  - alphanullmeric 3y ago
    You can’t steal an idea.
    - gavinhoward 3y ago
      Yes, you can. You can do that by making the monetary value of the idea zero when it used to be non-zero.
      - alphanullmeric 3y ago
        Stealing requires physical force, whether that be on the property or person. By your logic, independently discovering the idea is also stealing.
        gavinhoward 3y ago
        Stealing does not require physical force.

brookst 3y ago

> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.

samwillis 3y ago
Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction. If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static). With an ai.txt there is no potential downside to violating it - unless we get new legislation enforcing its legal standing. The nature of ML models is that it's opaque what content exactly it's trained on, there is no obvious retaliation or retribution.
- Wowfunhappy 3y ago
  > But my point is that it's not a system that we should copy for preventing the use of content in training AI. I don't think that's what OP is envisioning based on their post!
- BurningFrog 3y ago
  OP is trying to give helpful info to the AI, not set boundaries for it.
- capableweb 3y ago
  > But my point is that it's not a system that we should copy for preventing the use of content in training AI The purpose OP is suggesting in the submission is the opposite, help AI crawlers to understand what the page/website is about without actually having to infer the purpose from the content itself.
  - Xelynega 3y ago
    Isn't that the entire point of the semantic web?

prepend 3y ago

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... While I’m sure others than you share this opinion, I don’t think it’s as uniform as the more common “shorten/rationalize copyright terms and fair use” crowd “we.” I consider myself a knowledge worker and a pretty staunch proponent of floss and am perfectly fine with training AI on everything publicly available. While create stuff, I don’t make a living off selling particular copies of things I make, so my self preservation bias isn’t kicking in as much as someone who does want to sell items of their work. But I also made some pretty explicit choices in the 90s based on where I thought IP would go so I was never in a position where I had to sell copies to survive. My decision was more pragmatic first and philosophical second. I think someone entering the workforce now probably wants to align their livelihood with AI training on everything and not go against that. Even if US/Euro law limits training, there’s no way all other countries are going to, so it’s going to happen. And I don’t think it’s worth locking down the world to try to stop AIs from training on text, images, etc.

JohnFen 3y ago
Fair enough. But there should be some mechanism where people who don't want their works to contribute to AI training to be able to prevent that without having to resort to removing their works from the web.
- prepend 3y ago
  I think people who don’t want their content contributing to AI shouldn’t have it on the public web. There are many ways to restrict access. Use one of them. But if you respond to an anonymous http request with content then it shouldn’t matter if it’s a robot looking at it or a human (or a man or a woman or whatever). I think this both for simplicity and that I foresee a future where human consciousness is simulated and basically an AI. I don’t want to have rules that biological humans can view and digital humans can’t.
  - JohnFen 3y ago
    > I think people who don’t want their content contributing to AI shouldn’t have it on the public web. Practically speaking, that's the only effective solution. I just think that it's a shame that's necessary. It would be better for everyone if there wasn't a disincentive to making works publicly available. > I don’t want to have rules that biological humans can view and digital humans can’t. This is a point we disagree on. And "digital humans"? I would argue that such a thing can't exist, if you mean "human" in any way other than rough analogy.

mschuster91 3y ago

> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. That depends what you expect from it. For the purpose of limiting crawlers, at least the major search engines respect it.

majewsky 3y ago

> All a robots.txt is is a polite request to please follow the rules in it At least in my country (Germany), respecting robots.txt is a legal requirement for data mining. See German Copyright Code, section 44b: https://www.gesetze-im-internet.de/urhg/__44b.html https://www.gesetze-im-internet.de/urhg/__44b.html (IANAL)

FinnKuhn 3y ago

if you do data mining in the EU you are legally required to respect robots.txt afaik

LawTalkingGuy 3y ago
This is how it is a failed system. It doesn't really do what most people who have one think it does and yet it gets rolled into law and everyone now has to deal with it constantly while still not fixing the original issue. It's like the EU doesn't understand that bad law has a negative value.

IanCal 3y ago

The poster wants the opposite - a way of explicitly helping AI systems/etc to use their site. If people ignore it, they're just giving up a bit of help.

circuit10 3y ago
Yes, I feel like this person only read the title and not the text of the post and made an assumption
- benatkin 3y ago
  Not an assumption. Just a response to the title of the article.
  - circuit10 3y ago
    They said "Using robots.txt as a model for anything doesn't work." but it does work for the case described in the text
    - benatkin 3y ago
      What works? This is an idea, not anything that's been throughly tried.

none_to_remain 3y ago

Robots.txt is meant as an aid to crawlers. "This stuff is not useful to index," rather than a blocking mechanism

safety1st 3y ago

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... This gross generalization of other people's views on important issues is really offensive. My view is that the Copyright Act of 1976 had it about right when they established the duration of copyright. My view is that members of Congress were handsomely rewarded by a specific corporation to carve out special exceptions to this law because they wanted larger profits. "We" didn't call the Copyright Term Extension Act of 1998 the "Mickey Mouse Act" for nothing. It's also no coincidence that Disney is now the largest media company in the world. Reducing copyright term extension has everything to do with restoring competition and creativity to our economy, and reversing corruption that borders on white collar crime. It has nothing to do with AI. Don't recruit me into some bullshit argument that rewrites history and entrenches Disney's ill-gotten monopoly.

casey2 3y ago
Disney has a monopoly on media because they still have the copyright to their IP from the 20s? LOL! Companies that can leverage this new wave of AI will have, in reality, 1000x the advantage that you believe Disney has.
- safety1st 3y ago
  Yes. There's this little thing called brand value. Disney has one of the most valuable brands in the world. Forbes estimated it at being worth about $60 billion as I recall. That brand was built heavily over many decades on IP that dates back to the 1920s, such as the most recognizable Disney character, Mickey Mouse. They manipulated the law to enhance the value of that IP and thereby gained an edge over their competitors. That's a big part of why they now enjoy such a dominant position. None of this is especially controversial (you will get a very different spin from Disney of course). If you want to comment about how business works you should read history and learn how business works first. AI luminary that you are, if you choose to remain ignorant then I guess this whole cycle will happen again with AI.
- Spivak 3y ago
  The argument goes that copyright has allowed massive corporations to buy up and exert near total control over all of our shared stories. And when you own the cultural touchstones of whole generations that gives you power that no one else can ever wield. There is a massive amount of amazing stories based on ancient myths because it's one of the few large corpora that isn't copywritten. Once you see it in media you can't unsee it. The only space where that kind of creativity can thrive anymore is fan-fiction which lives in weird limbo where it's illegal but the copyright owners don't care. And when you want to bring any of it to the mainstream you have to hide it, all of Ali Hazelwoods books are reworked fanfics because she can't use the actual characters that inspired her -- her most famous book "The Love Hypothesis" is a Reylo fic. Go check out https://archiveofourown.org/media https://archiveofourown.org/media and see how many works are owned by a few large corporations.

Retric 3y ago

That’s factually incorrect. Germany for example does explicitly give machine readable permissions like robots.txt legal weight. In general without a fair use exemption or permission from robots.txt saving a copy of a website’s content to your own servers is copyright infringement. Purely factual information like Amazon’s prices isn’t protected by copyright, but if you want to save artwork or source files to train AI, that’s a copyright issue even before you get into the possibility of your AI being considered a derivative work.

deleted 3y ago

[deleted]

hoofhearted 3y ago

I don’t really agree with your sentiment. Robots.txt have served the simple purpose of directing bots like Google to the different parts of your website since the beginning of internet time. They still serve the same purpose, they tell bots where to go, and most importantly, they tell bots how to find your site map. Robots.txt is not there to prevent malicious crawlers from accessing pages as you have suggested. The robots.txt file acts simply like a garden gate. The good and honest people will honor the gate, while the more malicious might ignore it and hop the fence or something.

mort96 3y ago

Do you think there's a space between "you will never ever get to do anything at all with popular media until at least a hundred years after you're dead" and "anyone and any company can do anything they with everything I produce as long as it goes through an LLM"? Is it really so hard to think people may be against both of those extremes? There's a phrase I like which describes what you're doing. It's "vaguely gesturing at imagined hypocrisy".

shadowgovt 3y ago

I disagree it has failed as a system. While it does not substitute for authentication / authorization, reputable crawlers respect it, and there'd be a lot more traffic load on sites if they didn't have a way to tell reputable crawlers "please stop." Similarly, extending robots.txt to direct AI would have a similar effect: not sufficient, but useful (if for no other reason than to make it easy to distinguish reputable AI projects from ones that feel like they own the Internet to do with as they please).

FpUser 3y ago

>"All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative" Up until the point when some person / entity with the deep pockets will put a clear license / terms of use on their site that prohibits ignoring of robots.txt and would be willing to sue the ignorant.

hiddencost 3y ago

The point of robots.txt is to inform well behaved scrapers about how to behave. It is not designed nor intended to prevent bad actors. Which is good design: don't pretend to solve problems you can't.

drbawb 3y ago

It's not actually contradictory at all when you consider the root of the issue is about power asymmetry between individual creators and the corporations. Copyright terms were lobbied for, and primarily benefit the large corporations. They're symbolic of corporate overreach, that's why they're unpopular. Meanwhile, now that the laws are inconvenient for them, tech companies are straight up ignoring labeling their training data to respect IP law. Labeling the data would be expensive, thereby eroding profits. The loss of usable data would also harm the efficacy of their models, and the time spent classifying the data will hamper their iteration time. The ideas are only dissonant if you are looking at the trees (copyright term, DMCA, right to repair, etc.) and not the forest: which is a class struggle between a few thousand billionaires versus the rest of humanity.

waffletower 3y ago

I have the belief that models should be allowed to ingest everything, just as a human is allowed. We are not yet at the stage where AI is autonomous, they currently are designed to require human agency for input, human agency for evaluation of output, and finally human agency for the dissemination of select output. This last important stage is well understood in the field of photography, but currently ignored in AI stewardship dialogues. Ultimately, it is the responsibility of the human agent who selects AI information products to determine its legality and appropriateness, just as if they had snapped a photograph and are wrestling with the decision whether or not it should be distributed in a particular medium. It takes a fairly selfish consciousness to become obsessed with the desire to prevent AI models access to information and disregard the collective benefits of rich information availability to training.

musTY8 3y ago
I like your view. From my limited knowledge, I speculate that if AI was developed further, and given as much publicly available data across a variety of scholarly topics as reasonably possible, it could potentially use statistics and such to help us find correlations across many different fields that a human would never think of. Whether it would revolutionize analytics or not I don't know as I am not qualified to say, but it's fun to dream of positive change these tools could bring.

zitterbewegung 3y ago

Considering the state of ai right now you probably are better off deleting your robots.txt file

bachmeier 3y ago

> All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. I don't know that this is true for the US. As far back as I can remember, there have been questions about whether a robots.txt file means you don't have permission to engage in those activities. The CFAA is one law that has repeatedly come up. See for example https://www.natlawreview.com/article/doj-revises-policy-cfaa-prosecution-to-reflect-developments-web-scraping-and-other https://www.natlawreview.com/article/doj-revises-policy-cfaa... It might be the case that there is nothing there legally, but I don't think I'd describe the actions of search engines as being driven by a moral imperative.

Macha 3y ago

omoikane 3y ago

There is no legal agreement to follow robots.txt, but it appears to have came up a few times (from the first search result for "court cases involving robots.txt"): https://www.robotstxt.org/faq/legal.html https://www.robotstxt.org/faq/legal.html If an "ai.txt" were to exist, I hope it's a signal for opt-in rather than opt-out. Whereas "robots.txt" being an explicit signal for opt-out might be useful because people who build public websites generally want their websites to be discovered, it seemed unlikely that training unknown AI would be a use case that content creators had in mind, considering that most existing content predates current AI systems.

balaji1 3y ago

With search engines and other crawlers, there wasn't easy ways to monetize "copyright theft" at scale. Google, which had the biggest share of eyeballs, was much more equitable in sharing revenue to content producers (who wanted to monetize). And Google was probably more just in taking action against copyright theft. Individual high value IP was always much less accessible (not available as a webpage on the internet). Gen AI/LLMs with the internet scale data is too powerful and maybe easier to monetize.

User23 3y ago

> there is no "legal" agreement to follow those rules Yes there certainly is[1]. The robots.txt clearly specifies authorized use and violating it exceeds that authorization. Now granted good luck getting the FBI to doorkick their friends at Google and other politically connected tech companies, but as the law is written crawlers need to honor the site owner's robots.txt. [1] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act

quenix 3y ago

I think you fundamentally misunderstood the OP's point. They're not trying to use their ai.txt as any sort of deterrent, legal or otherwise. They are trying to use it as a form of extended metadata for training AIs. Essentially, "ah I see you're training using my website! Here's some extra info about it: [...]"

yafbum 3y ago

robots.txt is a successful coordination mechanism between website operators and crawlers. It is not in any way a security mechanism meant to address adversarial situations, as you seem to make it out to be.

wwweston 3y ago

> The thing I somewhat struggle with is that after 20-30 years of calls for shorter copyright terms, lesser restrictions on content you access publicly, and what you can do with it, we are now in the situation where the arguments are quickly leaning the other way. There've always been solid human arguments for sustaining copyright legally. The balance is the tricky part. On one hand we had a period where terms got too long, and some of the really aggressive legal enforcement from 20 years ago before stakeholders actually figured out how to get into digital markets were was entitled and useless. The pendulum also swung the other way with things like buffet streaming services essentially offering an economic bargain for creators with a sliver of compensatory difference from piracy but with none of piracy's actual benefits (people who simply pirate know they're not participating in a relationship of economic support with creators and might be persuaded to, someone who uses Spotify is under the illusion there's something fully legit on that front). But the fundamental copyright bargain -- creators can recoup investments of time and effort in proportion to how popular engagement with their work is -- has always made sense. > "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... Both these things can be true: (1) Using a work as training data for AI is a very novel use, it's entirely plausible there should be novel considerations and rights to go with it. (2) The incentive & benefits of copyrights have diminishing returns the longer the horizons are, while the cost in terms of social inaccessibility only increase. Where that's balanced out precisely is a debatable question, but something longer than a human lifespan is probably on the wrong side.

dragonwriter 3y ago

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... I don’t know who “we” are, but I absolutely don't want “stricter copyright law when it comes to AI”. More clarity? Sure. Narrowing fair use? No fucking way.