4 ms·

I don't think it's an exaggeration to say that LLMs might lead to the end of the open web, or at least a drastically reduced version of it. So much of these mod

by rurp 3y ago

I don't think it's an exaggeration to say that LLMs might lead to the end of the open web, or at least a drastically reduced version of it. So much of these model's utility is in directly competing with the producers of the training data. Content creators and aggregators are seeing more and more reason to restrict and limit access, to avoid having AI companies consume all of their data and then be the ones making money from it going forward. I fear that LLMs are going to cause the internet to be a much worse and less open space.

buildbot 3y ago
No it's the death of the corporate content hosting web - the open web was never about making money with your blog post/irc chat/usenet group/etc content, at least in my opinion. Let data be free! If someone wants to use it to make money, well, it's open, just like open source. It's still not okay to take open source work and claim it as your own, which is what copyright should be limited to. Stealing a photo or plagiarizing an essay is intrinsically different than just having a copy read by something, be it human or an mechanical process such as training a LLM.
- sundaeofshock 3y ago
  Maybe I don’t want my non-corporate art to be a part of some large corporation’s training data.
  - k_vi 3y ago
    User-agent: GPTBot Disallow: /
    - jamilton 3y ago
      That was my gut reaction too, but presumably unless it becomes regulated, at least some competitors to OpenAI won't respect any robots.txt and thus any open content might be training data.
    - prmoustache 3y ago
      ATMO you shouldn't have to maintain knowledge of what kind of crawler bot exist and having to maintain deny list. It should be the opposite, only expressedly allowed content should be crawled by mainaining allow lists.
      - wraptile 3y ago
        You can do the opposite since the inception of robots.txt: User-agent: * Disallow: / and then whitelist google bot and whatnot. Most of the web is already configured this way. Just check robots.txt of any major website, e.g. https://twitter.com/robots.txt https://twitter.com/robots.txt
        xnx 3y ago
        The Allow: directive was an extension to robots.txt added later.
    - gorbachev 3y ago
      User-Agent: <new technology category>Bot
  - andybak 3y ago
    Once again I have to point out that LLMs are not and will not be restricted to "large corporations". Even if initial training is prohibitively expensive, open base models already exist. "corporate this" and "corporate that" are handy rhetorical flourishes but they distort the debate.
- twelve40 3y ago
  if you want to give away your data for free, do it, but speak for yourself, not everyone is in position to be able to do it. Just like you can't force other people to give away their software (or anything else, really) for free if they choose to charge for it.
- deleted 3y ago
  [deleted]
lostmsu 3y ago
Let's assume that happens. How do I hedge against it? Is there a convenient way to mirror the bits of the web that are open now? Perhaps a mirror of archive/WayBack machine that could be viewed locally similar to Wikipedia dumps?
- paulmd 3y ago
  Yes, commoncrawl indexes
  - lostmsu 3y ago
    100TB. Dang! I will need to expand my storage significantly to mirror it.
    - paulmd 3y ago
      100TB so far ;) And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of the internet. the more practical thing is that (the royal) they have set up a rclone like utility that allows you to retrieve a slice of the whole snapshot, if you know your domain. it would be extremely funni if commoncrawl was a de-facto ipfs, allowing queries along domains (or arbitrary dimensions/clustering) of interest.
mike_hearn 3y ago
Really? Seems like it's not so different conceptually to search engines. They also make money by indexing "training data" of a sort. But the open web trundles on regardless, because the search engines found ways to cut the content producers in on it. I see no reason why that can't be the case here too, with AI companies training their models to act more like search engines when data comes from certain sources - i.e. they'll point you in the right direction but not directly answer you. That will suck for the AI users, but, for many questions the right answer will be available in open or bulk licensable training sets anyway. For example you can get legal access to nearly all books by doing deals with publishers, as Google Books has demonstrated, you can get access to map data by generating it yourself or doing deals with digital map companies etc.
- bryanrasmussen 3y ago
  >Really? Seems like it's not so different conceptually to search engines. They also make money by indexing "training data" of a sort. OK, so if a writer X has a blog to put up samples of their work to drive people to buy books and to get writing assignments and someone uses ChatGPT to write something in the style of X - this naively seems like a hit on that author's ability to sell their skills. And I don't think it is fixable by making the AI act more like a search engine.
  - mike_hearn 3y ago
    Seems like a rerun of the argument over snippets, or the "answer onebox" as Google used to call it where the info you need is directly inlined in the SERP rather than being behind a link.
    - bryanrasmussen 3y ago
      it is exactly a rerun of that argument, except that the UX of AI is different. Basically AI is structured to make every result an answer onebox whereas in search this only happens sometimes.
    - rurp 3y ago
wraptile 3y ago
Is this open web in the room with us? Cheekiness aside, the myth of beautiful open web always seems to be exagerated. Most content on the web is already generated, ugly and spammy. Most of the traffic is already owned by mega corporations. One could easily argue that it's unlikely it'll get worse - if anything, AI could empower competition as now a group of 3 passionate, free writers can compete with agenda-driven, for profit corporations on a similar level. This could very well make the web more open and free as it makes the web more accessible.
- ben_w 3y ago
  > Most content on the web is already generated, ugly and spammy. Now? Yes. I grew up when it was just ugly, and by ugly I mean animated gif backgrounds with obvious seams on the tile boundary. *old man shakes fist at The Cloud*

by rurp 3y ago

buildbot 3y ago

No it's the death of the corporate content hosting web - the open web was never about making money with your blog post/irc chat/usenet group/etc content, at least in my opinion. Let data be free! If someone wants to use it to make money, well, it's open, just like open source. It's still not okay to take open source work and claim it as your own, which is what copyright should be limited to. Stealing a photo or plagiarizing an essay is intrinsically different than just having a copy read by something, be it human or an mechanical process such as training a LLM.

sundaeofshock 3y ago
Maybe I don’t want my non-corporate art to be a part of some large corporation’s training data.
- k_vi 3y ago
  User-agent: GPTBot Disallow: /
  - jamilton 3y ago
    That was my gut reaction too, but presumably unless it becomes regulated, at least some competitors to OpenAI won't respect any robots.txt and thus any open content might be training data.
  - prmoustache 3y ago
    ATMO you shouldn't have to maintain knowledge of what kind of crawler bot exist and having to maintain deny list. It should be the opposite, only expressedly allowed content should be crawled by mainaining allow lists.
    - wraptile 3y ago
      You can do the opposite since the inception of robots.txt: User-agent: * Disallow: / and then whitelist google bot and whatnot. Most of the web is already configured this way. Just check robots.txt of any major website, e.g. https://twitter.com/robots.txt https://twitter.com/robots.txt
      - xnx 3y ago
        The Allow: directive was an extension to robots.txt added later.
  - gorbachev 3y ago
    User-Agent: <new technology category>Bot
- andybak 3y ago
  Once again I have to point out that LLMs are not and will not be restricted to "large corporations". Even if initial training is prohibitively expensive, open base models already exist. "corporate this" and "corporate that" are handy rhetorical flourishes but they distort the debate.
twelve40 3y ago
if you want to give away your data for free, do it, but speak for yourself, not everyone is in position to be able to do it. Just like you can't force other people to give away their software (or anything else, really) for free if they choose to charge for it.
deleted 3y ago
[deleted]

lostmsu 3y ago

Let's assume that happens. How do I hedge against it? Is there a convenient way to mirror the bits of the web that are open now? Perhaps a mirror of archive/WayBack machine that could be viewed locally similar to Wikipedia dumps?

paulmd 3y ago
Yes, commoncrawl indexes
- lostmsu 3y ago
  100TB. Dang! I will need to expand my storage significantly to mirror it.
  - paulmd 3y ago
    100TB so far ;) And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of the internet. the more practical thing is that (the royal) they have set up a rclone like utility that allows you to retrieve a slice of the whole snapshot, if you know your domain. it would be extremely funni if commoncrawl was a de-facto ipfs, allowing queries along domains (or arbitrary dimensions/clustering) of interest.

mike_hearn 3y ago

Really? Seems like it's not so different conceptually to search engines. They also make money by indexing "training data" of a sort. But the open web trundles on regardless, because the search engines found ways to cut the content producers in on it. I see no reason why that can't be the case here too, with AI companies training their models to act more like search engines when data comes from certain sources - i.e. they'll point you in the right direction but not directly answer you. That will suck for the AI users, but, for many questions the right answer will be available in open or bulk licensable training sets anyway. For example you can get legal access to nearly all books by doing deals with publishers, as Google Books has demonstrated, you can get access to map data by generating it yourself or doing deals with digital map companies etc.

bryanrasmussen 3y ago
>Really? Seems like it's not so different conceptually to search engines. They also make money by indexing "training data" of a sort. OK, so if a writer X has a blog to put up samples of their work to drive people to buy books and to get writing assignments and someone uses ChatGPT to write something in the style of X - this naively seems like a hit on that author's ability to sell their skills. And I don't think it is fixable by making the AI act more like a search engine.
- mike_hearn 3y ago
  Seems like a rerun of the argument over snippets, or the "answer onebox" as Google used to call it where the info you need is directly inlined in the SERP rather than being behind a link.
  - bryanrasmussen 3y ago
    it is exactly a rerun of that argument, except that the UX of AI is different. Basically AI is structured to make every result an answer onebox whereas in search this only happens sometimes.
  - rurp 3y ago

wraptile 3y ago

Is this open web in the room with us? Cheekiness aside, the myth of beautiful open web always seems to be exagerated. Most content on the web is already generated, ugly and spammy. Most of the traffic is already owned by mega corporations. One could easily argue that it's unlikely it'll get worse - if anything, AI could empower competition as now a group of 3 passionate, free writers can compete with agenda-driven, for profit corporations on a similar level. This could very well make the web more open and free as it makes the web more accessible.

ben_w 3y ago
> Most content on the web is already generated, ugly and spammy. Now? Yes. I grew up when it was just ugly, and by ugly I mean animated gif backgrounds with obvious seams on the tile boundary. *old man shakes fist at The Cloud*