3 ms·

I can’t help but think all this brouhaha over web scraping is just a rehash over the web scraping panic of the 1990s. Put it behind a robots.txt or better yet

I can’t help but think all this brouhaha over web scraping is just a rehash over the web scraping panic of the 1990s. Put it behind a robots.txt or better yet a login. This battle was fought and won. I’m not going back to the bad old days.

nicbou 3y ago
I want my website to be accessible to humans, and even indexed by search engines. I just don't want to be used as unpaid labour to train artificial intelligence. A bit like I will gladly build a bike for someone, but object to the bike being sold around the corner.
- puzzledobserver 3y ago
  I mostly agree with you, but as a counter-point: How would one precisely draw the lines between search engines and LLMs? One provides mostly airtight attributions, and the other is famous for hallucinating citations. But this doesn't sound like a distinction that can be used as the basis for law? Or are you less concerned with their design and more concerned with their purpose? Providing citations is fine, but creating content is not?
  - nicbou 3y ago
    One brings traffic to the original work, supporting its creators. The other takes uses the creators' labour without giving anything in return.
- jonathankoren 3y ago
  This was the same argument against search engines and in favor human curated hierarchies like original Yahoo and DMoz. How do you Wikipedia being used for products period, let alone used as data products, since that’s also repurposed unpaid labor?
  - nicbou 3y ago
    Wikipedia hosts people's hard work and makes it available to the world. Search engines help connect people to the things they want. At least they mostly do. AI inserts itself in the middle and strips the creators of credit, recognition or income.
    - jonathankoren 3y ago
      Given that you used the term “AI”, you may not know that there’s a lot more to general web search engine beyond a simple inverted index. There are multiple models (ie “AI”) used to determine things such as query intent, term disambiguation, query classification, and that’s even before we get to results ranking, or relevant snippet detection. So now, with that knowledge, do you believe that Wikipedia, or even the web crawl itself (because that’s literally what we’re talking about, a web crawl) should be used for anything beyond term naïve term indexing?
      - nicbou 3y ago
        Given the context, it's assumed that "AI" will not be used merely for providing better search results, but for keeping people on Google.com with "original" answers.
        jonathankoren 3y ago
        So just like the already existing info boxes, which I presume you have a moral problem with?
        nicbou 3y ago
        You can opt out of them. I'm fine with them, although they tend to be comically wrong.
  - djhn 3y ago
    If I want to read up more on this, could you throw some search terms at me that would help me find some more specific pieces of this conversation/debate around search engines' access to data? Any books, particular legal cases, prominent people or organisations engaged in this debate?
- troyvit 3y ago
  I feel the same way too, but I have to ask myself, why do I deserve a search engine to automatically index my content and then show it to others without my having to give anything in return? Let's say I actually wrote something of value, put an ad next to it, and it generated a hundred thousand organic search views. Those 100k views were basically given to me for free by a search engine. Do I owe them anything for that?
  - undrcvr95 3y ago
    No of course you don't owe them anything. They opted in to doing this and didn't even consult you before doing so. They did this because they have found a way to monetize this, which is selling their own ads. On the other hand you just posted a blog post. You didn't opt in to anything related to search engines. You just posted something, that's it.
    - troyvit 3y ago
      Yeah that's a good point. I didn't make a contract with them.
JohnFen 3y ago
> or better yet a login. That's what I've done along with a growing number of others -- but I'd much prefer to be able to make it available to the general public. I mourn that's not possible without also aiding the training of AI.

nicbou 3y ago

I want my website to be accessible to humans, and even indexed by search engines. I just don't want to be used as unpaid labour to train artificial intelligence. A bit like I will gladly build a bike for someone, but object to the bike being sold around the corner.

puzzledobserver 3y ago
I mostly agree with you, but as a counter-point: How would one precisely draw the lines between search engines and LLMs? One provides mostly airtight attributions, and the other is famous for hallucinating citations. But this doesn't sound like a distinction that can be used as the basis for law? Or are you less concerned with their design and more concerned with their purpose? Providing citations is fine, but creating content is not?
- nicbou 3y ago
  One brings traffic to the original work, supporting its creators. The other takes uses the creators' labour without giving anything in return.
jonathankoren 3y ago
This was the same argument against search engines and in favor human curated hierarchies like original Yahoo and DMoz. How do you Wikipedia being used for products period, let alone used as data products, since that’s also repurposed unpaid labor?
- nicbou 3y ago
  Wikipedia hosts people's hard work and makes it available to the world. Search engines help connect people to the things they want. At least they mostly do. AI inserts itself in the middle and strips the creators of credit, recognition or income.
  - jonathankoren 3y ago
    Given that you used the term “AI”, you may not know that there’s a lot more to general web search engine beyond a simple inverted index. There are multiple models (ie “AI”) used to determine things such as query intent, term disambiguation, query classification, and that’s even before we get to results ranking, or relevant snippet detection. So now, with that knowledge, do you believe that Wikipedia, or even the web crawl itself (because that’s literally what we’re talking about, a web crawl) should be used for anything beyond term naïve term indexing?
    - nicbou 3y ago
      Given the context, it's assumed that "AI" will not be used merely for providing better search results, but for keeping people on Google.com with "original" answers.
      - jonathankoren 3y ago
        So just like the already existing info boxes, which I presume you have a moral problem with?
        nicbou 3y ago
        You can opt out of them. I'm fine with them, although they tend to be comically wrong.
- djhn 3y ago
  If I want to read up more on this, could you throw some search terms at me that would help me find some more specific pieces of this conversation/debate around search engines' access to data? Any books, particular legal cases, prominent people or organisations engaged in this debate?
troyvit 3y ago
I feel the same way too, but I have to ask myself, why do I deserve a search engine to automatically index my content and then show it to others without my having to give anything in return? Let's say I actually wrote something of value, put an ad next to it, and it generated a hundred thousand organic search views. Those 100k views were basically given to me for free by a search engine. Do I owe them anything for that?
- undrcvr95 3y ago
  No of course you don't owe them anything. They opted in to doing this and didn't even consult you before doing so. They did this because they have found a way to monetize this, which is selling their own ads. On the other hand you just posted a blog post. You didn't opt in to anything related to search engines. You just posted something, that's it.
  - troyvit 3y ago
    Yeah that's a good point. I didn't make a contract with them.

JohnFen 3y ago

> or better yet a login. That's what I've done along with a growing number of others -- but I'd much prefer to be able to make it available to the general public. I mourn that's not possible without also aiding the training of AI.