6 ms·

Like google is allowed to scrape the whole internet but you’re not allowed to scrape google. Rules for thee but not for me

kgwgk 3y ago
What rules? Google won’t scrape your part of the internet if you don’t allow it, right?
- makeitdouble 3y ago
  Google respects the "robot.txt" and asks you to use it to opt out of their crawling. Parent's point is if your own scaping army respects the "scaping.txt" and goes down on Google as they don't opt-out in their scraping.txt, it probably wouldn't fly.
  - kgwgk 3y ago
    I don't understand. What does "Rules for thee but not for me" mean if "google is allowed to scrape" whatever people allows Google to scrape but "you’re not allowed to scrape google" because using the same rules google.com/robots.txt says User-agent: * Disallow: /search ....
    - makeitdouble 3y ago
      There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
      - kgwgk 3y ago
        > Google [...] won't let other people to make up their own self-beneficial rules in a similar way. What "other people"? If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you. If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to. To be clear, I understand the original comment as LLM companies say "I can use your content and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape your content and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine" and that doesn't seem a valid analogy.
        makeitdouble 3y ago
        You should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them". I see it as the equivalent of the spam mail that require the user to login to disable them.
rvnx 3y ago
Also the main business model of Google (and of search engines in general) is to republish rearranged snippets of copyrighted content and even serve whole copies of the content (googleusercontent cache), without prior authorization of the copyright holders, and for-profit. It’s completely illegal if you think about it. So why LLMs who crawl the internet to present snippets and information should be treated differently from Google ? (who also reproduce verbatim the same content without paying any compensation to the copyright owners (all types: text, image, code)
- az226 3y ago
  It wouldn’t. Facebook is delusional if they think the license can pass muster. Presumably you can’t build an LLM that is a competitor of LlaMA using its outputs. But AI weights are in legal gray zone for now. So it’s muddy waters and fair game for anyone who wants to take on the legal risks.
- panzi 3y ago
  Not wanting to defend the likes of Google, but search engines link the original source (in contrast to LLMs). Their basic idea is to direct people to your content. There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. (extremely simplified story)
  - pyrale 3y ago
    > Their basic idea is to direct people to your content. This is less and less true, as evidenced by the progression of 0-click searchs. > There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. This story screams antitrust.
    - account42 3y ago
      > This story screams antitrust. You're right, the number of news publishers that share a common owner is something that should be of concern to antitrust enforcers.

kgwgk 3y ago

What rules? Google won’t scrape your part of the internet if you don’t allow it, right?

makeitdouble 3y ago
Google respects the "robot.txt" and asks you to use it to opt out of their crawling. Parent's point is if your own scaping army respects the "scaping.txt" and goes down on Google as they don't opt-out in their scraping.txt, it probably wouldn't fly.
- kgwgk 3y ago
  I don't understand. What does "Rules for thee but not for me" mean if "google is allowed to scrape" whatever people allows Google to scrape but "you’re not allowed to scrape google" because using the same rules google.com/robots.txt says User-agent: * Disallow: /search ....
  - makeitdouble 3y ago
    There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
    - kgwgk 3y ago
      > Google [...] won't let other people to make up their own self-beneficial rules in a similar way. What "other people"? If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you. If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to. To be clear, I understand the original comment as LLM companies say "I can use your content and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape your content and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine" and that doesn't seem a valid analogy.
      - makeitdouble 3y ago
        You should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them". I see it as the equivalent of the spam mail that require the user to login to disable them.

rvnx 3y ago

Also the main business model of Google (and of search engines in general) is to republish rearranged snippets of copyrighted content and even serve whole copies of the content (googleusercontent cache), without prior authorization of the copyright holders, and for-profit. It’s completely illegal if you think about it. So why LLMs who crawl the internet to present snippets and information should be treated differently from Google ? (who also reproduce verbatim the same content without paying any compensation to the copyright owners (all types: text, image, code)

az226 3y ago
It wouldn’t. Facebook is delusional if they think the license can pass muster. Presumably you can’t build an LLM that is a competitor of LlaMA using its outputs. But AI weights are in legal gray zone for now. So it’s muddy waters and fair game for anyone who wants to take on the legal risks.
panzi 3y ago
Not wanting to defend the likes of Google, but search engines link the original source (in contrast to LLMs). Their basic idea is to direct people to your content. There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. (extremely simplified story)
- pyrale 3y ago
  > Their basic idea is to direct people to your content. This is less and less true, as evidenced by the progression of 0-click searchs. > There are countries where content companies didn't like what Google does: Google took them out of the index -> suddenly they where ok with it again so that Google put them in again. This story screams antitrust.
  - account42 3y ago
    > This story screams antitrust. You're right, the number of news publishers that share a common owner is something that should be of concern to antitrust enforcers.