2 ms·

Google respects the "robot.txt" and asks you to use it to opt out of their crawling. Parent's point is if your own scaping army respects the "scaping.txt" and

Google respects the "robot.txt" and asks you to use it to opt out of their crawling. Parent's point is if your own scaping army respects the "scaping.txt" and goes down on Google as they don't opt-out in their scraping.txt, it probably wouldn't fly.

kgwgk 3y ago
I don't understand. What does "Rules for thee but not for me" mean if "google is allowed to scrape" whatever people allows Google to scrape but "you’re not allowed to scrape google" because using the same rules google.com/robots.txt says User-agent: * Disallow: /search ....
- makeitdouble 3y ago
  There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
  - kgwgk 3y ago
    > Google [...] won't let other people to make up their own self-beneficial rules in a similar way. What "other people"? If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you. If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to. To be clear, I understand the original comment as LLM companies say "I can use your content and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape your content and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine" and that doesn't seem a valid analogy.
    - makeitdouble 3y ago
      You should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them". I see it as the equivalent of the spam mail that require the user to login to disable them.

kgwgk 3y ago

I don't understand. What does "Rules for thee but not for me" mean if "google is allowed to scrape" whatever people allows Google to scrape but "you’re not allowed to scrape google" because using the same rules google.com/robots.txt says User-agent: * Disallow: /search ....

makeitdouble 3y ago
There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
- kgwgk 3y ago
  > Google [...] won't let other people to make up their own self-beneficial rules in a similar way. What "other people"? If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you. If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to. To be clear, I understand the original comment as LLM companies say "I can use your content and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape your content and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine" and that doesn't seem a valid analogy.
  - makeitdouble 3y ago
    You should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them". I see it as the equivalent of the spam mail that require the user to login to disable them.