4 ms·
There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made
by makeitdouble 3y ago
There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
- kgwgk 3y ago> Google [...] won't let other people to make up their own self-beneficial rules in a similar way. What "other people"? If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you. If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to. To be clear, I understand the original comment as LLM companies say "I can use your content and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape your content and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine" and that doesn't seem a valid analogy.
- makeitdouble 3y agoYou should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them". I see it as the equivalent of the spam mail that require the user to login to disable them.