4 ms·

Seems like it's intended for content stealing from every place that doesn't immediately implement Google's New Web Order as an addition to robots.txt. "Your do

by voytec 3y ago

Seems like it's intended for content stealing from every place that doesn't immediately implement Google's New Web Order as an addition to robots.txt. "Your do not enter sign uses font we don't like, so we'll just ignore it"

saagarjha 3y ago
What makes you think this? Why do you think Google actually cares about your sign if all they want to do is steal from you?
- oneeyedpigeon 3y ago
  IIRC, Google has precedent on this - e.g. scanning full books for search unless the owner explicitly refused.
  - dylan604 3y ago
    They are the ultimate ask for forgiveness rather than permission. Copyright has been a thing for a long long time before googs developed their scanning. They were well aware that it should have been an opt-in, but knew they’d never gain traction for their little project. So they bull in a China shop’d their way to a point of too far to stop them.
    - philipov 3y ago
      They don't even ask for forgiveness. They are "don't admit you've done anything wrong to begin with."
    - remus 3y ago
      Copyright is to do with protecting reproduction of works, no? What google has done here is scanning the book and indexed the content, presumably so it makes it easier for users to search books for relevant material. Assuming they don't reproduce large sections of copyrighted works in their search results I don't feel like they're doing anything wrong here.
      - tpxl 3y ago
        > Assuming they don't reproduce large sections of copyrighted works They do (or did). They showed the text around the search term, around a page or so, which made it possible to reconstruct the whole book without that much effort.
    - extra88 3y ago
      Yet the keep getting sued and keep winning in the courts, at least in the U.S. Seems like they have a pretty good grasp of how the laws work.
  - 411111111111111 3y ago
    Your phrasing makes it sound like that's a negative. I'm honestly surprised they're required to abstain from doing so at the author's request. You can only read the context of the match after finding the search result after all, not the whole book. It's an example of significant overreach of intellectual property from how I see it. The robot.txt rational doesn't apply there either, as their scanning does not impact anyone's resources. And it's been published, which makes it public by definition.
    - oneeyedpigeon 3y ago
      Oh, I agree with you. I think the whole idea of legislating against machines accessing public content is a very slippery slope.
Ferret7446 3y ago
To be clear, robots.txt is not legally binding, Google is not bound to follow it, and in fact I believe that Google doesn't follow it and hasn't for a very long time, for the simple reason that many sites' robots.txt file is wrong. The intent of robots.txt is to help crawlers, for example, to keep crawlers from getting stuck in a recursive loop of dynamic pages, or from crawling pages with no value. robots.txt is not for banning, restricting, or hindering crawlers.
- superkuh 3y ago
  That's just because google is a corporate person who is more equal than a human person. Human persons, at least in the USA, get charged under the CFAA 1030 law if they're using non-browser tools to access the public website of someone with power and/if they happen to rock the boat (like weev w/wget). That's not to say that I disagree. In most cases robots.txt is not legally binding. It only becomes a legal danger to not follow it when the person running the site has power and can buy a DA to indict you.
  - rafark 3y ago
    If a tool can access a url, does that not make it a browser?
    - TeMPOraL 3y ago
      Not under any but most narrow of meanings, i.e. "can follow URLs / can talk HTTP". By itself, it's not a browser to users, it's not a browser to software developers, and it's definitely not a browser to lawyers and judges.
      - rafark 3y ago
        Is there a legal definition of a web browser though? I think it’s an interesting topic.
  - TeMPOraL 3y ago
    > like weev w/wget Speaking of this and other cases of trying to punish someone for every iteration of a for loop - I wonder if the result would be the same if the accused drove actual browser to click stuff in a for loop, vs. using curl directly. I imagine the same, but then... ... what if they paid N people some token amount of money, to have each of those people do one step of the loop and send them the result? Does executing a for loop entirely on in part on the human substrate instead of in silico is seen as abuse under CFAA? (I have a feeling that it might not be - there's lots of jobs online and offline that involve one company paying lots of people some money for gathering information from their competitors, in a way the latter very much don't like.)
LinuxBender 3y ago
AFAIK the only way to reduce content stealing by bots is to add authentication requirements to a page and to detect if a real persons authentication is being shared by bots then instantly and automatically rotate their password each time that occurs.

by voytec 3y ago

saagarjha 3y ago

What makes you think this? Why do you think Google actually cares about your sign if all they want to do is steal from you?

oneeyedpigeon 3y ago
IIRC, Google has precedent on this - e.g. scanning full books for search unless the owner explicitly refused.
- dylan604 3y ago
  They are the ultimate ask for forgiveness rather than permission. Copyright has been a thing for a long long time before googs developed their scanning. They were well aware that it should have been an opt-in, but knew they’d never gain traction for their little project. So they bull in a China shop’d their way to a point of too far to stop them.
  - philipov 3y ago
    They don't even ask for forgiveness. They are "don't admit you've done anything wrong to begin with."
  - remus 3y ago
    Copyright is to do with protecting reproduction of works, no? What google has done here is scanning the book and indexed the content, presumably so it makes it easier for users to search books for relevant material. Assuming they don't reproduce large sections of copyrighted works in their search results I don't feel like they're doing anything wrong here.
    - tpxl 3y ago
      > Assuming they don't reproduce large sections of copyrighted works They do (or did). They showed the text around the search term, around a page or so, which made it possible to reconstruct the whole book without that much effort.
  - extra88 3y ago
    Yet the keep getting sued and keep winning in the courts, at least in the U.S. Seems like they have a pretty good grasp of how the laws work.
- 411111111111111 3y ago
  Your phrasing makes it sound like that's a negative. I'm honestly surprised they're required to abstain from doing so at the author's request. You can only read the context of the match after finding the search result after all, not the whole book. It's an example of significant overreach of intellectual property from how I see it. The robot.txt rational doesn't apply there either, as their scanning does not impact anyone's resources. And it's been published, which makes it public by definition.
  - oneeyedpigeon 3y ago
    Oh, I agree with you. I think the whole idea of legislating against machines accessing public content is a very slippery slope.

Ferret7446 3y ago

To be clear, robots.txt is not legally binding, Google is not bound to follow it, and in fact I believe that Google doesn't follow it and hasn't for a very long time, for the simple reason that many sites' robots.txt file is wrong. The intent of robots.txt is to help crawlers, for example, to keep crawlers from getting stuck in a recursive loop of dynamic pages, or from crawling pages with no value. robots.txt is not for banning, restricting, or hindering crawlers.

superkuh 3y ago
That's just because google is a corporate person who is more equal than a human person. Human persons, at least in the USA, get charged under the CFAA 1030 law if they're using non-browser tools to access the public website of someone with power and/if they happen to rock the boat (like weev w/wget). That's not to say that I disagree. In most cases robots.txt is not legally binding. It only becomes a legal danger to not follow it when the person running the site has power and can buy a DA to indict you.
- rafark 3y ago
  If a tool can access a url, does that not make it a browser?
  - TeMPOraL 3y ago
    Not under any but most narrow of meanings, i.e. "can follow URLs / can talk HTTP". By itself, it's not a browser to users, it's not a browser to software developers, and it's definitely not a browser to lawyers and judges.
    - rafark 3y ago
      Is there a legal definition of a web browser though? I think it’s an interesting topic.
- TeMPOraL 3y ago
  > like weev w/wget Speaking of this and other cases of trying to punish someone for every iteration of a for loop - I wonder if the result would be the same if the accused drove actual browser to click stuff in a for loop, vs. using curl directly. I imagine the same, but then... ... what if they paid N people some token amount of money, to have each of those people do one step of the loop and send them the result? Does executing a for loop entirely on in part on the human substrate instead of in silico is seen as abuse under CFAA? (I have a feeling that it might not be - there's lots of jobs online and offline that involve one company paying lots of people some money for gathering information from their competitors, in a way the latter very much don't like.)

LinuxBender 3y ago

AFAIK the only way to reduce content stealing by bots is to add authentication requirements to a page and to detect if a real persons authentication is being shared by bots then instantly and automatically rotate their password each time that occurs.