3 ms·

How can they truly 'de-list'? If the content is still open and accessible, what's to stop a spider from indexing it? I'm not a lawyer, but I have to imagine tha

by enjo 17y ago

How can they truly 'de-list'? If the content is still open and accessible, what's to stop a spider from indexing it? I'm not a lawyer, but I have to imagine that falls well within the domain of fair use as long as your not presenting the content back for public consumption.

gojomo 17y ago
The idea you could index a site against its wishes remains to be tested; the custom so far has been not to do so. Google tried indexing books without rightsholder permission, then backed down. If NewsCorp, NYTimesCo, WashingtonPost, ComcastNBC, etc. are unified against the idea of indexing-as-fair-use, and if (for business strategic reasons), the deep-pocketed MSFT and Google decide they can live with paying-for-permission (as a useful barrier to upstarts), who will prove the case in the courts or legislature?
chollida1 17y ago
> what's to stop a spider from indexing it? a robots.txt that forbids a particular spider from accessing the content. This assumes that the spider does respect the robots.txt, which googlebot does.
- Tichy 17y ago
  Is there a law against it? I thought robots.txt is more of a convention. However, Google already said that they would heed the robots.txt.
  - chollida1 17y ago
    > Is there a law against it? I don't know. I'd imagine that there isn't a law specifically written for robots.txt and scraping. If a case went to court I'd imagine it would be settled based on laws meant for things like "keep out" and "No trespassing" signs:)
  - itgoon 17y ago
    I think not heeding robots.txt would cause them more problems than it would solve. Currently, it gets used for more than "deny/allow all" - it is selective. If I have to worry about stuff I don't want indexed being indexed, then I'm going to block all crawling. Yes, I know that wouldn't be easy, but I wouldn't be alone in my sentiment.
storborg 17y ago
Even if this was proven as fair use, it would be quite difficult to make a usable search engine that didn't present at least some of the content back for public consumption. Imagine if Google search results just had a bunch of links that said, "link 1", "link 2", etc instead of the title and summary of the page.
blasdel 17y ago
Even if the spider doesn't respect robots.txt, the act of indexing is really obvious -- your webserver does not have to respond to their HTTP Requests if you don't want it to. It's not as facile as 'lock your doors' -- instead just not answering when they knock.

by enjo 17y ago

gojomo 17y ago

The idea you could index a site against its wishes remains to be tested; the custom so far has been not to do so. Google tried indexing books without rightsholder permission, then backed down. If NewsCorp, NYTimesCo, WashingtonPost, ComcastNBC, etc. are unified against the idea of indexing-as-fair-use, and if (for business strategic reasons), the deep-pocketed MSFT and Google decide they can live with paying-for-permission (as a useful barrier to upstarts), who will prove the case in the courts or legislature?

chollida1 17y ago

> what's to stop a spider from indexing it? a robots.txt that forbids a particular spider from accessing the content. This assumes that the spider does respect the robots.txt, which googlebot does.

Tichy 17y ago
Is there a law against it? I thought robots.txt is more of a convention. However, Google already said that they would heed the robots.txt.
- chollida1 17y ago
  > Is there a law against it? I don't know. I'd imagine that there isn't a law specifically written for robots.txt and scraping. If a case went to court I'd imagine it would be settled based on laws meant for things like "keep out" and "No trespassing" signs:)
- itgoon 17y ago
  I think not heeding robots.txt would cause them more problems than it would solve. Currently, it gets used for more than "deny/allow all" - it is selective. If I have to worry about stuff I don't want indexed being indexed, then I'm going to block all crawling. Yes, I know that wouldn't be easy, but I wouldn't be alone in my sentiment.

storborg 17y ago

Even if this was proven as fair use, it would be quite difficult to make a usable search engine that didn't present at least some of the content back for public consumption. Imagine if Google search results just had a bunch of links that said, "link 1", "link 2", etc instead of the title and summary of the page.

blasdel 17y ago

Even if the spider doesn't respect robots.txt, the act of indexing is really obvious -- your webserver does not have to respond to their HTTP Requests if you don't want it to. It's not as facile as 'lock your doors' -- instead just not answering when they knock.