3 ms·

Has Rupert heard of /robots.txt?

by mthreat 17y ago

jister 17y ago
This is exactly the first thing that comes to my mind when I read about "indexing" in the article.
philk 17y ago
The point is not that he wants to cut Google off, it's that he wants leverage. Simply updating robots.txt won't effect Google at all and will blow what little leverage he has left.
alain94040 17y ago
Are you saying that if your content doesn't have a robots.txt, then everyone is allowed to copy the content of the site with no restrictions whatsoever, and that copyright law doesn't apply? I'm not a big fan of Murdoch, but it's an interesting discussion that should go beyond the obvious technical arguments.
- corruption 17y ago
  Imagine the world with strict copyright. Search engines would be illegal!
  - deleted 17y ago
    [deleted]
  - alain94040 17y ago
    I guess I'll ask again, differently. What do you mean by strict? Are you saying that today, there is an implicit understanding among a class of people (judges, CEOs of web 2.0 companies...?) to not apply the law strictly? That doesn't sound very likely. If Google started copying a full news site verbatim and offer its content when people search for news, that would definitely be wrong, robots.txt or not. Strict copyright includes fair use exceptions and so on. Where fair use stops and infringment starts is an interesting discussion for me.
- romland 17y ago
  You get moderated downwards for that comment? What is this, Slashdot?
  - dasil003 17y ago
    Uh, Google doesn't do just "anything" with the content, they index, summarize, and link back to the original article. There's no obvious copyright violation there. The GP was hyberbole that doesn't add anything to the debate. Ironically that's exactly what happens all the time on Slashdot.
  - alain94040 17y ago
    Yes, I got modded down. I didn't list my credentials on the topic, so maybe I should have. I did get sued and went through significant $$$ expenses to defend frivolous lawsuits, so anyone who blankly states "oh, use robots.txt and don't worry about a thing" sounds like a beginner in the intricacies of the law to me.
- Herring 17y ago
  That's kind of a red herring, because Murdoch wants his stuff to be copied by google regardless of how that argument is resolved.
- philk 17y ago
  It's not a particularly onerous requirement and far cheaper than calling your lawyers. If you're making the argument that the company that is automatically spidering your content should stop then your case is somewhat undermined by failing to take the simple steps to prevent aforementioned spidering.
- Confusion 17y ago
  Are you saying that if your content doesn't have a robots.txt, then everyone is allowed to copy the content of the site with no restrictions whatsoever No, he is saying that if your content doesn't have a robots.txt, then everyone is allowed to index the content of the site with no restrictions whatsoever. After all, you are publishing it to the world. The fact that technology and automation are involved is irrelevant: the process is identical to me telling you that paper newspaper X had an article on Y in it's Jan 23rd, 2005 edition.

Has Rupert heard of /robots.txt?

by mthreat 17y ago

Has Rupert heard of /robots.txt?

jister 17y ago

This is exactly the first thing that comes to my mind when I read about "indexing" in the article.

philk 17y ago

The point is not that he wants to cut Google off, it's that he wants leverage. Simply updating robots.txt won't effect Google at all and will blow what little leverage he has left.

alain94040 17y ago

Are you saying that if your content doesn't have a robots.txt, then everyone is allowed to copy the content of the site with no restrictions whatsoever, and that copyright law doesn't apply? I'm not a big fan of Murdoch, but it's an interesting discussion that should go beyond the obvious technical arguments.

corruption 17y ago
Imagine the world with strict copyright. Search engines would be illegal!
- deleted 17y ago
  [deleted]
- alain94040 17y ago
  I guess I'll ask again, differently. What do you mean by strict? Are you saying that today, there is an implicit understanding among a class of people (judges, CEOs of web 2.0 companies...?) to not apply the law strictly? That doesn't sound very likely. If Google started copying a full news site verbatim and offer its content when people search for news, that would definitely be wrong, robots.txt or not. Strict copyright includes fair use exceptions and so on. Where fair use stops and infringment starts is an interesting discussion for me.
romland 17y ago
You get moderated downwards for that comment? What is this, Slashdot?
- dasil003 17y ago
  Uh, Google doesn't do just "anything" with the content, they index, summarize, and link back to the original article. There's no obvious copyright violation there. The GP was hyberbole that doesn't add anything to the debate. Ironically that's exactly what happens all the time on Slashdot.
- alain94040 17y ago
  Yes, I got modded down. I didn't list my credentials on the topic, so maybe I should have. I did get sued and went through significant $$$ expenses to defend frivolous lawsuits, so anyone who blankly states "oh, use robots.txt and don't worry about a thing" sounds like a beginner in the intricacies of the law to me.
Herring 17y ago
That's kind of a red herring, because Murdoch wants his stuff to be copied by google regardless of how that argument is resolved.
philk 17y ago
It's not a particularly onerous requirement and far cheaper than calling your lawyers. If you're making the argument that the company that is automatically spidering your content should stop then your case is somewhat undermined by failing to take the simple steps to prevent aforementioned spidering.
Confusion 17y ago
Are you saying that if your content doesn't have a robots.txt, then everyone is allowed to copy the content of the site with no restrictions whatsoever No, he is saying that if your content doesn't have a robots.txt, then everyone is allowed to index the content of the site with no restrictions whatsoever. After all, you are publishing it to the world. The fact that technology and automation are involved is irrelevant: the process is identical to me telling you that paper newspaper X had an article on Y in it's Jan 23rd, 2005 edition.