4 ms·

I disagree with this post almost as strongly as I agree with it. Robots.txt is a suicide note. It's utter short-sighted hubris to say "this is MY information a

by sgentle 15y ago

I disagree with this post almost as strongly as I agree with it. Robots.txt is a suicide note. It's utter short-sighted hubris to say "this is MY information and I don't want you spidering it". Are you volunteering to maintain that information forever? Are you promising to never go out of business? Never be ordered to remove it by the government? Never be bought out by Oracle? Right now there seems to be a lot of confusion over the morality of information. People are possessed by the strange idea that you, mister content provider, own that content and have an inalienable right to control it any way you can get away with. But someday you will die, and your company will die, just like Geocities, Google Video, and the Library of Alexandria. Society should have a right to keep that information after you're gone. Of course, the law disagrees. And without the efforts of criminals like geohot, the iPhone DevTeam, The Nomad, Muslix64 and, yes, The Archive Team, people of the future will have no way to access the information we've locked up through our own paranoia. You don't have to cast your mind to a thousand years in the future - it's happening right now. Vast swathes of data are disappearing as DRM servers go dark only a few years after they appear (thanks, MSN Music, Yahoo Music Store). I believe that we owe it to our descendants to give them access to their history. I believe it's not our decision whether the things we make are too valuable or too uncomfortable to be preserved. And I believe that robots.txt is a suicide note, a product of the diseased minds that think our short-term desire for control outweighs our legacy. But I don't know what the fuck the article's talking about. It seems to be making a bunch of points that don't matter. Use robots.txt to prevent technical problems if you like, I don't care. Just don't use it to stop people from crawling your content or you're shitting on the future.

jamie_ca 15y ago
While I'm sure the "content provider" would disagree, in many cases it's perfectly legal to make a copy of their content - the laws in place only prevent redistribution for a certain time period. Personally, I'd be ecstatic if there was some organization set up that is manufacturing high-quality archive copies of books, music, film and the like, and storing them with the date of initial publication, as well as the date they will enter public domain (where it is known - not the case for living authors) in various locales. Then have a site live-tracking the release of that content. There's a site that keeps getting linked every year that posts things that would be public domain this year were it not for US copyright extensions, and something like that with accurate PD status and archive copies would be marvelous.

by sgentle 15y ago

jamie_ca 15y ago

While I'm sure the "content provider" would disagree, in many cases it's perfectly legal to make a copy of their content - the laws in place only prevent redistribution for a certain time period. Personally, I'd be ecstatic if there was some organization set up that is manufacturing high-quality archive copies of books, music, film and the like, and storing them with the date of initial publication, as well as the date they will enter public domain (where it is known - not the case for living authors) in various locales. Then have a site live-tracking the release of that content. There's a site that keeps getting linked every year that posts things that would be public domain this year were it not for US copyright extensions, and something like that with accurate PD status and archive copies would be marvelous.