3 ms·
Because if you ask Google to remove you from their index, they will. Rupert Murdoch has threatened this in the past. Each time he's backed down, because he kno
by debacle 14y ago
Because if you ask Google to remove you from their index, they will.
Rupert Murdoch has threatened this in the past. Each time he's backed down, because he knows that being removed from Google's index is akin to being removed from the Internet.
- bluesnowmonkey 14y agoCraigslist doesn't needs to be removed from the search index. They just need to prevent caching of posts, which is what 3Taps was abusing. Noarchive in robots.txt ought to take care of that.
- IanDrake 14y agoTo your knowledge, is it against the law to not follow a site's robots.txt? I'm not aware of one, but I am curious.
- debacle 14y agoI don't know if it's illegal, but Google would follow the site's robots.txt.
- BenOfTomorrow 14y agoAmbiguous. See http://news.ycombinator.com/item?id=4286857 http://news.ycombinator.com/item?id=4286857.
- coliveira 14y agoWhich shows just how much Google profits from their virtual monopoly. If content companies knew that Google would make billions of dollars selling advertisement based on the content that they generated, it would be very hard for Google to pull the massive Internet scrapping that they managed to do from day one. It is interesting that people understand the illegitimacy of taking data from a single content source (Craigslist), but fail to see how unethical is to take data from everyone and sell advertisements based on that.
- nostromo 14y agoInterestingly, Craigslist is being consistent when it comes to not allowing scrapers, including Google. http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt edit: oops! they're not actually blocking google from the apartment listings. thanks smackfu.
- smackfu 14y agoAren't they only disallowing particular pages there? Like apartments are at *.craigslist.org/apa/number.html which doesn't seem blocked by that robots.txt.
- callmevlad 14y agoI can imagine some clever hacks where Padmapper uses Google targeted searches and Google's cache exclusively to get Craigslist data. Since Craigslist would never ask Google to remove them from their index, I wonder if Craigslist would still go after those who get their data as proxied by Google.
- debacle 14y ago> Padmapper uses...Google's cache exclusively to get Craigslist data. That's what Padmapper is doing right now. They're being sued for it.
- callmevlad 14y agoI thought Craigslist is suing 3taps because they scrape their site directly?