3 ms·
Isn't the way for Craigslist to remove its posts from search results via http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt?
by tjpd 14y ago
Isn't the way for Craigslist to remove its posts from search results via http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt?
- testing12341234 14y agoIf you take a look at a page like http://portland.craigslist.org/mlt/eng/3189881249.html http://portland.craigslist.org/mlt/eng/3189881249.html You can see that they added a <meta name="robots" content="NOARCHIVE,NOFOLLOW"> tag. So Google will index it and show it, but it won't cache the full text. This change means that companies like 3taps can no longer get the full text from the Google cache, while still allowing search results to show in searches.
- sigmadelta 14y agohttp://www.seroundtable.com/archives/019533.html http://www.seroundtable.com/archives/019533.html That article is from 2009, noting that CL had added the meta tag to posting pages. So I think this is old news. I think storborg has it right in http://news.ycombinator.com/item?id=4353120 http://news.ycombinator.com/item?id=4353120 when saying that "3taps seems to be claiming that Craigslist has cut off Google, but I think it's just that Craigslist has cut off 3taps."