3 ms·
Craigslist reportedly removes listings from search engines
- sigmadelta 14y agoI just tried the following: visited http://sfbay.craigslist.org/bia/ http://sfbay.craigslist.org/bia/ saw ad for "Burning Man Beach Cruiser Bicycle" http://sfbay.craigslist.org/eby/bik/3189652975.html http://sfbay.craigslist.org/eby/bik/3189652975.html performed the following search on google http://www.google.com/search?q=site%3Acraigslist.org+neon+green+cruiser+bike&btnG=Search http://www.google.com/search?q=site%3Acraigslist.org+neon+gr... first result returned is that ad, with the following snippet: Burning Man Beach Cruiser Bicycle sfbay.craigslist.org/eby/bik/3189652975.html 22 minutes ago ... Neon Green cruiser bicycle, perfect for the playa, or cruising around town. Single speed, great shape. Available nights and weekends. -- "22 minutes ago" sounds like Google is still actively crawling content from craigslist to me.
- griffindy 14y agoam I the only one who uses craiggers? although I suppose they'll have to go the way of the dodo soon enough now
- tjpd 14y agoIsn't the way for Craigslist to remove its posts from search results via http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt?
- testing12341234 14y agoIf you take a look at a page like http://portland.craigslist.org/mlt/eng/3189881249.html http://portland.craigslist.org/mlt/eng/3189881249.html You can see that they added a <meta name="robots" content="NOARCHIVE,NOFOLLOW"> tag. So Google will index it and show it, but it won't cache the full text. This change means that companies like 3taps can no longer get the full text from the Google cache, while still allowing search results to show in searches.
- sigmadelta 14y agohttp://www.seroundtable.com/archives/019533.html http://www.seroundtable.com/archives/019533.html That article is from 2009, noting that CL had added the meta tag to posting pages. So I think this is old news. I think storborg has it right in http://news.ycombinator.com/item?id=4353120 http://news.ycombinator.com/item?id=4353120 when saying that "3taps seems to be claiming that Craigslist has cut off Google, but I think it's just that Craigslist has cut off 3taps."