3 ms·
Ask HN: Mar. 2 - Craigslist ads no longer indexed by Google?
It looks like for the past day or so, google has no longer been indexing craigslist postings. See these two searches for example:
All postings in the Cars by Owner category over the past week:
https://www.google.com/search?q=inurl:cto+-intitle:classifieds+site:craigslist.org&num=20&hl=en&safe=off&tbs=qdr:w,sbd:1
And over the past day:
https://www.google.com/search?q=inurl:cto+-intitle:classifieds+site:craigslist.org&num=20&hl=en&safe=off&tbs=qdr:d,sbd:1
Actually, since I first noticed it, it may have been fixed. Note that both searches return results from the past 31 minutes (as of now, 4:18pm PST), but nothing else for the rest of today.
Question is, anyone know what caused this, and more generally, whether these sorts of 'gaps' are common? Given my understanding of how Google works, I can't see how a gap would persist once whatever was preventing scraping was removed. There should still be links from the CL index pages to most of the missing ads from that period, so even if Google was temporarily unable to access them, why wouldn't they be filled in after the fact? Or has it just not happened yet?
- btbuildem 14y agoJerks finally decided to cut off indexing -- there's been a spate of lawsuits involving housing maps mashups in the recent months. One of the arguments used in defense of such mashups was "but google scrapes craigslist". Fairly recently CL rolled out their own housing maps (finally, they figured out they need to keep up with the times), perhaps this is a second salvo?
- phasevar 14y agoHow exactly does that make them jerks? It's their web site. They can decide to run it how ever they please. You don't have to use their site if you don't like them.
- tempestn 14y agoAppears it's been un-fixed again. Not only are new results once again not being indexed, but the ones from that brief period of time when it appeared fixed have been removed.