4 ms·
They no longer need the exclusive since PadMapper's workaround has been effectively killed by the changes Craigslist made to its robots.txt, Google is no longe
by gagabity 14y ago
They no longer need the exclusive since PadMapper's workaround has been effectively killed by the changes Craigslist made to its robots.txt, Google is no longer allowed to index apartments thus PadMapper can not scrape the Google cache.
- sigmadelta 14y agocraigslist didn't change robots.txt: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT