4 ms·
CL's robot.txt file does not 'disallow' crawling the real estate section. So Padmapper was likely as compliant as Google. http://www.craigslist.org/robots.txt
by operand 14y ago
CL's robot.txt file does not 'disallow' crawling the real estate section. So Padmapper was likely as compliant as Google.
http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt
- eli 14y agoNo, this is just semantics. The point is Google respects publishers' desire not to be indexed. Padmapper does not. Robots.txt is simply a common method for conveying that message. It's not like Padmapper could argue they didn't know CL was unhappy; they got a certified letter! This is just semantics. Google respects publishers who do not want their sites listed.