3 ms·

Google obeys robots.txt file. Padmapper does not. That's a critical difference.

• malandrew 14y ago
  True, but that's irrelevant to the comment I was replying to. We are talking about user expectations with regard to content they posted being made available on other sites.
• operand 14y ago
  CL's robot.txt file does not 'disallow' crawling the real estate section. So Padmapper was likely as compliant as Google. http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt
  • eli 14y ago
    No, this is just semantics. The point is Google respects publishers' desire not to be indexed. Padmapper does not. Robots.txt is simply a common method for conveying that message. It's not like Padmapper could argue they didn't know CL was unhappy; they got a certified letter! This is just semantics. Google respects publishers who do not want their sites listed.
• dschobel 14y ago
  Good luck explaining that to a judge. Would you gamble a company based on that distinction?