Scraping is clearly covered by robots.txt, one of the earliest conventions on the web. In their case (http://www.leboncoin.fr/robots.txt http://www.leboncoin.fr

Scraping is clearly covered by robots.txt, one of the earliest conventions on the web. In their case (http://www.leboncoin.fr/robots.txt http://www.leboncoin.fr/robots.txt) they allow the spiders from most commercial search engine, but disallow all others. I think it's a terrible idea from Schibsted (the Norwegian owners of the company) to react like this; but then again, lawyers...