3 ms·
CL has every right to do that, especially since scraping CL listings creates a very high amount of load on their servers. Higher load --> slower response time -
by achompas 14y ago
CL has every right to do that, especially since scraping CL listings creates a very high amount of load on their servers. Higher load --> slower response time --> deteriorated user experience.
- wpietri 14y agoDid I say they don't have the legal right to do that? No, I didn't. I doubt anybody did. There's no particular reason to think that spiders are a giant problem for them. Any high-volume site deals with crawlers all the time. If it were a problem because of some site architecture weirdness, it'd be easy enough to set up something that wouldn't be problematic. For example, the RSS feeds they already offer. Worst case, they could charge people like Padmapper a fee to license the data. Which in fact they already do, but they insist on it being mobile only. So Craigslist's behavior is purely a business choice, not driven by any technical necessity.
- tempestn 14y agoPadmapper (and 3taps) don't scrape craigslist though. They get all info from public sources such as search engine caches.