4 ms·
In the end, client side scraping of the data actually increases the traffic on CL's servers, because multiple people have to scrape the same posts, rather than
by chimi 14y ago
In the end, client side scraping of the data actually increases the traffic on CL's servers, because multiple people have to scrape the same posts, rather than a central server. If padmapper remained small, then they probably scrape more posts than they have unique views. Of the 100 they scrape, maybe only 10 are seen. As padmapper et al grow, it may reduce the bandwidth on cl.
In 10 years, if all clients looking for apartments use a plug-in however, now you have cl paying for a lot more bandwidth than outsourcing this functionality to a centralized service, because for every 100 posts they scrape, they may only really look at one and they want the ability to show thousands on a single map.
I really don't understand why cl doesn't just do it themselves. It's not hard. I've built these maps countless times on top of all the popular mapping services and using different geocoders.
The hardest part is figuring out what the poster means in their location: where is "forth and main" [sic] or do you mean "East 4th St at Main St?", but I can see a simple address validator that updates a map with their location until they get something that will geocode correctly. That's the best solution and everyone wins here, except padmapper and the like, but there are other sources of data to map that aren't from craigslist too.
It's really trivial. The only other issue is cost, but those can be covered easily with all the money CL leaves on the table.
- jack-r-abbit 14y agoNot only that, but for every housing page with a list on it, the scraper is going to pull all those pages to parse the data and display on a map. I suspect a person who is doing a lot of apartment hunting with this extension runs the risk of just getting their IP address banned. I assume CL has that capability.
- loumf 14y agoI think CL is worried that their bandwidth would start going down as people start putting their apartment listings in PadMapper instead of CL because CL doesn't have any apartment listings any more.
- welder 14y agoSince users will be browsing the CL website, CL will still control the distribution of their content. I'm betting tech details are just details as long as users are browsing the CL website.
- joe_the_user 14y agoYes, I don't CL ever made the argument that the problem was traffic. The whole "we're not hurting your servers" thing seems like a strawman to justify violating CL's TOS.
- aidenn0 14y agoCL repeatedly claimed that traffic was a major issue with scraping.
- olog-hai 14y ago> client side scraping of the data actually increases the traffic on CL's servers Of course, but not in PadMapper's case. "Padmapper gets its data from 3Taps, a site that collects and republishes Craigslist's content. 3Taps, in turn, says it gets its data from cached copies supplied by Google and Bing, rather than Craigslist directly." Source: http://arstechnica.com/tech-policy/2012/08/craigslist-tightens-grip-demanding-exclusive-ownership-of-ads/ http://arstechnica.com/tech-policy/2012/08/craigslist-tighte...
- jeremyarussell 14y agoIs that argument in consideration of the fact that padmapper started off by scraping CL directly and forwarding everyone back over to CL?
- jack-r-abbit 14y agoThey used to do that. They were told to stop. They stopped. They found a new way that didn't affect CL bandwidth. So what they started off doing is not all that relevant to what they are doing now. However, what this Chrome Extension is doing is pretty much what PadMapper was doing... so it would follow that this Chrome Extension will get smacked. Or worse... the extension users each get smacked.
- cdibona 14y agoI'm sorry, but you should really ask yourself if this is true. Bing maybe, as they can do what ddg does and pull a feed from Yahoo/Bing BOSS (maybe?) but to scrape Google would be a very bad and unreliable (and counter to the Google TOS) way of getting this information. This whole situation is very sketchy, honestly.
- cbr 14y agoPadmapper used to scrape Craigslist, but stopped after a cease and desist letter. While buying the data from a service that scrapes search engine caches may be unreliable, my understanding is that it's legally much more defensible.
- pavel_lishin 14y ago> I really don't understand why cl doesn't just do it themselves. It's not hard. It's not hard to do a mediocre job, but it is super hard to build something really great. And it's super easy to build something so shitty, that it'll drive away the users that you have now.