5 ms·

That's a pretty serious claim; any pointers? If you've got some I'll be sure to nag Craig about it next time I see him.

by aaronsw 18y ago

fragmede 18y ago
listpic? http://mashable.com/2007/06/08/listpic-craigslist/ http://mashable.com/2007/06/08/listpic-craigslist/
- vesterr 18y ago
  Ha, with EC2 you can get around that now. Separate your website IP from your crawling IP, and every time CL blocks an IP, switch to a new one. Eventually they'll have to block the entire AWS range, but that's okay, you can crawl over cablemodem/DSL connections that use DHCP. What are they going to do, block Verizon, Comcast and Time Warner? Then you can get around referer [sic] blocks for your links to CL that the user clicks on by using a redirect, I think. You can slurp the entirety of CL daily without causing them traffic problems. I mean it's equivalent to each page getting one page view per day, which is nothing. Just keep track of URLs so you only slurp new content, and serve thumbnails off your own hosts (it's fair use).
  - nuggien 18y ago
    Why would you go to all that effort if they don't want to play with you in the first place? If they really wanted to block you, they'll do it through their lawyers.
    - vesterr 18y ago
      That stuff is really easy. And you may convince them to join you after all, like YouTube and the media companies they parasite off of.
      - nuggien 18y ago
        I didn't say it was hard. It's also very easy to differentiate between a crawler and a user. But they don't even have to do that. Blocking the entire AWS ip range isn't hard either. I doubt any craigslist users browse the site through AWS :)
- aaronsw 18y ago
  So the issue is: 1) they were putting ads on top of Craigslist, and 2) they were using up too much bandwidth. I expect if you actually wanted to build something useful you could just email Craig and he'd get you a dump.
  - gscott 18y ago
    There have been a few others, one fellow made a search system (using the rss feeds) where you can search across cities (by using checkboxes to select the different areas). Craiglist didn't like that. I guess the bigger question is, what would Craigslist like? I am not sure, I just have not seen anything survive yet because Craigslist hasn't liked anything anyone has done so far.
aquateen 18y ago
One example, I used to want to search for craigslist items regardless of location. I found some sites that briefly offered this service, but they were requested to stop by CL.

That's a pretty serious claim; any pointers? If you've got some I'll be sure to nag Craig about it next time I see him.

by aaronsw 18y ago

That's a pretty serious claim; any pointers? If you've got some I'll be sure to nag Craig about it next time I see him.

fragmede 18y ago

listpic? http://mashable.com/2007/06/08/listpic-craigslist/ http://mashable.com/2007/06/08/listpic-craigslist/

vesterr 18y ago
Ha, with EC2 you can get around that now. Separate your website IP from your crawling IP, and every time CL blocks an IP, switch to a new one. Eventually they'll have to block the entire AWS range, but that's okay, you can crawl over cablemodem/DSL connections that use DHCP. What are they going to do, block Verizon, Comcast and Time Warner? Then you can get around referer [sic] blocks for your links to CL that the user clicks on by using a redirect, I think. You can slurp the entirety of CL daily without causing them traffic problems. I mean it's equivalent to each page getting one page view per day, which is nothing. Just keep track of URLs so you only slurp new content, and serve thumbnails off your own hosts (it's fair use).
- nuggien 18y ago
  Why would you go to all that effort if they don't want to play with you in the first place? If they really wanted to block you, they'll do it through their lawyers.
  - vesterr 18y ago
    That stuff is really easy. And you may convince them to join you after all, like YouTube and the media companies they parasite off of.
    - nuggien 18y ago
      I didn't say it was hard. It's also very easy to differentiate between a crawler and a user. But they don't even have to do that. Blocking the entire AWS ip range isn't hard either. I doubt any craigslist users browse the site through AWS :)
aaronsw 18y ago
So the issue is: 1) they were putting ads on top of Craigslist, and 2) they were using up too much bandwidth. I expect if you actually wanted to build something useful you could just email Craig and he'd get you a dump.
- gscott 18y ago
  There have been a few others, one fellow made a search system (using the rss feeds) where you can search across cities (by using checkboxes to select the different areas). Craiglist didn't like that. I guess the bigger question is, what would Craigslist like? I am not sure, I just have not seen anything survive yet because Craigslist hasn't liked anything anyone has done so far.

aquateen 18y ago

One example, I used to want to search for craigslist items regardless of location. I found some sites that briefly offered this service, but they were requested to stop by CL.