8 ms·

Craigslist Casualty: The Story of ApartmentsNearMetro.com

29 points by wlindner 14y ago

derwiki 14y ago
`http://trycourier.com/?utm_source=anm-blog` http://trycourier.com/?utm_source=anm-blog` linked in the post seems to 403.
- wlindner 14y ago
  Sorry about that, it's just http://trycourier.com http://trycourier.com, I fixed the link. Thanks!
jdavis703 14y ago
I had that exact same pain point in 2011. It's too bad the Craigslist refuses to innovate or let others innovate on their platform.
- wlindner 14y ago
  Yeah! It almost seems like a source of pride for them... What were you working on that got squashed by Craigslist?
  - jdavis703 14y ago
    I was simply trying to locate an apartment near Metro. I toyed with the idea of scraping their site but remembered this Wired story from 2009 [0] so I just gave up on using them for my search. 0: http://www.wired.com/entertainment/theweb/magazine/17-09/ff_craigslist?currentPage=all http://www.wired.com/entertainment/theweb/magazine/17-09/ff_...
- jzawodn 14y ago
  What does 'innovate' mean in this context? Ditto for 'platform'? Both terms are terribly over-used and make discussion challenging, since both sides end up assuming what they words probably mean (without every discussing the meaning).
homedog 14y ago
It's the inevitable fate for anything that uses CL for data. CL is too afraid of competition, yet refuses to innovate. It's a shame.
- a5seo 14y ago
  In defense of CL (which I have never before offered), I noticed some UI improvements recently: pic view, grid view, and map view. These are the first noticeable innovations on CL in many years. Might they be the result of the on-going parade of sites improving upon CL w/ their own data, not to mention the criticisms on HN and elsewhere? Whether they 'hate' competition is debatable. Clearly they view their data as THEIR property. They occasionally let things like PadMapper go while putting the ban hammer down on other things, very arbitrarily. The Wired story on them about 5-6 years back suggested they are just very skeptical of third parties and human nature in general. It left me with the impression that Craig is just deeply selfish and authoritarian.
  - wlindner 14y ago
    I've seen the UI improvements, but haven't needed to use them so I don't know how good they are. If those work: Awesome! In a way, that's a victory. If CL incorporates the best things about 3rd party sites into their own site, I think that's a big win for consumers. If CL refuses to allow 3rd parties to exist because they are skeptical of human nature. Talk about the ultimate irony: a site that lives off of people's willingness to trust strangers being unwilling to trust strangers themselves.
    - gamblor956 14y ago
      CraigsList doesn't care if other websites use the same data to power their own services, as long as those other websites obtain the data directly from the source (i.e., the user) and not by scraping Craiglist.
    - a5seo 14y ago
      Good point. Here's the quote from the end of the article: "These all signal Newmark and Buckmaster's wariness about what humans, including themselves, might do if given the chance. There may be a peace sign on every page, but the implicit political philosophy of craigslist has a deeply conservative, even a tragic cast. Every day the choristers of the social web chirp their advice about openness and trust; craigslist follows none of it, and every day it grows." HN discussion: http://news.ycombinator.com/item?id=784663 http://news.ycombinator.com/item?id=784663
pm90 14y ago
If the application is blocked, can the author please openSource the script? If everyone starts using the script, I believe it will be quite impossible for CL to block it.
- wlindner 14y ago
  Yeah, that's a good idea. Craigslist basically blocked any connections from Google App Engine to their domain. But, if you ran the site from your own computer, they wouldn't really have any way to block it.
TimMontague 14y ago
FYI: this is unreadable on my WP8 device. The screen width is set smaller than the text width so I have so scroll left and right for every line.
- alanctgardner2 14y ago
  I think the easy fix for this (for the benefit of the poster) is to use a meta tag to set the "viewport" attribute to 100%.
- wlindner 14y ago
  Thanks for letting me know, I implemented a quick fix for mobile devices. Unfortunately I don't have a WP8 to test it on, but it shouldn't require scrolling left and right anymore to read the text.
  - TimMontague 14y ago
    Thank for fixing!
typicalbender 14y ago
Similar story as https://www.padmapper.com/ https://www.padmapper.com/. It's sad that craigslist isn't willing to either update their application or allow someone to make it more useful for their customers.
danso 14y ago
> Had around 1,500 unique visitors with an average time spent on the site of about 7 and a half minutes. With no disrespect to the OP, that's not a ton of users, even for a single day...so I'm kind of surprised Craigslist noticed enough to block them. There must be tons of unapproved scraper scripts checking CL on a daily basis.
- wlindner 14y ago
  That's a very good point. 1,500 unique visitors is tiny. Hell, I will probably get more visitors to this single blog post. Forgoing everything about CL blocking me and such, I never did enough to promote the site. I started the site in 2011 and had little experience with promoting content. As a newb developer, I was leery of promoting something I created. What if it broke? What if people didn't like it? When in reality, I learned over time, that most people frankly don't care. And that's not a bad thing. The past few years I have been learning over and over: ship it, get feedback, improve, repeat. So, getting back to how they blocked me. If I understand correctly, craigslist blocked any HTTP requests from the Google App Engine User Agent. HTTP Requests from Google App Engine always append 'AppEngine-Google' to the end of the User Agent (http://code.google.com/p/googleappengine/issues/detail?id=342#c12 http://code.google.com/p/googleappengine/issues/detail?id=34...). You can recreate this block by doing this in python (you need the amazing python requests library, so install that): import requests r = requests.get('http://craigslist.com http://craigslist.com) print r.status_code This should print 200 and you're good to go. Then, try it with the 'AppEngine-Google' in the user agent: r = requests.get('http://craigslist.com http://craigslist.com, headers={'User-Agent': 'AppEngine-Google'}) and r.status_code will return 404 bingo bango, you've been blocked. I could have just moved off of Google App Engine, etc. but I would probably be faced with a similar fate as Padmapper if I ever got any kind of sizable user base. I created this gist: https://gist.github.com/wlindner/4752122 https://gist.github.com/wlindner/4752122 since the python code formatting is wonky
  - jzawodn 14y ago
    craigslist is hiring engineers. Interested?
oniTony 14y ago
So there's Walk Score. Apartments near (Washington) metro example — http://walk.sc/YPUq1a http://walk.sc/YPUq1a This doesn't really solve the problem of Craigslist hoarding all their data to themselves (as pointed out numerous times already), but there are enough other sources to make it usable.
ScottWhigham 14y ago
I don't get why people say things like, "It's too bad the Craigslist won't ... let other people innovate on their platform." It's just childish in my view. Here's another child's toys - and, granted, he has a LOT of toys[1] - and he's chosen to share them to every child in the world regardless of their background, yet you're upset because he's not sharing them in such a way that would only allow people to play with the toys on another playground. "He" doesn't want that - he wants to play, play, play with everyone. So he decides to say "No!" to anyone who wants to take his toys off of his playground. If you don't like the way "he" plays with his toys, then tough cookies - "he" doesn't care and neither does anyone else except those who want to also complain about him not sharing in the way they want him to. You'll see this damn near every time someone posts a "Help! My API provider just changed the rules!" type of post. There are two types of responses: "I hate that - they should be more open", or "It's ridiculous to even think that you can build a business based off of leaching from someone else and that that other entity would be okay with that forever." [1]Unlike children (who typically are given their toys), these toys are the result of "his" work.
- largesse 14y ago
  Unlike children (who typically are given their toys), these toys are the result of "his" work. Er, no. They are the work of everyone who posts on CL.
vampirechicken 14y ago
If you want to compete with craigslist, build a better product and convince people that there is value for them to enter their data. That's what craigslist did.

29 points by wlindner 14y ago

`http://trycourier.com/?utm_source=anm-blog` http://trycourier.com/?utm_source=anm-blog` linked in the post seems to 403.

wlindner 14y ago
Sorry about that, it's just http://trycourier.com http://trycourier.com, I fixed the link. Thanks!

jdavis703 14y ago

I had that exact same pain point in 2011. It's too bad the Craigslist refuses to innovate or let others innovate on their platform.

wlindner 14y ago
Yeah! It almost seems like a source of pride for them... What were you working on that got squashed by Craigslist?
- jdavis703 14y ago
  I was simply trying to locate an apartment near Metro. I toyed with the idea of scraping their site but remembered this Wired story from 2009 [0] so I just gave up on using them for my search. 0: http://www.wired.com/entertainment/theweb/magazine/17-09/ff_craigslist?currentPage=all http://www.wired.com/entertainment/theweb/magazine/17-09/ff_...
jzawodn 14y ago
What does 'innovate' mean in this context? Ditto for 'platform'? Both terms are terribly over-used and make discussion challenging, since both sides end up assuming what they words probably mean (without every discussing the meaning).

homedog 14y ago

It's the inevitable fate for anything that uses CL for data. CL is too afraid of competition, yet refuses to innovate. It's a shame.

a5seo 14y ago
In defense of CL (which I have never before offered), I noticed some UI improvements recently: pic view, grid view, and map view. These are the first noticeable innovations on CL in many years. Might they be the result of the on-going parade of sites improving upon CL w/ their own data, not to mention the criticisms on HN and elsewhere? Whether they 'hate' competition is debatable. Clearly they view their data as THEIR property. They occasionally let things like PadMapper go while putting the ban hammer down on other things, very arbitrarily. The Wired story on them about 5-6 years back suggested they are just very skeptical of third parties and human nature in general. It left me with the impression that Craig is just deeply selfish and authoritarian.
- wlindner 14y ago
  I've seen the UI improvements, but haven't needed to use them so I don't know how good they are. If those work: Awesome! In a way, that's a victory. If CL incorporates the best things about 3rd party sites into their own site, I think that's a big win for consumers. If CL refuses to allow 3rd parties to exist because they are skeptical of human nature. Talk about the ultimate irony: a site that lives off of people's willingness to trust strangers being unwilling to trust strangers themselves.
  - gamblor956 14y ago
    CraigsList doesn't care if other websites use the same data to power their own services, as long as those other websites obtain the data directly from the source (i.e., the user) and not by scraping Craiglist.
  - a5seo 14y ago
    Good point. Here's the quote from the end of the article: "These all signal Newmark and Buckmaster's wariness about what humans, including themselves, might do if given the chance. There may be a peace sign on every page, but the implicit political philosophy of craigslist has a deeply conservative, even a tragic cast. Every day the choristers of the social web chirp their advice about openness and trust; craigslist follows none of it, and every day it grows." HN discussion: http://news.ycombinator.com/item?id=784663 http://news.ycombinator.com/item?id=784663

pm90 14y ago

If the application is blocked, can the author please openSource the script? If everyone starts using the script, I believe it will be quite impossible for CL to block it.

wlindner 14y ago
Yeah, that's a good idea. Craigslist basically blocked any connections from Google App Engine to their domain. But, if you ran the site from your own computer, they wouldn't really have any way to block it.

TimMontague 14y ago

FYI: this is unreadable on my WP8 device. The screen width is set smaller than the text width so I have so scroll left and right for every line.

alanctgardner2 14y ago
I think the easy fix for this (for the benefit of the poster) is to use a meta tag to set the "viewport" attribute to 100%.
wlindner 14y ago
Thanks for letting me know, I implemented a quick fix for mobile devices. Unfortunately I don't have a WP8 to test it on, but it shouldn't require scrolling left and right anymore to read the text.
- TimMontague 14y ago
  Thank for fixing!

typicalbender 14y ago

Similar story as https://www.padmapper.com/ https://www.padmapper.com/. It's sad that craigslist isn't willing to either update their application or allow someone to make it more useful for their customers.

danso 14y ago

> Had around 1,500 unique visitors with an average time spent on the site of about 7 and a half minutes. With no disrespect to the OP, that's not a ton of users, even for a single day...so I'm kind of surprised Craigslist noticed enough to block them. There must be tons of unapproved scraper scripts checking CL on a daily basis.

wlindner 14y ago
That's a very good point. 1,500 unique visitors is tiny. Hell, I will probably get more visitors to this single blog post. Forgoing everything about CL blocking me and such, I never did enough to promote the site. I started the site in 2011 and had little experience with promoting content. As a newb developer, I was leery of promoting something I created. What if it broke? What if people didn't like it? When in reality, I learned over time, that most people frankly don't care. And that's not a bad thing. The past few years I have been learning over and over: ship it, get feedback, improve, repeat. So, getting back to how they blocked me. If I understand correctly, craigslist blocked any HTTP requests from the Google App Engine User Agent. HTTP Requests from Google App Engine always append 'AppEngine-Google' to the end of the User Agent (http://code.google.com/p/googleappengine/issues/detail?id=342#c12 http://code.google.com/p/googleappengine/issues/detail?id=34...). You can recreate this block by doing this in python (you need the amazing python requests library, so install that): import requests r = requests.get('http://craigslist.com http://craigslist.com) print r.status_code This should print 200 and you're good to go. Then, try it with the 'AppEngine-Google' in the user agent: r = requests.get('http://craigslist.com http://craigslist.com, headers={'User-Agent': 'AppEngine-Google'}) and r.status_code will return 404 bingo bango, you've been blocked. I could have just moved off of Google App Engine, etc. but I would probably be faced with a similar fate as Padmapper if I ever got any kind of sizable user base. I created this gist: https://gist.github.com/wlindner/4752122 https://gist.github.com/wlindner/4752122 since the python code formatting is wonky
- jzawodn 14y ago
  craigslist is hiring engineers. Interested?

oniTony 14y ago

So there's Walk Score. Apartments near (Washington) metro example — http://walk.sc/YPUq1a http://walk.sc/YPUq1a This doesn't really solve the problem of Craigslist hoarding all their data to themselves (as pointed out numerous times already), but there are enough other sources to make it usable.

ScottWhigham 14y ago

I don't get why people say things like, "It's too bad the Craigslist won't ... let other people innovate on their platform." It's just childish in my view. Here's another child's toys - and, granted, he has a LOT of toys[1] - and he's chosen to share them to every child in the world regardless of their background, yet you're upset because he's not sharing them in such a way that would only allow people to play with the toys on another playground. "He" doesn't want that - he wants to play, play, play with everyone. So he decides to say "No!" to anyone who wants to take his toys off of his playground. If you don't like the way "he" plays with his toys, then tough cookies - "he" doesn't care and neither does anyone else except those who want to also complain about him not sharing in the way they want him to. You'll see this damn near every time someone posts a "Help! My API provider just changed the rules!" type of post. There are two types of responses: "I hate that - they should be more open", or "It's ridiculous to even think that you can build a business based off of leaching from someone else and that that other entity would be okay with that forever." [1]Unlike children (who typically are given their toys), these toys are the result of "his" work.

largesse 14y ago
Unlike children (who typically are given their toys), these toys are the result of "his" work. Er, no. They are the work of everyone who posts on CL.

vampirechicken 14y ago

If you want to compete with craigslist, build a better product and convince people that there is value for them to enter their data. That's what craigslist did.