7 ms·

Craigslist blocks Yahoo Pipes

93 points by slay2k 17y ago

jacoblyles 17y ago
It hurts me when people chose to keep things broken (Craigslist) for irrational reasons, especially when it wastes hours of time for millions of people.
- jimboyoungblood 17y ago
  their decision seems perfectly rational. if services like this one become popular, they are a huge threat to their business. they're not morally obligated to provide unfettered access to their data to anyone who wants it- i fail to see how it is "broken"
  - seiji 17y ago
    Brokenness: Show me all apartments or rooms-for-rent in mountain view with dish washers, clothes washers, and covered parking. There's no way to (legitimately) outwardly index, categorize, or aggregate craigslist content. (Related gripe: apartment searching in google is useless too: http://www.google.com/search?q=mountain+view+apartments http://www.google.com/search?q=mountain+view+apartments It gets slightly better with searching on maps, but not by much.)
    - ams6110 17y ago
      Many apartment/house finding sites have these features: search pages with dozens of checkboxes and dropdowns, and equally complex data entry pages for the people listing the apartments/houses. Yet Craigslist dominates. Simplicity wins.
      - seiji 17y ago
        Craigslist isn't a "simplicity wins" situation, it's a lockin-by-failure-of-innovation situation (see: IE6).
  - jeremymims 17y ago
    It's broken because the people who go out of their way to create software for craigslist aren't largely duplicating functionality. Want a map view so you can see all the relevant listings in one view? Want craigslist to deliberately filter the same text or picture used for different listed apartments or jobs so you don't waste your time or get scammed? Want to search by squarefoot or by actual bedrooms or by picture? Apple would say "There's an app for that." Craigslist says "screw you".
    - jimboyoungblood 17y ago
      It just means there's a huge opportunity to build a great craigslist competitor by building a service that has all these features.
      - Kaizyn 17y ago
        Have you ever heard of the network effect? Craigslist has such a lock on its market that you would have to have something extremely compelling to dislodge them. A few flashy maps and a web 2.0ish appearance wouldn't be nearly enough to do it.
        deleted 17y ago
        [deleted]
        elai 17y ago
        Classified sites gain a regional dominance. Craigslist in canada for example, is only really dominant in vancouver, one of their biggest foreign markets. Kijiji is dominant in other towns.
  - samwise 17y ago
    Their data? The data should belong to the people that produce the content Not the people that host it. Craigslist gains value from it's users not they other way around.
  - jacoblyles 17y ago
    How are these people a threat to Craigslist business? The more people use its data, the more valuable it becomes to post to Craigslist, and the more market share Craigslist gains. Craigslist does not lose a dime because of third-party mash-up sites.
- forkandwait 17y ago
  It seems to me that Craig is trying to balance making CL wide-openly available to individual people while preventing a business ecology from growing on top of it. He is all for the latter, and really hates the former. In contrast EBay thrives on the business ecology it has created. (But then I think Craig is a socialist at heart -- and bless him for it!) So, while I am sorry that a bunch of cool hacks got shut down, it might be a pretty reasonable decision on CL's part to cut off Yahoo Pipes. Plus, it's his site to do with it what he wants... (There is some great interview with him in Wired that got linked here not too long ago.)
  - alaskamiller 17y ago
    I think you mean he's for the former and against the latter. But the sentence is moot since you already stated the facts previously.
antidaily 17y ago
Unfortunate, but predictable... as pointed out in your previous thread: http://news.ycombinator.com/item?id=963634 http://news.ycombinator.com/item?id=963634
- slay2k 17y ago
  Craig Newmark: "as a rule of thumb, okay to use RSS feeds for noncommercial purposes." What's predictable here ?
  - hack_edu 17y ago
    "A rule of thumb is a principle with broad application that is not intended to be strictly accurate or reliable for every situation." http://en.wikipedia.org/wiki/Rule_of_thumb http://en.wikipedia.org/wiki/Rule_of_thumb
  - rms 17y ago
    Predictable is that craigslist seems to have some vendetta against external web apps, and you showed them a big data hole they had. I'd ask Craig again on Twitter about the Pipes thing, he's better at responding to tweets than emails sometimes. http://twitter.com/craignewmark http://twitter.com/craignewmark
    - slay2k 17y ago
      Not today he isn't, asked him 3 hours ago, no response.
jeremymims 17y ago
For the life of me, I can't understand why craigslist has had this mentality. I basically have to chalk it up to Craig being stubborn. He was burned by ebay grabbing 20%, he hasn't changed the design ever, and may constantly live in fear of being co-opted by the man. But like it or not, Craigslist is the man. And when your users go out of their way to create pieces of software that actually improve the experience for your users (like letting me sort through the pile of duplicate, spam, and misleading listings) don't shut them down. Open up and help them. Craigslist lets people post their information for free in many cases, but for some strange reason won't let people take information out. I had hoped that the culture might become more developer friendly after Jeremy Zawodny (formerly at Yahoo) joined up. Zawodny even wrote an article about how all this terrific data was available via RSS, but there was no good way to piece it together: http://jeremy.zawodny.com/blog/archives/008513.html http://jeremy.zawodny.com/blog/archives/008513.html Craig is a programmer and he comes across as a normal, humble sort of guy. We like him and we like his service. I don't understand his obsession with keeping craigslist a) locked in the dark ages for functionality and b) not letting anyone build something that adds useful features. It really starts to make you think. Is he just strange or (if you don't give him the benefit of the doubt like I do) evil?
- forkandwait 17y ago
  "For the life of me, I can't understand why craigslist has had this mentality." You mean the mentality that has created one of the top five websites ever? I mean EVER? ... Oh, that poor Craig, silly idealistic fellow...
  - jeremymims 17y ago
    No, not the mentality that created a large website. The mentality that keeps people from creating ways to view the data that benefit the website's users. Craigslist won the market. Now it's time to open up. If craigslist were solely out to monetize this website, that'd be fine. But they don't. They keep some things free and Craig has made a big deal about making money in the categories that needed a charge to prevent spamming. So there's a disconnect between a guy who wants to fight the "man" and someone who clearly just went out of his way to squash a bunch of little ones. The correct moves in this situation for a top five website are: - build the feature yourself. It sucks for the little guy who made his website but your users get the benefits anyway. - don't build the feature because this feature is trivial and your users don't want it anyway. So this external website languishes and provides no benefit and no harm. - buy the feature. Buy the company for a little money and implement the feature. The top 5 websites do this all the time. As an added bonus, you can acquire this talent who can create lots of great features for your users. - hire the guys who made the feature and ask them to build it for you. They'd probably leap at the chance. - Give the external website your blessing. Invest a little money in them so that if they strike it big you'll benefit. - Make them pay a data access fee. Charge them per query. Any of these would have been better options. This would be how a top five company should behave.
maurycy 17y ago
Sorry to say but company that treats developers like dirt is doomed.
- misuba 17y ago
  Craigslist has continued to thrive over at least ten years of neglect and hostility towards developers as well as interaction design. They seem to disprove your claim.
  - jeremymims 17y ago
    I think a big part of this is the misattribution of the .org label combined with overly simple design. People believe that craigslist is altruistic and trustworthy because it has given the appearance that it can only just keep the lights on and pay the bills. A real company could afford a logo designer! So they've gotten huge (not in number of employees), but they're behaving just as arrogantly.
  - maurycy 17y ago
    It is rather blunt to assume so. Nearly every week we discuss whether Google can be defeated. Yet, you think that Craigslist cannot? My point is that their closeness will be major factor in their failure. EDIT: I don't buy Craigslist hype. To me, their service is poor. Outside the US, especially in the East Europe, there's a lot of better alternatives. There was a lot of discussions why they succeed. I point out how they're likely to fail.
deleted 17y ago
[deleted]
deleted 17y ago
[deleted]
justinhj 17y ago
Surely if you build something on a stack of services that belong to other people, this is a hazard you have to face. Craigslist have no obligation to let you use their data however you like.
sh1mmer 17y ago
For what it's worth YQL still works (it uses a different UA, "Pipes 2.0") so the YQL module in Pipes will probably work too. That is until they turn that off too.
callmeed 17y ago
People here on HN often make comments about the dangers of building your business/app/service on top someone else's business/app/service (be it Twitter, FB, etc.). If your app depends on Yahoo! Pipes AND Craigslist, well ... that's probably extra dangerous.
nym 17y ago
This is not the first time a site has been shut down when an enthusiastic developer emailed Craig saying "Hey check out what I built!" ...nor will it be the last I suspect.
- madair 17y ago
  References? [Edit:] Thanks!
  - nym 17y ago
    A close friend... he asked me not to use his name.
  - biotech 17y ago
    http://www.codinghorror.com/craigslist/ http://www.codinghorror.com/craigslist/
scelerat 17y ago
Perhaps it's as simple as CL has no way of determining who is on the other side of the pipe. In Craig's quoted statement about RSS feeds, it seems that they are concerned about commercial use of CL data. If Yahoo Pipes essentially masks the origin of the request, then it makes CL's job more difficult. As for determining motive -- profit or otherwise -- I suspect it's more about keeping others' greedy mitts off their community than cold cash. CL has been about the community of people using it since it was just a mailing list, and publicly Craig and Craigslist have been pretty clear about what they value. So, they probably see pipes as a threat to that community.
luckydude 17y ago
I know Craig personally, slightly, we've been on panels together. I also know Jim and Susan, been to their place in the city, hung out. I'm old, 47, been in the valley for 20+ years. Seen a lot of shit, seen a lot of people get screwed over in stock options, seen the good and seen the bad. What I'm trying to do is establish credibility (prolly didn't do a good job). Whatever. You can listen or not. Craig is good people. Really really good people. He cares, he does the right thing as much as he can do it. Craigslist could be at least 2 orders of magnitude more rich if they wanted to fuck you over. They haven't done that, they repeatedly try and take care of their users. If you are unhappy with craig I pity you. Real life is going to fuck you.
- natrius 17y ago
  This comment is a giant logical fallacy.
  - novum 17y ago
    appeal to authority.
- slay2k 17y ago
  Thank you for the life lesson luckydude, but I'd prefer an explanation from Craig or Jim themselves.
- tezza 17y ago
  Thanks for your comment. It's hard to establish your credentials (account created 7 hours ago, just like your post). Normally I would have looked through your previous comments/blog and tried to establish veracity of your claims. Freshness of account means thats not possible. But in my mind there's two options 1) You're claims are not legit, you're a stooge Shame on you. 2) You are legit, and you are mentioning your personal opinion Thank you! It's nice to hear someone vouching for someone. The HN startup community do it all the time. They just happen to be 20 somethings looking (rightly) to fit into the YC community. I don't think you are appealing to authority as your comment will be judged on its own merits. Fuck those argument technicality Nazis anyway... you're not claiming to be a doctor, or the head of some company. Nor are you claiming to have special information that other people don't have... I'm sure lots of people have met this Craig (not me I'm in London) and could deny your view hiding behind screen_names. I'll wait to see whether you continue to post to see if its 1) or 2). In case you are 2) please don't be discouraged by ignorant haters.
hyperbovine 17y ago
Hold on here, let's call a spade a spade. Your project has its own doofy .com name, calls itself "alpha", and was rolled out on a web site whose primary function is to connect people who are trying to launch online business ventures. So to wave the bloody shirt after Craig stated that RSS feeds for noncommercial purposes are golden, is a little disingenuous. As others have pointed out, Craig's whole m.o. appears to be preventing people from building for-profit businesses on top of his site. Craig is anti-business. That's his prerogative. You obviously disagree with his philosophy, but a difference of opinion is not cause to go around calling somebody evil. He's done more good through his site and the $$ he generates off his site (which gets plowed into his foundation) than you, me and the next 100 people who read this combined.
- slay2k 17y ago
  It would only be disingenuous if there was any kind of commercial aspect to the site as it stands. There was none, and "rolling it out" on HN is merely asking for feedback, not trying to sign up paid users. In the email exchange I've had with Craig, not all of which I've posted, I've asked him to clarify on this point: "Oh, and about being noncommercial. A bunch of the RSS mashup sites I've seen, like Craiglook, throw up advertising presumably to cover server costs or make a few bucks. If I didn't want to go that route, would asking for donations be acceptable ?" In his response, he said he can ask. I said please do, and that was the end of it. So for CL to break things when there is absolutely no sign of commercial purpose is a little unnecessary in my eyes.
  - cdibona 17y ago
    I hate to break it to you, but it doesn't matter if there is no 'sign' of commercial purpose. Are you a non-profit? If you are, let them know. In my experience they'll hear you out.
  - jshspencer 17y ago
    Slay, Just don't post any advertising from your site. I think that worked fine for Craiglook. Futhermore, Yahoo pipes seem to be working fine with craigslist. Are you sure it's not your site being blocked ?
mmagin 17y ago
As a random datapoint: Back when I used bloglines, I had subscribed to some searches with them, and after a few weeks, it stopped working. Craigslist seems to block anyone who hits a given rss feed much.
skolor 17y ago
You know, I was playing with a couple scripts (mainly Markov chain implementation), that used Craigslist as a source for data. I got a good ways into it, and got a working model (I could generate text based on the handful of inputs I had set up for it). It wasn't until that point (I know, stupid) that I went back through all the ToS and realized I really couldn't do that. Now, I know I might be able to get some where with them if I tried emailing them, and asked for the data I wanted. Then again, it sounds like I probably wouldn't. Either way, while the data was surprising well suited for what I wanted to do, it isn't really available, at least not in any way I could actually use it. It makes me kind of sad every time I see someone do something like this. You've created something super cool, something with tons and tons of data. For what I was using it for (localized text samples), I can't think of a better, more complete place to find data from. But when you see something cool, something you never thought of/never thought was important enough to implement, and go out of your way to squash it, it makes the part of me that loves data feel bad about it. Craiglist has, at any given time, literally gigs of written text with all sorts of metadata on it. In the short bit I processed, I took all of the data from just a handful of boards, and ended up with over 150mb of data. Considering that most of that was <2 weeks old, They likely process up to a terabyte of data every year, with fairly good metadata on it. Seeing all of that only being used at face value seems like such a waste. (As a side note, anyone with a bunch of written text samples, with information as to where each of them was written, I would really like to talk to you. Actually, that goes for anyone with any sizeable amount of data. Just because you don't see a use for it doesn't meant that there isn't someone out there who would absolutely love to get their hands on it.)
luckydude 17y ago
You guys. Get a grip. It's his company, he built it, you didn't. He makes the rules, you don't. If you don't like that, then go build your own site that has the same level of success and define your own rules. And deal with all the people who tell you you are doing it wrong, that's a lot of fun. For the record, I have no connection to craigslist other than admiring it. I'm flaming because I've got my own company and I deal with this crap as well and it's annoying. If you don't like the way someone else is doing it then either go do it better or shut the fuck up.
- skolor 17y ago
  If someone is complaining about your service, it may be a good idea to listen to them. All of the "Blah, blah, if you want it you should be able to create your own super successful site" posts are really rather pointless. Yes, site/company owners have to make some tough decisions about their product, and how users can access/use it. There are going to be complaints, and there are going to be users who disagree. Telling your users to "shut the fuck up" when they're saying they disagree when you've gone from supporting them to destroying what they've built without telling them anything about it seems to be rather harsh. Its easy to come out and say "Oh, he can do anything he wants because its his company, and if you don't like it just leave". It takes a hell of a lot more effort to actually put the finger on why a move like this (blocking an entire site which hosts various mashups, over 2000 targeting just Craigslist, without notifying the users or having any real method of notifying the "developers" why it was done, or what they can do about it) and trying to point out to the people who actually could do something about it. As a tip, if you're getting that much flak from your users that you have to start flaming users on the internet about it, you're probably doing something wrong.
  - luckydude 17y ago
    Thanks for the tip. The pointer to your successful site is? Sorry, cheap shot but... And your claim that you or the other folks are a user is based on what? Craig takes care of the _actual_ users, the people who buy/sell stuff there, use it for job listings/postings, etc. Your point about listening to your users would be valid if you actually were one. Or better yet represented a whole class of users. The whining here comes from a set of people who are leaches. They are people who want to leverage craigslist.org. Whether it is for profit or for fun or for fame, it's leveraging the work of someone else. In no way are those people "users" of craigslist.
petercooper 17y ago
Back when I was running a service called Feed Digest (which I went on to sell - mostly due to the issue I'm raising here - now known as Feed Informer) this was a common problem. Not just with Craigslist, but with feeds from Google and all over the place. We wouldn't hit a single feed any more than once every 30 minutes but getting whitelisted everywhere was an exhausting task and ensuring you could always get the feeds for your users to reprocess was so difficult that I was heavily turned off of getting into a "man in the middle" situation like that again.
leelin 17y ago
Maybe this isn't as reactive as you think. I noticed just in the last few days CL blocks any listing containing the string "bit.ly" (a net good for CL users). Try creating a post and you will get an error on the screen where you normally solve a captcha. Maybe your correspondence coincided with a more general security tune-up they decided to roll out on Thanksgiving?
xtrmntr 17y ago
so how is http://housingmaps.com http://housingmaps.com still allowed?
- ig1 17y ago
  they got special permission from craigslist
mcantelon 17y ago
Suck the RSS to a VPS and read it via Pipes through that.
fizx 17y ago
http://housingmaps.com http://housingmaps.com has been up for the last, what, three years? I guess some mashups are ok.
blasdel 17y ago
Your problem is that in their eyes Yahoo Pipes doesn't have any legitimate uses against their feeds -- it's a naked proxy. They'd never block Google Reader -- they'd sooner just disable RSS entirely.

93 points by slay2k 17y ago

It hurts me when people chose to keep things broken (Craigslist) for irrational reasons, especially when it wastes hours of time for millions of people.

jimboyoungblood 17y ago
their decision seems perfectly rational. if services like this one become popular, they are a huge threat to their business. they're not morally obligated to provide unfettered access to their data to anyone who wants it- i fail to see how it is "broken"
- seiji 17y ago
  Brokenness: Show me all apartments or rooms-for-rent in mountain view with dish washers, clothes washers, and covered parking. There's no way to (legitimately) outwardly index, categorize, or aggregate craigslist content. (Related gripe: apartment searching in google is useless too: http://www.google.com/search?q=mountain+view+apartments http://www.google.com/search?q=mountain+view+apartments It gets slightly better with searching on maps, but not by much.)
  - ams6110 17y ago
    Many apartment/house finding sites have these features: search pages with dozens of checkboxes and dropdowns, and equally complex data entry pages for the people listing the apartments/houses. Yet Craigslist dominates. Simplicity wins.
    - seiji 17y ago
      Craigslist isn't a "simplicity wins" situation, it's a lockin-by-failure-of-innovation situation (see: IE6).
- jeremymims 17y ago
  It's broken because the people who go out of their way to create software for craigslist aren't largely duplicating functionality. Want a map view so you can see all the relevant listings in one view? Want craigslist to deliberately filter the same text or picture used for different listed apartments or jobs so you don't waste your time or get scammed? Want to search by squarefoot or by actual bedrooms or by picture? Apple would say "There's an app for that." Craigslist says "screw you".
  - jimboyoungblood 17y ago
    It just means there's a huge opportunity to build a great craigslist competitor by building a service that has all these features.
    - Kaizyn 17y ago
      Have you ever heard of the network effect? Craigslist has such a lock on its market that you would have to have something extremely compelling to dislodge them. A few flashy maps and a web 2.0ish appearance wouldn't be nearly enough to do it.
      - deleted 17y ago
        [deleted]
      - elai 17y ago
        Classified sites gain a regional dominance. Craigslist in canada for example, is only really dominant in vancouver, one of their biggest foreign markets. Kijiji is dominant in other towns.
- samwise 17y ago
  Their data? The data should belong to the people that produce the content Not the people that host it. Craigslist gains value from it's users not they other way around.
- jacoblyles 17y ago
  How are these people a threat to Craigslist business? The more people use its data, the more valuable it becomes to post to Craigslist, and the more market share Craigslist gains. Craigslist does not lose a dime because of third-party mash-up sites.
forkandwait 17y ago
It seems to me that Craig is trying to balance making CL wide-openly available to individual people while preventing a business ecology from growing on top of it. He is all for the latter, and really hates the former. In contrast EBay thrives on the business ecology it has created. (But then I think Craig is a socialist at heart -- and bless him for it!) So, while I am sorry that a bunch of cool hacks got shut down, it might be a pretty reasonable decision on CL's part to cut off Yahoo Pipes. Plus, it's his site to do with it what he wants... (There is some great interview with him in Wired that got linked here not too long ago.)
- alaskamiller 17y ago
  I think you mean he's for the former and against the latter. But the sentence is moot since you already stated the facts previously.

antidaily 17y ago

Unfortunate, but predictable... as pointed out in your previous thread: http://news.ycombinator.com/item?id=963634 http://news.ycombinator.com/item?id=963634

slay2k 17y ago
Craig Newmark: "as a rule of thumb, okay to use RSS feeds for noncommercial purposes." What's predictable here ?
- hack_edu 17y ago
  "A rule of thumb is a principle with broad application that is not intended to be strictly accurate or reliable for every situation." http://en.wikipedia.org/wiki/Rule_of_thumb http://en.wikipedia.org/wiki/Rule_of_thumb
- rms 17y ago
  Predictable is that craigslist seems to have some vendetta against external web apps, and you showed them a big data hole they had. I'd ask Craig again on Twitter about the Pipes thing, he's better at responding to tweets than emails sometimes. http://twitter.com/craignewmark http://twitter.com/craignewmark
  - slay2k 17y ago
    Not today he isn't, asked him 3 hours ago, no response.

jeremymims 17y ago

For the life of me, I can't understand why craigslist has had this mentality. I basically have to chalk it up to Craig being stubborn. He was burned by ebay grabbing 20%, he hasn't changed the design ever, and may constantly live in fear of being co-opted by the man. But like it or not, Craigslist is the man. And when your users go out of their way to create pieces of software that actually improve the experience for your users (like letting me sort through the pile of duplicate, spam, and misleading listings) don't shut them down. Open up and help them. Craigslist lets people post their information for free in many cases, but for some strange reason won't let people take information out. I had hoped that the culture might become more developer friendly after Jeremy Zawodny (formerly at Yahoo) joined up. Zawodny even wrote an article about how all this terrific data was available via RSS, but there was no good way to piece it together: http://jeremy.zawodny.com/blog/archives/008513.html http://jeremy.zawodny.com/blog/archives/008513.html Craig is a programmer and he comes across as a normal, humble sort of guy. We like him and we like his service. I don't understand his obsession with keeping craigslist a) locked in the dark ages for functionality and b) not letting anyone build something that adds useful features. It really starts to make you think. Is he just strange or (if you don't give him the benefit of the doubt like I do) evil?

forkandwait 17y ago
"For the life of me, I can't understand why craigslist has had this mentality." You mean the mentality that has created one of the top five websites ever? I mean EVER? ... Oh, that poor Craig, silly idealistic fellow...
- jeremymims 17y ago
  No, not the mentality that created a large website. The mentality that keeps people from creating ways to view the data that benefit the website's users. Craigslist won the market. Now it's time to open up. If craigslist were solely out to monetize this website, that'd be fine. But they don't. They keep some things free and Craig has made a big deal about making money in the categories that needed a charge to prevent spamming. So there's a disconnect between a guy who wants to fight the "man" and someone who clearly just went out of his way to squash a bunch of little ones. The correct moves in this situation for a top five website are: - build the feature yourself. It sucks for the little guy who made his website but your users get the benefits anyway. - don't build the feature because this feature is trivial and your users don't want it anyway. So this external website languishes and provides no benefit and no harm. - buy the feature. Buy the company for a little money and implement the feature. The top 5 websites do this all the time. As an added bonus, you can acquire this talent who can create lots of great features for your users. - hire the guys who made the feature and ask them to build it for you. They'd probably leap at the chance. - Give the external website your blessing. Invest a little money in them so that if they strike it big you'll benefit. - Make them pay a data access fee. Charge them per query. Any of these would have been better options. This would be how a top five company should behave.

maurycy 17y ago

Sorry to say but company that treats developers like dirt is doomed.

misuba 17y ago
Craigslist has continued to thrive over at least ten years of neglect and hostility towards developers as well as interaction design. They seem to disprove your claim.
- jeremymims 17y ago
  I think a big part of this is the misattribution of the .org label combined with overly simple design. People believe that craigslist is altruistic and trustworthy because it has given the appearance that it can only just keep the lights on and pay the bills. A real company could afford a logo designer! So they've gotten huge (not in number of employees), but they're behaving just as arrogantly.
- maurycy 17y ago
  It is rather blunt to assume so. Nearly every week we discuss whether Google can be defeated. Yet, you think that Craigslist cannot? My point is that their closeness will be major factor in their failure. EDIT: I don't buy Craigslist hype. To me, their service is poor. Outside the US, especially in the East Europe, there's a lot of better alternatives. There was a lot of discussions why they succeed. I point out how they're likely to fail.

deleted 17y ago

[deleted]

deleted 17y ago

[deleted]

justinhj 17y ago

Surely if you build something on a stack of services that belong to other people, this is a hazard you have to face. Craigslist have no obligation to let you use their data however you like.

sh1mmer 17y ago

For what it's worth YQL still works (it uses a different UA, "Pipes 2.0") so the YQL module in Pipes will probably work too. That is until they turn that off too.

callmeed 17y ago

People here on HN often make comments about the dangers of building your business/app/service on top someone else's business/app/service (be it Twitter, FB, etc.). If your app depends on Yahoo! Pipes AND Craigslist, well ... that's probably extra dangerous.

nym 17y ago

This is not the first time a site has been shut down when an enthusiastic developer emailed Craig saying "Hey check out what I built!" ...nor will it be the last I suspect.

madair 17y ago
References? [Edit:] Thanks!
- nym 17y ago
  A close friend... he asked me not to use his name.
- biotech 17y ago
  http://www.codinghorror.com/craigslist/ http://www.codinghorror.com/craigslist/

scelerat 17y ago

Perhaps it's as simple as CL has no way of determining who is on the other side of the pipe. In Craig's quoted statement about RSS feeds, it seems that they are concerned about commercial use of CL data. If Yahoo Pipes essentially masks the origin of the request, then it makes CL's job more difficult. As for determining motive -- profit or otherwise -- I suspect it's more about keeping others' greedy mitts off their community than cold cash. CL has been about the community of people using it since it was just a mailing list, and publicly Craig and Craigslist have been pretty clear about what they value. So, they probably see pipes as a threat to that community.

luckydude 17y ago

I know Craig personally, slightly, we've been on panels together. I also know Jim and Susan, been to their place in the city, hung out. I'm old, 47, been in the valley for 20+ years. Seen a lot of shit, seen a lot of people get screwed over in stock options, seen the good and seen the bad. What I'm trying to do is establish credibility (prolly didn't do a good job). Whatever. You can listen or not. Craig is good people. Really really good people. He cares, he does the right thing as much as he can do it. Craigslist could be at least 2 orders of magnitude more rich if they wanted to fuck you over. They haven't done that, they repeatedly try and take care of their users. If you are unhappy with craig I pity you. Real life is going to fuck you.

natrius 17y ago
This comment is a giant logical fallacy.
- novum 17y ago
  appeal to authority.
slay2k 17y ago
Thank you for the life lesson luckydude, but I'd prefer an explanation from Craig or Jim themselves.
tezza 17y ago
Thanks for your comment. It's hard to establish your credentials (account created 7 hours ago, just like your post). Normally I would have looked through your previous comments/blog and tried to establish veracity of your claims. Freshness of account means thats not possible. But in my mind there's two options 1) You're claims are not legit, you're a stooge Shame on you. 2) You are legit, and you are mentioning your personal opinion Thank you! It's nice to hear someone vouching for someone. The HN startup community do it all the time. They just happen to be 20 somethings looking (rightly) to fit into the YC community. I don't think you are appealing to authority as your comment will be judged on its own merits. Fuck those argument technicality Nazis anyway... you're not claiming to be a doctor, or the head of some company. Nor are you claiming to have special information that other people don't have... I'm sure lots of people have met this Craig (not me I'm in London) and could deny your view hiding behind screen_names. I'll wait to see whether you continue to post to see if its 1) or 2). In case you are 2) please don't be discouraged by ignorant haters.

hyperbovine 17y ago

Hold on here, let's call a spade a spade. Your project has its own doofy .com name, calls itself "alpha", and was rolled out on a web site whose primary function is to connect people who are trying to launch online business ventures. So to wave the bloody shirt after Craig stated that RSS feeds for noncommercial purposes are golden, is a little disingenuous. As others have pointed out, Craig's whole m.o. appears to be preventing people from building for-profit businesses on top of his site. Craig is anti-business. That's his prerogative. You obviously disagree with his philosophy, but a difference of opinion is not cause to go around calling somebody evil. He's done more good through his site and the $$ he generates off his site (which gets plowed into his foundation) than you, me and the next 100 people who read this combined.

slay2k 17y ago
It would only be disingenuous if there was any kind of commercial aspect to the site as it stands. There was none, and "rolling it out" on HN is merely asking for feedback, not trying to sign up paid users. In the email exchange I've had with Craig, not all of which I've posted, I've asked him to clarify on this point: "Oh, and about being noncommercial. A bunch of the RSS mashup sites I've seen, like Craiglook, throw up advertising presumably to cover server costs or make a few bucks. If I didn't want to go that route, would asking for donations be acceptable ?" In his response, he said he can ask. I said please do, and that was the end of it. So for CL to break things when there is absolutely no sign of commercial purpose is a little unnecessary in my eyes.
- cdibona 17y ago
  I hate to break it to you, but it doesn't matter if there is no 'sign' of commercial purpose. Are you a non-profit? If you are, let them know. In my experience they'll hear you out.
- jshspencer 17y ago
  Slay, Just don't post any advertising from your site. I think that worked fine for Craiglook. Futhermore, Yahoo pipes seem to be working fine with craigslist. Are you sure it's not your site being blocked ?

mmagin 17y ago

As a random datapoint: Back when I used bloglines, I had subscribed to some searches with them, and after a few weeks, it stopped working. Craigslist seems to block anyone who hits a given rss feed much.

skolor 17y ago

You know, I was playing with a couple scripts (mainly Markov chain implementation), that used Craigslist as a source for data. I got a good ways into it, and got a working model (I could generate text based on the handful of inputs I had set up for it). It wasn't until that point (I know, stupid) that I went back through all the ToS and realized I really couldn't do that. Now, I know I might be able to get some where with them if I tried emailing them, and asked for the data I wanted. Then again, it sounds like I probably wouldn't. Either way, while the data was surprising well suited for what I wanted to do, it isn't really available, at least not in any way I could actually use it. It makes me kind of sad every time I see someone do something like this. You've created something super cool, something with tons and tons of data. For what I was using it for (localized text samples), I can't think of a better, more complete place to find data from. But when you see something cool, something you never thought of/never thought was important enough to implement, and go out of your way to squash it, it makes the part of me that loves data feel bad about it. Craiglist has, at any given time, literally gigs of written text with all sorts of metadata on it. In the short bit I processed, I took all of the data from just a handful of boards, and ended up with over 150mb of data. Considering that most of that was <2 weeks old, They likely process up to a terabyte of data every year, with fairly good metadata on it. Seeing all of that only being used at face value seems like such a waste. (As a side note, anyone with a bunch of written text samples, with information as to where each of them was written, I would really like to talk to you. Actually, that goes for anyone with any sizeable amount of data. Just because you don't see a use for it doesn't meant that there isn't someone out there who would absolutely love to get their hands on it.)

luckydude 17y ago

You guys. Get a grip. It's his company, he built it, you didn't. He makes the rules, you don't. If you don't like that, then go build your own site that has the same level of success and define your own rules. And deal with all the people who tell you you are doing it wrong, that's a lot of fun. For the record, I have no connection to craigslist other than admiring it. I'm flaming because I've got my own company and I deal with this crap as well and it's annoying. If you don't like the way someone else is doing it then either go do it better or shut the fuck up.

skolor 17y ago
If someone is complaining about your service, it may be a good idea to listen to them. All of the "Blah, blah, if you want it you should be able to create your own super successful site" posts are really rather pointless. Yes, site/company owners have to make some tough decisions about their product, and how users can access/use it. There are going to be complaints, and there are going to be users who disagree. Telling your users to "shut the fuck up" when they're saying they disagree when you've gone from supporting them to destroying what they've built without telling them anything about it seems to be rather harsh. Its easy to come out and say "Oh, he can do anything he wants because its his company, and if you don't like it just leave". It takes a hell of a lot more effort to actually put the finger on why a move like this (blocking an entire site which hosts various mashups, over 2000 targeting just Craigslist, without notifying the users or having any real method of notifying the "developers" why it was done, or what they can do about it) and trying to point out to the people who actually could do something about it. As a tip, if you're getting that much flak from your users that you have to start flaming users on the internet about it, you're probably doing something wrong.
- luckydude 17y ago
  Thanks for the tip. The pointer to your successful site is? Sorry, cheap shot but... And your claim that you or the other folks are a user is based on what? Craig takes care of the _actual_ users, the people who buy/sell stuff there, use it for job listings/postings, etc. Your point about listening to your users would be valid if you actually were one. Or better yet represented a whole class of users. The whining here comes from a set of people who are leaches. They are people who want to leverage craigslist.org. Whether it is for profit or for fun or for fame, it's leveraging the work of someone else. In no way are those people "users" of craigslist.

petercooper 17y ago

Back when I was running a service called Feed Digest (which I went on to sell - mostly due to the issue I'm raising here - now known as Feed Informer) this was a common problem. Not just with Craigslist, but with feeds from Google and all over the place. We wouldn't hit a single feed any more than once every 30 minutes but getting whitelisted everywhere was an exhausting task and ensuring you could always get the feeds for your users to reprocess was so difficult that I was heavily turned off of getting into a "man in the middle" situation like that again.

leelin 17y ago

Maybe this isn't as reactive as you think. I noticed just in the last few days CL blocks any listing containing the string "bit.ly" (a net good for CL users). Try creating a post and you will get an error on the screen where you normally solve a captcha. Maybe your correspondence coincided with a more general security tune-up they decided to roll out on Thanksgiving?

xtrmntr 17y ago

so how is http://housingmaps.com http://housingmaps.com still allowed?

ig1 17y ago
they got special permission from craigslist

mcantelon 17y ago

Suck the RSS to a VPS and read it via Pipes through that.

fizx 17y ago

http://housingmaps.com http://housingmaps.com has been up for the last, what, three years? I guess some mashups are ok.

blasdel 17y ago

Your problem is that in their eyes Yahoo Pipes doesn't have any legitimate uses against their feeds -- it's a naked proxy. They'd never block Google Reader -- they'd sooner just disable RSS entirely.