12 ms·

"(Craigslist) said it offered a license that would have allowed PadMapper to use its data on mobile applications but that the competitor did not accept the term

"(Craigslist) said it offered a license that would have allowed PadMapper to use its data on mobile applications but that the competitor did not accept the terms." But on the other hand, "users and developers are exasperated with Craigslist’s insistence on preserving an outdated interface and design." I don't know the legal merits of either party's position, but from an ethical perspective, I tend to side with Craigslist here. Dissatisfaction with a commercial site's UI is not just cause for using their data without permission, particularly if they had made an effort to offer a licensing agreement, whatever the terms might have been.

wickedchicken 14y ago
I see this from the opposite viewpoint, that PadMapper is operating as a search engine designed to help you find the right Craigslist posts for your apartment hunt. Claiming that PadMapper's success is from Craigslist failing to 'provide a good UI' is like suggesting that Google News' success is because newspaper companies failed to provide good article SEO. Instead, it's a different modality of content discovery, and it ultimately redirects you to the original source. In my (personal) opinion, I would argue that the content providers or aggregators and search engines benefit synergistically. CL and Belgian newspapers appear to disagree.
- eli 14y ago
  I don't think this has any legal bearing for anyone, but it's worth noting that Google respects publishers who do not wish to be indexed.
  - hardtke 14y ago
    Here's the abstract for a nice review of search engine law: http://works.bepress.com/james_grimmelmann/13/ http://works.bepress.com/james_grimmelmann/13/ If I remember correctly, indexing a site that asks to not be indexed might be illegal as an illegal tresspass, but it is not settled law. The argument is that you are stealing resources (computer time) from the site owner.
    - learc83 14y ago
      >The argument is that you are stealing resources (computer time) from the site owner. That's why 3taps is getting the data from google's cache without touching craigslist servers.
      - smackfu 14y ago
        And makes it impossible to block them as a side effect.
      - unreal37 14y ago
        It's not impossible. You just tell google not to cache.
      - slig 14y ago
        How the heck are they scraping Google without being banned or rate limited?
        TomAnthony 14y ago
        There are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies. The two that jump to mind are Authority Labs and SEOmoz. I guess: a shed load of proxies. :)
        nmridul 14y ago
        Amazon/ other clouds out there. Just auto provision your instances (lots of them), scrap, sleep, wake, scrap, sleep...
    - malandrew 14y ago
      IANAL but AFAIK it is only a civil matter (i.e. not illegal) since it is a usually prosecuted as a tort of trespass to chattels. For such a case to succeed the prosecution needs to show that the actions of the defendant deprived them of use of the good they were trespassing on. i.e. they need to cause enough of a burden on the servers that the claimant or their customers could not use the service.
  - abecedarius 14y ago
    What I'd care more about is whether the people posting the ads wish to be indexed; I'd be surprised if many don't. (I don't know about the legal bearing either.)
    - SilasX 14y ago
      Perhaps craigslist could put up a checkbox (like they do with the never-checked "It's okay to contact me about products ...") that says something like: "I'm okay with people finding this listing through another service." Perhaps because they don't want people to get to their listings that way even if they want to, and thus don't want your opinion? (Edit: and perhaps they really aren't as interested as they claim in making it easier for buyers and sellers to find each other?)
  - flinty 14y ago
    So eventually this comes down to CL trying to protect it's monopoly as the listing site of the internet
- Kadrith 14y ago
  Does PadMapper respect the robots.txt file?
  - IanDrake 14y ago
    PadMapper uses 3Taps.com to get the data. 3Taps gets Craigslist data through Google. Google's bot respects robots.txt. So, yes. [edit for clarity, and for misspelling clarity]
    - alex_c 14y ago
      Well... "sort of", then. PadMapper respects the robots.txt for GoogleBot, not for PadMapper.
    - ig1 14y ago
      Presumably 3Taps violates Google's ToS to get the data from them ?
      - stfu 14y ago
        I would really love to know that kind of shady proxy bot army they have implemented in order to scrape Google on such a scale.
        user345 14y ago
        This type of comment is common. Not only on HN but on forums in general. While I agree the web is full of lowlifes engaged in web development, many of them in porn or some other area that appeals to base instincts, I find this comment perplexing. Because it is so subjective, yet it tries to seem objective by focusing on some random criteria. Google employs a "bot army" to scrape the entire web. So what? If the comment was something like "I don't like Company X." Or even "I don't like Company X because...", it would make sense to me. But that is not how this common type of comment goes. Instead it suggests that bot=evil, i.e. any sort of automation or any sort of data collection by anyone other than [your favorite company] is "shady". That's crazy. IMO. It's what a company does with the data that matters. Anyway, I'm not keen on 3Taps because they are not provinding bulk data, only API's that require "developer keys". Why? Either you are going to democratise data, or you are just another schemer trying to find ways to collect infromation about people, in this case people using "your API". I don't want API's I want the data. I can make my own interfaces thank you.
- waterlesscloud 14y ago
  For that analogy to hold, PadMapper would have to be pulling data from a large number of sources. The value that Google News provides is that it aggregates a very large number of sources. PadMapper does not.
  - fireflash38 14y ago
    Padmapper collates data from multiple sources: Craigslist, Rent.com, Apartments.com, and "Others" (not sure which those would be).
    - unreal37 14y ago
      It's pretty clear PadMapper was BUILT off of Craigslist that CL's poor UI is isn't entire reason for being, and it never would have been successful without CL. It's disingenuous to ignore that by listing "rent.com" as an alternate source.
  - wickedchicken 14y ago
    > PadMapper does not. It does.
    - huhtenberg 14y ago
      But without the Craigslist data it has no value.
      - jlgreco 14y ago
        Total nonsense. I found my current place with them, without craigslist.
    - waterlesscloud 14y ago
      It does not. One main source and a handful of secondary sources is not at all comparable to a true aggregator like Google News. Look, it's obvious that even PadMapper believes Craigslist is THE data source they must have. Google News could drop any given data source and not blink. PadMapper simply is not in the same position.
jacoblyles 14y ago
The problem is that Craigslist has a monopoly. They use that monopoly to shut down competitors, resulting in millions of hours of lost time for users every month.
soup10 14y ago
I don't like Craigslist because of lock in. Sellers must sell there because they are the biggest market, buyers must buy there because they are the biggest market. Everyone is locked in. Lock-in is super common(and often unintentional or unavoidable), but it goes against the ideal of equal opportunity and competitive markets that rewards companies for a continuous commitment to quality and innovation. I feel that craigslist has done nothing wrong except for standing in the doorway when other people who want to innovate and improve things are trying to get through (like PadMapper).
- eli 14y ago
  True, but it's also very easy as a buyer (or seller) to use more than one site. Nothing about Craigslist is exclusive and in that sense the lock-in is very weak. A Craigslist competitor doesn't need a huge market share to be viable. For me as a seller, it just has to expand my audience of buyers enough to justify the small amount of time it takes to post and manage a second listing.
- res0nat0r 14y ago
  The reason why Craigslist is valuable is because of it's large network. It is obvious that smaller sites like PadMapper want to slowly try and chip away at Craiglist content (see PadLister) after they hopefully gain enough traction to supplant them. Padmapper / Padlister are fine to start their own sites and have users come and post their listings there, but not to copy content from Craigslist after they specifically asked them to stop.
  - stcredzero 14y ago
    > It is obvious that smaller sites like PadMapper want to slowly try and chip away at Craiglist content It's not Craigslist's content. It's our content, the general public's. Craigslist is just a caretaker. Right now, it's a caretaker acting in its own interests against the public's.
    - Kadrith 14y ago
      It also does not mean that CL has any obligation to provide that data to another for free. If you want the data in multiple locations, post it to those other locations.
iamwil 14y ago
The license offered by craigslist was for a mobile application only. There was nothing offered for websites. You can see this as either, 'license offered with terms' or 'no license offered at all'
dclowd9901 14y ago
How do people not see this for exactly what it is: The same situation as we are finding with cable providers not wanting to be dumb pipes. The only difference is people don't pay money to CL. They just give CL all their information, and CL decides what to do with it. Craigslist has no divine right to its users' information; it just happens to be the only viable option. If Craigslist didn't exist, someone else would, and would probably do it better. If Craigslist wants to continue existing, they should take advantage of the fact that they are the go-to for online classifieds, create a modest subscription-based API and let the information flow.

wickedchicken 14y ago

I see this from the opposite viewpoint, that PadMapper is operating as a search engine designed to help you find the right Craigslist posts for your apartment hunt. Claiming that PadMapper's success is from Craigslist failing to 'provide a good UI' is like suggesting that Google News' success is because newspaper companies failed to provide good article SEO. Instead, it's a different modality of content discovery, and it ultimately redirects you to the original source. In my (personal) opinion, I would argue that the content providers or aggregators and search engines benefit synergistically. CL and Belgian newspapers appear to disagree.

eli 14y ago
I don't think this has any legal bearing for anyone, but it's worth noting that Google respects publishers who do not wish to be indexed.
- hardtke 14y ago
  Here's the abstract for a nice review of search engine law: http://works.bepress.com/james_grimmelmann/13/ http://works.bepress.com/james_grimmelmann/13/ If I remember correctly, indexing a site that asks to not be indexed might be illegal as an illegal tresspass, but it is not settled law. The argument is that you are stealing resources (computer time) from the site owner.
  - learc83 14y ago
    >The argument is that you are stealing resources (computer time) from the site owner. That's why 3taps is getting the data from google's cache without touching craigslist servers.
    - smackfu 14y ago
      And makes it impossible to block them as a side effect.
    - unreal37 14y ago
      It's not impossible. You just tell google not to cache.
    - slig 14y ago
      How the heck are they scraping Google without being banned or rate limited?
      - TomAnthony 14y ago
        There are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies. The two that jump to mind are Authority Labs and SEOmoz. I guess: a shed load of proxies. :)
      - nmridul 14y ago
        Amazon/ other clouds out there. Just auto provision your instances (lots of them), scrap, sleep, wake, scrap, sleep...
  - malandrew 14y ago
    IANAL but AFAIK it is only a civil matter (i.e. not illegal) since it is a usually prosecuted as a tort of trespass to chattels. For such a case to succeed the prosecution needs to show that the actions of the defendant deprived them of use of the good they were trespassing on. i.e. they need to cause enough of a burden on the servers that the claimant or their customers could not use the service.
- abecedarius 14y ago
  What I'd care more about is whether the people posting the ads wish to be indexed; I'd be surprised if many don't. (I don't know about the legal bearing either.)
  - SilasX 14y ago
    Perhaps craigslist could put up a checkbox (like they do with the never-checked "It's okay to contact me about products ...") that says something like: "I'm okay with people finding this listing through another service." Perhaps because they don't want people to get to their listings that way even if they want to, and thus don't want your opinion? (Edit: and perhaps they really aren't as interested as they claim in making it easier for buyers and sellers to find each other?)
- flinty 14y ago
  So eventually this comes down to CL trying to protect it's monopoly as the listing site of the internet
Kadrith 14y ago
Does PadMapper respect the robots.txt file?
- IanDrake 14y ago
  PadMapper uses 3Taps.com to get the data. 3Taps gets Craigslist data through Google. Google's bot respects robots.txt. So, yes. [edit for clarity, and for misspelling clarity]
  - alex_c 14y ago
    Well... "sort of", then. PadMapper respects the robots.txt for GoogleBot, not for PadMapper.
  - ig1 14y ago
    Presumably 3Taps violates Google's ToS to get the data from them ?
    - stfu 14y ago
      I would really love to know that kind of shady proxy bot army they have implemented in order to scrape Google on such a scale.
      - user345 14y ago
        This type of comment is common. Not only on HN but on forums in general. While I agree the web is full of lowlifes engaged in web development, many of them in porn or some other area that appeals to base instincts, I find this comment perplexing. Because it is so subjective, yet it tries to seem objective by focusing on some random criteria. Google employs a "bot army" to scrape the entire web. So what? If the comment was something like "I don't like Company X." Or even "I don't like Company X because...", it would make sense to me. But that is not how this common type of comment goes. Instead it suggests that bot=evil, i.e. any sort of automation or any sort of data collection by anyone other than [your favorite company] is "shady". That's crazy. IMO. It's what a company does with the data that matters. Anyway, I'm not keen on 3Taps because they are not provinding bulk data, only API's that require "developer keys". Why? Either you are going to democratise data, or you are just another schemer trying to find ways to collect infromation about people, in this case people using "your API". I don't want API's I want the data. I can make my own interfaces thank you.
waterlesscloud 14y ago
For that analogy to hold, PadMapper would have to be pulling data from a large number of sources. The value that Google News provides is that it aggregates a very large number of sources. PadMapper does not.
- fireflash38 14y ago
  Padmapper collates data from multiple sources: Craigslist, Rent.com, Apartments.com, and "Others" (not sure which those would be).
  - unreal37 14y ago
    It's pretty clear PadMapper was BUILT off of Craigslist that CL's poor UI is isn't entire reason for being, and it never would have been successful without CL. It's disingenuous to ignore that by listing "rent.com" as an alternate source.
- wickedchicken 14y ago
  > PadMapper does not. It does.
  - huhtenberg 14y ago
    But without the Craigslist data it has no value.
    - jlgreco 14y ago
      Total nonsense. I found my current place with them, without craigslist.
  - waterlesscloud 14y ago
    It does not. One main source and a handful of secondary sources is not at all comparable to a true aggregator like Google News. Look, it's obvious that even PadMapper believes Craigslist is THE data source they must have. Google News could drop any given data source and not blink. PadMapper simply is not in the same position.

jacoblyles 14y ago

The problem is that Craigslist has a monopoly. They use that monopoly to shut down competitors, resulting in millions of hours of lost time for users every month.

soup10 14y ago

I don't like Craigslist because of lock in. Sellers must sell there because they are the biggest market, buyers must buy there because they are the biggest market. Everyone is locked in. Lock-in is super common(and often unintentional or unavoidable), but it goes against the ideal of equal opportunity and competitive markets that rewards companies for a continuous commitment to quality and innovation. I feel that craigslist has done nothing wrong except for standing in the doorway when other people who want to innovate and improve things are trying to get through (like PadMapper).

eli 14y ago
True, but it's also very easy as a buyer (or seller) to use more than one site. Nothing about Craigslist is exclusive and in that sense the lock-in is very weak. A Craigslist competitor doesn't need a huge market share to be viable. For me as a seller, it just has to expand my audience of buyers enough to justify the small amount of time it takes to post and manage a second listing.
res0nat0r 14y ago
The reason why Craigslist is valuable is because of it's large network. It is obvious that smaller sites like PadMapper want to slowly try and chip away at Craiglist content (see PadLister) after they hopefully gain enough traction to supplant them. Padmapper / Padlister are fine to start their own sites and have users come and post their listings there, but not to copy content from Craigslist after they specifically asked them to stop.
- stcredzero 14y ago
  > It is obvious that smaller sites like PadMapper want to slowly try and chip away at Craiglist content It's not Craigslist's content. It's our content, the general public's. Craigslist is just a caretaker. Right now, it's a caretaker acting in its own interests against the public's.
  - Kadrith 14y ago
    It also does not mean that CL has any obligation to provide that data to another for free. If you want the data in multiple locations, post it to those other locations.

iamwil 14y ago

The license offered by craigslist was for a mobile application only. There was nothing offered for websites. You can see this as either, 'license offered with terms' or 'no license offered at all'

dclowd9901 14y ago

How do people not see this for exactly what it is: The same situation as we are finding with cable providers not wanting to be dumb pipes. The only difference is people don't pay money to CL. They just give CL all their information, and CL decides what to do with it. Craigslist has no divine right to its users' information; it just happens to be the only viable option. If Craigslist didn't exist, someone else would, and would probably do it better. If Craigslist wants to continue existing, they should take advantage of the fact that they are the go-to for online classifieds, create a modest subscription-based API and let the information flow.