4 ms·

I see this from the opposite viewpoint, that PadMapper is operating as a search engine designed to help you find the right Craigslist posts for your apartment h

I see this from the opposite viewpoint, that PadMapper is operating as a search engine designed to help you find the right Craigslist posts for your apartment hunt. Claiming that PadMapper's success is from Craigslist failing to 'provide a good UI' is like suggesting that Google News' success is because newspaper companies failed to provide good article SEO. Instead, it's a different modality of content discovery, and it ultimately redirects you to the original source. In my (personal) opinion, I would argue that the content providers or aggregators and search engines benefit synergistically. CL and Belgian newspapers appear to disagree.

eli 14y ago
I don't think this has any legal bearing for anyone, but it's worth noting that Google respects publishers who do not wish to be indexed.
- hardtke 14y ago
  Here's the abstract for a nice review of search engine law: http://works.bepress.com/james_grimmelmann/13/ http://works.bepress.com/james_grimmelmann/13/ If I remember correctly, indexing a site that asks to not be indexed might be illegal as an illegal tresspass, but it is not settled law. The argument is that you are stealing resources (computer time) from the site owner.
  - learc83 14y ago
    >The argument is that you are stealing resources (computer time) from the site owner. That's why 3taps is getting the data from google's cache without touching craigslist servers.
    - smackfu 14y ago
      And makes it impossible to block them as a side effect.
    - unreal37 14y ago
      It's not impossible. You just tell google not to cache.
    - slig 14y ago
      How the heck are they scraping Google without being banned or rate limited?
      - TomAnthony 14y ago
        There are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies. The two that jump to mind are Authority Labs and SEOmoz. I guess: a shed load of proxies. :)
      - nmridul 14y ago
        Amazon/ other clouds out there. Just auto provision your instances (lots of them), scrap, sleep, wake, scrap, sleep...
  - malandrew 14y ago
    IANAL but AFAIK it is only a civil matter (i.e. not illegal) since it is a usually prosecuted as a tort of trespass to chattels. For such a case to succeed the prosecution needs to show that the actions of the defendant deprived them of use of the good they were trespassing on. i.e. they need to cause enough of a burden on the servers that the claimant or their customers could not use the service.
- abecedarius 14y ago
  What I'd care more about is whether the people posting the ads wish to be indexed; I'd be surprised if many don't. (I don't know about the legal bearing either.)
  - SilasX 14y ago
    Perhaps craigslist could put up a checkbox (like they do with the never-checked "It's okay to contact me about products ...") that says something like: "I'm okay with people finding this listing through another service." Perhaps because they don't want people to get to their listings that way even if they want to, and thus don't want your opinion? (Edit: and perhaps they really aren't as interested as they claim in making it easier for buyers and sellers to find each other?)
- flinty 14y ago
  So eventually this comes down to CL trying to protect it's monopoly as the listing site of the internet
Kadrith 14y ago
Does PadMapper respect the robots.txt file?
- IanDrake 14y ago
  PadMapper uses 3Taps.com to get the data. 3Taps gets Craigslist data through Google. Google's bot respects robots.txt. So, yes. [edit for clarity, and for misspelling clarity]
  - alex_c 14y ago
    Well... "sort of", then. PadMapper respects the robots.txt for GoogleBot, not for PadMapper.
  - ig1 14y ago
    Presumably 3Taps violates Google's ToS to get the data from them ?
    - stfu 14y ago
      I would really love to know that kind of shady proxy bot army they have implemented in order to scrape Google on such a scale.
      - user345 14y ago
        This type of comment is common. Not only on HN but on forums in general. While I agree the web is full of lowlifes engaged in web development, many of them in porn or some other area that appeals to base instincts, I find this comment perplexing. Because it is so subjective, yet it tries to seem objective by focusing on some random criteria. Google employs a "bot army" to scrape the entire web. So what? If the comment was something like "I don't like Company X." Or even "I don't like Company X because...", it would make sense to me. But that is not how this common type of comment goes. Instead it suggests that bot=evil, i.e. any sort of automation or any sort of data collection by anyone other than [your favorite company] is "shady". That's crazy. IMO. It's what a company does with the data that matters. Anyway, I'm not keen on 3Taps because they are not provinding bulk data, only API's that require "developer keys". Why? Either you are going to democratise data, or you are just another schemer trying to find ways to collect infromation about people, in this case people using "your API". I don't want API's I want the data. I can make my own interfaces thank you.
waterlesscloud 14y ago
For that analogy to hold, PadMapper would have to be pulling data from a large number of sources. The value that Google News provides is that it aggregates a very large number of sources. PadMapper does not.
- fireflash38 14y ago
  Padmapper collates data from multiple sources: Craigslist, Rent.com, Apartments.com, and "Others" (not sure which those would be).
  - unreal37 14y ago
    It's pretty clear PadMapper was BUILT off of Craigslist that CL's poor UI is isn't entire reason for being, and it never would have been successful without CL. It's disingenuous to ignore that by listing "rent.com" as an alternate source.
- wickedchicken 14y ago
  > PadMapper does not. It does.
  - huhtenberg 14y ago
    But without the Craigslist data it has no value.
    - jlgreco 14y ago
      Total nonsense. I found my current place with them, without craigslist.
  - waterlesscloud 14y ago
    It does not. One main source and a handful of secondary sources is not at all comparable to a true aggregator like Google News. Look, it's obvious that even PadMapper believes Craigslist is THE data source they must have. Google News could drop any given data source and not blink. PadMapper simply is not in the same position.

eli 14y ago

I don't think this has any legal bearing for anyone, but it's worth noting that Google respects publishers who do not wish to be indexed.

hardtke 14y ago
Here's the abstract for a nice review of search engine law: http://works.bepress.com/james_grimmelmann/13/ http://works.bepress.com/james_grimmelmann/13/ If I remember correctly, indexing a site that asks to not be indexed might be illegal as an illegal tresspass, but it is not settled law. The argument is that you are stealing resources (computer time) from the site owner.
- learc83 14y ago
  >The argument is that you are stealing resources (computer time) from the site owner. That's why 3taps is getting the data from google's cache without touching craigslist servers.
  - smackfu 14y ago
    And makes it impossible to block them as a side effect.
  - unreal37 14y ago
    It's not impossible. You just tell google not to cache.
  - slig 14y ago
    How the heck are they scraping Google without being banned or rate limited?
    - TomAnthony 14y ago
      There are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies. The two that jump to mind are Authority Labs and SEOmoz. I guess: a shed load of proxies. :)
    - nmridul 14y ago
      Amazon/ other clouds out there. Just auto provision your instances (lots of them), scrap, sleep, wake, scrap, sleep...
- malandrew 14y ago
  IANAL but AFAIK it is only a civil matter (i.e. not illegal) since it is a usually prosecuted as a tort of trespass to chattels. For such a case to succeed the prosecution needs to show that the actions of the defendant deprived them of use of the good they were trespassing on. i.e. they need to cause enough of a burden on the servers that the claimant or their customers could not use the service.
abecedarius 14y ago
What I'd care more about is whether the people posting the ads wish to be indexed; I'd be surprised if many don't. (I don't know about the legal bearing either.)
- SilasX 14y ago
  Perhaps craigslist could put up a checkbox (like they do with the never-checked "It's okay to contact me about products ...") that says something like: "I'm okay with people finding this listing through another service." Perhaps because they don't want people to get to their listings that way even if they want to, and thus don't want your opinion? (Edit: and perhaps they really aren't as interested as they claim in making it easier for buyers and sellers to find each other?)
flinty 14y ago
So eventually this comes down to CL trying to protect it's monopoly as the listing site of the internet

Kadrith 14y ago

Does PadMapper respect the robots.txt file?

IanDrake 14y ago
PadMapper uses 3Taps.com to get the data. 3Taps gets Craigslist data through Google. Google's bot respects robots.txt. So, yes. [edit for clarity, and for misspelling clarity]
- alex_c 14y ago
  Well... "sort of", then. PadMapper respects the robots.txt for GoogleBot, not for PadMapper.
- ig1 14y ago
  Presumably 3Taps violates Google's ToS to get the data from them ?
  - stfu 14y ago
    I would really love to know that kind of shady proxy bot army they have implemented in order to scrape Google on such a scale.
    - user345 14y ago
      This type of comment is common. Not only on HN but on forums in general. While I agree the web is full of lowlifes engaged in web development, many of them in porn or some other area that appeals to base instincts, I find this comment perplexing. Because it is so subjective, yet it tries to seem objective by focusing on some random criteria. Google employs a "bot army" to scrape the entire web. So what? If the comment was something like "I don't like Company X." Or even "I don't like Company X because...", it would make sense to me. But that is not how this common type of comment goes. Instead it suggests that bot=evil, i.e. any sort of automation or any sort of data collection by anyone other than [your favorite company] is "shady". That's crazy. IMO. It's what a company does with the data that matters. Anyway, I'm not keen on 3Taps because they are not provinding bulk data, only API's that require "developer keys". Why? Either you are going to democratise data, or you are just another schemer trying to find ways to collect infromation about people, in this case people using "your API". I don't want API's I want the data. I can make my own interfaces thank you.

waterlesscloud 14y ago

For that analogy to hold, PadMapper would have to be pulling data from a large number of sources. The value that Google News provides is that it aggregates a very large number of sources. PadMapper does not.

fireflash38 14y ago
Padmapper collates data from multiple sources: Craigslist, Rent.com, Apartments.com, and "Others" (not sure which those would be).
- unreal37 14y ago
  It's pretty clear PadMapper was BUILT off of Craigslist that CL's poor UI is isn't entire reason for being, and it never would have been successful without CL. It's disingenuous to ignore that by listing "rent.com" as an alternate source.
wickedchicken 14y ago
> PadMapper does not. It does.
- huhtenberg 14y ago
  But without the Craigslist data it has no value.
  - jlgreco 14y ago
    Total nonsense. I found my current place with them, without craigslist.
- waterlesscloud 14y ago
  It does not. One main source and a handful of secondary sources is not at all comparable to a true aggregator like Google News. Look, it's obvious that even PadMapper believes Craigslist is THE data source they must have. Google News could drop any given data source and not blink. PadMapper simply is not in the same position.