4 ms·

CL has also removed themselves from search engines, so PadMapper can no longer use the google cache workaround. I don't think they can go back to scraping eith

CL has also removed themselves from search engines, so PadMapper can no longer use the google cache workaround. I don't think they can go back to scraping either, but I'm not certain of that aspect of it. This may be the real reason behind the change. Perhaps CL just doesn't need it any more.

danso 14y ago
I think this is correct, though the EFF post is too vague, and doesn't quote the passage in its current state. Also, CL has not removed itself from search engines. They're using NOARCHIVE.
- dredmorbius 14y ago
  Last I checked, robots.txt excludes all crawlers from at least several major classifications, including housing listings (/hhh): User-agent: * Disallow: /cgi-bin Disallow: /cgi-secure Disallow: /forums Disallow: /search Disallow: /res/ Disallow: /post Disallow: /email.friend Disallow: /eaf Disallow: /reply Disallow: /?flagCode Disallow: /ccc Disallow: /hhh Disallow: /sss Disallow: /bbb Disallow: /ggg Disallow: /jjj Disallow: /*rss$ http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt 'jjj' is jobs, 'ggg' is gigs, 'bbb' is services, 'sss' is for sale, 'ccc' is community, 'res' is resumes.
  - danso 14y ago
    Right...but not apartment listings (aap) http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/sub/ http://newyork.craigslist.org/sub/ http://www.google.com/search?client=safari&rls=en&q=bronx+all+apartments+classifieds&ie=UTF-8&oe=UTF-8 http://www.google.com/search?client=safari&rls=en&q=...
    - dredmorbius 14y ago
      Point. I mistakenly assumed /aaa covered all of housing. Looks like the same principle applies to a number of other subcategories. So while overview indices aren't spiderable, subindices are.
defen 14y ago
Google still provides a screenshot of CL pages - I doubt it would take much work for an enterprising hacker or 3taps to hook an OCR library up to that.
- SilasX 14y ago
  But wouldn't that be insane on bandwidth?
  - rabidsnail 14y ago
    Inbound bandwidth is free.
    - SilasX 14y ago
      I'm talking about for downloading all those pictures.
      - defen 14y ago
        That's the point he was making - inbound traffic to AWS is $0/GB.
jobu 14y ago
That sucks, searching is how I double-checked dubious craigslist posts. Select a couple phrases and search google. If identical posts pop up under various other cities on craigslist, then I knew for sure it was spam/fraud.
- IanDrake 14y ago
  Just one of the many ways third parties fix broken things on Craigslist. Things Craigslist refuses to recognize.

danso 14y ago

I think this is correct, though the EFF post is too vague, and doesn't quote the passage in its current state. Also, CL has not removed itself from search engines. They're using NOARCHIVE.

dredmorbius 14y ago
Last I checked, robots.txt excludes all crawlers from at least several major classifications, including housing listings (/hhh): User-agent: * Disallow: /cgi-bin Disallow: /cgi-secure Disallow: /forums Disallow: /search Disallow: /res/ Disallow: /post Disallow: /email.friend Disallow: /eaf Disallow: /reply Disallow: /?flagCode Disallow: /ccc Disallow: /hhh Disallow: /sss Disallow: /bbb Disallow: /ggg Disallow: /jjj Disallow: /*rss$ http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt 'jjj' is jobs, 'ggg' is gigs, 'bbb' is services, 'sss' is for sale, 'ccc' is community, 'res' is resumes.
- danso 14y ago
  Right...but not apartment listings (aap) http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/sub/ http://newyork.craigslist.org/sub/ http://www.google.com/search?client=safari&rls=en&q=bronx+all+apartments+classifieds&ie=UTF-8&oe=UTF-8 http://www.google.com/search?client=safari&rls=en&q=...
  - dredmorbius 14y ago
    Point. I mistakenly assumed /aaa covered all of housing. Looks like the same principle applies to a number of other subcategories. So while overview indices aren't spiderable, subindices are.

defen 14y ago

Google still provides a screenshot of CL pages - I doubt it would take much work for an enterprising hacker or 3taps to hook an OCR library up to that.

SilasX 14y ago
But wouldn't that be insane on bandwidth?
- rabidsnail 14y ago
  Inbound bandwidth is free.
  - SilasX 14y ago
    I'm talking about for downloading all those pictures.
    - defen 14y ago
      That's the point he was making - inbound traffic to AWS is $0/GB.

jobu 14y ago

That sucks, searching is how I double-checked dubious craigslist posts. Select a couple phrases and search google. If identical posts pop up under various other cities on craigslist, then I knew for sure it was spam/fraud.

IanDrake 14y ago
Just one of the many ways third parties fix broken things on Craigslist. Things Craigslist refuses to recognize.