3 ms·
I think this is correct, though the EFF post is too vague, and doesn't quote the passage in its current state. Also, CL has not removed itself from search engi
by danso 14y ago
I think this is correct, though the EFF post is too vague, and doesn't quote the passage in its current state.
Also, CL has not removed itself from search engines. They're using NOARCHIVE.
- dredmorbius 14y agoLast I checked, robots.txt excludes all crawlers from at least several major classifications, including housing listings (/hhh): User-agent: * Disallow: /cgi-bin Disallow: /cgi-secure Disallow: /forums Disallow: /search Disallow: /res/ Disallow: /post Disallow: /email.friend Disallow: /eaf Disallow: /reply Disallow: /?flagCode Disallow: /ccc Disallow: /hhh Disallow: /sss Disallow: /bbb Disallow: /ggg Disallow: /jjj Disallow: /*rss$ http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt 'jjj' is jobs, 'ggg' is gigs, 'bbb' is services, 'sss' is for sale, 'ccc' is community, 'res' is resumes.
- danso 14y agoRight...but not apartment listings (aap) http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/sub/ http://newyork.craigslist.org/sub/ http://www.google.com/search?client=safari&rls=en&q=bronx+all+apartments+classifieds&ie=UTF-8&oe=UTF-8 http://www.google.com/search?client=safari&rls=en&q=...
- dredmorbius 14y agoPoint. I mistakenly assumed /aaa covered all of housing. Looks like the same principle applies to a number of other subcategories. So while overview indices aren't spiderable, subindices are.