5 ms·
LinkedIn sitemap.xml
- mooism2 14y agoWhat about it?
- wslh 14y agoAn easier scraping?
- mylittlepony 14y agoBig news.
- pjbrunet 14y agoIt's against their TOS.
- adanto6840 14y agoThe Clickwrap based TOS you mean? ;-) It's frustrating, at least for me, that the legality here is still so gray (at least IMO it is). Much of their content is likely in the public domain (facts / basic non-creative information) although there is definitely plenty that is not; the lack of 'black-and-white' is what frustrates me... Clickwrap - http://en.wikipedia.org/wiki/Clickwrap http://en.wikipedia.org/wiki/Clickwrap
- binarysolo 14y agoIs this actually for all 150M users? I guess they really want crawlers to go the shortURLs easier...
- icebraining 14y agoNot even close to 150M: $ curl http://www.linkedin.com/sitemap.xml | grep -c '<url>' 47785 I wonder why were these selected. Probably the most searched.
- arcatek 14y agouser@/tmp > grep lastmod sitemap.xml | cut -d - -f 1 | uniq -c 47785 <lastmod>2006 They were probably sitemaping all their users when they started the service, then they have decided to stop shortly after.
- binarysolo 14y agoYeah I didn't bother downloading the file -- but looks like they bailed on it after awhile.
- sleepyhead 14y agoThey might want to consider using sitemap index files: http://www.sitemaps.org/protocol.html#index http://www.sitemaps.org/protocol.html#index Edit: "If you want to list more than 50,000 URLs, you must create multiple Sitemap files." According to 'icebraining' they have 47785 urls.
- tszming 14y agoThey already have it: http://www.linkedin.com/robots.txt http://www.linkedin.com/robots.txt
- sleepyhead 14y agoAh, ok, strange that they have /sitemap.xml in addition to that then. I would assume any robot that would understand sitemap also understands that it can be specified in robots.txt
- JackWebbHeller 14y agoIt would be helpful if an editor could provide some kind of warning of this in the title. My Netbook is still trying to open this over dialup, and xPUD has ground to a halt.
- jrajav 14y agoMaybe this comment will be enough? Size of file: 8.2mb Transfer time over ~10Mbit broadband: 18.24s Firefox onload (from GET): 49.21s Chrome onload: N/A, still hung at 100% CPU after 15 minutes (Chrome attempts to style raw XML, whereas Firefox just makes a foldable tree like IE) ... I recommend: curl -s http://www.linkedin.com/sitemap.xml | less
- gaving 14y agoI agree, it just beach-balled Safari here.
- navs 14y agoI've noticed it doesn't take much to beach-ball Safari, at least for me. OS 10.8, 2008 model MBP, Safari 6.x Canary kept on chugging at the expense of the whole OS hanging.
- message 14y agoFound in robots.txt link to Sitemap: http://partner.linkedin.com/sitemaps/smindex.xml.gz http://partner.linkedin.com/sitemaps/smindex.xml.gz
- WatchDog 14y agoHere is one for google plus https://ssl.gstatic.com/s2/sitemaps/profiles-sitemap.xml https://ssl.gstatic.com/s2/sitemaps/profiles-sitemap.xml
- mokash 14y agoWow, this is a sitemap of .gz files containing the links to the actual profiles. I did a wget to download all the .gz files. It's still going 10 minutes later.
- sahaskatta 14y agoDo you have a script to fetch them all?
- sheraz 14y agoSomeone is way ahead of us on this. Claims 35MM+ profiles http://thepiratebay.se/torrent/7031839/35_Million_Google_Profiles_Database_(SQL) http://thepiratebay.se/torrent/7031839/35_Million_Google_Pro...
- deleted 14y ago[deleted]
- mokash 14y agoManaged to get a list of all only the URLs. https://dl.dropbox.com/u/8433360/linkedinusersurls.txt https://dl.dropbox.com/u/8433360/linkedinusersurls.txt
- bennny 14y agoThis crashes mobile safari!
- deleted 14y ago[deleted]
- makmanalp 14y agoI've actually been working on my own site's sitemap, and I've figured out a bunch of things: You can use a sitemap index [0] to split a sitemap into multiple ones. This is beneficial because you can change part of it and not have to have google re-crawl everything, since index updates happen right after a whole sitemap has been crawled. As a result, changes you make to a smaller sitemap get indexed way faster. I've switched to grouping my bulk pages into sitemaps alphabetically, and then putting important ones (front page, about) into different sitemaps, and specific landing pages into yet another different one. [0] http://en.wikipedia.org/wiki/Sitemap_index http://en.wikipedia.org/wiki/Sitemap_index
- krob 14y agoThis document just locked up my browser, i7 w/ 8gb ram using chrome.
- scottmcleod 14y agoInteresting, thanks for the share.
- pjbrunet 14y agoI'm not In the sitemap :( In case you were looking for me: http://linkedin.com/in/pjbrunet http://linkedin.com/in/pjbrunet :)