3 ms·
Huh, I was not aware of the archive at all. However, looking at the archive, it appears that there are a lot of community, user-added posts so I'm unsure of the
by minimaxir 12y ago
Huh, I was not aware of the archive at all. However, looking at the archive, it appears that there are a lot of community, user-added posts so I'm unsure of the reliability of those posts.
- danso 12y agoYeah it took me a bit of time to find the archive. Unlike you, I have a little more faith in BuzzFeed's SEO expertise and figured they would have to have some kind of sitemap ;) Don't know about the community posts or how they're structured with the main content...it might be possible on a second pass to filter out non-Buzzfeed-staff posts based on the URL (which I assume contains the username of the author). (Also, I don't know if they changed this since you attempted your last scrape via pagination...but now if you paginate too far, you will be rewarded with a Boyz II Men video)
- aw3c2 12y agohttp://www.buzzfeed.com/sitemap.xml http://www.buzzfeed.com/sitemap.xml