3 ms·
I'm teaching a Unix/data course this quarter, and inspired by the OP (who I believe did an analysis/visualization prior to this) and other BuzzFeed-title analys
by danso 12y ago
I'm teaching a Unix/data course this quarter, and inspired by the OP (who I believe did an analysis/visualization prior to this) and other BuzzFeed-title analysts, I put together an exercise in doing a frequency distribution of BuzzFeed titles (really, an exercise in basic HTML parsing and regexes) http://www.compciv.org/homework/assignments/buzzfeed-listicle-title-parser/ http://www.compciv.org/homework/assignments/buzzfeed-listicl...
I gathered the links through their archive pages, which go back to 2006...I didn't run the exercise yet myself, but at one point did a quick count of total titles since 2006 and counted more than 200,000. That didn't seem right (and maybe my selector was off)...but the other day, I tried it with just the 2014 year and came up with more than 60,000 articles.
...that sounded clearly off (the OP said he scraped only ~60,000 distinct articles)...but when I visited one of the archive pages to eyeball it...yeah, 60,000 articles in a single year looks about right: http://www.buzzfeed.com/archive/2014/11/18 http://www.buzzfeed.com/archive/2014/11/18
It's even more astonishing when you go back to the 2006 pages and see that there were days/regular weekends in which BuzzFeed published no content at all.
- minimaxir 12y agoHuh, I was not aware of the archive at all. However, looking at the archive, it appears that there are a lot of community, user-added posts so I'm unsure of the reliability of those posts.
- danso 12y agoYeah it took me a bit of time to find the archive. Unlike you, I have a little more faith in BuzzFeed's SEO expertise and figured they would have to have some kind of sitemap ;) Don't know about the community posts or how they're structured with the main content...it might be possible on a second pass to filter out non-Buzzfeed-staff posts based on the URL (which I assume contains the username of the author). (Also, I don't know if they changed this since you attempted your last scrape via pagination...but now if you paginate too far, you will be rewarded with a Boyz II Men video)
- aw3c2 12y agohttp://www.buzzfeed.com/sitemap.xml http://www.buzzfeed.com/sitemap.xml