4 ms·
I spent 3 months scraping 6 million pages from an extremely old political web forum (I think their backend was written in perl and hasn't been updated since 200
by fibers 9y ago
I spent 3 months scraping 6 million pages from an extremely old political web forum (I think their backend was written in perl and hasn't been updated since 2001 and the response times for these pages were extremely high) in an effort to learn NLP stuff with a working database. Its cool because you get to see how old Boomers digested huge events during 9/11 up until now and the slow decline of web forums as a medium. The pain is trying to parse the actual pages since the dev never learned how to liberally use div tags like we do now.
- ioddly 9y agoI once scraped a large forum off Yuku, the HTML was easily the most bizarre I've ever seen (i.e. multiple different HTML formats for posts on the same page for no apparent reason).
- mistermann 9y agoHave you hosted it anywhere, that might be an interesting read.
- fibers 9y ago1. no i haven't since it will most likely result in the site owner coercing me since its his content 2. i can def. make a blog about it which is the point of this
- mistermann 9y agoIf you have any interesting observations from it I think a lot of people would be interested in that.