4 ms·
Ask HN: Is anyone working on a Reddit archive?
Although I disagree with most of the criticisms leveraged towards the platform, I still readily depend on it for a lot of day-to-day resources and general questions, many times less technically oriented.
While I wouldn't mind losing the system there in the long run, I think the state of posts before this upheaval was very valuable as a reference.
Like the title says - has anyone done anything like a Reddit "takeout" yet?
- minimaxir 3y agoPushshift was the Reddit archive but apparently recent agreements with Reddit may have changed that. Anyone else creating a Reddit archive will likely get a C&D.
- speedgoose 3y agoPushift is supposed to come back with a new agreement with Reddit. No precise date was given though, and it’s likely not going to be accessible to everyone but only moderators, researchers, and perhaps a few businesses.
- uniqueuid 3y agoYes. There is the pushift dataset covering posts and comments through 2022 [1]. And the ArchiveTeam has begun crawling reddit as well some time ago [2] [1] https://old.reddit.com/r/pushshift/comments/10bwxke/updated_torrent_of_dump_files_through_december/ https://old.reddit.com/r/pushshift/comments/10bwxke/updated_... [2] https://news.ycombinator.com/item?id=36254172 https://news.ycombinator.com/item?id=36254172
- cookiengineer 3y agoI was focussing mostly on cyber security related subreddits because the vulnerability and exploit discussions were of great value to me. I built a little scraper in golang that stores the JSON data (instead of the HTML which the archive warrior stores) to save hdd storage. [1] The problem with reddit's API is that it only shows 1000 entries over 10 pages in every api. Meaning hot/top/new, and search results are limited. If you have more links related to the keyword, you won't discover more. So you need a very specific keyword list to be able to discover more posts, and search each subreddit for each entry in the keyword list. [1] https://github.com/cookiengineer/reddit-archivar https://github.com/cookiengineer/reddit-archivar
- simonblack 3y agoIt's next on my list after I finish the MySpace archive. Seriously, why would anybody do this? Reddit has such a high noise-to-signal ratio that it would be a waste of resources. There may be value in keeping an archive of some individual subreddits, but not the main bulk of Reddit itself.
- eimrine 3y agoProbably the most of the noise is inside obvious places like r/fun or r/music. And most of the signal is in some micro subreddits with few hundreds of subscribers.