15 ms·
ArchiveTeam has saved over 11.2B Reddit links
- chinaman425 3y ago[dead]
- guraf 3y agoI have read the linked post and they seem to be only saving links, possibly titles, but not comments or text posts? Reddit's value is in the discussions (the links are usually shared on all social platforms, so that's the only differentiator). So what value is there in a collection of links going back years (meaning most of them are likely already broken)?
- gammarays_ 3y agoThe archive includes all information in the webpage including text posts, comments, images and is updated to archive.org.
- doctoboggan 3y agoThe OP said this: > By Reddit links I mean posts/comments/images, I should’ve been a bit clearer.
- arthurcolle 3y agoYou are misinterpreting across multiple categories
- deleted 3y ago[deleted]
- nXqd 3y agoThe link here is everything in that link including post, comments, likes ...
- alboaie 3y agoFrom discussions it seems that the problem is that many subreddits will be going private to protest the recent Reddit API costs changes. Some will not come back unless the change is reverted. If the change is never reverted, they will be gone forever and this project is trying to save old posts so they can still be seen even though the subreddits are private. Not sure how usefull it will be... but somehow interesting as an example of reaction of policies changes of centralised social networks
- reaperman 3y agohttps://www.reddit.com/r/ltsc/ https://www.reddit.com/r/ltsc/ Did that quite awhile ago, I forget in response to what. I lost a lot of really great info on how to use Windows 10 LTSC / Server / Enterprise / IoT when r/LTSC went permanently private.
- OJFord 3y agoCouldn't you (request to) join if that's interesting/valuable to you? Or it went private as a sort of pseudo-deletion, and doesn't really exist any more?
- reaperman 3y agoI’ve tried that with multiple alts and haven’t been approved. Again, I forget the story. It was in protest to some Reddit policy.
- pjc50 3y agoReddit were threatening to de-private the subs and replace the mods. When they can do, but will likely kill the community anyway for small communities. Perhaps big ones are anonymous enough for that to work.
- echelon 3y ago> Reddit were threatening to de-private the subs and replace the mods. Is there a link to this? That's crazy!
- agilob 3y agoLive leaderboard of archived links https://tracker.archiveteam.org/reddit/ https://tracker.archiveteam.org/reddit/ I've been contributing to this project for ~2 years now and I've never seen it running so fast
- mozman 3y agoHow can I help?
- arcanemachiner 3y agoRun this software in a VM: https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior It appears to make you into a sort of node that backups up a share of reddit and sends a copy to a group that's archiving all of reddit.
- fsflover 3y agoI'm using Qubes OS. I wonder why I can't simply run some application and instead have to install and manage Docker for that. I care about my security myself.
- FastEatSlow 3y agoThey mention this in their docs: https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior#Why_a_virtual_machine/container_in_the_first_place https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior#W...?
- codetrotter 3y agoToo bad they don’t support running on ARM. > Can I run the Warrior on ARM or some other unusual architecture? > No, currently we do not allow ARM (used on Raspberry Pi and M1 Macs) or other non-x86 architectures. This is because we have previously discovered questionable practices in the Wget archive-creating components and are not confident it runs under different endiannesses etc. If you still want to run it apparently Docker can emulate x86_64.
- tus666 3y ago[flagged]
- Operyl 3y agoThere are plenty of communities on Reddit that provide truly useful information, tons of niches. We should be archiving this supposed "rubbish."
- black_puppydog 3y agoFunnily, my main beef with reddit has always been that it is "yet another data silo" and now that they seem hell-bent on proving my point, this project might actually change that to an extent. Of course, the move will still kill the platform or at least gut it of its actual value (the many niche communities built on it) but at least the data will be free. :)
- poutrathor 3y agohow does compliance with RGPD will be hold by Archive Teams ? do they remove all personal information while scraping ?
- londons_explore 3y agoGoogle used to keep a mirror of all reddit posts and comments as a demo for their cloud bigquery product: SELECT * FROM fh-bigquery.reddit.subreddits Unfortunately they stopped updating it in 2016.
- IanCal 3y agoComments data is available until the end of 2019 in fh-bigquery.reddit_comments There are torrents available it seems for comments that cover later times than that.
- rovr138 3y agoThere's archived data until March, https://archive.org/details/pushshift-reddit-2023-03 https://archive.org/details/pushshift-reddit-2023-03 The rest of the data can be found on AcademicTorrents.
- rkwasny 3y agoHow can I download the data they archived? Asking for a friendly AGI :)
- iruoy 3y agohttps://archive.org/details/archiveteam_reddit https://archive.org/details/archiveteam_reddit
- malermeister 3y agoCould one import all of this into, say, a Lemmy instance to kickstart a reddit alternative?
- PythagoRascal 3y agoHaven't tried it, but this comment on /r/DataHoarder mentioned these two repos: https://github.com/rileynull/RedditLemmyImporter https://github.com/rileynull/RedditLemmyImporter https://github.com/LemmyNet/lemmy https://github.com/LemmyNet/lemmy
- jannes 3y agoIt seems that the ArchiveTeam servers are overloaded. I constantly get errors like these: @ERROR: max connections (-1) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1817) [sender=3.2.3]
- capableweb 3y agoThis is common and just means that the rsync server your warrior tried to upload to was too busy. It'll retry and try another upload host if you leave it to do its thing.
- lucb1e 3y agoGot my threads stuck on that as well. Increased concurrency now to the maximum of 6 so that those 60 seconds delays don't choke it completely, or at least less. Working reasonably so far, 5 threads waiting for the upload to complete, 1 still going. (Can't imagine that 1 thread continuously working will remain unbanned 24/7 anyway.)
- jannes 3y agoIt also happens every time it retries, so it spends most of the time doing nothing. Retrying after 60 seconds... Is that really normal? The downloads from reddit work fine, but if the upload doesn't work then I don't see the point of running this.
- toomuchtodo 3y agoPlease allow it time to catch up. It always does.
- capableweb 3y agoYes, it is really normal when lots of people try to upload at the same time. Bandwidth is limited, so when lots of people start to run the warrior, the servers need some space to do their thing. Also, IA has limited bandwidth, so sometimes that's the bottleneck too. If you give it time, it'll work eventually. Up the concurrency to max, so you can have more items in the upload state, as long as you don't start hitting rate limits from reddit, it'll be fine.
- Rhaomi 3y agoI really hope they're defaulting to Old Reddit, because I seem to recall Archive.org choking on the redesign and not actually showing anything readable for archived pages. (Also, is this including Reddit-hosted images/video?)
- deleted 3y ago[deleted]
- amelius 3y agoWe need someone like SciHub to say the ownership of those Reddit posts does not belong to Reddit, and simply fork the entire thing.
- capableweb 3y agoThe reddit user agreement used to say something like "You retain the rights to your copyrighted content or information that you submit to reddit ('user content') except as described below." so reddit inc never really owned those comments in the first place, whatever "ownership of comments on reddit" really means.
- zX41ZdbW 3y agoHere is an example of analyzing 20+ billion Reddit comments in ClickHouse: https://clickhouse.com/docs/en/getting-started/example-datasets/reddit-comments https://clickhouse.com/docs/en/getting-started/example-datas...
- doommius 3y agoI setup the agent a few days ago just for fun, however it seems to have stalled/not getting new jobs.
- capableweb 3y agoAre you running the watchtower (for automatic updates) as well? Otherwise, a restart should update it.
- social_ism 3y agoFirst some facts and then some news. The first rule of social networks is you can't touch them or they will die. Proof: Aol MySpace Twitter Facebook (Zuckerberg won't touch it now) Reddit This is obvious except to billionaires. Now for the news: the only content you see on social networks is designed to reinforce your supposedly persistent self and sell ads. There is no other purpose for social media. All social media content on the internet since the internet began will be deleted, lost and forgotten as there will be no profit motive to do otherwise.
- throwawaaarrgh 3y agoWhich is fine. These people are digital hoarders, desperate to save threads of puns on pictures of cats. None of it matters. You're not going to go back and look at all the pictures and threads you've posted. Life goes on.
- social_ism 3y agoThe US congress would never bother regulating "icanhazcheezburger.com" These things and by things I mean the dumpsterfication of social media content or internet content because it has no value. Those same things don't "influence elections" and traumatized children and supposedly destroy society. Social media is a fancy dumpster fire and that's all it will ever be. It burns out and starts anew from some other source.
- 8organicbits 3y agoI disagree, there's tons of valuable content on Reddit that people regularly reference years later. Tech support, product reviews, and similar. You'll often see people on HN recommend searching with `site:reddit.com` based on the valuable information it holds and the degradation of search result quality. Saving content from forums is the same idea. Reddit is just a collection of forums.
- social_ism 3y agoI agree with the hoarding bit. I'm not an fan of that either. But I'm much less of a fan of the self importance of tech pretending this "data" has any more use than temporary ad surveillance until they delete it or resell it.
- armchairhacker 3y agoI plan to stop using Reddit for social media. But I also add “reddit” to a lot of my searches, using Reddit posts for “Buy it for Life” products and obscure knowledge among other things, and I don’t want that to do away. So I’m glad ArchiveTeam managed to archive almost all of them. I also use Reddit for tech updates and discussions from subreddits like r/rust and r/ProgrammingLanguages, but most of these already have alternative sites (e.g. discourse, Discord, SE), and I’m more hopeful most of the new posts migrate to one of these than other subreddits or Reddit migrating in general.
- lvncelot 3y ago> but most of these already have alternative sites (e.g. discourse, Discord, SE) I've seen Discord used as a platform for project-based discussion, and I just can't get into it. It just seems so fundamentally not built for long-term threads and the aggregation of information (like a wiki, for instance), and it kind of frustrates me to see some projects that are hell-bent on keeping all related discussion on Discord. Granted, Reddit is not the greatest place for what I'm talking about either, since it also prioritizes newer self-posts/links, but I feel the clunkyness is even worse in Discord.
- A71C103D115 3y agoSame, I really don't like Discord. It probably doesn't help that I only use it when I have to but it makes me feel tired trying to find specific information even in smaller Discord groups, and even when I know for sure it's there.
- social_ism 3y ago[flagged]
- mewmew07 3y agoare TOS terms law though?
- capableweb 3y agoTerms of Service applies to users of the service. You can run this without being a user of the service. And worse case scenario, they can ban you for breaking the Terms of Service, it's not law, but dictates how you can use the service. If you're not actually breaking any laws, you don't have to worry about lawsuits.
- smarx007 3y agohttps://techcrunch.com/2022/04/18/web-scraping-legal-court/ https://techcrunch.com/2022/04/18/web-scraping-legal-court/ https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- mirages 3y agoIt's me or the data they get is not in a searchable format nor indexed ?
- capableweb 3y agoUsually it goes something like this: - Grab the data in a raw format - Upload to Internet Archive - Figure out how to extract structured data from raw dump - Upload structured data to IA
- ravenstine 3y agoAlthough I'm not exactly opposed to the archiving, I personally think this is a waste of bytes. Reddit is not like a regular web forum. Posts/threads are designed to be short-lived and quickly forgotten. Because threads can't be bumped, many communities are a dumping ground of rehashed ideas, memes, sanctimony, and "im new here how do i get started". Many people like(d) Reddit because one can easily sign up, post some stuff anonymously, and not have it effectively hang around in the high consciousness of the internet. Although there are niche subreddits with some good information in them, this doesn't mean that the vast-vast majority of Reddit posts reflect that quality. I can't say there's anything I've ever posted on Reddit that I've wanted to continue existing in perpetuity, and there's nothing I've ever read on there that I've wanted or expected to last forever. IMO, the archive team should do what it's doing but keep the archive and stay out of the issue to give negative sentiment towards Reddit enough of a chance to create some actual change. I don't have much confidence said change will take place, but I don't think what the archive team has done here will do anything but subvert the energy to fight back against Reddit. Only when Reddit has been defeated should they expose their archive of Reddit posts. Then again, the money needed to store Reddit links would be better spent on the archive team's legal budget right now.
- capableweb 3y ago> Posts/threads are designed to be short-lived and quickly forgotten No matter what they are designed for, the fact is that links around the net links to them, some of them 10 days old, others 10 years old. All those links might stop working, which sucks, no matter what the content itself is.
- ravenstine 3y agoThat's fair, but I don't think it sucks, personally. Just because it exists doesn't mean it necessarily has the universal value to be archived. Others will disagree, and perhaps most will, but the way I see it is that there's better things to spend time and effort on than backing up Reddit. As an alternative, I think it would be better for there to be good tools for individual Reddit communities to back up their own content, which would also allow them the control to archive what's valuable and leave behind the flotsam. Perhaps such a tool would even support migrating to other forum systems. If no one would be willing to use such a tool, then that would essentially prove my point that most Reddit content isn't highly worthy of being archived. Like I tried to suggest in my first sentence in my original comment, I don't necessarily think there's no argument for or no value at all in archiving Reddit. It just doesn't seem like the best choice, especially for an organization like the Archive Team who could be spending that time and effort to fight for their own existence, even if it's only costing them $20 in total. The effort would be better left to communities and individuals to decide what should be archived and how to host it.
- lopkeny12ko 3y agoLooks like someone configured their server connection limit incorrectly ;) @ERROR: max connections (-1) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1817) [sender=3.2.3]
- capableweb 3y agoSeemingly, Internet Archive is overloaded with upload requests from Archive Team. That error is hinting that the upload slots are all currently used.
- toomuchtodo 3y agoInternet Archive Wayback outbound archiving slots are distinct from ArchiveTeam. There are per host concurrent limits at Wayback to be polite.
- journey_how 3y ago[dead]
- journey_how 3y ago[dead]