7 ms·
ArchiveTeam Warrior backing up Reddit
- donkarma 5y agoWhat a waste of the space of the Internet Archive. Just because something exists doesn't mean it should be backed up, I would be surprised if anyone actually needed something from Usenet for example. Things like this are going to kill the Archive eventually.
- darknavi 5y ago> Just because something exists doesn't mean it should be backed up. /r/DataHoarder disagrees. It's pretty crazy the what people will back up these days.
- donkarma 5y agoThere is a limit to storage, especially when you don't own it, and the usefulness of the content collected should be considered when you're backing up what is primarily a news and meme aggregator. They aren't backing up the Library of Babel so where does the line get drawn?
- rout39574 5y agoI think the line is drawn where nobody cares enough to archive it. How interesting would it be to trace the evolution of e.g. TheDonald from parody to party to sewer, without the retrospective editings and deletions? Yeah, there's a limit to storage, but it's growing faster than, perhaps, you warrant.
- Pmop 5y agoI'm thinking about becoming a data hoarder. One of my favorite hobbies is listening to music on YouTube and reading people's comments about how that song was important in some moment of their lives. Most of the videos are unofficial, since the official ones have comments turned off. From time to time, one of these music videos gets dmca'd and sadly the comments are all lost.
- beamatronic 5y agoI would love to lookup my old usenet posts from the early 90's
- donkarma 5y agoMy point was that most people wouldn't get much value out of it aside sentimental reasons, and you can back up your own Reddit posts if this matters to you instead of backing up the entirety of it as this is the only part that should matter and should have a smaller fingerprint. If Reddit goes down for whatever reason you could try to merge with others, too.
- simonw 5y agoIf you try telling a historian in a hundred years time that there wasn't anything of value worth saving on a site where millions of regular people had conversations about pretty much everything of relevance to life in the early 21st century they will robustly disagree with you.
- oenetan 5y agoThe groups.google.com search is working again. Why not?
- simonw 5y ago> I would be surprised if anyone actually needed something from Usenet for example Totally disagree. Usenet is the only place recording the history of a huge number of influential projects from the 80s and 90s. That history deserves to be recorded.
- serf 5y agoabsolutely disagree. Some of the most valuable and insightful anthropological artifacts are merely shop ledgers and discourse on bathroom walls -- and we've never been better equipped to document/store/search/access the entirety of the saved artifacts from the modern age -- and presumably our mastery over information technology as a domain will only improve and make things easier.
- zarzavat 5y agoArchive Team is not the same as the Internet Archive.
- nojito 5y agoWhere do you think all of this data is being pushed to exactly?
- deadbunny 5y agoTo Archive Team contributors hard drives. You should probably check out how Archive Team works, it's pretty cool.
- nojito 5y agoThe end result is the data flowing to the IA...sure they are "not affiliated" but the data is sent there which is what the GP is talking about.
- sp332 5y agoThe "Warrior" program only downloads the data temporarily, and then uploads it to a staging server run by AT members. The staging server packs the WARCs into MegaWARCs and sends them to live at the Internet Archive.
- romwell 5y ago...why? It's a pretty high bar to say that something should not be archived and it's a waste. I am not even going to state my opinion on the issue now, I am just disappointed about the level of discourse comments like this create. No justification, no logic. TL;DR is that you don't like reddit. This is not useful to anyone.
- deleted 5y ago[deleted]
- totetsu 5y agono one cared about the snap shot of Bill Clinton dancing with an intern either until later.
- _jal 5y agoThis is why individual people should never be left to make such decisions alone. They're likely to throw away things of enormous value for parochial reasons.
- wl 5y agoYou never know what will be useful to the future. To give an example, the field of papyrology is largely built around trying to construct a view of the past using scraps of texts excavated from ancient dumps.
- karlding 5y agoThe 20 Newsgroups dataset is a collection of ~20k newsgroup documents and is super popular for experimentation in text applications of various machine learning techniques. Without Usenet (and archives of Usenet) that probably wouldn't exist.
- textfiles 5y agoThank you, drive through
- mnd999 5y agoI think this requires more explanation. Is this some kind of cloud archiving? Are they using ipfs or something like that?
- timdaub 5y agoI'm also not totally familiar with what's going on but I discovered it on /r/datahoarder and I think it's because redditors are scared that content starts vanishing now that reddit files for IPO. Anyways, here's a further description: http://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior http://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior
- Spooky23 5y agoOf course it will. Reddit is probably one the largest porn sites.
- jcrawfordor 5y agoArchiveTeam usually sends all the data they collect into Internet Archive. The two are independent, unrelated organizations, but IA is pretty open about accepting hoards of this type.
- deleted 5y ago[deleted]
- smarx007 5y agoVolunteers run Docker containers or Virtualbox VMs at home so that the traffic looks residential and does not get banned. For example: https://imgur.com/a/QXrhudA https://imgur.com/a/QXrhudA Most useful content gets packaged by the ArchiveTeam and send to the Internet Archive (no affiliation between the two).
- bshipp 5y agobut it's not really necessary for Reddit. their API is fairly robust and there are numerous options for scraping the site.
- jerheinze 5y agoFor context: https://teddit.net/r/DataHoarder/comments/rhttac/with_reddit_plans_to_go_public_soon_is_it_time_to/hosyxrf/ https://teddit.net/r/DataHoarder/comments/rhttac/with_reddit...
- ajsfoux234 5y agoWhere does ArchiveTeam find all the reddit posts and comments to archive? Do they have a script automatically going through the "New" section or are they finding posts through Google or link crawling?
- Gigachad 5y agoIn general, ArchiveTeam has scripts which hit random links to see if there is any content. They have coordination servers which share info on which slugs have been checked before to avoid duplicate effort.
- giantrobot 5y agoBesides their Archive Warrior distributed crawler I imagine PushShift[0] is probably a starting point for them. [0] https://files.pushshift.io/ https://files.pushshift.io/
- ortusdux 5y agoThere are several sites that let you view deleted reddit comments. I wonder if they have complete text backups, or if they get the comments from somewhere else.
- elpocko 5y agoThe ones I know are powered by https://pushshift.io/ https://pushshift.io/
- rhaksw 5y agoThat's right. I'm the author of the one called reveddit and maintain a FAQ about how it works if anyone's interested, https://www.reveddit.com/about/faq/ https://www.reveddit.com/about/faq/
- bloqs 5y agoInused removeddit for years then it suddenly stopped working due to some sort of reddit API change, is this true for yours too?
- rhaksw 5y agoReveddit works fine with the current workings of reddit and the archive service's APIs. Removeddit's domain lapsed. Prior to that it was working, except on Firefox with Tracking Protection (see https://www.reveddit.com/about/faq/#firefox https://www.reveddit.com/about/faq/#firefox ).
- bloqs 5y agoMany thanks for your efforts!!
- pronoiac 5y agoArchiveTeam wiki on Reddit: https://wiki.archiveteam.org/index.php/Reddit https://wiki.archiveteam.org/index.php/Reddit
- jl6 5y agoA 2018-era article suggests the Internet Archive held 46PB of content. It’s probably much more now. This warrior has extracted ~880TB so far. I wouldn’t be surprised if the result occupies a material proportion of the IA’s capacity, at significant cost. Still, better than letting it all get burnt by the shareholders a few years down the line.
- deleted 5y ago[deleted]
- romwell 5y ago>Still, better than letting it all get burnt by the shareholders You wish. By the looks of it, it will be burned down by yet another "redesign" when they inevitably shut down the sane UI of https://old.reddit.com https://old.reddit.com because it's not pushing ads and "social" "features" strongly enough. (If you don't use https://old.reddit.com/ https://old.reddit.com/ instead of the "new", aka default, reddit, treat yourself to some sanity. Imagine if Reddit was more like HackerNews. Wait, you don't need to imagine, that's what the link goes to.) I really like how they are pushing so hard for video, while struggling to display more than a thousand text comments or even links. Feels like reddit is digging it own grave with these moves, and the shareholders' money with it. I really hope to revisit this comment in 10 years and say how utterly stupid and wrong I was, because I've got an incredible amount out of the communities on reddit, and poured a lot into them to. Particularly, support groups.
- alexanderskates 5y agoTheir video player is also garbage, especially on mobile. I'll have to open and close a video multiple times to get it to play, the quality will take a nosedive midway through and just stay that way for the rest of the video, and they take forever to load.
- hogFeast 5y agoThey are going to get popped by journos. Twitter got popped left and right by journos and this was despite the platform being a huge source of revenue for the media, and the owner's politics. The week they go public, the stories will start getting published about the content on there, the weak moderation (community moderation...lol), journos sending emails to companies asking whether they like their ads appearing along X or Y figure, etc. The line that Huffman has taken before on this stuff (basically live and let live) works if you are private. It will go down extremely badly after the IPO. Sorry. Reddit is already dead. (Btw, I have no idea why this is...it makes no logical sense because being a public company changes nothing. But the media seems to understand that they can print something, that thing can potentially move the stock, and then they can bounce management into doing something).
- throwawaysea 5y agoThis is so very necessary. Reddit has banned great content from many subs whose ideology they didn’t agree with. Unfortunately it isn’t even possible to know the URLs of posts from banned subreddits to look them up in the first place. Does anyone know if there are backups of banned subreddits already?
- NaturalPhallacy 5y agoAnd with "permanently suspended accounts" nothing they wrote is visible. And reddit is very liberal with their banhammers.
- fnord77 5y agoin some cases it is positively ridiculous. I got banned from /r/coronavirus for posting a scientific article suggesting breakthrough cases in fully vaccinated people was a possibility (this was fairly early, Mar 2021 maybe). The mod denounced me as an antivaxxer which I am certainly am not. Lo and behold - breakthrough cases are a real thing.
- PragmaticPulp 5y ago> I got banned from /r/coronavirus for ... Reddit doesn't moderate individual subreddits. Non-employee moderators do. Most subreddits are just an extension of whatever their moderators want to see. They're a huge shaping influence on the content and character of Reddit, but their actions are mostly invisible.
- nashashmi 5y ago.
- octopoc 5y agoIs it possible to write a program that connects to an HTTPS server and archives content, but it keeps track of the session keys and the encrypted data coming from the server, and then records all that session traffic in a file. Replaying the file will allow anyone to observe that the data truly did come from that specific server, because it's signed with the cert for that server. In other words, is it possible to get any HTTPS website to give you what is essentially a digitally signed copy of content you want to prove originated with that site? And is it true that that digital signature is easily verified to belong to the original website?
- ajsfoux234 5y agoTLSNotary tries to solve this problem: https://tlsnotary.org/ https://tlsnotary.org/ It was posted here a while ago: https://news.ycombinator.com/item?id=29090604 https://news.ycombinator.com/item?id=29090604
- paxys 5y agoThe problem is when you are trying to verify the data down the line the original certificate (up to and including the root cert or even the authority itself) will have expired, so it won’t be possible to trust it.
- rentnorove 5y agoWouldn't you want to validate it for the time at which it was served? In which case you'd just an archive of the CA roots at the time.