4 ms·
ArchiveTeam Warrior
- xnx 2y agoVery easy to set up. Glad to contribute back in some small way now that I my Internet plan is "unlimited". Was trying to figure out if I could run ArchiveTeam Warrior scripts on an unused Android phone, but that's not directly supported and above my skillset.
- NortySpock 2y agoAgree, I found it trivial to run in a docker-composed container on an x86 box. I wish I had spare time to try to figure out how to get it set up inside QEMU on a Raspberry Pi... Seems like publishing such a compose file would unlock it for even more people.
- idunnoman1222 2y ago[dead]
- jeroenhd 2y agoNo need to mess with QEMU. You may need to build the container for ARM, though: https://github.com/ArchiveTeam/warrior-dockerfile/tree/master https://github.com/ArchiveTeam/warrior-dockerfile/tree/maste... There's a specific "wget-lua.raspberry" file in the repo, so Raspberry Pi seems to be supported almost natively.
- tech234a 2y agoI believe that file isn't actually supported because of potential data consistency issues in how wget runs on ARM devices.
- TheTechRobo 2y agoUnfortunately the Raspberry Pi version isn't supported anymore because their version of Wget isn't well-tested on ARM. I'm surprised it's still there in the repo. You can get Docker to emulate x86, though.
- simlevesque 2y agoIs it possible to work on multiple projects at the same time ?
- luuurker 2y agoNot with the Warrior, as far as I'm aware, but you can do it with docker. It's usually the repos with -grab at the end: https://github.com/orgs/ArchiveTeam/repositories https://github.com/orgs/ArchiveTeam/repositories
- diggan 2y agoYou can run multiple warriors, select different projects for each one you run (or run both with "auto"). I think it's advisable to only run one warrior per IP though, if you're doing many warriors for same project, as otherwise it's a lot easier to get rate-limited by whatever website/service you're helping to archive. https://github.com/ArchiveTeam/warrior-dockerfile https://github.com/ArchiveTeam/warrior-dockerfile makes it pretty easy to setup.
- hnbear 2y agoI created a VM and then just duped the VM. Definitely overkill for this purpose, but nicely segregates them out. Right now I run three instances. Pretty low resource utilization and they're totally segregated into their own instances so just boot them and they run, shut them down and they stop. Given they're running arbitrary external commands I wanted them kept on their own machines as much as possible.
- myself248 2y agoNot in a single warrior, but you can run multiple warriors. I do this; I have one running Telegrab and one running URLTeam. Once in a while, a site will tolerate a large number of connections, and since the Warrior VM only supports a concurrency of 6, it can make sense to run multiple warriors on the same project. But this is almost always a bad idea, and many sites will 429 you with a concurrency of anything more than 2 or 3, always check the project-specific IRC channel for concurrency recommendations.
- toomuchtodo 2y agoPlease consider running a warrior appliance or docker container to contribute to digital preservation efforts.
- dang 2y agoRelated: ArchiveTeam Warrior: archiving as much of imgur as possible - https://news.ycombinator.com/item?id=35983510 https://news.ycombinator.com/item?id=35983510 - May 2023 (2 comments) Help preserve the internet with Archiveteam's warrior - https://news.ycombinator.com/item?id=30524842 https://news.ycombinator.com/item?id=30524842 - March 2022 (51 comments) ArchiveTeam Warrior backing up Reddit - https://news.ycombinator.com/item?id=29584622 https://news.ycombinator.com/item?id=29584622 - Dec 2021 (71 comments)
- rglullis 2y agoThese projects are very much needed, but I also believe that we need to go beyond simple "archiving" and have a way to make this data actually available to the public. For example: I don't think that Reddit ever felt threatened by the fact that a bunch of people are pulling their data and creating a backup that can not be easily accessible. But I think that Reddit would be very much afraid of a distributed network of nodes running partial copies of Reddit and making it available for local-first clients and/or Lemmy mirrors.
- tech234a 2y agoMost projects output WARC files which are batched, uploaded to archive.org, and go into the Wayback Machine: https://web.archive.org/ https://web.archive.org/ (Note that Archive Team is separate from Internet Archive)
- lopkeny12ko 2y agoThat still doesn't solve the discoverability problem. The Wayback Machine is only useful if you happen to know the URL where a piece of content used to live.
- rglullis 2y agoThat, and the fact that the data is "frozen". There is no way to add a comment to a Reddit thread that has been archived without going to Reddit first. I'd like to have a two-way mirror.
- userbinator 2y agoIsn't read-only one of the main points of being "archived"?
- rglullis 2y agoYeah, but I am saying that I don't want it merely "archived". I want it "copied and available outside of Reddit's control".
- tech234a 2y agoSee also the page on the Archive Team wiki, with more detailed instructions: https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior
- myself248 2y agoThis is a much better link and really should be what the headline points to.
- throwaway888abc 2y agoI can run docker for this. So looked up the repo and sweet surprise - it's all there. https://github.com/ArchiveTeam/warrior-dockerfile/tree/master https://github.com/ArchiveTeam/warrior-dockerfile/tree/maste... https://github.com/ArchiveTeam/warrior-dockerfile/blob/master/docker-compose.yml https://github.com/ArchiveTeam/warrior-dockerfile/blob/maste...
- stavros 2y agoA few months ago I designed a system that will allow people to contribute their disk space towards a service like archive.org. Basically, you'd say "I want to host 100 GB of archive.org content" and my system would talk to archive.org, figure out which content is currently the most rare, and push it to you. It also comes with a retrieval function, where you can say "I want to get X content from this network" and it will find it for you. It's a fairly thin wrapper over torrents, but it something that doesn't currently exist. Unfortunately, it wasn't met with much interest when I contacted some archivists.
- agilob 2y agolibgen has a tracker like this for books https://phillm.net/libgen-seeds-needed.php https://phillm.net/libgen-seeds-needed.php looks like it's currently broken :(
- immibis 2y agoArchive Team had a defunct project called INTERNETARCHIVE.BAK (they use slightly whimsical project names) which is basically this idea. Note that by using this tool, you're breaking the law (copyright infringement).
- tvshtr 2y agoWasn't IPFS supposed to fill that role?
- stavros 2y agoNo, IPFS only lets you retrieve and store a file you want, not a bunch of files someone upstream wants you to, and not with a specific space target.
- Kye 2y agoThat sounds like a BOINC-like file system.
- theblazehen 2y agoRunning an IPFS Cluster follower node gets part of the way there. I'm intending to do this with the data from Anna's Archive. See some existing projects: https://collab.ipfscluster.io/ https://collab.ipfscluster.io/
- Yeri 2y agoWould love to contribute using my Raspberry Pi, but it not supporting ARM is a bit of a pain (and no easy way around it; I want to download the container, run it and done): https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior#Can_I_run_the_Warrior_on_ARM_or_some_other_unusual_architecture https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior#C...?
- theblazehen 2y agoThere are some consistency issues when wget-at is compiled for ARM. You can run in under QEMU user emulation however, and I do that on a couple hosts
- NortySpock 2y agoDo you have a write up or link to a page that describes how to run it under QEMU emulation? I've been wanting to try it / publish a docker-compose file that does it in order to make it easy for Raspberry Pi users to spin up a Warrior...
- theblazehen 2y agoI just used https://github.com/dbhi/qus https://github.com/dbhi/qus to enable it for all docker containers on the system (alternatively you can do the binfmt config manually, eg https://wiki.debian.org/QemuUserEmulation https://wiki.debian.org/QemuUserEmulation) However to have it more tightly integrated, you'd want an arm image at /, mount the original warrior image at /warrior, then you could do something like `qemu-user-amd64 /warrior/bin/chroot /warrior/entrypoint` - Updating all paths as relevant
- DandyDev 2y agoI don’t understand the advantage of people downloading sites and then uploading them to the archive setup by archive team. What is the advantage over archive team directly downloading sites into their archive?
- extra88 2y agoA) It requires fewer resources from the archive team. B) Probably more importantly, it distributes where the requests are coming from so they're less likely to be throttled or blocked.
- mattl 2y agoArchive Team isn’t Internet Archive. They need somewhere to download things before pushing them to archive.org
- nh2 2y agoHow does this project prevent that I as an attacker insert random garbage or rewrite the downloaded pages to be wrong in specific ways? For HTTP this seems impossible. For HTTPS, who does the TLS termination? It could be safe if the warrior was just at TCP proxy and Archive was the TLS client, but https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior doesn't seem to explain that clearly.
- kalleboo 2y ago> How does this project prevent that I as an attacker insert random garbage or rewrite the downloaded pages to be wrong in specific ways? It doesn't, but in practice, it's not a problem. The goal is to with minimum turnaround rescue as much data as possible before a site goes down. Getting 1% garbage is more preferable than losing 30% to inefficiency in archiving because you're paranoid. > For HTTPS, who does the TLS termination? The Warrior literally just runs wget and saves to warc files that are uploaded to a tracker server.
- Alifatisk 2y agoHow will running Warrior in the background affect the pc? Will it degrade the disks overtime? I remember running the Siacoin app degraded my disks a bit. Or was it IPFS?
- Technetium 2y agoI decided to read the README and saw the following notice: Version 4 is now available at https://warriorhq.archiveteam.org/downloads/warrior4/ https://warriorhq.archiveteam.org/downloads/warrior4/ Which then contains a README with a link to the source for the new version: https://github.com/ArchiveTeam/warrior4-vm https://github.com/ArchiveTeam/warrior4-vm This new version is also mentioned on the homepage of the Wiki: An updated Warrior virtual appliance (v3.2, v4.0) is now available with better support for newer projects that utilize wget-at. ArchiveTeam should update the appliance download URL.