10 ms·

ArchiveBox is evolving: the future of self-hosted internet archives

669 points by nikisweeting 2y ago

We've been pushing really hard over the last 6mo to develop this release. I'd love to hear feedback from people who've worked on big plugin systems in the past, or anyone who's tried our betas!

wongarsu 2y ago
Does this mean it's now possible to write plugins that dismiss cookie popups, solve captchas, scroll web pages etc.?
- nikisweeting 2y ago
  I have a private plugin with puppeteer support for stuff like this, currently charging clients money to use it to fund the open source development. The clients are people who are already legally allowed to evade CAPTCHAS (e.g. governments, NGOs doing research, lawyers collecting evidence, etc.) Unfortunatley I cant open source the CAPTCHA solving stuff myself, because it opens me up to liability, but if someone wants to contribute a plugin to the ecosystem I cant stop them ;).
  - 0x1ch 2y ago
    Legally allowed to evade CAPTCHAs? LOL. What world do we live in where evading a captcha is an illegal offense?
    - nikisweeting 2y ago
      It doesn't matter whether or not it's actually legal, what matters is that the big platforms will sue you for trying, so you need a big bankroll to stand your ground. At the very least they can bar you from accessing their sites as you're violating ToS that you accept upon signup.
  - pbronez 2y ago
    That’s a pretty savvy open-core strategy: paywall features that create risk of legal liability so they are only used when there’s a paying customer/operator to assume that risk.
- xiconfjs 2y ago
  You mean ArchiveBox still doesn’t deal with cookie popups? If so, it’s quasi not useful for EU based web sites.
  - nikisweeting 2y ago
    It does, you just have to set up a chrome profile that has an extension to hide cookie popups, or use a profile where you've already accepted/closed them and have a session. You can archive with any chrome profile with arbitrary extensions enabled, so you can use uBlock, I still Don't care about cookies, Ghostery, etc.
    - ajvs 2y ago
      How do you set this up? I found this relevant issue[1] but it doesn't explain how to get it working. [1] https://github.com/ArchiveBox/ArchiveBox/issues/211 https://github.com/ArchiveBox/ArchiveBox/issues/211
      - nikisweeting 2y ago
        https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Insta... https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration/#chrome_user_data_dir https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration/... https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archiving-content-behind-log-ins--advanced-users-only https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv...
deleted 2y ago
[deleted]
grinch5751 2y ago
This looks like a really wonderful set of developments. Already making plans to use an old laptop of mine as an achivebox machine.
the_gorilla 2y ago
I don't know how anyone manages to use archivebox. I've tried it twice in the last 3 years and its site compatibility is bad, it quietly leaks everything you archive to archive.org by default, and whenever it fails on a download it stops archiving anything even after deleting and resubmitting all the jobs. I'm sure it works for some people, but not me.
- nikisweeting 2y ago
  These are legitimate gripes that have plagued specific past releases, I hear your frustration. Please keep in mind this was a solo effort of a single developer, only worked on in my spare time over the last 7 years (up until very recently). The new v0.8 adds a BG queue specifically to deal with the issue of stalling when some sites fail. There was a system to do this in the past, but it was imperfect and mostly optimized for the docker setup where a scheduler is running `archivebox update` every few hours to retry failed URLs. Site compability is much improved with the new BETA, but it's a perpetual cat and mouse game to fix specific sites, which is why we think the new plugin system is the way forward. It's just not sustainable for a single company (really just me right now) to maintain hundreds of workarounds for each individual site. I'm also discussing with the Webrecorder and Archive.org teams how we can to share these site-specific workarounds as cross-compatible plugins (aka "behaviors") between our various software. > it quietly leaks everything you archive to archive.org by default It's prominently mentioned many times (at least 4) on our homepage that this is the default, and archiving public-only sites (which are already fair game for Archive.org) is a default for good reason. Archiving private content requires several important changes and security considerations. More context: https://news.ycombinator.com/item?id=26866689 https://news.ycombinator.com/item?id=26866689
  - freedomben 2y ago
    Yeah, I'm not sure whether archive.org should be defaulted to on or off (I see both sides of that one), but its existence is definitely surfaced. I love Archive Box btw, thank you for your effort! It's filling a very important need.
  - the_gorilla 2y ago
toomuchtodo 2y ago
https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to enable a distributed archive. CDX indexes can be used for rapidly querying the underlying WARC archives. You might support cryptographically signing WARC archives; Wayback is particular about archive provenance and integrity, for example. https://www.loc.gov/preservation/digital/formats/fdd/fdd000590.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0005... ("CDX Internet Archive Index File") https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... ("WARC, Web ARChive file format") https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server https://github.com/internetarchive/wayback/tree/master/wayba... ("Wayback CDX Server API - BETA")
- nikisweeting 2y ago
  I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ. ArchiveBox is also gearing up to support real cryptographic signing of archives using https://tlsnotary.org/ https://tlsnotary.org/ in an upcoming plugin. (in a way that actually solves the TLS non-repudation issue, which traditional "signing a WARC" does not, more info: https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_09-4_Ritzdorf_paper.pdf https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd...)
  - deleted 2y ago
    [deleted]
  - toomuchtodo 2y ago
    Keep in mind, what signing methodology you use is a function of who accepts it. If I can confirm "ArchiveTeam ripped this", that is is superior to whatever tlsnotary is doing with MPC, blockchain, distributed ledger, whatever (in my use case). Have to trust someone at the end of the day. ArchiveTeam's Warrior doesn't use tlsnotary, for example, and rips entire sites just fine.
treyd 2y ago
Is this a project that could be developed to support a distributed mirror of archive.org similar to how Anna's Archive works?
- nikisweeting 2y ago
  Yeah that's what we're aiming for eventually, but with the addition of fine-grained permissions controls so you don't have to share everything 100% publically, you can choose a subset. https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap
nfriedly 2y ago
I've been using an instance of https://readeck.org/ https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too. I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I don't use it for personal content.
- nikisweeting 2y ago
  I love ArchiveTeam warrior, it's such a good idea! We run several instances ourselves, and it's part of our Good Karma Kit for computers with spare capacity: https://github.com/ArchiveBox/good-karma-kit https://github.com/ArchiveBox/good-karma-kit There are a bunch of other alternatives like ReadDeck listed on our wiki too, we encourage people to check it out! https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#other-archivebox-alternatives https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-...
- venusenvy47 2y ago
  I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution. https://addons.mozilla.org/en-US/firefox/addon/single-file/ https://addons.mozilla.org/en-US/firefox/addon/single-file/
  - nfriedly 2y ago
    I haven't looked at the on-disk format, I just use the browser interface. (It's fairly common for me to save something from my phone that I'll want to review on a computer later.) Here's an example of an Amazon "review" I recently archived that has instructions for using a USB tester I have: https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw And, for comparison, here's the original: https://www.amazon.com/gp/customer-reviews/R3EF0QW6MAJ0VP https://www.amazon.com/gp/customer-reviews/R3EF0QW6MAJ0VP It'd be nice if I could edit out the extra junk near the top, but the important bits are all there.
sagz 2y ago
Do y'all support archiving pages that are behind logins? Like using browser cookies?
- markerz 2y ago
  Yes, but there's security concerns where you might accidentally leak your credentials / cookies if you publish your archive to the public. https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archiving-content-behind-log-ins--advanced-users-only https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv... PS. I'm an archivebox user, not a dev or maintainer.
  - nikisweeting 2y ago
    Yes this is correct, with plans to make this easier in the near future via setup wizard that guides you through creating dedicated credentials for archiving.
millvalleydev 2y ago
For devs like us, archivebox? or browsertrix-crawler? for scraping entire sites for our own uses, maybe to keep contents behind pay walls while we have subscriptions or maybe to feed them to local LLMs to ask?
- nikisweeting 2y ago
  For scraping entire sites browserteix is currently more suited until we add full depth recursive crawling in v0.9. For feeding to LLMs ArchiveBox MIGHT BE better (imho) because we extract the raw content and you likely don't need the whole WARC.
bravura 2y ago
@nikisweeting ArchiveBox is awesome and we'd really love it to be more awesome. And sustainable! I've posted issues and PRs for showstopper issues that took months to get merged in: https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/pull/1026 https://github.com/ArchiveBox/ArchiveBox/pull/1026 You have the opportunity for the community to lean in on ArchiveBox. I understand it's hard to do everything as a solo dev, we've seen many cases in the community where solo devs get burned out or have personal challenges that take priority etc. It's hard for us users to lean in on ArchiveBox when after a happy month of archiving, things start break and you're left with maintaining a branch of your own fixes that aren't in main. Meanwhile, your solution of soliciting one time donations just makes the whole project feel more rickety and fly-by-night. How about thinking bigger? We NEED ArchiveBox to be a real thing. Decentralized tooling for archiving is SO IMPORTANT. I care about it and I suspect many people do. I'm posting this so other people who care about it can also comment and chime in and suggest how it can become something we can rely on. Because archiving isn't just about the past, it's about the future. Maybe it needs to be a dev org of three committed part-time maintainers, and a small foundation that people recurrently support is what grants it? IDK. I'm not an expert at how to make open source resilient. There have been discussions about this in the past, but I think it's worth a serious look because ArchiveBox is IMPORTANT and I want it to work any month I decide to re-activate my interest in it. I invite people to discuss ways to make this valuable project more sustianable and resilient.
- nikisweeting 2y ago
  Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space. It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history. (I should also mention that if you're a commercial entity relying on ArchiveBox, you can hire us for dedicated support and uptime guarantees. We have a closed source fork that has a much better test suite and lots of other goodies)
  - giancarlostoro 2y ago
thomasking004 2y ago
[flagged]
thomasking004 2y ago
[flagged]
404mm 2y ago
Somewhat similar topic, anyone has recommendations for a self-hosted internet website change monitoring system? I’ve been running Huginn for many years and it works well; however, I have a feeling the project is on its last leg. Also, it’s based on either text scraping (XPath/CSS/HTML and rss but it struggles with newer JS-based sites.
- nikisweeting 2y ago
  Changedetection.io
  - 404mm 2y ago
    Thank you! That looks great!
- arminiusreturns 2y ago
  Why do you feel like Huginn is on its last leg? It's been in my list of things to play with for years now, but I never got around to it...
  - 404mm 2y ago
    It looks like it’s being maintained by a single remaining developer. No new features are being added, just some basic maintenance. The product as a whole still works well, so unless you find something better, I do recommend it. I run it in k3s and the image is probably the easiest way of maintaining it.
- pabs3 2y ago
  I recommend urlwatch, you run it from a terminal and send the output wherever you want, such as email via cron. https://thp.io/2008/urlwatch/ https://thp.io/2008/urlwatch/
Acrobatic_Road 2y ago
The subline mentions "Auto-login", but the article never elaborates on this. Does this mean we will be able to more easily archive non-public websites? Also, how do you plan to ensure data authenticity across a distributed archive? For example, if I archive someone's blog, what is stopping me from inserting inflammatory posts that they never wrote, and passing them off as the real deal? Slight update: I see you're using TLS Notary! That's exactly what I would have suggested!
- nikisweeting 2y ago
  Auto log in is currently a service I provide for paying clients, and you can do it in the open source version manually with some extra config. Working hard on making it more accessible in the future, and plugins should help!
FiniteField 2y ago
Disappointing that a project that should ostensibly care about preserving the open, non-centralised internet takes the time to namedrop and talk about making "compromises" against preserving a well-known, medium-sized clearnet forum legally operated from a US-based LLC. Still-living independent forum sites in this day and age have unrivalled SNR of actual human-to-human communication, there should be no better candidate for archival. It's sad that a self-hosted archival tool has to apologise for any "evil" content it might be used for in the first place. Tape recorders do not require a disclaimer about people saying "hate speech" into them.
- nikisweeting 2y ago
  Sorry which medium sized forum are you referring to? I love forums and want them to continue, I'm not sure where you got the idea that I dislike them as a medium. I was just pointing out that public sites in general have started to see some attrition a bit lately for a variety of reasons, and the tooling needs to keep with new mediums as they appear. I also make no apology for the content, in fact ArchiveBox is explicitly designed to archive the most vile stuff for lawyers and governments to use for long term storage or evidence collection. One of our first prospective clients was the UN wanting to use it to document Syrian war crimes. The point there was that we can save stuff without amplifying it, and that's sometimes useful in niche scenarios. Lawyers/LE especially don't want to broadcast to the world (or tip off their suspect) that they are investigating or endorsing a particular person, so the ability to capture without publicly announcing/mirroring every capture is vital.
  - dark-star 2y ago
    I guess he's talking about K_wi F_rms which was mentioned in one of the screenshots...
    - nikisweeting 2y ago
      Ahh that makes sense. Well all I can say to that is that it's not up to me what's evil. The point I was trying to make is: sometimes you want to archive something that you don't endorse / don't want to be publicly linked. You might not want to amplify and broadcast the fact that you're archiving it to the world.
rodolphoarruda 2y ago
> "In an era where fear of public scrutiny is very tangible, people are afraid of archiving things for eternity. As a result, people choose not to archive at all, effectively erasing that history forever." Really? I don't get that feeling at all. I use Evernote to archive anything I consider worth keeping. I wonder where such "fear of archiving" comes from.
- nikisweeting 2y ago
  A lot of people are retreating off public free-for-all platforms like Twitter to more siloed spaces like Discord, for many reasons, not just fear of archiving. It all has the same effect of making it harder to archive though.
rcarmo 2y ago
This is nice. I'm actually much more excited about the REST API (which will let me do searches and pull information out, I hope) than the plugin ecosystem, since the last thing I need is for another tool to have a half-baked LLM integration -- I prefer to do that myself and have full control. Being able to do RAG on my ArchiveBox is something that I have very much wanted to do for over a year now, and it might finally be within reach without my going and hacking at the archived content tree... Edit: Just looked at the API schema at https://demo.archivebox.io/api/v1/docs https://demo.archivebox.io/api/v1/docs. No dedicated search endpoint? This looks like a HUGE missed opportunity. I was hoping to be able to query an FTS index on the SQLlite database... Have I missed something?
- nikisweeting 2y ago
  The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip. As for the AI stuff don't worry, none of it is touching core, it's all in an optional community plugin only for those who want it. I'm not personally a huge AI person but I have clients who are already using it and getting massive value from it, so it's worth mentioning. (They're doing some automated QA on thousands of collected captured and feeding results into spreadsheets)
  - rcarmo 2y ago
    Thanks, I'll have a look. My use for this is very different--I want to be able to use a specific subset of my archived pages (which is mostly reference documentation) to "chat" with, providing different LLM prompts depending on subset and fetching plaintext chunks as reference info for the LLM to summarize (and point me back to the archived pages if I need more info).
    - nikisweeting 2y ago
      Ok that makes sense, I think archivebox works as the first step in a pipeline there, with some other tool doing the LLM analysis and query stuff.
favorited 2y ago
As someone who was archiving a doomed website earlier today using wget, I was reminded that really need to get ArchiveBox working... I used to rely on my Pinboard subscription, but apparently archive exports haven't worked for years, so those days are over.
- nikisweeting 2y ago
  Pocket also doesn't offer archived page exports (or even RSS export). I feel like both are really dropping the ball in this area!
- VTimofeenko 2y ago
  I recently found omnivore.app through HN comments -- works great for sharing a reading list across machines. I am exporting articles through obsidian, but there is an API option. I don't think it supports outbound RSS, but they have inbound RSS(i.e. omnivore as RSS reader) in beta.
- pronoiac 2y ago
  Oh, writing my own Pinboard archive exporter is somewhere on my too-long to-do list. I should find out what would be good for importing into Archivebox. (WARC?)
- lgvld 2y ago
  FYI I am able to export (as JSON/HTML/XML) my Pinboard bookmarks.
orblivion 2y ago
Have you (and I wonder the same about archive.org) considered making a Merkle tree of the data that gets archived? Since data (including photos and videos) are getting easier to fake, it may be nice to have a provable record that at least a certain version of the data existed at a certain time. It would be most useful in case of some sort of oppressive regime down the line that wants to edit history. You'd want to publish the tip somewhere that records the time, and a blockchain seems to make the most sense to me but maybe you don't like blockchains.
- nikisweeting 2y ago
  Yup, already doing that in the betas. Thats what I'm referring to as the beginnings of a "content addressable store" in the article. In the closed source fork we currently store a merkle tree summary of each dir in a dotfile containing the sha256 and blake3 hash of all entries / subdirs. When a result is "sealed" the summary is generated, and the final salted hash can be submitted to Solana or ETH or some other network to attest to the time of capture and the content. (That part is coming via a plugin later)
  - orblivion 2y ago
    Wow that's great!
  - zvr 2y ago
    You might be interested in taking a look at SWHID (Software Hash IDentifiers), which defines a way (on its way to become an ISO standard) to reference files and directories with content-based identifiers, like swh:1:dir:d198bc9d7a6bcf6db04f476d29314f157507d505. Yes, it uses Merkle trees for filesystem hierarchy. https://www.swhid.org/specification/v1.1/5.Core_identifiers/ https://www.swhid.org/specification/v1.1/5.Core_identifiers/
- beefnugs 2y ago
  Not just all that nonsense, but also it makes a lot of sense to share just the parts from a website that matter like a single video etc without having to download an entire archive or the rest of the site
petertodd 2y ago
You really should add timestamping to ArchiveBox. The easiest way to do that would be via my OpenTimestamps protocol, https://opentimestamps.org https://opentimestamps.org It's open source and free to use, and uses Bitcoin for the actual timestamps. Users of it do not need to make Bitcoin transactions themselves as a set of community calendar servers do that for you. You also don't need a Bitcoin node to create an OTS timestamp, and you can validate an OTS timestamp without a Bitcoin node as well by trusting someone else to do that for you. The big thing that ArchiveBox can't do, and the Internet Archive can, is attest to the accuracy of the archive. Being at least able to prove that the archive was created in the past, prior to there being a reason to tamper it, is the best we can realistically do with current cryptography. So it'd be really good if support for timestamping was added. IIUC ArchiveBox is written in Python; OTS has a Python library that should work fine for you: https://github.com/opentimestamps/python-opentimestamps https://github.com/opentimestamps/python-opentimestamps
- nikisweeting 2y ago
  We're going to add TLSNotary support for real cryptographic signing, see my comments below :) Timestamping is also on my roadmap, definitely as a plugin (and likely paid) as it's more corporate users that really need it. We need to keep some of the really advanced attestation features paid to be able to support the rest of the business.
  - mikae1 2y ago
    Thanks for the box! Any examples of other possible really advanced features that might go for-pay? Is there any chance you will make current free features for-pay? That'd be rather off-putting for me as a home user.
    - nikisweeting 2y ago
      No, everything currently free will stay free. The paid stuff currently is: - per-user permissions & groups - audit logging - auto CAPTCHA solving - burner credential management for FB/Insta/Twitter/etc. w/ auto phone based account verification ability - custom JS scripts for expanding comments, hiding pop ups, etc. - managed hosting + support Some of this stuff ^ is going to become free in upcoming releases, some will stay paid. What I decide to make free is mostly based on abuse potential and legal ramifications, I'd rather have a say in how the risky stuff is used so that it doesn't become a tool weaponized for botting.
dark-star 2y ago
Some time ago I installed ArchiveBox on a RaspberryPi 4 running k3s (a lightweight Kubernetes distro). I have documented that here: https://darkstar.github.io/2022/02/07/k3s-on-raspberrypi-at-home-part6.html https://darkstar.github.io/2022/02/07/k3s-on-raspberrypi-at-... Note that this was a rather old version and some things have probably changed compared to now, so YMMV, but it might still provide a good reference for those who want to try
- nikisweeting 2y ago
  Thanks for making that tutorial! Happy to report that most of the quirks you cover have been improved: - uid 999 is no longer just enforced, you can pass any PUID:GUID now (like Linuxserver.io containers) - it now accepts ADMIN_USERNAME + ADMIN_PASSWORD env vars to create an initial admin user on first start without having to exec - archivebox/archivebox:latest is 0.7.2 (yearly stable release) and :dev is the 0.8.x pre-release updated daily. All Images are all amd64 & arm64 compatible. - singlefile and sonic are now included in all images & available on all platforms amd64/arm64
  - dark-star 2y ago
    yeah I really need to update that guide. Since I published it I have updated ArchiveBox locally to a newer version but never bothered to update the guide :)
A4ET8a8uTh0 2y ago
Those additions are welcome, but if I could request one -- I and one that it is very consistently requested -- feature: - backing up an entire page Yes, it is hard. Yes, for non-pure html pages is extra kind of painful, but that would honestly making archivebox go from nice to have to.. yes, I have an actual archive I can use when stuff goes down.
- nikisweeting 2y ago
  Do you mean backing up an entire domain? Like example.com/* If so that's starting to roll out in v0.8.5rc50, check out the archivebox/crawls/ folder. If you mean archiving a single page more thoroughly, what do you find is missing in Archivebox? Are you able to get singlefile/chrome/wget html when archiving?
  - A4ET8a8uTh0 2y ago
    Edit: The first option. ( previous stuff removed ) Lemme check my current version ( edit: 0.7.2 -- ty, I will update and test soon :D)
    - nikisweeting 2y ago
      Ah ok. One caveat: it's only available via the 'archivebox shell' / Python API currently, the CLI & web UIs for full depth crawling will come later. You can play around with the models and tasks, but I would wait a few weeks for it to stabilize and check again, it's still under heavy active development Check archivebox/archivebox:dev periodically
      - A4ET8a8uTh0 2y ago
        No worries. I can do that. You guys probably hear it all the time, but you are doing lords work. If I thought I could be of use in that project, I would be trying to contribute myself ( in fact, let me see if there a way I can participate in a useful manner ).
bityard 2y ago
So, after reading through the comments and website, I just realized I used ArchiveBox a month or two ago for a very specific purpose. You see, I inherited a boat. This boat belonged to my father. He was not materialistic but he took very good care of the things he cared about, and he cared about this boat. It's an old 18' aluminum fishing/cruising boat built in the early 1960's. It's not particularly valuable as a collectible but it is fairly rare and has some unique modifications. I spent a lot of time trying to dig up all of the info that I could on it, but this is one of those situations where most of the companies involved have been gone for decades and most everyone who was around when these were made are either dead or not really on the Internet. It's a shame that I waited so long to start my research because 10 or 20 years ago, there were quite a few active web forums containing informational/tutorial threads from the proud owners of these old boats. I know because I have seen references to them. Some of the URLs are in archive.org, some are not. But the forums are gone, so a large chunk of knowledge on these boats is too, probably forever. I did manage to dig up some interesting articles, pictures, and forum threads and needed a way to save them so that they didn't disappear from the web as well. There is probably an easier way to go about it, but in the end I ran ArchiveBox via Docker and set it to fetching what I could find and then downloaded the resulting pages as self-contained HTML pages.
- shiroiushi 2y ago
  >because 10 or 20 years ago, there were quite a few active web forums containing informational/tutorial threads from the proud owners of these old boats. ... But the forums are gone, so a large chunk of knowledge on these boats is too, probably forever. These days, that kind of info would be locked up in a closed Discord chat somewhere, so you can forget about people 20 years from now ever seeing it.
  - stavros 2y ago
    Or people today ever discovering it.
  - Magnets 2y ago
    Lots of private groups on facebook too
pabs3 2y ago
Unfortunately ArchiveBox uses wget, so it produces non-standard WARC files. Sadly there are lots of things like this in the WARC ecosystem. https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem
- nikisweeting 2y ago
  Yes, this is true currently. If you need nice WARCs I recommend Browsertrix by our friends at Webrecorder instead. Its on my roadmap to improve this eventually, but currently I'm focused on saving raw files to a filesystem, because it's more accessible to most users, and easier to pipe into other tools. I encourage people to use ZFS to do deduping and compression at the filesystem layer.
  - TheTechRobo 2y ago
    Browsertrix (and Webrecorder tools in general) also violate the standard by modifying response data. It's supposed to be the raw bytes as they are sent over the network (minus TLS). The entire WARC ecosystem is kind of a mess.
    - ikreymer 2y ago
      This isn't really true, our tools do not just modify response data for no reason! Our tools do the best that we can with an old format that is in use by many institutions. The WARC format does not account for H2/H3 data, which is used by most sites nowadays. The goal of our (Webreocrder) tools is to preserve interactive web content with as much fidelity as possible and make them accessible/viewable in the browser. That means stripping TLS, H2/H3, sometimes forcing a certain video resolution, etc.. while preserving the authenticity and interactivity of the site. It can be a tricky balance. If the goal is to preserve 'raw bytes sent over the network' you can use Wireshark / packet capture, but your archive won't necessarily be useful to a human.
      - CorentinB 2y ago
newman314 2y ago
@nikisweeting Is abx-dl already available or is it coming? I took a quick dive and didn't see a repo under the org. I'm happy to help package this up once it is available.
- nikisweeting 2y ago
  Not currently available, it should be out soon after v0.9 is released. Currently `mkdir tmp_data && cd tmp_data; archivebox install; archivebox add ...` is effectively equivalent to what `abx-dl` will do.
  - nikisweeting 2y ago
    Check here for progress: https://github.com/ArchiveBox/abx-dl https://github.com/ArchiveBox/abx-dl
chillfox 2y ago
Awesome, I am really looking forward to the new api and plugins. I have been running an instance for almost 2 years now that I use for archiving articles that I reference in my notes.
joeross 2y ago
I have no programming skill at all and I don’t know a ton about ArchiveBox except I set it up and ran it for myself for a while, so I’m asking as an innocent, ignorant and curious geek, but is this something that could be adapted to peer to peer distribution or some other means of making it simultaneously as private and local as you want it and as distributed and bulletproof, uptime wise, as possible?
agnishom 2y ago
This is interesting. I personally use Omnivore + backup to Obsidian for this purpose.
dewey 2y ago
I've tried to get started with ArchiveBox many times, but it was always quite buggy (Not working in Safari, a bit clunky to run,...) but I've noticed a lot of updates in the past months so I'm excited about it moving forward and giving it another shot.
hooverd 2y ago
Does anyone have recommendations for the hardware side of self-hosting something like that? How do I avoid bit rot? Also I see django-ninja! Very cool.

669 points by nikisweeting 2y ago

We've been pushing really hard over the last 6mo to develop this release. I'd love to hear feedback from people who've worked on big plugin systems in the past, or anyone who's tried our betas!

wongarsu 2y ago

Does this mean it's now possible to write plugins that dismiss cookie popups, solve captchas, scroll web pages etc.?

nikisweeting 2y ago
I have a private plugin with puppeteer support for stuff like this, currently charging clients money to use it to fund the open source development. The clients are people who are already legally allowed to evade CAPTCHAS (e.g. governments, NGOs doing research, lawyers collecting evidence, etc.) Unfortunatley I cant open source the CAPTCHA solving stuff myself, because it opens me up to liability, but if someone wants to contribute a plugin to the ecosystem I cant stop them ;).
- 0x1ch 2y ago
  Legally allowed to evade CAPTCHAs? LOL. What world do we live in where evading a captcha is an illegal offense?
  - nikisweeting 2y ago
    It doesn't matter whether or not it's actually legal, what matters is that the big platforms will sue you for trying, so you need a big bankroll to stand your ground. At the very least they can bar you from accessing their sites as you're violating ToS that you accept upon signup.
- pbronez 2y ago
  That’s a pretty savvy open-core strategy: paywall features that create risk of legal liability so they are only used when there’s a paying customer/operator to assume that risk.
xiconfjs 2y ago
You mean ArchiveBox still doesn’t deal with cookie popups? If so, it’s quasi not useful for EU based web sites.
- nikisweeting 2y ago
  It does, you just have to set up a chrome profile that has an extension to hide cookie popups, or use a profile where you've already accepted/closed them and have a session. You can archive with any chrome profile with arbitrary extensions enabled, so you can use uBlock, I still Don't care about cookies, Ghostery, etc.
  - ajvs 2y ago
    How do you set this up? I found this relevant issue[1] but it doesn't explain how to get it working. [1] https://github.com/ArchiveBox/ArchiveBox/issues/211 https://github.com/ArchiveBox/ArchiveBox/issues/211
    - nikisweeting 2y ago
      https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Insta... https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration/#chrome_user_data_dir https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration/... https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archiving-content-behind-log-ins--advanced-users-only https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv...

deleted 2y ago

[deleted]

grinch5751 2y ago

This looks like a really wonderful set of developments. Already making plans to use an old laptop of mine as an achivebox machine.

the_gorilla 2y ago

I don't know how anyone manages to use archivebox. I've tried it twice in the last 3 years and its site compatibility is bad, it quietly leaks everything you archive to archive.org by default, and whenever it fails on a download it stops archiving anything even after deleting and resubmitting all the jobs. I'm sure it works for some people, but not me.

nikisweeting 2y ago
These are legitimate gripes that have plagued specific past releases, I hear your frustration. Please keep in mind this was a solo effort of a single developer, only worked on in my spare time over the last 7 years (up until very recently). The new v0.8 adds a BG queue specifically to deal with the issue of stalling when some sites fail. There was a system to do this in the past, but it was imperfect and mostly optimized for the docker setup where a scheduler is running `archivebox update` every few hours to retry failed URLs. Site compability is much improved with the new BETA, but it's a perpetual cat and mouse game to fix specific sites, which is why we think the new plugin system is the way forward. It's just not sustainable for a single company (really just me right now) to maintain hundreds of workarounds for each individual site. I'm also discussing with the Webrecorder and Archive.org teams how we can to share these site-specific workarounds as cross-compatible plugins (aka "behaviors") between our various software. > it quietly leaks everything you archive to archive.org by default It's prominently mentioned many times (at least 4) on our homepage that this is the default, and archiving public-only sites (which are already fair game for Archive.org) is a default for good reason. Archiving private content requires several important changes and security considerations. More context: https://news.ycombinator.com/item?id=26866689 https://news.ycombinator.com/item?id=26866689
- freedomben 2y ago
  Yeah, I'm not sure whether archive.org should be defaulted to on or off (I see both sides of that one), but its existence is definitely surfaced. I love Archive Box btw, thank you for your effort! It's filling a very important need.
- the_gorilla 2y ago

toomuchtodo 2y ago

https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to enable a distributed archive. CDX indexes can be used for rapidly querying the underlying WARC archives. You might support cryptographically signing WARC archives; Wayback is particular about archive provenance and integrity, for example. https://www.loc.gov/preservation/digital/formats/fdd/fdd000590.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0005... ("CDX Internet Archive Index File") https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... ("WARC, Web ARChive file format") https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server https://github.com/internetarchive/wayback/tree/master/wayba... ("Wayback CDX Server API - BETA")

nikisweeting 2y ago
I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ. ArchiveBox is also gearing up to support real cryptographic signing of archives using https://tlsnotary.org/ https://tlsnotary.org/ in an upcoming plugin. (in a way that actually solves the TLS non-repudation issue, which traditional "signing a WARC" does not, more info: https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_09-4_Ritzdorf_paper.pdf https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd...)
- deleted 2y ago
  [deleted]
- toomuchtodo 2y ago
  Keep in mind, what signing methodology you use is a function of who accepts it. If I can confirm "ArchiveTeam ripped this", that is is superior to whatever tlsnotary is doing with MPC, blockchain, distributed ledger, whatever (in my use case). Have to trust someone at the end of the day. ArchiveTeam's Warrior doesn't use tlsnotary, for example, and rips entire sites just fine.

treyd 2y ago

Is this a project that could be developed to support a distributed mirror of archive.org similar to how Anna's Archive works?

nikisweeting 2y ago
Yeah that's what we're aiming for eventually, but with the addition of fine-grained permissions controls so you don't have to share everything 100% publically, you can choose a subset. https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap

nfriedly 2y ago

I've been using an instance of https://readeck.org/ https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too. I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I don't use it for personal content.

nikisweeting 2y ago
I love ArchiveTeam warrior, it's such a good idea! We run several instances ourselves, and it's part of our Good Karma Kit for computers with spare capacity: https://github.com/ArchiveBox/good-karma-kit https://github.com/ArchiveBox/good-karma-kit There are a bunch of other alternatives like ReadDeck listed on our wiki too, we encourage people to check it out! https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#other-archivebox-alternatives https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-...
venusenvy47 2y ago
I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution. https://addons.mozilla.org/en-US/firefox/addon/single-file/ https://addons.mozilla.org/en-US/firefox/addon/single-file/
- nfriedly 2y ago
  I haven't looked at the on-disk format, I just use the browser interface. (It's fairly common for me to save something from my phone that I'll want to review on a computer later.) Here's an example of an Amazon "review" I recently archived that has instructions for using a USB tester I have: https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw And, for comparison, here's the original: https://www.amazon.com/gp/customer-reviews/R3EF0QW6MAJ0VP https://www.amazon.com/gp/customer-reviews/R3EF0QW6MAJ0VP It'd be nice if I could edit out the extra junk near the top, but the important bits are all there.

sagz 2y ago

Do y'all support archiving pages that are behind logins? Like using browser cookies?

markerz 2y ago
Yes, but there's security concerns where you might accidentally leak your credentials / cookies if you publish your archive to the public. https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archiving-content-behind-log-ins--advanced-users-only https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv... PS. I'm an archivebox user, not a dev or maintainer.
- nikisweeting 2y ago
  Yes this is correct, with plans to make this easier in the near future via setup wizard that guides you through creating dedicated credentials for archiving.

millvalleydev 2y ago

For devs like us, archivebox? or browsertrix-crawler? for scraping entire sites for our own uses, maybe to keep contents behind pay walls while we have subscriptions or maybe to feed them to local LLMs to ask?

nikisweeting 2y ago
For scraping entire sites browserteix is currently more suited until we add full depth recursive crawling in v0.9. For feeding to LLMs ArchiveBox MIGHT BE better (imho) because we extract the raw content and you likely don't need the whole WARC.

bravura 2y ago

@nikisweeting ArchiveBox is awesome and we'd really love it to be more awesome. And sustainable! I've posted issues and PRs for showstopper issues that took months to get merged in: https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/pull/1026 https://github.com/ArchiveBox/ArchiveBox/pull/1026 You have the opportunity for the community to lean in on ArchiveBox. I understand it's hard to do everything as a solo dev, we've seen many cases in the community where solo devs get burned out or have personal challenges that take priority etc. It's hard for us users to lean in on ArchiveBox when after a happy month of archiving, things start break and you're left with maintaining a branch of your own fixes that aren't in main. Meanwhile, your solution of soliciting one time donations just makes the whole project feel more rickety and fly-by-night. How about thinking bigger? We NEED ArchiveBox to be a real thing. Decentralized tooling for archiving is SO IMPORTANT. I care about it and I suspect many people do. I'm posting this so other people who care about it can also comment and chime in and suggest how it can become something we can rely on. Because archiving isn't just about the past, it's about the future. Maybe it needs to be a dev org of three committed part-time maintainers, and a small foundation that people recurrently support is what grants it? IDK. I'm not an expert at how to make open source resilient. There have been discussions about this in the past, but I think it's worth a serious look because ArchiveBox is IMPORTANT and I want it to work any month I decide to re-activate my interest in it. I invite people to discuss ways to make this valuable project more sustianable and resilient.

nikisweeting 2y ago
Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space. It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history. (I should also mention that if you're a commercial entity relying on ArchiveBox, you can hire us for dedicated support and uptime guarantees. We have a closed source fork that has a much better test suite and lots of other goodies)
- giancarlostoro 2y ago

thomasking004 2y ago

[flagged]

thomasking004 2y ago

[flagged]

404mm 2y ago

Somewhat similar topic, anyone has recommendations for a self-hosted internet website change monitoring system? I’ve been running Huginn for many years and it works well; however, I have a feeling the project is on its last leg. Also, it’s based on either text scraping (XPath/CSS/HTML and rss but it struggles with newer JS-based sites.

nikisweeting 2y ago
Changedetection.io
- 404mm 2y ago
  Thank you! That looks great!
arminiusreturns 2y ago
Why do you feel like Huginn is on its last leg? It's been in my list of things to play with for years now, but I never got around to it...
- 404mm 2y ago
  It looks like it’s being maintained by a single remaining developer. No new features are being added, just some basic maintenance. The product as a whole still works well, so unless you find something better, I do recommend it. I run it in k3s and the image is probably the easiest way of maintaining it.
pabs3 2y ago
I recommend urlwatch, you run it from a terminal and send the output wherever you want, such as email via cron. https://thp.io/2008/urlwatch/ https://thp.io/2008/urlwatch/

Acrobatic_Road 2y ago

The subline mentions "Auto-login", but the article never elaborates on this. Does this mean we will be able to more easily archive non-public websites? Also, how do you plan to ensure data authenticity across a distributed archive? For example, if I archive someone's blog, what is stopping me from inserting inflammatory posts that they never wrote, and passing them off as the real deal? Slight update: I see you're using TLS Notary! That's exactly what I would have suggested!

nikisweeting 2y ago
Auto log in is currently a service I provide for paying clients, and you can do it in the open source version manually with some extra config. Working hard on making it more accessible in the future, and plugins should help!

FiniteField 2y ago

Disappointing that a project that should ostensibly care about preserving the open, non-centralised internet takes the time to namedrop and talk about making "compromises" against preserving a well-known, medium-sized clearnet forum legally operated from a US-based LLC. Still-living independent forum sites in this day and age have unrivalled SNR of actual human-to-human communication, there should be no better candidate for archival. It's sad that a self-hosted archival tool has to apologise for any "evil" content it might be used for in the first place. Tape recorders do not require a disclaimer about people saying "hate speech" into them.

nikisweeting 2y ago
Sorry which medium sized forum are you referring to? I love forums and want them to continue, I'm not sure where you got the idea that I dislike them as a medium. I was just pointing out that public sites in general have started to see some attrition a bit lately for a variety of reasons, and the tooling needs to keep with new mediums as they appear. I also make no apology for the content, in fact ArchiveBox is explicitly designed to archive the most vile stuff for lawyers and governments to use for long term storage or evidence collection. One of our first prospective clients was the UN wanting to use it to document Syrian war crimes. The point there was that we can save stuff without amplifying it, and that's sometimes useful in niche scenarios. Lawyers/LE especially don't want to broadcast to the world (or tip off their suspect) that they are investigating or endorsing a particular person, so the ability to capture without publicly announcing/mirroring every capture is vital.
- dark-star 2y ago
  I guess he's talking about K_wi F_rms which was mentioned in one of the screenshots...
  - nikisweeting 2y ago
    Ahh that makes sense. Well all I can say to that is that it's not up to me what's evil. The point I was trying to make is: sometimes you want to archive something that you don't endorse / don't want to be publicly linked. You might not want to amplify and broadcast the fact that you're archiving it to the world.

rodolphoarruda 2y ago

> "In an era where fear of public scrutiny is very tangible, people are afraid of archiving things for eternity. As a result, people choose not to archive at all, effectively erasing that history forever." Really? I don't get that feeling at all. I use Evernote to archive anything I consider worth keeping. I wonder where such "fear of archiving" comes from.

nikisweeting 2y ago
A lot of people are retreating off public free-for-all platforms like Twitter to more siloed spaces like Discord, for many reasons, not just fear of archiving. It all has the same effect of making it harder to archive though.

rcarmo 2y ago

This is nice. I'm actually much more excited about the REST API (which will let me do searches and pull information out, I hope) than the plugin ecosystem, since the last thing I need is for another tool to have a half-baked LLM integration -- I prefer to do that myself and have full control. Being able to do RAG on my ArchiveBox is something that I have very much wanted to do for over a year now, and it might finally be within reach without my going and hacking at the archived content tree... Edit: Just looked at the API schema at https://demo.archivebox.io/api/v1/docs https://demo.archivebox.io/api/v1/docs. No dedicated search endpoint? This looks like a HUGE missed opportunity. I was hoping to be able to query an FTS index on the SQLlite database... Have I missed something?

nikisweeting 2y ago
The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip. As for the AI stuff don't worry, none of it is touching core, it's all in an optional community plugin only for those who want it. I'm not personally a huge AI person but I have clients who are already using it and getting massive value from it, so it's worth mentioning. (They're doing some automated QA on thousands of collected captured and feeding results into spreadsheets)
- rcarmo 2y ago
  Thanks, I'll have a look. My use for this is very different--I want to be able to use a specific subset of my archived pages (which is mostly reference documentation) to "chat" with, providing different LLM prompts depending on subset and fetching plaintext chunks as reference info for the LLM to summarize (and point me back to the archived pages if I need more info).
  - nikisweeting 2y ago
    Ok that makes sense, I think archivebox works as the first step in a pipeline there, with some other tool doing the LLM analysis and query stuff.

favorited 2y ago

As someone who was archiving a doomed website earlier today using wget, I was reminded that really need to get ArchiveBox working... I used to rely on my Pinboard subscription, but apparently archive exports haven't worked for years, so those days are over.

nikisweeting 2y ago
Pocket also doesn't offer archived page exports (or even RSS export). I feel like both are really dropping the ball in this area!
VTimofeenko 2y ago
I recently found omnivore.app through HN comments -- works great for sharing a reading list across machines. I am exporting articles through obsidian, but there is an API option. I don't think it supports outbound RSS, but they have inbound RSS(i.e. omnivore as RSS reader) in beta.
pronoiac 2y ago
Oh, writing my own Pinboard archive exporter is somewhere on my too-long to-do list. I should find out what would be good for importing into Archivebox. (WARC?)
lgvld 2y ago
FYI I am able to export (as JSON/HTML/XML) my Pinboard bookmarks.

orblivion 2y ago

Have you (and I wonder the same about archive.org) considered making a Merkle tree of the data that gets archived? Since data (including photos and videos) are getting easier to fake, it may be nice to have a provable record that at least a certain version of the data existed at a certain time. It would be most useful in case of some sort of oppressive regime down the line that wants to edit history. You'd want to publish the tip somewhere that records the time, and a blockchain seems to make the most sense to me but maybe you don't like blockchains.

nikisweeting 2y ago
Yup, already doing that in the betas. Thats what I'm referring to as the beginnings of a "content addressable store" in the article. In the closed source fork we currently store a merkle tree summary of each dir in a dotfile containing the sha256 and blake3 hash of all entries / subdirs. When a result is "sealed" the summary is generated, and the final salted hash can be submitted to Solana or ETH or some other network to attest to the time of capture and the content. (That part is coming via a plugin later)
- orblivion 2y ago
  Wow that's great!
- zvr 2y ago
  You might be interested in taking a look at SWHID (Software Hash IDentifiers), which defines a way (on its way to become an ISO standard) to reference files and directories with content-based identifiers, like swh:1:dir:d198bc9d7a6bcf6db04f476d29314f157507d505. Yes, it uses Merkle trees for filesystem hierarchy. https://www.swhid.org/specification/v1.1/5.Core_identifiers/ https://www.swhid.org/specification/v1.1/5.Core_identifiers/
beefnugs 2y ago
Not just all that nonsense, but also it makes a lot of sense to share just the parts from a website that matter like a single video etc without having to download an entire archive or the rest of the site

petertodd 2y ago

You really should add timestamping to ArchiveBox. The easiest way to do that would be via my OpenTimestamps protocol, https://opentimestamps.org https://opentimestamps.org It's open source and free to use, and uses Bitcoin for the actual timestamps. Users of it do not need to make Bitcoin transactions themselves as a set of community calendar servers do that for you. You also don't need a Bitcoin node to create an OTS timestamp, and you can validate an OTS timestamp without a Bitcoin node as well by trusting someone else to do that for you. The big thing that ArchiveBox can't do, and the Internet Archive can, is attest to the accuracy of the archive. Being at least able to prove that the archive was created in the past, prior to there being a reason to tamper it, is the best we can realistically do with current cryptography. So it'd be really good if support for timestamping was added. IIUC ArchiveBox is written in Python; OTS has a Python library that should work fine for you: https://github.com/opentimestamps/python-opentimestamps https://github.com/opentimestamps/python-opentimestamps

nikisweeting 2y ago
We're going to add TLSNotary support for real cryptographic signing, see my comments below :) Timestamping is also on my roadmap, definitely as a plugin (and likely paid) as it's more corporate users that really need it. We need to keep some of the really advanced attestation features paid to be able to support the rest of the business.
- mikae1 2y ago
  Thanks for the box! Any examples of other possible really advanced features that might go for-pay? Is there any chance you will make current free features for-pay? That'd be rather off-putting for me as a home user.
  - nikisweeting 2y ago
    No, everything currently free will stay free. The paid stuff currently is: - per-user permissions & groups - audit logging - auto CAPTCHA solving - burner credential management for FB/Insta/Twitter/etc. w/ auto phone based account verification ability - custom JS scripts for expanding comments, hiding pop ups, etc. - managed hosting + support Some of this stuff ^ is going to become free in upcoming releases, some will stay paid. What I decide to make free is mostly based on abuse potential and legal ramifications, I'd rather have a say in how the risky stuff is used so that it doesn't become a tool weaponized for botting.

dark-star 2y ago

Some time ago I installed ArchiveBox on a RaspberryPi 4 running k3s (a lightweight Kubernetes distro). I have documented that here: https://darkstar.github.io/2022/02/07/k3s-on-raspberrypi-at-home-part6.html https://darkstar.github.io/2022/02/07/k3s-on-raspberrypi-at-... Note that this was a rather old version and some things have probably changed compared to now, so YMMV, but it might still provide a good reference for those who want to try

nikisweeting 2y ago
Thanks for making that tutorial! Happy to report that most of the quirks you cover have been improved: - uid 999 is no longer just enforced, you can pass any PUID:GUID now (like Linuxserver.io containers) - it now accepts ADMIN_USERNAME + ADMIN_PASSWORD env vars to create an initial admin user on first start without having to exec - archivebox/archivebox:latest is 0.7.2 (yearly stable release) and :dev is the 0.8.x pre-release updated daily. All Images are all amd64 & arm64 compatible. - singlefile and sonic are now included in all images & available on all platforms amd64/arm64
- dark-star 2y ago
  yeah I really need to update that guide. Since I published it I have updated ArchiveBox locally to a newer version but never bothered to update the guide :)

A4ET8a8uTh0 2y ago

Those additions are welcome, but if I could request one -- I and one that it is very consistently requested -- feature: - backing up an entire page Yes, it is hard. Yes, for non-pure html pages is extra kind of painful, but that would honestly making archivebox go from nice to have to.. yes, I have an actual archive I can use when stuff goes down.

nikisweeting 2y ago
Do you mean backing up an entire domain? Like example.com/* If so that's starting to roll out in v0.8.5rc50, check out the archivebox/crawls/ folder. If you mean archiving a single page more thoroughly, what do you find is missing in Archivebox? Are you able to get singlefile/chrome/wget html when archiving?
- A4ET8a8uTh0 2y ago
  Edit: The first option. ( previous stuff removed ) Lemme check my current version ( edit: 0.7.2 -- ty, I will update and test soon :D)
  - nikisweeting 2y ago
    Ah ok. One caveat: it's only available via the 'archivebox shell' / Python API currently, the CLI & web UIs for full depth crawling will come later. You can play around with the models and tasks, but I would wait a few weeks for it to stabilize and check again, it's still under heavy active development Check archivebox/archivebox:dev periodically
    - A4ET8a8uTh0 2y ago
      No worries. I can do that. You guys probably hear it all the time, but you are doing lords work. If I thought I could be of use in that project, I would be trying to contribute myself ( in fact, let me see if there a way I can participate in a useful manner ).

bityard 2y ago

So, after reading through the comments and website, I just realized I used ArchiveBox a month or two ago for a very specific purpose. You see, I inherited a boat. This boat belonged to my father. He was not materialistic but he took very good care of the things he cared about, and he cared about this boat. It's an old 18' aluminum fishing/cruising boat built in the early 1960's. It's not particularly valuable as a collectible but it is fairly rare and has some unique modifications. I spent a lot of time trying to dig up all of the info that I could on it, but this is one of those situations where most of the companies involved have been gone for decades and most everyone who was around when these were made are either dead or not really on the Internet. It's a shame that I waited so long to start my research because 10 or 20 years ago, there were quite a few active web forums containing informational/tutorial threads from the proud owners of these old boats. I know because I have seen references to them. Some of the URLs are in archive.org, some are not. But the forums are gone, so a large chunk of knowledge on these boats is too, probably forever. I did manage to dig up some interesting articles, pictures, and forum threads and needed a way to save them so that they didn't disappear from the web as well. There is probably an easier way to go about it, but in the end I ran ArchiveBox via Docker and set it to fetching what I could find and then downloaded the resulting pages as self-contained HTML pages.

shiroiushi 2y ago
>because 10 or 20 years ago, there were quite a few active web forums containing informational/tutorial threads from the proud owners of these old boats. ... But the forums are gone, so a large chunk of knowledge on these boats is too, probably forever. These days, that kind of info would be locked up in a closed Discord chat somewhere, so you can forget about people 20 years from now ever seeing it.
- stavros 2y ago
  Or people today ever discovering it.
- Magnets 2y ago
  Lots of private groups on facebook too

pabs3 2y ago

Unfortunately ArchiveBox uses wget, so it produces non-standard WARC files. Sadly there are lots of things like this in the WARC ecosystem. https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem

nikisweeting 2y ago
Yes, this is true currently. If you need nice WARCs I recommend Browsertrix by our friends at Webrecorder instead. Its on my roadmap to improve this eventually, but currently I'm focused on saving raw files to a filesystem, because it's more accessible to most users, and easier to pipe into other tools. I encourage people to use ZFS to do deduping and compression at the filesystem layer.
- TheTechRobo 2y ago
  Browsertrix (and Webrecorder tools in general) also violate the standard by modifying response data. It's supposed to be the raw bytes as they are sent over the network (minus TLS). The entire WARC ecosystem is kind of a mess.
  - ikreymer 2y ago
    This isn't really true, our tools do not just modify response data for no reason! Our tools do the best that we can with an old format that is in use by many institutions. The WARC format does not account for H2/H3 data, which is used by most sites nowadays. The goal of our (Webreocrder) tools is to preserve interactive web content with as much fidelity as possible and make them accessible/viewable in the browser. That means stripping TLS, H2/H3, sometimes forcing a certain video resolution, etc.. while preserving the authenticity and interactivity of the site. It can be a tricky balance. If the goal is to preserve 'raw bytes sent over the network' you can use Wireshark / packet capture, but your archive won't necessarily be useful to a human.
    - CorentinB 2y ago

newman314 2y ago

@nikisweeting Is abx-dl already available or is it coming? I took a quick dive and didn't see a repo under the org. I'm happy to help package this up once it is available.

nikisweeting 2y ago
Not currently available, it should be out soon after v0.9 is released. Currently `mkdir tmp_data && cd tmp_data; archivebox install; archivebox add ...` is effectively equivalent to what `abx-dl` will do.
- nikisweeting 2y ago
  Check here for progress: https://github.com/ArchiveBox/abx-dl https://github.com/ArchiveBox/abx-dl

chillfox 2y ago

Awesome, I am really looking forward to the new api and plugins. I have been running an instance for almost 2 years now that I use for archiving articles that I reference in my notes.

joeross 2y ago

I have no programming skill at all and I don’t know a ton about ArchiveBox except I set it up and ran it for myself for a while, so I’m asking as an innocent, ignorant and curious geek, but is this something that could be adapted to peer to peer distribution or some other means of making it simultaneously as private and local as you want it and as distributed and bulletproof, uptime wise, as possible?

agnishom 2y ago

This is interesting. I personally use Omnivore + backup to Obsidian for this purpose.

dewey 2y ago

I've tried to get started with ArchiveBox many times, but it was always quite buggy (Not working in Safari, a bit clunky to run,...) but I've noticed a lot of updates in the past months so I'm excited about it moving forward and giving it another shot.

hooverd 2y ago

Does anyone have recommendations for the hardware side of self-hosting something like that? How do I avoid bit rot? Also I see django-ninja! Very cool.