7 ms·
ArchiveBox: Open-source self-hosted web archiving
- chrisweekly 6y agoWow, great! Self-hosted, open-source, solid UI, tie-ins to the broader ecosystem... seems to check all the right boxes. Looking fwd to trying it and if all goes well, maybe see about integrating it into AthensResearch. Thanks for sharing!
- pacifika 6y agoNo mention of search?
- dugite-code 6y agoI literally just spun up a copy but it looks like it has Sonic full-text integration, however I'm not 100% sure if it's working via the UI as there isn't much feed back letting you know why a site has displayed in the results.
- nikisweeting 6y agoIt has full-text search using ripgrep and sonic.
- gravypod 6y agoI can't wait for the API to be completed. I want to build something to archive HN (article + comments) and turn it into epub go read offline. Hard to do currently.
- dugite-code 6y agoLast time I saw this you could only view the archive via the UI. It's really come a long way. Could be very useful now.
- sleavey 6y agoEven works on the Raspberry Pi, apparently. This would be nice in combination with a Pi-Hole.
- dehrmann 6y agoAm I the only one who spins up a new VM in VMWare ESXi for things like this?
- thies226j 6y agoYeah, why don’t you just use a container?
- Karunamon 6y agoVIC containers for me - a full VM is a bit overkill for something this light :)
- gzer0 6y agoWhat's your setup like? I use VMWare workstation pro- I have Windows Enterprise 2019 LTSC N installed and snapshotted to a base VM. Any time I need to do anything, I will full clone the base; with a decent SSD it takes maybe 10 seconds for the full clone and I have a full OS.
- Karunamon 6y agoActual ESXI on a decently powerful (33 cores, 512 gigs) machine. VMWare’s been really good to me, minus some points for occasional stupidity on upgrades.
- weekay 6y agoIt saves pages to archive.org as well. You might want to be careful while using this to archive personal content.
- kenniskrag 6y agoCan be disabled but on in default mode. https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#submit_archive_dot_org https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#...
- Proven 6y ago"Democratizing Tragedy of Commons at scale"
- asaddhamani 6y agoarchive.org will only archive publicly visible content and it respects robots.txt
- tyingq 6y ago"and it respects robots.txt" Not since 2017. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... They now have a clunky manual process to exclude your site. https://help.archive.org/hc/en-us/articles/360004651732-Using-The-Wayback-Machine https://help.archive.org/hc/en-us/articles/360004651732-Usin... ("How can I exclude...") They don't spoof user agents, but blocking them actively doesn't remove their history.
- gogopuppygogo 6y agoWould be great to allow for it to save to worm storage.
- asaddhamani 6y agoWorm storage?
- kinard 6y agowrite once read many
- nikisweeting 6y agoYou might like one of ArchiveBox's peers, https://ArchiveWeb.page https://ArchiveWeb.page by Ilya Kreymer / Webrecorder.io, it has an option to save directly to IPFS.
- elric 6y agoIf you're interested in this sort of thing, you might also be interested in Archivy [1], which is somewhat similar but it (thankfully) doesn't upload your stuff to archive.org [1] https://archivy.github.io/ https://archivy.github.io/
- dugite-code 6y ago> doesn't upload your stuff to archive.org FIY You just have to set the environment variable SUBMIT_ARCHIVE_DOT_ORG=False
- pseudalopex 6y agoIt's an accident waiting to happen.
- nikisweeting 6y agoSee my answer here for why it's the default: https://news.ycombinator.com/item?id=26866689 https://news.ycombinator.com/item?id=26866689
- faitswulff 6y agoWhat’s the danger in uploading to archive.org?
- elric 6y agoAside from the can of worms that is copyright infringement? There was a recent HN discussion about how much of a pain it is to get something removed from archive.org. Not uploading people's stuff to permanent, public archives seems like a good rule of thumb.
- tashbarg 6y agoIsn't this just passing the URL to archive.org which then does the actual archiving? If it isn't already public (i.e., reachable by archive.org), it won't be afterwards?
- rcMgD2BwE72F 6y agoHow does it differ from https://wallabag.org https://wallabag.org?
- hlasdjlfhalwjk 6y agoWallabag only stores links. ArchiveBox archives a snapshot of the actual content of a page at a specific point in time.
- ernesth 6y agoWallabag extracts the content and stores it. Not just the links.
- hlasdjlfhalwjk 6y agoDidn't know that. Thanks for clarifying.
- coffeeri 6y agoIt is quiet convenient to combine the two. You are able to export your wallabag list via RSS and import it on a schedule in archivebox.
- unicornporn 6y agohttps://www.linkace.org/ https://www.linkace.org/ Just a heads-up. Found that a while ago and much prefer it over wallabag.
- Kovah 6y agoThanks for sharing LinkAce! Maintainer here. If you or others have any questions, feel free to ask.
- pseudalopex 6y agoWhat makes it better?
- thedanbob 6y agoI’ve been using ArchiveBox since the last time it popped up on HN and I like it a lot. It recently got a significant UI upgrade.
- ramraj07 6y agoCan I export my history from my iPhone web browsing?
- rakoo 6y agoIf you can somehow extract your browsing history, archivebox can ingest a list of links
- nikisweeting 6y agoArchiveBox comes with a script that exports Safari history to a text file (which can then be imported into AB): ./bin/export_browser_history.sh --safari https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/export_browser_history.sh https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/export...
- dewey 6y agoI'm not sure how the iCloud Sync works in this case but if you use Desktop Safari where it has your iOS history too you might be able to get it out of the Safari sqlite DB on your computer: https://stackoverflow.com/questions/28628385/sqlite-safari-history-db https://stackoverflow.com/questions/28628385/sqlite-safari-h...
- nojito 6y agoSafari reading list already does this https://support.apple.com/guide/safari/keep-a-reading-list-sfri35905/mac https://support.apple.com/guide/safari/keep-a-reading-list-s...
- throwaway823882 6y agoLast time I tried to do this same thing, I didn't know about these, and ended up spending a couple days on wget and httrack. Do all these alternatives work from the command line, or are they their own little proprietary ecosystem?
- kwhitefoot 6y agoHow do tools like this cope with pages that are rendered by Javascript. What do the tools actually save? For instance if I save a Quora page using Firefox I can open it but if Quora is not accessible it doesn't work.
- nikisweeting 6y agoArchiveBox is a wrapper around ~12 different extractor modules, each of which saves the page or its assets in a different way. The most relevant to JS is Singlefile, which renders the page in headless chrome and then snapshots the DOM with all assets inlined after a few seconds of JS execution. It's not perfect, but it works well even for the majority of JS-heavy sites. For the very complex sites that really rely on a ton of interactive JS or dynamic requests to APIs to render their content, check out https://ArchiveWeb.page https://ArchiveWeb.page + https://ReplayWeb.page https://ReplayWeb.page by https://webrecorder.io https://webrecorder.io.
- skerit 6y agoThis is cool! Didn't know there were so many archiving options either, gonna check them all out.