7 ms·

Show HN: Kage – Shadow any website to a single binary for offline viewing

712 points by tamnd 4mo ago

lolpython 4mo ago
This is cool. I could see myself downloading the articles behind the first couple pages of hacker news with this, for viewing on a flight or long distance train ride with spotty internet
maxloh 4mo ago
I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli https://github.com/gildas-lormeau/single-file-cli
- tamnd 4mo ago
  It seems this repo only saves one web page? What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.
  - sdevonoes 4mo ago
    [flagged]
    - sermah 4mo ago
      Um. Whose website are you on right now?
      - ivangelion 4mo ago
        Don't come here to laugh but always great when it happens anyways.
  - maxloh 4mo ago
    Oh, I see. In that case, feature-wise, it is actually a modern alternative to HTTrack. I think the misunderstanding stems from the browser's "Save As" reference in the description. It is misleading. You use "Save As" to save a single page, not an entire website. Also, the description lacks a clear explanation of the project's purpose. It would be helpful to include a sentence explaining that the program downloads an entire website, not just a single page.
  - sillysaurusx 4mo ago
    > For example, all essays from paulgraham.com Not the same thing, but I made a clone of pg’s website which can be used for exactly that: https://github.com/shawwn/pg https://github.com/shawwn/pg https://shawwn.github.io/pg/ https://shawwn.github.io/pg/ If you want to read all essays, just clone the repo and open any of the .html files. Or any of the .page files which generated them.
  - nikisweeting 4mo ago
    Singlefile supports scoped recursive crawls too: https://github.com/gildas-lormeau/single-file-cli#:~:text=and%20crawl%20its-,internal%20links,-with%20the%20query https://github.com/gildas-lormeau/single-file-cli#:~:text=an... I highly recommend reading the singlefile source or https://archiveweb.page/ https://archiveweb.page/ to see how they handle closed shadow DOMs, cross-origin iframes, websockets, media urls, deduping large assets, etc.
- tamnd 4mo ago
  And thanks for the link. Let me implement this single HTML feature, it looks nice to have!
  - maxloh 4mo ago
    Yeah. An idea on top of that is to bundle an entire website into a single HTML page, with vendored JavaScript to enable client-side routing (all of the original pages' JS is still stripped out). That way, the page is self-contained as it is, but requires no bundled binary code to serve the site. It is actually safer security-wise. The vendored script can be as simple as this: const site = { "path-1": "<!DOCTYPE html><html> ... </html>", "path-2": "<!DOCTYPE html><html> ... </html>", // More paths } function attachListeners() { for (const [path, html] of Object.entries(site)) { document.querySelector(`a[href=${path}]`).onclick = () => { document.documentElement.outerHTML = html attachListeners() } } } document.addEventListeners("DOMContentLoaded", attachListeners)
- HelloUsername 4mo ago
  What's the difference with, any webbrowser on a computer, File -> Save as ?
  - nmstoker 4mo ago
    That's for a single page, this handles the whole site. Also the browser Save As options often work poorly.
  - dmazzoni 4mo ago
    Save As works fine for simple websites with static content. Let's say you have a site that fetches content from a database. If you Save As, then at best you'll get a local copy of an HTML page with JS that loads the content from the same remote database. It might not work (since the local copy has a different origin), or if it does, it requires you to be online, which defeats half of the purpose. What this project, and SingleFile, both do is save a snapshot of what the rendered page actually looks like at that moment in time. The scripts are stripped out so it runs locally and has no external dependencies.
- deleted 4mo ago
  [deleted]
- wamatt 4mo ago
  Love love love SingleFile too. The FF extension works pretty well for a clean save. That said, Kage looks promising if OP can combine SingleFile reproduction quality with the HTTPTrack spidering approach. SPA's are kinda tricky with archiving and do wonder how well Kage would handle that
  - initramfs 4mo ago
    I've seen the option in IE- .mhtml. For some reason it displays in IE better but I don't recall seeing this option in chrome of Firefox recently..
- arikrahman 4mo ago
  This is what I first thought and it's a very elegant solution, and not needlessly overcomplicated.
gregwebs 4mo ago
This seems like it has potential to create a lot of load on a site- are there settings to set how fast it clones or avoid images/videos? Is there a way to only get a subset of a website?
- tamnd 4mo ago
  Could you help create a new issue for that? I will do it later. It is already 1:00 AM my time, but I am happy that anyone is interested in it. : )
- ares623 4mo ago
  Just pretend you're an AI crawler problem solved
sanqui 4mo ago
Cool concept. I would like to see this combined with mitmproxy for archive grade fidelity. You could be saving exactly the data served and at the same time a representation by a modern (contemporary) browser, with all JS having run. This combination would be my perfect replacement for the WARC format.
- tamnd 4mo ago
  I'm working on WARC too, with format from Common Crawl! By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli https://github.com/tamnd/ccrawl-cli
  - sanqui 4mo ago
    That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a potentially nice new archival format.
    - tamnd 4mo ago
      I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support. For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!
      - sanqui 4mo ago
        I'm a fan of compatibility with established formats! Sounds awesome. There is a lot of untapped potential with respect to efficiently archiving and indexing websites. I saw the impressive things Marginalia Search is doing in this area (the blog is great when it gets technical). There is also a lot of very complete archives of websites out there which are not being indexed at all, and I would love to make them available for researchers. In any case, I'm interested in your project!
rahimnathwani 4mo ago
So this is like using wget --mirror except that it works on pages that require javascript, right?
- tamnd 4mo ago
  Yeah, it is. For example, openai.com is rendered with Next.js, so I will try to mirror it tomorrow.
wolttam 4mo ago
One use I'd have for this is company wikis that you want to give folks easy offline access to (maybe the wiki has documentation that's useful at sites that don't have cellular coverage). Cool! It would be especially cool to have a version that didn't require the separate serving process - even though it's nifty you can package up a whole site as a single binary. Maybe a single HTML entrypoint shim with a bit of javascript that could index into an archive (potentially embedded) of the site's content?
- tamnd 4mo ago
  Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :) Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.
  - mgiampapa 4mo ago
    I think the zim flow was perfect for offline use. I know I will be making use of it as soon as I can figure out how to pass chrome the cookies so I can be signed into the site. Didn't see it in the page, but I didn't look closely yet.
    - tamnd 4mo ago
      Not yet supporting cookies, since I created this tool for shadowing public websites first. I will add options to pass cookies later. It will pass them to the underlying Chrome/Chromium process, so it should not be hard to do.
  - smeej 4mo ago
    I would use the shit out of this. I'm a heavy user of Logseq (OG, the md file-based version). Would LOVE to save my favorite web resources this way.
  - mcdonje 4mo ago
    Not to load you up with too many ideas, but a markdown folder sounds a lot like obsidian, which has a plugin system now. Epub would also be a great target.
grahamstanes17 4mo ago
nice
dimiprasakis 4mo ago
Neat project, I like the idea. One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on! In any case, cool stuff :)
- nikisweeting 4mo ago
  --no-sandbox is needed in docker, maybe they assume it will mostly run in docker?
  - tamnd 4mo ago
    Exactly. For downloading, Kage requires Chrome or Chromium. Running it inside Docker makes setup easier and keeps cleanup simple: https://github.com/tamnd/kage/blob/main/Dockerfile https://github.com/tamnd/kage/blob/main/Dockerfile Btw, let me think the way to only enable this when running inside Docker.
    - nikisweeting 4mo ago
      Docker is designed to be undetectable by default, the best way I have found is to set env IN_DOCKER=True manually in your Dockerfile + check that there is no $DISPLAY configured + that you're on linux. Usually if all/most of those are true you can safely add --no-sandbox --disable-setuid-sandbox --disable-dev-shm-usage etc. all the docker-specific flags. Thats what we do in https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile#L46 https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile...
      - tamnd 4mo ago
        It should be fixed by https://github.com/tamnd/kage/pull/12 https://github.com/tamnd/kage/pull/12 Thanks for nice trick.
      - dimiprasakis 4mo ago
        Cool approach. But, a compromise still lands on host's kernel, Docker doesn't provide kernel isolation (well it does on a macOS because it runs in Docker machine but thats a side effect). I wonder if a better solution would be to play with seccomp or Linux capabilities so that Chrome is sandboxed even in Docker. Not sure how this would work tbh. Answering here to get ideas, I saw your fix on Git and request for feedback (will try to review and give it some thought once I find some time)
daviding 4mo ago
Nice idea! fwiw, false positives and all, but the Windows 11 default Windows Security doesn't like it: `leakless.exe: Operation did not complete successfully because the file contains a virus or potentially unwanted software.`
mococa 4mo ago
curl can do this
Igor_Wiwi 4mo ago
This is quite useful tool, especially for the cases where internet access is limited (the flights for example). I implemented it as a separate feature in mdview.io: for example you can export a document as a html file for offline usage, with all the presentation features like reach tables, mermaid and etc built in. Example https://mdview.io/s/why-markdown-became-default-format-for-ai https://mdview.io/s/why-markdown-became-default-format-for-a... then try to Export - Export HTML
deleted 4mo ago
[deleted]
deleted 4mo ago
[deleted]
telesilla 4mo ago
I've been using httrack (https://www.httrack.com https://www.httrack.com) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.
- throwaway219450 4mo ago
  Specifically for wikis, is there a reason you wouldn't use Kiwix? For non "official" releases it's more complicated, but there are some services to generate the ZIM files. The desktop reader app is pretty good in my experience. https://wiki.openzim.org/wiki/Build_your_ZIM_file https://wiki.openzim.org/wiki/Build_your_ZIM_file EDIT: https://get.kiwix.org/en/solutions/applications/kiwix-reader/ https://get.kiwix.org/en/solutions/applications/kiwix-reader...
  - telesilla 4mo ago
    Thanks, never knew about this and great to hear about it.
  - tamnd 4mo ago
    Kiwix has readers for almost every platform, Android, desktop, iPhone. That's why I made Kage produce ZIM file. The executable file is mostly for people who don't have Kiwix installed yet, or just want to run the archive directly.
- nikisweeting 4mo ago
  https://github.com/archiveteam/grab-site https://github.com/archiveteam/grab-site or browsertrix may be easier to use for some, it's what was used to save a lot of the data.gov stuff before it got taken down.
- tamnd 4mo ago
  This brings back memories. Around twenty years ago, internet was still expensive dial-up, so I used to go to an internet cafe, run HTTrack to download websites and manga, copy everything onto my tiny 128MB USB stick (felt very large at that time), then bring it home and read offline ;))
latexr 4mo ago
For those with an eReader, one thing that works really well is using pandoc to download and convert a webpage to EPUB that you can then load to your reader. pandoc --from html --to epub --output /PATH/TO/FILE.epub https://example.com
- arikrahman 4mo ago
  Thanks, will try this out on the Kobo later.
ninalanyon 4mo ago
> kage serve $HOME/data/kage/paulgraham.com If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like: $ firefox $HOME/data/kage/paulgraham.com Then the result would be useable on machines without kage nstalled.
- doctoboggan 4mo ago
  Usually JavaScript is blocked when you load pages that way.
  - pixelatedindex 4mo ago
    I thought all the JS was stripper?
  - embedding-shape 4mo ago
    Since when? You won't be able to make HTTP requests to localhost, as it'd be a different Origin, but I don't think any mainstream browser blocks JS outright when you use file:// to load and view HTML files.
    - rzzzt 4mo ago
      Somewhere around 2019, each document loaded from file:// became its own origin in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1500453 https://bugzilla.mozilla.org/show_bug.cgi?id=1500453 (I didn't check when this happened in Chromium) Related WHATWG discussion: https://github.com/whatwg/html/issues/3099 https://github.com/whatwg/html/issues/3099
      - embedding-shape 4mo ago
        Yeah, but that's fine, the document is .html, and it can load ./app.js or ./style.css just fine even if loaded by file:// (as long as it isn't initiated by JS itself, then Origin starts to matter a lot more), otherwise basically every single local HTML file would suddenly be broken, I don't think anyone would have accepted that even with the origin changes.
simonw 4mo ago
I was intrigued to see how the demo GIF in the README was generated: https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c630610112b211964d/docs/static/demo.gif https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif https://github.com/tamnd/ascii-gif The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c630610112b211964d/docs/demo/kage.tape https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it: ascii-gif render docs/demo/kage.tape -o docs/static/demo.gif Looks like it's an opinionated wrapper around https://github.com/charmbracelet/vhs https://github.com/charmbracelet/vhs
- alterom 4mo ago
  FYI, on other platforms (Windows/MacOS), LiceCAP is a fantastic tool to record screen into compact GIFs by the author of Winamp and Reaper DAW: https://www.cockos.com/licecap/ https://www.cockos.com/licecap/
- jubilanti 4mo ago
  Have you heard the good news about the terminal savior asciinema -- https://asciinema.org/ https://asciinema.org/
  - embedding-shape 4mo ago
    It's a cool tool/platform, but very different. Asciinema tries to make the "multimedia" itself better by making it actual text instead of being video/images, while the CLI command above turns actual text into multimedia supported by platforms already. Both are useful, both have their use cases :)
- stavros 4mo ago
  VHS is fantastic for scripting cli video generation.
- vqtska 4mo ago
  You can also do an animated svg which is way smaller than a gif because it's just text keyframes (https://github.com/vytskalt/pseudoc/blob/main/assets/factorial.svg https://github.com/vytskalt/pseudoc/blob/main/assets/factori...)
chinnyys 4mo ago
The readme is AI slop, and incredibly grating to read. The disgust I felt while reading it almost put me off trying the project. Is the code also AI slop?
shinryuu 4mo ago
Reminds me of this. https://gwern.net/gwtar https://gwern.net/gwtar Compared to that is there anything kage does better?
nitotm 4mo ago
I was looking for something like this the other day, it can be very helpful.
soulofmischief 4mo ago
Cool project! I know it's written in go, but it would be cool to see something like this which uses Cosmopolitan Libc + redbean or something similar to create a binary which runs anywhere. Would be fun to be able to pass around self-executable website archives. https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan https://justine.lol/cosmopolitan/index.html https://justine.lol/cosmopolitan/index.html https://redbean.dev https://redbean.dev (Certificates just expired for justine's website, just ignore the warning.)
- tamnd 4mo ago
  This could be a nice code golf project. It only needs a webview, a ZIM reader, and a way to append data to an existing binary and read it back. I did something like that a very long time ago (Of course, I have forgotten)
- jokethrowaway 4mo ago
  I never understood the appeal for cosmopolitan. I'd rather have platform specific minimal binaries than a single binary with hacks. Installing packages is a solved problem
  - soulofmischief 4mo ago
    Installing packages is a completely different activity than passing around self-executable archives among friends. Not everything needs to go through a CI pipeline and distribution platform before you can share it with others. On top of that, I really enjoy being able to write quick little utilities and then pass them around without worrying about what operating system anyone who stumbles upon it has. It's fine if you don't personally find it useful for your workflow, but I think it's mad cool, especially since you can zip together multiple binaries into one, along with data.
KellyCriterion 4mo ago
Sounds like .MCH-files re-invented? (-:
Onavo 4mo ago
How does it handle websites with client side paywalls? Can you run it with extensions like bypass paywalls and ublock origin?
coffeecoders 4mo ago
I've accumulated a bunch of old website archives over the years. The funny thing is the ugly HTML dumps have been more useful than the "perfect" archive. It's one of the reasons I've become a bigger fan of RSS over time. A feed from 10-ish years ago is often more usable today than a carefully preserved (application) website.
- tamnd 4mo ago
  I have a project for creating and archiving RSS feeds, keeping the full history from the time the crawler starts. I need to clean up a bit, then will open source it soon.
- couscouspie 4mo ago
  Maybe it is just me, but by far most of the time, when I want to archive something from the internet, it is information and information is best served in an absolutely minimal text format like html or md.
cynicalsecurity 4mo ago
Binary app is a really bad way of storing data. No one would ever want to run a binary shared with them or found online.
- tamnd 4mo ago
  For sharing, better use the html folder or zim format, Kage supports both of them.
- sablekit 4mo ago
  [flagged]
calrizien 4mo ago
Does this work for the Apple Docs website? Really tricky to get those offline.
- tamnd 4mo ago
  Making docs available offline was one of my main motivations for building this tool. I will try Apple Docs too. I previously downloaded the Snowflake docs, and it was something like tens or even hundreds of thousands of pages, I do not remember exactly. The output ended up being very large. By the way, I forgot to add zstd compression support to my ZIM reader/writer. I will implement that in the next version.
- tamnd 4mo ago
  Good news for you: here is the command to clone Apple Docs: ```bash bin/kage clone https://developer.apple.com/documentation/ https://developer.apple.com/documentation/ \ --scope-prefix /documentation/ \ --out /Users/apple/data/apple-docs \ --chrome "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \ --max-pages 0 --max-depth 0 \ --workers 3 --browser-pages 3 --asset-workers 6 \ --render-timeout 60s --settle 2s --timeout 30s \ 2>&1 | tee -a /Users/apple/apple-docs.log ``` Adjust it to your needs :) I smoke-tested it, and all the content and CSS work, but I stripped all the JS, so the sidebar won't work. If you run into any problems, feel free to create new issues in the repo. It helps me prioritize and know what should be fixed.
k4rnaj1k 4mo ago
[dead]
sneak 4mo ago
The README is LLM slop. This makes me assume the code is the same.
chfritz 4mo ago
how is this different from using puppeteer to load the page and save the DOM as HTML?
kadhirvelm 4mo ago
This is awesome, we wanted an offline copy of someone’s prototype (as built on Lovable, etc) so we could do version control and sharing in an easier format. Wrote our approach here: https://productnow.ai/blogs/extracting-html-from-ai-prototyping-tools https://productnow.ai/blogs/extracting-html-from-ai-prototyp... But will look into this now, see if we can swap some stuff out. We’ve really liked the idea of an offline mirror, makes a lot of collaboration use cases simpler
jyscao 4mo ago
I tried to clone a HTTP (not HTTPS) site, and it's giving me `navigation failed: net::ERR_NAME_NOT_RESOLVED`. Even when I explicitly included the protocol with `http://<FQDN>`.
godot 4mo ago
the readme uses paulgraham.com as an example (which is text articles mostly) and I never use "Save As" for a web page (for the reasons the author states), I always just print as PDF and save the PDF file. for an entire website though of many pages I can see this can be useful.
amatecha 4mo ago
Suddenly remembering the days of dialup and your browser serving a fully-functional cached copy of a webpage when you try to access it and you're not online...
eventinbox 4mo ago
[flagged]
G_o_D 4mo ago
How its different then MHTML ??
rickylin 4mo ago
It seems like https://github.com/tw93/pake https://github.com/tw93/pake is better.
- italiancheese 4mo ago
  Both of these projects have completely different purposes and use cases. Have you even read the first line of the readme of the project you're commenting on?
smusamashah 4mo ago
What if I wanted to download all Confluence docs at work?
carsonye 4mo ago
This is interesting. Is the intended use case mostly read-only websites like blogs/docs/essays? How well does it handle sites where navigation, search, dropdowns, or other UI interactions depend on JavaScript?
- tamnd 4mo ago
  Currently, all of that is broken. At one point, I had a traumatic experience where an archived HTML file kept redirecting to the live site, even though I already had all the content rendered, so I ended up disabling all JavaScript entirely.
- tamnd 4mo ago
  If there's more demand for that, maybe I will implement a more relaxed version.
jokethrowaway 4mo ago
Amazing stuff! I would recommend an add-on or new feature to detect and remove cookie banners / annoying popups that open on load (eg. sign up to my mailing list). listing a few examples form fastText could help you. You might also have the opposite problem though: some websites have content in the base html (so it's searchable by Google and they get views) and remove it on load (so you have to pay). Capturing the initial html and comparing it to the final version could give you some hints and allow you to repair the removed content. Best of luck with the project!
sails 4mo ago
What is the best way to give coding agent a full website so that it can see what I see? With animation and design I’m never sure what it gets when I save the website in the browser. Maybe this is suitable?
endorphine 4mo ago
Anyone remembers Teleport Pro?
xlii 4mo ago
> No tracking, no network calls, no surprises. Won't comment on a project (though idea seems interesting) but this in README is a tell for me ;)
- xd1936 4mo ago
  It's not just no tracking — It's no surprises.
- praveer13 4mo ago
  Somehow 'Kage' is the first name claude suggests to me for any new project as well
  - timerol 4mo ago
    The Japanese word for shadow?
c7b 4mo ago
Probably a stupid question, but could this archive embedded videos as well?
- tamnd 4mo ago
  Possible, but currently I disable all large files, including videos. For video downloading, I suggest wrapping around yt-dlp. It's an awesome tool.
Sathwickp 4mo ago
I'm still trying to cope with your github profile, 68k commits a year is crazyy
aa-jv 4mo ago
I've been using "Print to PDF" as my principle bookmarks management tool, since 1998, and I have over 90,000+ such PDF's sitting on my system, easily re-read and discovered. So I don't quite get whats the point of kage? What does it do that print-to-PDF won't already do? The resulting .pdf's contain all the content, and also include the original URL and creation date, etc. How is kage an improvement?
snowflaxxx 4mo ago
Meet Teleport Pro
netdevphoenix 4mo ago
[dead]
ekianjo 4mo ago
Curious about "keep it for a decade" claim. Can something possibly break down the road?
kjmh 4mo ago
I was floored by the idea of browsing docs offline but disappointed that recreating the demo of archiving Paul Graham’s essays gave me a ZIM with broken images and broken Unicode symbols when viewed in Kiwix.
Departed7405 4mo ago
What's the advantage compared to mhtml?
ralferoo 4mo ago
It sounds like a nice idea, but "It drives a real browser, lets the page finish doing whatever it does, grabs the finished result, and then rips every script out of it." sounds like it'd fail for a lot of websites that do things like display some kind of modal banner inviting you to click something to close it, and that prevents scrolling while that is open, whilst also delaying loading the rest of the article until a certain scroll point is hit.
deleted 4mo ago
[deleted]
b0dhimind 4mo ago
I've had a subscription to this site: app.dancevision.com for a while. It requires cookies to login and view and each video (3 directories down) is hosted on Vimeo. Could Kage actually do an offline mirror of the site (and the videos)?
aynite 4mo ago
nice job and good inspiration, i'm just looking for something like this for some local processing
aivanise 4mo ago
Was a bit disappointed when I realized that .zim files do not have an index and thus are not searchable. Limits a lot of use cases, e.g. I wanted to crawl my corporate wiki to have a wiki to know how to fix wiki when wiki is down ;)

712 points by tamnd 4mo ago

This is cool. I could see myself downloading the articles behind the first couple pages of hacker news with this, for viewing on a flight or long distance train ride with spotty internet

maxloh 4mo ago

I find SingleFile [0] to be a much more robust version of this. It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings. They also offer a CLI powered by Puppeteer. [1] [0]: https://github.com/gildas-lormeau/singlefile https://github.com/gildas-lormeau/singlefile [1]: https://github.com/gildas-lormeau/single-file-cli https://github.com/gildas-lormeau/single-file-cli

tamnd 4mo ago
It seems this repo only saves one web page? What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.
- sdevonoes 4mo ago
  [flagged]
  - sermah 4mo ago
    Um. Whose website are you on right now?
    - ivangelion 4mo ago
      Don't come here to laugh but always great when it happens anyways.
- maxloh 4mo ago
  Oh, I see. In that case, feature-wise, it is actually a modern alternative to HTTrack. I think the misunderstanding stems from the browser's "Save As" reference in the description. It is misleading. You use "Save As" to save a single page, not an entire website. Also, the description lacks a clear explanation of the project's purpose. It would be helpful to include a sentence explaining that the program downloads an entire website, not just a single page.
- sillysaurusx 4mo ago
  > For example, all essays from paulgraham.com Not the same thing, but I made a clone of pg’s website which can be used for exactly that: https://github.com/shawwn/pg https://github.com/shawwn/pg https://shawwn.github.io/pg/ https://shawwn.github.io/pg/ If you want to read all essays, just clone the repo and open any of the .html files. Or any of the .page files which generated them.
- nikisweeting 4mo ago
  Singlefile supports scoped recursive crawls too: https://github.com/gildas-lormeau/single-file-cli#:~:text=and%20crawl%20its-,internal%20links,-with%20the%20query https://github.com/gildas-lormeau/single-file-cli#:~:text=an... I highly recommend reading the singlefile source or https://archiveweb.page/ https://archiveweb.page/ to see how they handle closed shadow DOMs, cross-origin iframes, websockets, media urls, deduping large assets, etc.
tamnd 4mo ago
And thanks for the link. Let me implement this single HTML feature, it looks nice to have!
- maxloh 4mo ago
  Yeah. An idea on top of that is to bundle an entire website into a single HTML page, with vendored JavaScript to enable client-side routing (all of the original pages' JS is still stripped out). That way, the page is self-contained as it is, but requires no bundled binary code to serve the site. It is actually safer security-wise. The vendored script can be as simple as this: const site = { "path-1": "<!DOCTYPE html><html> ... </html>", "path-2": "<!DOCTYPE html><html> ... </html>", // More paths } function attachListeners() { for (const [path, html] of Object.entries(site)) { document.querySelector(`a[href=${path}]`).onclick = () => { document.documentElement.outerHTML = html attachListeners() } } } document.addEventListeners("DOMContentLoaded", attachListeners)
HelloUsername 4mo ago
What's the difference with, any webbrowser on a computer, File -> Save as ?
- nmstoker 4mo ago
  That's for a single page, this handles the whole site. Also the browser Save As options often work poorly.
- dmazzoni 4mo ago
  Save As works fine for simple websites with static content. Let's say you have a site that fetches content from a database. If you Save As, then at best you'll get a local copy of an HTML page with JS that loads the content from the same remote database. It might not work (since the local copy has a different origin), or if it does, it requires you to be online, which defeats half of the purpose. What this project, and SingleFile, both do is save a snapshot of what the rendered page actually looks like at that moment in time. The scripts are stripped out so it runs locally and has no external dependencies.
deleted 4mo ago
[deleted]
wamatt 4mo ago
Love love love SingleFile too. The FF extension works pretty well for a clean save. That said, Kage looks promising if OP can combine SingleFile reproduction quality with the HTTPTrack spidering approach. SPA's are kinda tricky with archiving and do wonder how well Kage would handle that
- initramfs 4mo ago
  I've seen the option in IE- .mhtml. For some reason it displays in IE better but I don't recall seeing this option in chrome of Firefox recently..
arikrahman 4mo ago
This is what I first thought and it's a very elegant solution, and not needlessly overcomplicated.

gregwebs 4mo ago

This seems like it has potential to create a lot of load on a site- are there settings to set how fast it clones or avoid images/videos? Is there a way to only get a subset of a website?

tamnd 4mo ago
Could you help create a new issue for that? I will do it later. It is already 1:00 AM my time, but I am happy that anyone is interested in it. : )
ares623 4mo ago
Just pretend you're an AI crawler problem solved

sanqui 4mo ago

Cool concept. I would like to see this combined with mitmproxy for archive grade fidelity. You could be saving exactly the data served and at the same time a representation by a modern (contemporary) browser, with all JS having run. This combination would be my perfect replacement for the WARC format.

tamnd 4mo ago
I'm working on WARC too, with format from Common Crawl! By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli https://github.com/tamnd/ccrawl-cli
- sanqui 4mo ago
  That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a potentially nice new archival format.
  - tamnd 4mo ago
    I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support. For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!
    - sanqui 4mo ago
      I'm a fan of compatibility with established formats! Sounds awesome. There is a lot of untapped potential with respect to efficiently archiving and indexing websites. I saw the impressive things Marginalia Search is doing in this area (the blog is great when it gets technical). There is also a lot of very complete archives of websites out there which are not being indexed at all, and I would love to make them available for researchers. In any case, I'm interested in your project!

rahimnathwani 4mo ago

So this is like using wget --mirror except that it works on pages that require javascript, right?

tamnd 4mo ago
Yeah, it is. For example, openai.com is rendered with Next.js, so I will try to mirror it tomorrow.

wolttam 4mo ago

One use I'd have for this is company wikis that you want to give folks easy offline access to (maybe the wiki has documentation that's useful at sites that don't have cellular coverage). Cool! It would be especially cool to have a version that didn't require the separate serving process - even though it's nifty you can package up a whole site as a single binary. Maybe a single HTML entrypoint shim with a bit of javascript that could index into an archive (potentially embedded) of the site's content?

tamnd 4mo ago
Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :) Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.
- mgiampapa 4mo ago
  I think the zim flow was perfect for offline use. I know I will be making use of it as soon as I can figure out how to pass chrome the cookies so I can be signed into the site. Didn't see it in the page, but I didn't look closely yet.
  - tamnd 4mo ago
    Not yet supporting cookies, since I created this tool for shadowing public websites first. I will add options to pass cookies later. It will pass them to the underlying Chrome/Chromium process, so it should not be hard to do.
- smeej 4mo ago
  I would use the shit out of this. I'm a heavy user of Logseq (OG, the md file-based version). Would LOVE to save my favorite web resources this way.
- mcdonje 4mo ago
  Not to load you up with too many ideas, but a markdown folder sounds a lot like obsidian, which has a plugin system now. Epub would also be a great target.

grahamstanes17 4mo ago

nice

dimiprasakis 4mo ago

Neat project, I like the idea. One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on! In any case, cool stuff :)

nikisweeting 4mo ago
--no-sandbox is needed in docker, maybe they assume it will mostly run in docker?
- tamnd 4mo ago
  Exactly. For downloading, Kage requires Chrome or Chromium. Running it inside Docker makes setup easier and keeps cleanup simple: https://github.com/tamnd/kage/blob/main/Dockerfile https://github.com/tamnd/kage/blob/main/Dockerfile Btw, let me think the way to only enable this when running inside Docker.
  - nikisweeting 4mo ago
    Docker is designed to be undetectable by default, the best way I have found is to set env IN_DOCKER=True manually in your Dockerfile + check that there is no $DISPLAY configured + that you're on linux. Usually if all/most of those are true you can safely add --no-sandbox --disable-setuid-sandbox --disable-dev-shm-usage etc. all the docker-specific flags. Thats what we do in https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile#L46 https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile...
    - tamnd 4mo ago
      It should be fixed by https://github.com/tamnd/kage/pull/12 https://github.com/tamnd/kage/pull/12 Thanks for nice trick.
    - dimiprasakis 4mo ago
      Cool approach. But, a compromise still lands on host's kernel, Docker doesn't provide kernel isolation (well it does on a macOS because it runs in Docker machine but thats a side effect). I wonder if a better solution would be to play with seccomp or Linux capabilities so that Chrome is sandboxed even in Docker. Not sure how this would work tbh. Answering here to get ideas, I saw your fix on Git and request for feedback (will try to review and give it some thought once I find some time)

daviding 4mo ago

Nice idea! fwiw, false positives and all, but the Windows 11 default Windows Security doesn't like it: `leakless.exe: Operation did not complete successfully because the file contains a virus or potentially unwanted software.`

mococa 4mo ago

curl can do this

Igor_Wiwi 4mo ago

This is quite useful tool, especially for the cases where internet access is limited (the flights for example). I implemented it as a separate feature in mdview.io: for example you can export a document as a html file for offline usage, with all the presentation features like reach tables, mermaid and etc built in. Example https://mdview.io/s/why-markdown-became-default-format-for-ai https://mdview.io/s/why-markdown-became-default-format-for-a... then try to Export - Export HTML

deleted 4mo ago

[deleted]

deleted 4mo ago

[deleted]

telesilla 4mo ago

I've been using httrack (https://www.httrack.com https://www.httrack.com) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.

throwaway219450 4mo ago
Specifically for wikis, is there a reason you wouldn't use Kiwix? For non "official" releases it's more complicated, but there are some services to generate the ZIM files. The desktop reader app is pretty good in my experience. https://wiki.openzim.org/wiki/Build_your_ZIM_file https://wiki.openzim.org/wiki/Build_your_ZIM_file EDIT: https://get.kiwix.org/en/solutions/applications/kiwix-reader/ https://get.kiwix.org/en/solutions/applications/kiwix-reader...
- telesilla 4mo ago
  Thanks, never knew about this and great to hear about it.
- tamnd 4mo ago
  Kiwix has readers for almost every platform, Android, desktop, iPhone. That's why I made Kage produce ZIM file. The executable file is mostly for people who don't have Kiwix installed yet, or just want to run the archive directly.
nikisweeting 4mo ago
https://github.com/archiveteam/grab-site https://github.com/archiveteam/grab-site or browsertrix may be easier to use for some, it's what was used to save a lot of the data.gov stuff before it got taken down.
tamnd 4mo ago
This brings back memories. Around twenty years ago, internet was still expensive dial-up, so I used to go to an internet cafe, run HTTrack to download websites and manga, copy everything onto my tiny 128MB USB stick (felt very large at that time), then bring it home and read offline ;))

latexr 4mo ago

For those with an eReader, one thing that works really well is using pandoc to download and convert a webpage to EPUB that you can then load to your reader. pandoc --from html --to epub --output /PATH/TO/FILE.epub https://example.com

arikrahman 4mo ago
Thanks, will try this out on the Kobo later.

ninalanyon 4mo ago

> kage serve $HOME/data/kage/paulgraham.com If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like: $ firefox $HOME/data/kage/paulgraham.com Then the result would be useable on machines without kage nstalled.

doctoboggan 4mo ago
Usually JavaScript is blocked when you load pages that way.
- pixelatedindex 4mo ago
  I thought all the JS was stripper?
- embedding-shape 4mo ago
  Since when? You won't be able to make HTTP requests to localhost, as it'd be a different Origin, but I don't think any mainstream browser blocks JS outright when you use file:// to load and view HTML files.
  - rzzzt 4mo ago
    Somewhere around 2019, each document loaded from file:// became its own origin in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1500453 https://bugzilla.mozilla.org/show_bug.cgi?id=1500453 (I didn't check when this happened in Chromium) Related WHATWG discussion: https://github.com/whatwg/html/issues/3099 https://github.com/whatwg/html/issues/3099
    - embedding-shape 4mo ago
      Yeah, but that's fine, the document is .html, and it can load ./app.js or ./style.css just fine even if loaded by file:// (as long as it isn't initiated by JS itself, then Origin starts to matter a lot more), otherwise basically every single local HTML file would suddenly be broken, I don't think anyone would have accepted that even with the origin changes.

simonw 4mo ago

I was intrigued to see how the demo GIF in the README was generated: https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c630610112b211964d/docs/static/demo.gif https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif https://github.com/tamnd/ascii-gif The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c630610112b211964d/docs/demo/kage.tape https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it: ascii-gif render docs/demo/kage.tape -o docs/static/demo.gif Looks like it's an opinionated wrapper around https://github.com/charmbracelet/vhs https://github.com/charmbracelet/vhs

alterom 4mo ago
FYI, on other platforms (Windows/MacOS), LiceCAP is a fantastic tool to record screen into compact GIFs by the author of Winamp and Reaper DAW: https://www.cockos.com/licecap/ https://www.cockos.com/licecap/
jubilanti 4mo ago
Have you heard the good news about the terminal savior asciinema -- https://asciinema.org/ https://asciinema.org/
- embedding-shape 4mo ago
  It's a cool tool/platform, but very different. Asciinema tries to make the "multimedia" itself better by making it actual text instead of being video/images, while the CLI command above turns actual text into multimedia supported by platforms already. Both are useful, both have their use cases :)
stavros 4mo ago
VHS is fantastic for scripting cli video generation.
vqtska 4mo ago
You can also do an animated svg which is way smaller than a gif because it's just text keyframes (https://github.com/vytskalt/pseudoc/blob/main/assets/factorial.svg https://github.com/vytskalt/pseudoc/blob/main/assets/factori...)

chinnyys 4mo ago

The readme is AI slop, and incredibly grating to read. The disgust I felt while reading it almost put me off trying the project. Is the code also AI slop?

shinryuu 4mo ago

Reminds me of this. https://gwern.net/gwtar https://gwern.net/gwtar Compared to that is there anything kage does better?

nitotm 4mo ago

I was looking for something like this the other day, it can be very helpful.

soulofmischief 4mo ago

Cool project! I know it's written in go, but it would be cool to see something like this which uses Cosmopolitan Libc + redbean or something similar to create a binary which runs anywhere. Would be fun to be able to pass around self-executable website archives. https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan https://justine.lol/cosmopolitan/index.html https://justine.lol/cosmopolitan/index.html https://redbean.dev https://redbean.dev (Certificates just expired for justine's website, just ignore the warning.)

tamnd 4mo ago
This could be a nice code golf project. It only needs a webview, a ZIM reader, and a way to append data to an existing binary and read it back. I did something like that a very long time ago (Of course, I have forgotten)
jokethrowaway 4mo ago
I never understood the appeal for cosmopolitan. I'd rather have platform specific minimal binaries than a single binary with hacks. Installing packages is a solved problem
- soulofmischief 4mo ago
  Installing packages is a completely different activity than passing around self-executable archives among friends. Not everything needs to go through a CI pipeline and distribution platform before you can share it with others. On top of that, I really enjoy being able to write quick little utilities and then pass them around without worrying about what operating system anyone who stumbles upon it has. It's fine if you don't personally find it useful for your workflow, but I think it's mad cool, especially since you can zip together multiple binaries into one, along with data.

KellyCriterion 4mo ago

Sounds like .MCH-files re-invented? (-:

Onavo 4mo ago

How does it handle websites with client side paywalls? Can you run it with extensions like bypass paywalls and ublock origin?

coffeecoders 4mo ago

I've accumulated a bunch of old website archives over the years. The funny thing is the ugly HTML dumps have been more useful than the "perfect" archive. It's one of the reasons I've become a bigger fan of RSS over time. A feed from 10-ish years ago is often more usable today than a carefully preserved (application) website.

tamnd 4mo ago
I have a project for creating and archiving RSS feeds, keeping the full history from the time the crawler starts. I need to clean up a bit, then will open source it soon.
couscouspie 4mo ago
Maybe it is just me, but by far most of the time, when I want to archive something from the internet, it is information and information is best served in an absolutely minimal text format like html or md.

cynicalsecurity 4mo ago

Binary app is a really bad way of storing data. No one would ever want to run a binary shared with them or found online.

tamnd 4mo ago
For sharing, better use the html folder or zim format, Kage supports both of them.
sablekit 4mo ago
[flagged]

calrizien 4mo ago

Does this work for the Apple Docs website? Really tricky to get those offline.

tamnd 4mo ago
Making docs available offline was one of my main motivations for building this tool. I will try Apple Docs too. I previously downloaded the Snowflake docs, and it was something like tens or even hundreds of thousands of pages, I do not remember exactly. The output ended up being very large. By the way, I forgot to add zstd compression support to my ZIM reader/writer. I will implement that in the next version.
tamnd 4mo ago
Good news for you: here is the command to clone Apple Docs: ```bash bin/kage clone https://developer.apple.com/documentation/ https://developer.apple.com/documentation/ \ --scope-prefix /documentation/ \ --out /Users/apple/data/apple-docs \ --chrome "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \ --max-pages 0 --max-depth 0 \ --workers 3 --browser-pages 3 --asset-workers 6 \ --render-timeout 60s --settle 2s --timeout 30s \ 2>&1 | tee -a /Users/apple/apple-docs.log ``` Adjust it to your needs :) I smoke-tested it, and all the content and CSS work, but I stripped all the JS, so the sidebar won't work. If you run into any problems, feel free to create new issues in the repo. It helps me prioritize and know what should be fixed.

k4rnaj1k 4mo ago

[dead]

sneak 4mo ago

The README is LLM slop. This makes me assume the code is the same.

chfritz 4mo ago

how is this different from using puppeteer to load the page and save the DOM as HTML?

kadhirvelm 4mo ago

This is awesome, we wanted an offline copy of someone’s prototype (as built on Lovable, etc) so we could do version control and sharing in an easier format. Wrote our approach here: https://productnow.ai/blogs/extracting-html-from-ai-prototyping-tools https://productnow.ai/blogs/extracting-html-from-ai-prototyp... But will look into this now, see if we can swap some stuff out. We’ve really liked the idea of an offline mirror, makes a lot of collaboration use cases simpler

jyscao 4mo ago

I tried to clone a HTTP (not HTTPS) site, and it's giving me `navigation failed: net::ERR_NAME_NOT_RESOLVED`. Even when I explicitly included the protocol with `http://<FQDN>`.

godot 4mo ago

the readme uses paulgraham.com as an example (which is text articles mostly) and I never use "Save As" for a web page (for the reasons the author states), I always just print as PDF and save the PDF file. for an entire website though of many pages I can see this can be useful.

amatecha 4mo ago

Suddenly remembering the days of dialup and your browser serving a fully-functional cached copy of a webpage when you try to access it and you're not online...

eventinbox 4mo ago

[flagged]

G_o_D 4mo ago

How its different then MHTML ??

rickylin 4mo ago

It seems like https://github.com/tw93/pake https://github.com/tw93/pake is better.

italiancheese 4mo ago
Both of these projects have completely different purposes and use cases. Have you even read the first line of the readme of the project you're commenting on?

smusamashah 4mo ago

What if I wanted to download all Confluence docs at work?

carsonye 4mo ago

This is interesting. Is the intended use case mostly read-only websites like blogs/docs/essays? How well does it handle sites where navigation, search, dropdowns, or other UI interactions depend on JavaScript?

tamnd 4mo ago
Currently, all of that is broken. At one point, I had a traumatic experience where an archived HTML file kept redirecting to the live site, even though I already had all the content rendered, so I ended up disabling all JavaScript entirely.
tamnd 4mo ago
If there's more demand for that, maybe I will implement a more relaxed version.

jokethrowaway 4mo ago

Amazing stuff! I would recommend an add-on or new feature to detect and remove cookie banners / annoying popups that open on load (eg. sign up to my mailing list). listing a few examples form fastText could help you. You might also have the opposite problem though: some websites have content in the base html (so it's searchable by Google and they get views) and remove it on load (so you have to pay). Capturing the initial html and comparing it to the final version could give you some hints and allow you to repair the removed content. Best of luck with the project!

sails 4mo ago

What is the best way to give coding agent a full website so that it can see what I see? With animation and design I’m never sure what it gets when I save the website in the browser. Maybe this is suitable?

endorphine 4mo ago

Anyone remembers Teleport Pro?

xlii 4mo ago

> No tracking, no network calls, no surprises. Won't comment on a project (though idea seems interesting) but this in README is a tell for me ;)

xd1936 4mo ago
It's not just no tracking — It's no surprises.
praveer13 4mo ago
Somehow 'Kage' is the first name claude suggests to me for any new project as well
- timerol 4mo ago
  The Japanese word for shadow?

c7b 4mo ago

Probably a stupid question, but could this archive embedded videos as well?

tamnd 4mo ago
Possible, but currently I disable all large files, including videos. For video downloading, I suggest wrapping around yt-dlp. It's an awesome tool.

Sathwickp 4mo ago

I'm still trying to cope with your github profile, 68k commits a year is crazyy

aa-jv 4mo ago

I've been using "Print to PDF" as my principle bookmarks management tool, since 1998, and I have over 90,000+ such PDF's sitting on my system, easily re-read and discovered. So I don't quite get whats the point of kage? What does it do that print-to-PDF won't already do? The resulting .pdf's contain all the content, and also include the original URL and creation date, etc. How is kage an improvement?

snowflaxxx 4mo ago

Meet Teleport Pro

netdevphoenix 4mo ago

[dead]

ekianjo 4mo ago

Curious about "keep it for a decade" claim. Can something possibly break down the road?

kjmh 4mo ago

I was floored by the idea of browsing docs offline but disappointed that recreating the demo of archiving Paul Graham’s essays gave me a ZIM with broken images and broken Unicode symbols when viewed in Kiwix.

Departed7405 4mo ago

What's the advantage compared to mhtml?

ralferoo 4mo ago

It sounds like a nice idea, but "It drives a real browser, lets the page finish doing whatever it does, grabs the finished result, and then rips every script out of it." sounds like it'd fail for a lot of websites that do things like display some kind of modal banner inviting you to click something to close it, and that prevents scrolling while that is open, whilst also delaying loading the rest of the article until a certain scroll point is hit.

deleted 4mo ago

[deleted]

b0dhimind 4mo ago

I've had a subscription to this site: app.dancevision.com for a while. It requires cookies to login and view and each video (3 directories down) is hosted on Vimeo. Could Kage actually do an offline mirror of the site (and the videos)?

aynite 4mo ago

nice job and good inspiration, i'm just looking for something like this for some local processing

aivanise 4mo ago

Was a bit disappointed when I realized that .zim files do not have an index and thus are not searchable. Limits a lot of use cases, e.g. I wanted to crawl my corporate wiki to have a wiki to know how to fix wiki when wiki is down ;)