5 ms·

From the the project page it says: > The offline content is stored in ~/.cache/offpunk/ as plain .gmi/.html files. The structure of the Gemini-space is tentati

From the the project page it says: > The offline content is stored in ~/.cache/offpunk/ as plain .gmi/.html files. The structure of the Gemini-space is tentatively recreated. One key element of the design is to avoid any database. The cache can thus be modified by hand, content can be removed, used or added by software other than offpunk. One ambition I have it to setup https://github.com/davidfstr/webcrystal https://github.com/davidfstr/webcrystal > An archiving HTTP proxy and on-disk archival format for websites. so that all my regular web browsing is auto archived at some level. It would sure be neat if the archive formats could be compatible. It would allow for a setup where everything I’ve seen with my eyes is then immediately accessible programmatically or in a terminal. I feel that could open some significant productive advantages, especially in the age of LLMs also in the terminal.

ploum 3y ago
Offpunk author here: the goal of "netcache" is to allow access to the cache by more tools. It would be quite easy to build a proxy like webcrystal but for any URL. Something like https://localhost:666/news.ycombinator.com/ https://localhost:666/news.ycombinator.com/
- alchemist1e9 3y ago
  I’ll confess to two crimes a) I’ve come across offpunk at least 3 times already and haven’t tried it yet, it looks really neat! b) I often have ideas for interesting hacks that I write down but then drop the ball on. This is one of them but in case you’re interested and it by chance helps with the awesome work you are doing. Here is my vision: - mitmproxy is running in transparent mode with a plugin/custom code that does continuous archiving on all traffic, perhaps with help of your netcache tool instead of webcrystal I mentioned. it’s set as the gateway. - terminal user can then retrieve quickly history list, fzf select, page/dump text for cut and paste, or perhaps some direct to LLM tools pipeline, for summarizing or extracting. Perhaps I’m stating the obvious above and can be implied by my prior comment. I guess the one detail I’m adding is I think using mitmproxy transparent mode it could be completely transparent to the regular desktop user. No per website configs, no proxy settings. I run my desktops in a VM using vfio-pci GPU passthrough on the physical workstations and then put all my traffic through another pfSense VM, so re-routing traffic for me is already a fundamental part of my setup. Perhaps this all sounds a bit complicated and yes it is. But there are numerous reasons I feel it can have a huge payoff. Thank you for your open source work and I hope I’ll have some time soon to play around with offpunk. It looks like a fantastic tool with many potential uses.
- karmakaze 3y ago
  Is (or could) Offpunk be made as two parts: the browser and persistent caching proxy? The persistence of the proxy could be enabled and any content browsed by any browser cached. There could be issues with modern web/js pages not being able to naively run from cache, so safer to store the rendered documents.
  - ploum 3y ago
    It is made of 3 different parts: - netcache (caching and network) - ansicat (terminal rendering) - offpunk (browsing) So you could use only the cache part
livrem 3y ago
I used to daydream about having a web proxy that could store every page I visited (instead of having to manually save interesting pages... something I do a lot). But I never had the storage space for that, and the bloat in web pages has grown faster than the size of disk I can afford. Since I started using Offpunk for Gemini some time ago I at least get a complete saved record of all the Gemini pages I read. 65 MB in one year. Far more realistic to maintain than with web pages.
- alchemist1e9 3y ago
  > But I never had the storage space for that, and the bloat in web pages has grown faster than the size of disk I can afford. Is this really true for you? Seems surprising to me given how cheap large HDD storage is now. I have trouble believing even bloated web pages are that large relatively speaking. I guess I should try to do it and find out, probably I’m wrong and it’s much more data than I’m expecting. Gemini seems really neat, I should have investigated earlier. ssh kiosk@gemini.circumlunar.space Let’s one get a sense of what it is about. 1 bookmarks found me some spaces.
  - ploum 3y ago
    Two years of using nearly exclusively offpunk without ever trimming the cache: 2.5G gemini 1.3G gopher 1.5G http 23G https With the exception of dynamic webpages, every single page I’ve read in those two years is there. With pictures. And with every single page linked by those webpages. And all the pages linked on HN.
    - alchemist1e9 3y ago
      That’s very awesome and by itself a compelling argument for one to consider adopting offpunk. I’d imagine putting the archive in git repo with annex/lfs and along with builtin time-stamping it means an activity archive also. Lot’s of interesting use cases if combined with LLMs and RAG for example. “A few weeks ago I was researching technology XYZ and there was an open source python package that looked neat, but I can’t remember what it was. Could you use my browsing cache and prepare a report on what I was reading about and summarize it please?”
unforgivenpasta 3y ago
I've looked into archiving all the pages i visit as well and warcprox[1] has been bookmarked for a while now Hard drive storage space being so cheap in the ~$15/TB range makes this more feasible even for video archival [1] https://github.com/internetarchive/warcprox https://github.com/internetarchive/warcprox
- alchemist1e9 3y ago
  Excellent pointer with warcprox, I hadn’t seen it. I’m noticing mitmproxy, warcprox, webcrystal, and also obviously offpunk are all python. It seems there should be some mashup of them all that can produce a solution. One that also involves using offpunk to access the archive in the terminal. Mitmproxy caught my eye with transparent mode [1] and the idea that the client/user VM may not even need configuration in my setup, the vfio-pci GPU passthough desktop OS approach. The archiver VM produced archive/cache could just be NFS mounted over a private bridge interface between the desktop VM and archive VM. [1] http://docs.mitmproxy.org.s3-website-us-west-2.amazonaws.com/dev/howto-transparent-vms/ http://docs.mitmproxy.org.s3-website-us-west-2.amazonaws.com...
  - ploum 3y ago
    You can now start accessing the raw cache with "netcache --offline". Or you can access it by hand: the cache is only made of files stored in folders.

ploum 3y ago

Offpunk author here: the goal of "netcache" is to allow access to the cache by more tools. It would be quite easy to build a proxy like webcrystal but for any URL. Something like https://localhost:666/news.ycombinator.com/ https://localhost:666/news.ycombinator.com/

alchemist1e9 3y ago
I’ll confess to two crimes a) I’ve come across offpunk at least 3 times already and haven’t tried it yet, it looks really neat! b) I often have ideas for interesting hacks that I write down but then drop the ball on. This is one of them but in case you’re interested and it by chance helps with the awesome work you are doing. Here is my vision: - mitmproxy is running in transparent mode with a plugin/custom code that does continuous archiving on all traffic, perhaps with help of your netcache tool instead of webcrystal I mentioned. it’s set as the gateway. - terminal user can then retrieve quickly history list, fzf select, page/dump text for cut and paste, or perhaps some direct to LLM tools pipeline, for summarizing or extracting. Perhaps I’m stating the obvious above and can be implied by my prior comment. I guess the one detail I’m adding is I think using mitmproxy transparent mode it could be completely transparent to the regular desktop user. No per website configs, no proxy settings. I run my desktops in a VM using vfio-pci GPU passthrough on the physical workstations and then put all my traffic through another pfSense VM, so re-routing traffic for me is already a fundamental part of my setup. Perhaps this all sounds a bit complicated and yes it is. But there are numerous reasons I feel it can have a huge payoff. Thank you for your open source work and I hope I’ll have some time soon to play around with offpunk. It looks like a fantastic tool with many potential uses.
karmakaze 3y ago
Is (or could) Offpunk be made as two parts: the browser and persistent caching proxy? The persistence of the proxy could be enabled and any content browsed by any browser cached. There could be issues with modern web/js pages not being able to naively run from cache, so safer to store the rendered documents.
- ploum 3y ago
  It is made of 3 different parts: - netcache (caching and network) - ansicat (terminal rendering) - offpunk (browsing) So you could use only the cache part

livrem 3y ago

I used to daydream about having a web proxy that could store every page I visited (instead of having to manually save interesting pages... something I do a lot). But I never had the storage space for that, and the bloat in web pages has grown faster than the size of disk I can afford. Since I started using Offpunk for Gemini some time ago I at least get a complete saved record of all the Gemini pages I read. 65 MB in one year. Far more realistic to maintain than with web pages.

alchemist1e9 3y ago
> But I never had the storage space for that, and the bloat in web pages has grown faster than the size of disk I can afford. Is this really true for you? Seems surprising to me given how cheap large HDD storage is now. I have trouble believing even bloated web pages are that large relatively speaking. I guess I should try to do it and find out, probably I’m wrong and it’s much more data than I’m expecting. Gemini seems really neat, I should have investigated earlier. ssh kiosk@gemini.circumlunar.space Let’s one get a sense of what it is about. 1 bookmarks found me some spaces.
- ploum 3y ago
  Two years of using nearly exclusively offpunk without ever trimming the cache: 2.5G gemini 1.3G gopher 1.5G http 23G https With the exception of dynamic webpages, every single page I’ve read in those two years is there. With pictures. And with every single page linked by those webpages. And all the pages linked on HN.
  - alchemist1e9 3y ago
    That’s very awesome and by itself a compelling argument for one to consider adopting offpunk. I’d imagine putting the archive in git repo with annex/lfs and along with builtin time-stamping it means an activity archive also. Lot’s of interesting use cases if combined with LLMs and RAG for example. “A few weeks ago I was researching technology XYZ and there was an open source python package that looked neat, but I can’t remember what it was. Could you use my browsing cache and prepare a report on what I was reading about and summarize it please?”

unforgivenpasta 3y ago

I've looked into archiving all the pages i visit as well and warcprox[1] has been bookmarked for a while now Hard drive storage space being so cheap in the ~$15/TB range makes this more feasible even for video archival [1] https://github.com/internetarchive/warcprox https://github.com/internetarchive/warcprox

alchemist1e9 3y ago
Excellent pointer with warcprox, I hadn’t seen it. I’m noticing mitmproxy, warcprox, webcrystal, and also obviously offpunk are all python. It seems there should be some mashup of them all that can produce a solution. One that also involves using offpunk to access the archive in the terminal. Mitmproxy caught my eye with transparent mode [1] and the idea that the client/user VM may not even need configuration in my setup, the vfio-pci GPU passthough desktop OS approach. The archiver VM produced archive/cache could just be NFS mounted over a private bridge interface between the desktop VM and archive VM. [1] http://docs.mitmproxy.org.s3-website-us-west-2.amazonaws.com/dev/howto-transparent-vms/ http://docs.mitmproxy.org.s3-website-us-west-2.amazonaws.com...
- ploum 3y ago
  You can now start accessing the raw cache with "netcache --offline". Or you can access it by hand: the cache is only made of files stored in folders.