12 ms·

There's no reason why a web browser bookmark action doesn't automatically create a WARC (web archive) format. Heck, with the cost of storage so low, recording

• railmeat 7y ago
  This comment misses the point. The point is to make a webpage that lasts. So people can link to it and get the page. That means making a maintainable webpage and a url that does not change. It is great that you can archive every page you visit for yourself, but that is not the same as making a lasting web. Lets make something that others can use too.
• arjie 7y ago
  In practice how is this different from MHTML? I think most browsers have built-in support for MHTML so it should be possible to build that part easily.
  • est31 7y ago
    According to the MHTML entry on Wikipedia, Chrome requires an extension, Firefox doesn't support it, and only Internet Explorer supports MHTML.
    • scrollaway 7y ago
      I mean, it's not any worse than WARC support…
      • est31 7y ago
        Maybe MAFF's are best as they use compression instead of base64 encoding: https://en.wikipedia.org/wiki/Mozilla_Archive_Format https://en.wikipedia.org/wiki/Mozilla_Archive_Format
        • deleted 7y ago
          [deleted]
        • gildas 7y ago
          Or SingleFileZ files which can be viewed without installing any extension https://github.com/gildas-lormeau/SingleFileZ https://github.com/gildas-lormeau/SingleFileZ. Edit: it can also auto-save pages, like SingleFile.
    • mrspuratic 7y ago
      I used the excellent unMHT plugin for Firefox, but it got dropped some time ago, failed to meet "enhanced security requirements" :( I still keep an old ESR with this plugin for archiving, and accessing MHTs.
  • Laforet 7y ago
    The state of mhtml support is fairly pathetic at the moment. Firefox broke mhtml compatibility with the quantum overhaul. Chrome's mht support had been a hit and miss over the years, sometimes removing the GUI option entirely and requiring one to manually launch the browser with a special tag to enable it. The only browser with a history of consistent mhtml support happens to be....Internet Explorer, followed by a bunch of even more obscure vendors that nobody really uses. I am currently dealing with the problem of parsing large mht files (several megabytes and up). A regular web browser would hang and crash upon opening these files and most ready made tools I could find struggle with the number of embedded images. It's very much a neglected format with very little support in 2019.
    • deleted 7y ago
      [deleted]
• fouc 7y ago
  How is viewing the WARC after? Is it the same quality as archive.is/ or archive.org/ ?
  • edraferi 7y ago
    Pretty good, depending on the tooling. I’m having good luck with https://webrecorder.io/ https://webrecorder.io/ and their related open-source tools.
  • shakna 7y ago
    archive.org use WARC if I remember correctly, and offer guides on creating and reading the format.
  • cookiecaper 7y ago
    It's a pain, like most of the WARC ecosystem. It's been several months since I dug in, so maybe it's had some spitshine the last little bit, but I usually end up using combinations of wpull, grab-site, and a smattering of other utilities to reliably capture a page/set of pages, and have had to make some quick hacks as well as manually merging in some PRs to get things to work with Python3. Once I have the WARC, I typically end up using warcat to extract the contents into a local directory and explore that way. WARC as a format seems promising, but at least last I checked, open-source tooling to make it a pleasant and/or transparent experience is not really there, and worse, at least as of several months ago, doesn't really seem actively worked on. Definitely an area you'd expect to be further along.
• romwell 7y ago
  >There's no reason why a web browser bookmark action doesn't automatically create a WARC (web archive) format. Indeed. And I still remember the modem days where I would download entire websites because the ISP charged by the hour, and I'd read them offline to save money.
  • JoeSmithson 7y ago
    I can't put my finger on it but this has a sort of Dickensian quality to me. I think this says something kind of profound about information and capitalism and whatnot.
    • thomasz 7y ago
      No, it hasn't. Technology just wasn't there back then which caused significant cost per time unit, which makes it only fair to charge per time unit.
      • majewsky 7y ago
        Yeah. In the dialup days, layer 1 and 2 of a home internet connection was a long-running phone call between your own modem and a modem of an ISP. You payed via your phone bill, for the duration of the call.
    • alentist 7y ago
      It says scarce resources are pricier. Welcome to the real world!
• pvg 7y ago
  You can ask Safari to do that by enabling 'Reading List: Save articles for offline reading automatically'. It's not WARC but it is an offline archive. The shortcut is cmd-shift-D which is almost the same as the bookmark one. It's also the only way I know of to get Safari to show you bookmarks in reverse chronological order. And it syncs to iOS devices. This could be done in better and more specialized ways, one problem is browser extension APIs don't provide very good access to the browser's webpage-saving features.
• steveklabnik 7y ago
  https://www.gwern.net/Archiving-URLs https://www.gwern.net/Archiving-URLs describes extracting brower history to create an archive via a batch job.
• bloopernova 7y ago
  I actually this week asked the author of SingleFile if he could implement a save-on-bookmark feature for SingleFile, and he was amenable: https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile/issues/320 https://github.com/gildas-lormeau/SingleFile/issues/320
  • j88439h84 7y ago
    Nice. FYI there's also SingleFileZ > SingleFileZ is a fork of SingleFile that allows you to save a webpage as a self-extracting HTML file. This HTML file is also a valid ZIP file which contains the resources (images, fonts, stylesheets and frames) of the saved page. https://github.com/gildas-lormeau/SingleFileZ https://github.com/gildas-lormeau/SingleFileZ
    • gildas 7y ago
      I'll implement the "save bookmark page" feature in both extensions :)
      • bloopernova 7y ago
        oh, hello! It's funny how this subject has popped up again.
        • gildas 7y ago
          Hi! I think it confirms that there's a real interest in this feature.
          • SirYandi 7y ago
            Off topic, but could I ask how you knew your software was being talked about? Did you just happen by or have you some monitoring agent looking for mentions? Just curious
            • 7y ago
• kccqzy 7y ago
  Safari used to (and still) do this automatically but in a limited way. In the browsing history view (Command Y), you can search visited pages by its content, and this is extremely useful. But there's no way(†) to tell Safari to display that saved content. If you revisit a URL in the history, Safari fetches it again, losing the original saved content. (†): short of direct plist manipulation
  • saagarjha 7y ago
    Does it actually? I thought history was stored in a SQLite database and only kept the URL and page title.
• cosarara 7y ago
  Firefox actually purges history automatically. For instance, the oldest history I have on this browser right now is from January 2018. I found about this the hard way.
  • kccqzy 7y ago
    Safari does the same, even though I tell it to never clear browsing history.
    • Avamander 7y ago
      I think Chrome(ium) does as well. Very annoying tbh.
      • heavyset_go 7y ago
        Chrome was the first browser I encountered that deletes history without being instructed to.
        • Thorrez 7y ago
          It looks like Firefox has been doing it since 2010[1]. I wonder how long Chrome has been doing it, since launch, 2008? Here's a Chrome bug discussing it[2]. [1] https://web.archive.org/web/20151229082536/http://blog.bonardo.net/2010/01/20/places-got-async-expiration https://web.archive.org/web/20151229082536/http://blog.bonar... [2] https://bugs.chromium.org/p/chromium/issues/detail?id=500239 https://bugs.chromium.org/p/chromium/issues/detail?id=500239
  • 6510 7y ago
    Mosaic had full text history search.
  • nayuki 7y ago
    I noticed this behavior in Firefox too. So I started writing personal Python scripts to scrape FF's SQLite database where it stores all the browsing history information.
• tempestn 7y ago
  This was what made me convert from bookmarking to clipping pages into Evernote around 6-7 years ago. I realized I had this huge archive of reference bookmarks that were almost useless because 1) I could rarely find what I was looking for, if I even remembered I'd bookmarked something in the first place, and 2) if I did, it was likely gone anyway. With Evernote I can full text search anything I've clipped in the past (and also add additional notes or keywords to ease in finding or add reference info). Since starting with replacing bookmarks, I've moved other forms of reference info in there, and now have a whole GTD setup there as well, which is extremely handy since I can search in one place for reference info and personal tasks (past and future). Only downside is I'm dependent on Evernote, but hopefully it manages to stick around in some form for a good while, and if it ever doesn't, I expect I'll be able to migrate to something similar.
  • charlesdaniels 7y ago
    Shout out to https://joplinapp.org/ https://joplinapp.org/ I was an Evernote user when I was on macOS. When I switched to Linux, a proper web clipper was something I really missed. I'm now on Joplin and it does everything I used to use Evernote for and then some. It even has vim bindings now! As far as longevity goes, I think they got their archive / backup format right - it's just a tarball with markdown in it.
    • dragonsh 7y ago
      No need of proprietary code and apps why not build it into browsers. I have seen Firefox and Chrome can download web pages. So it will be nicer if they can download the bookmarked pages and store in a local html, css, image folder. I think it's pretty easy to achieve. Also people need to move away from those esoteric reactjs, angular, vuejs and plethora of CMS as API or static site generators relying on some js framework which won't last even 2-3 years. Use a static site generator which can generate a plain html, like static site generators built on pandoc, python docutils or similar. Personally I like restructuredText as the preferred format for content as its a complete specification and plain text. So the only thing in this article I will change is that content can also be in rst format and then generate html from it. Markdown is not a specification as each site implements their own markdown directives unlike restructuredtext specification and most of the parsers and tooling are little different from each other.
• sneak 7y ago
  It would be nice if we had browsers that were actually user-agents that allow full pluggable customizability for all cookie, header, UI, request, and history behavior. Then, this would just be a plugin that anyone could install.
  • Polylactic_acid 7y ago
    And then people install garbage extensions that break the browser and people think "Wow, this firefox browser is so buggy and slow" and switch to chrome. And then your extensions break with every single browser update because they are tampering with internal code. Everyone is free to fork a browser and apply any changes they want. Allowing extensions to change anything at all essentially is the same as forking and merging your changes with upstream every update.
    • sneak 7y ago
      Somehow that doesn’t seem to happen with editors; we have a ton of them and they are very customizable. I guess “with a reasonably stable hook api” was supposed to be implicit in my statement.
• reificator 7y ago
  This is what started me clipping everything to OneNote instead of bookmarking. Unfortunately, it becomes difficult to maintain, the formatting is off, things subtly break, pages clipped on mobile use different fonts for god knows what reason, some content is discarded silently because the clipper deems it's not part of the main article, I could go on. It's better than nothing but it's also increasingly frustrating to deal with.
• jimbo1qaz 7y ago
  WorldBrain's Memex (https://addons.mozilla.org/en-US/firefox/addon/worldbrain/ https://addons.mozilla.org/en-US/firefox/addon/worldbrain/) has an option to perform a full-text index (not archive) of bookmarks, or pages you visited for 5 seconds (default) down to 1 second (no option to index all pages). It stores this stuff into a giant Local Storage (etc) database, which Firefox implements as a sqlite file.
• boring_twenties 7y ago
  That would be even more useful if a search warrant is ever executed on my house. Not useful to me personally, but useful to someone!
• ryanfox 7y ago
  I’ve been building an application to do this, except for everything on your computer! It’s called APSE[0], short for A Personal Search Engine. [0] https://apse.io https://apse.io
  • hk__2 7y ago
    Having to pay $15/month ($180/yr!) to be able to search stuff on my own computer for years seems awfully expensive. I'd rather depend on some simple open-source piece of software that I can understand and maintain if necessary.
    • abraae 7y ago
      Decades ago there was an amazing piece of software from lotus when I worked there called magellan. I remember the first time I saw someone search, and find results in text documents, spreadsheets and many other of the common formats of the day. That was in 1989 and today I mostly search my computer using find and grep commands, since that's what just keeps working.
      • nukst 7y ago
        I should try adopting find and grep, but on Windows I'm currently using this and I'm very happy with it: https://www.voidtools.com/downloads/ https://www.voidtools.com/downloads/
        • Multicomp 7y ago
          I use Void Tools Everything to find files by name, and AstroGrep for finding information in them.
    • tasogare 7y ago
      Yeah, the sheer idea of paying a subscription for software that is running on my computer to index local resources is crazy. This kind of software should be should sold as one-time buy license.
    • 7y ago
• 3xblah 7y ago
  "Imagine having the last 30 years of web browsing history saved on your local machine." I believe the name for that experience was/is "Microsoft Windows".
• btrettel 7y ago
  Years ago I used the old Firefox addon Shelve to automatically archive the vast majority of web pages I visited. http://shelve.sourceforge.net/ http://shelve.sourceforge.net/ The main disadvantage was disk space. This is particularly true when some pages are 10 MB or larger. I would periodically prune the archive directory for this reason. I stopped using Shelve when I started running out of disk space, and now I can't use Shelve because the addon is no longer supported. The author of Shelve has some suggestions for software with similar functionality: https://groups.google.com/forum/#!topic/shelve-firefox-addon/CSjDYj89Txw https://groups.google.com/forum/#!topic/shelve-firefox-addon...
  • dbtx 7y ago
    It installs and loads in Waterfox, but of course it still hasn't been touched in 3.5 years. I used Scrapbook in the past (which also still works) but I usually just save random things in ~/webpages/ since (apparently) 2011. The earliest is a copy of the landing page at bigthink.com. Of course now almost every link is broken, excluding social media buttons, About Us, Contact Us, RSS, Privacy, Terms of Use, Login, and the header logo pointing at the same page.
• chaostheory 7y ago
  This problem has been solved a long time ago if you use Pinboard. https://pinboard.in/ https://pinboard.in/ Just pay the yearly subscription so pinboard can cache your bookmarks.
  • simongr3dal 7y ago
    Does pinboard caching work with sites that are behind a login or paywall?
    • L_Rahman 7y ago
      It does not. See FAQ here: https://pinboard.in/upgrade/ https://pinboard.in/upgrade/
      • larrywright 7y ago
        This is the advantage to Evernote. Since it’s a browser extension, it has access to anything you have access to.
        • majewsky 7y ago
          The downside is, since it’s a browser extension, it has access to anything you have access to.
          • larrywright 7y ago
            Agreed that there’s a tradeoff. I don’t think there’s really an alternative solution though.
  • akho 7y ago
    I do not see how using a web thing is a solution to web things going away.
    • L_Rahman 7y ago
• 3xblah 7y ago
  Better than a bookmark action would be a commandline option, similar to Firefox's -screenshot, which will work without starting X11. Something like -archive:warc
• soulofmischief 7y ago
  I use the Zotero extension for this feature. https://www.zotero.org/ https://www.zotero.org/
  • eavotan 7y ago
    came here to say this. zotero also saves metadata as an extra. as long as it is used from within a browser.
• pjmlp 7y ago
  I always save pages instead of only bookmarking them. Most websites I used to visit during demoscene high days, are now gone.
• jakeogh 7y ago
  Personal wayback machins should be standard computing kit. I have had one since around 2013. Very bare bones demo: https://bpaste.net/show/3FBH6 https://bpaste.net/show/3FBH6 it does much more than that. file:// is supported for example, so you can recursively import a folder tree, and re-export it later if you wanted to. Or in some random script: "from iridb import iri_index" "data = iri_index.get('https://some/url')" https://some/url')" I'm skipping lots, you can ref by hash, url, url+timestamp. It hands back a fh, you dont know if the data you are reffing even fits in memory. Extensive caching, all the iri/url quirks, punycode, PSL etc. Some random pdf in ~/Downloads, "import doc.pdf" and dmenu pops up, you type a tag, hit enter and the pdf disappears into the hash tree, tagged, and you never need to remember where you put it. Later on you only need to remember part of the tag, and a tag is just a sequence of unicode words. Chunks are on my github (jakeogh/uhashfs, it's heka-outdated dont use it yet), I'll be posting the full thing sometime soonish.
• temporaryvector 7y ago
  >Heck, with the cost of storage so low, recording every webpage you ever visit in searchable format is also very realistic. I tend to do that, I also save a lot of scientific papers, ebooks and personal notes. I've found that doing so does not help me at all. The main problem I have is that when I need to look something up (an article, a book, a bit of info) I reach for google first, usually end up finding the answer and go to save it, only to find that I had already found the answer beforehand (and perhaps already made clarifying notes to go along with it) and then forgot about it. This, and not dead links, is the fundamental problem with bookmarks for me. Not only bookmarks, it extends to my physical notes and pretty much everything I do. If I haven't actively worked on something for a couple of months, I forget all about it and when I come back to it I usually have to start from scratch until I (hopefully) refresh my memory. Some of it is also usually outdated information. I think this is a big, unsolved problem and I'm not even sure how to go about starting to solve it. I can envision some form of AI-powered research assistant, but only in abstract terms. I can't envision how it would actually work to make my life better or easier. It would need to be something that would help blur the line between things I know and things that are on my computer somehow. If I think of my brain like it has RAM and cache, things I'm working on right now are in the cache and things I've worked on recently or work on a lot are in RAM, but what's for me lacking is a way to easily move knowledge from my brain-RAM to long term storage and then move that knowledge back into working memory faster than I can do so now. I'm not even talking about brain uploading or mind-machine interfaces, but just something that can remind me of things I already know but forgot about faster than I can do so by myself. I am convinced that figuring out how to do this will lead to the next leap in technological development speed and efficiency. Not quite the singularity that transhumanists like to talk about, but a substantial advancement.
  • hikarudo 7y ago
    I have exactly the same problem. What I've found is that I need to spend more time deciding what is important, and less time consuming frivolous information. That's hardly a technology problem. For things I really don't want to forget, I'm using Anki [0], a Spaced Repetition System (SRS). Anki is supremely good at knowing when you're about to forget an item and prompting you to review it. Spaced practice and retrieval practice, both of which are used in SRS, are two learning techniques for which there is ample evidence that they actually work [1]. You still need to decide what is worth remembering, but that's something technology can't help with, I think. [0] https://apps.ankiweb.net/ https://apps.ankiweb.net/ [1] https://www.learningscientists.org/ https://www.learningscientists.org/
• BiteCode_dev 7y ago
  I like the idea, but wanted to know how realistic it would be so I made a quick and dirty Python script to download all my bookmarks. If you want to make the same experiment, you can get it from here: https://gist.github.com/ksamuel/fb3af1345626cb66c474f38d8f037405 https://gist.github.com/ksamuel/fb3af1345626cb66c474f38d8f03... It requires Python 3.8 (just the stdlib) and wget. I have 3633 bookmarks, for a total of 1.5 Go unziped, 1.0 Go zipped (and we know we can get more from better algo and using links to files with the same checksum like for JS and css deps). This seems acceptable IMO, espacially since I used to consider myself a heavy bookmarker and I was stunned by how few I actually had and how little disk they occupied. Here are the types of the files: 31396 text/plain 3034 application/octet-stream 1316 text/x-c++ 1123 text/x-po 865 text/x-python 384 text/html 227 application/gzip 218 inode/x-empty 178 text/x-pascal 113 image/png 44 application/zlib 29 text/x-c 28 text/x-shellscript 14 application/xml 13 application/x-dosexec 12 text/troff 5 text/x-makefile 4 text/x-asm 3 application/zip 2 image/jpeg 2 image/gif 2 application/x-elc 1 text/x-ruby 1 text/x-diff 1 text/rtf 1 image/x-xcf 1 image/x-icon 1 image/svg+xml 1 application/x-shockwave-flash 1 application/x-mach-binary 1 application/x-executable 1 application/x-dbf 1 application/pdf It should probably be opt-in though, like a double click on the "save as bookmark icon" to download the entire file, and the star becomes a different color. Mobile phones, chrome books and raspy may not want to use the spaces, not to mention there are some bookmark content that you don't want your OS to index, and show you preview of in every search. But it would be fantastic: by doing this experiment I noticed that many bookmarks were 404 now, and I will never get their content back. Beside, searching bookmark, and referencing them is a huge pain. So definitely something I wish mozilla would consider.
  • asdasdasdasdwd 7y ago
    > definitely something I wish mozilla would consider There used to be this neat little extension called Read It Later that let you do just that. Bookmark and save it so you could read it when you were offline or the page disappeared. Later they changed their name and much later Mozilla bought it and added it to Firefox without a way to opt out. It was renamed to Pocket.
• franga2000 7y ago
  I've actually been saving every page I visit for a good two years now and it has barely caused in dent in my NAS storage space. As usual though, I wrote a crappy extension and Python script to do that because I never bothered to look online. Thanks for introducing me to WarcProxy - I'll probably be making the switch very soon.
• crazypython 7y ago
  I use DEVONThink on my mac, which has web archiving, full-text search, and auto-categorization.
• dredmorbius 7y ago
  Yes so much this. There are a few issues to consider: - Any comprehensive archive of your activity is itself going to be a tremendously "interesting" resource for others -- advertisers, law enforcement, business adversaries, and the like. Baking in strong crypto and privacy protections from the start would be exceedingly strongly advised. - That's also an excellent reason to have this outside the browser, by default, or otherwise sandboxed. - Back when I was foolish enough to think that making suggestions to Browser Monopoly #1 was remotely useful, I pointed out that the ability to search within the set of pages I currently have open or have visited would be immensely useful. It's (generally) a smaller set than the entire Web, and comprises a set of at least putatively known, familiar, and/or vetted references. I may as well have been writing in Linear A. - Context of references matters a lot to me. A reason I have a huge number of tabs open, in Firefox, using Tree-Style Tabs, is that the arrangement and relationships between tabs (and windows) is itself significant information. This is of course entirely lost in traditional bookmarks. - A classification language for categorising documents would be useful. I've been looking at various of these, including the Library of Congress Subject Headings. A way of automatically mapping 1-6 high-probability subjects to a given reference would be good, as well as, of course, tools for mapping between these. - I've an increasing difference of opinion with the Internet Archive over both the utility and ultimately advisability of saving Web content in precisely the format originally published. Often this is fragile and idiosyncratic. Upconverting to a standardised representation -- say, a strictly semantic, minimal-complexity HTML5, Markdown, or LaTeX, is often superior. Both have their place. On that last, I've been continuing to play with the suggestion a few days ago for a simplified Washington Post article scrubber, and now have a suite of simple scripts which read both WashPo articles and the homepage, fetching links from the homepage for local viewing. These tend to reduce the total page size to about 3-5% of the original, are easier to read than the source, and are much more robust. I'm reading HN at the moment from w3m (which means I've got vim as my comment editor, yay!), and have found that passing the source to pandoc and regenerating HTML from that (scrubbing some elements) is actually much preferable, for the homepage. Discussion pages are ... more difficult to process, and the default view in w3m is unpleasant, though vaguely usable. Upshot: saving a WARC strictly for archival purposes is probably useful, but generating useful formats as noted above would be generally preferable in addition. With the increasing untenability of mainstream Web design and practices, a Rococco catastrophe of mainstream browsers, the emergence of lightweight and alternative browsers and user-agents (though many based on mainstream rendering engines), the tyranny of the minimum viable user attacking any level of online informational access beyond simple push-stream based consumption, and more, it seems that at the very least there's a strongly favourable environment to rethinking what the Web is and what access methods it should support. Peaks in technological complexity tend to lead to a recapitulation phase in which former, simpler, ideas are resurrected, returned to, and become the basis of further development.
• tripzilch 7y ago
  Does this also strip the megabytes of superfluous tracking JS? It's probably what'll be the bulk of the size on the modern web, and I don't feel any particular need to store it. (I believe that for historical purposes, enough complaining about ads and tracking will survive that future historians can easily deduce the existence of this practice)