5 ms·

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solve

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all. The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of this sort. Unfortunately, the most popular browser in the world is controlled by the same people who control our protocols. If I ever get the energy, I will attempt to write a browser extension for this.

fooker 7y ago
wget -r ?
- gambler 7y ago
  No. There are some cases where it is useful to download many pages in a batch, but what I am talking about is, effectively, partial local replication. Bookmarking I describe should create a tiny (but useful) version of the web on your computer. It needs to be seamless. It needs to be searchable. It would be incredibly useful if it would capture relationships between pages (links) in addition to pages themselves to navigate offline. I can see many use cases for a kind of "super bookmark mode" where the browser automatically stores all the pages you visit within a certain domain.
  - deleted 7y ago
    [deleted]
  - kevin_thibedeau 7y ago
    You can't seamlessly replicate a REST server.
  - idle_zealot 7y ago
    HTTrack does most of that, minus "seamless". You could probably write an extension that automatically sent bookmarked urls to HTTrack for mirroring/archival, then they would all be browsable offline.
  - asdff 7y ago
    I would loooove this. Open one of your bookmarks and you can't tell it's the local copy unless you looked at the local path sitting in the URL bar. Maybe make an option to fetch a live copy though for sites like the front page of HN that are in constant turnover, but it would be great. Storage has never been cheaper, let me hoard! You can even take it a step further and create a viewable timeline of all the pages you've visited in case you have something at the tip of your tongue, but need to retrace your steps and logic to get there. Browsing history is kinda lackluster for this. My last ten entries on firefox are three hackernews articles with "Add Comment" dispersed half a dozen times in that list. If I had something along the lines of zoteros timeline for papers I could probably find stuff way easier.
- mwest 7y ago
  Don't forget --warc! :-) https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive
donmcronald 7y ago
https://historio.us https://historio.us AFAIK there’s no self hosted equivalent. Ive been using it for years. Whenever I put a link in my docs / notes I use that site to cache a copy as well. When I first signed up I wrote a script to convert my bookmarks to a text list so I could import them. Around 25% of the pages I’d bookmarked were gone.
fibonachos 7y ago
Tangentially related, but the reader view of a page seems ideally suited for archival bookmarking. It's unfortunate that so many sites seem to be intentionally breaking the reader view functionality in order to maximize newsletter subscriptions and eyeball monetization. Recipe sites are the worst offenders IMO.
- asdff 7y ago
  Pocket seems to have luck where other reader views and outlining services fail, and you can set up IFTTT to download pocket articles as html files to dropbox, but that's certainly not automatic and completely out of your way. I would have preferred it to archive as a plain text document but I couldn't figure it out through IFTTT.
  - dredmorbius 7y ago
    Are the downloaded pages in original or Pocket/Reader view? You can export (with some pain) your Pocket list. Fetching sources from there is ... somewhat painful. I've done it a couple of times, but it's not something I relish doing.
  - teknico 7y ago
    Pocket Premium stores the web pages on a server. If there was a way to batch export and access them locally, I'd gladly pay for it.
fwip 7y ago
Check out Beaker Browser. Solves this problem on the protocol level and bookmarking automatically stores a copy.
basch 7y ago
Best use of time and money is probably http://pinboard.in/ http://pinboard.in/ ? Seems like someone I can trust not to sell out and shut down.
- ianlevesque 7y ago
  But if they die?
  - nine_k 7y ago
    ...or just run out of money? Storage must be local, physicslly user-controlled. All online stuff is a backup only.
    - rjf72 7y ago
      On top of this there is currently a push for a mix of censorship and 'compulsory removal' on the internet. For instance the Canadian Supreme Court ruling Canada can force Google to remove worldwide results for something, the UK censoring increasingly arbitrary things, or various 'right to be forgotten' type laws. And of course there are also intellectual property lobbyists and special interests pushing for ever more draconian rules on that front. Any centralized online site that aggregates other data has an, at best, unclear future ahead of it. Really though, centralized is the key word. The internet so desperately needs to migrate to decentralization but we're collectively about as inertial as an 882 foot long, 46,328 ton tanker, built in 1909.
    - basch 7y ago
      why cant backup be local, user controlled? revenue looks like hes doing just fine. https://blog.pinboard.in/2017/07/eight_years_of_victory/ https://blog.pinboard.in/2017/07/eight_years_of_victory/ https://news.ycombinator.com/user?id=idlewords https://news.ycombinator.com/user?id=idlewords
papln 7y ago
Saving data locally is not a protocol problem. Chromium is almost there. Chrome had “chrome://flags/#show-saved-copy” which would do you want if you could make Chrome's cache arbitrarily large and persistent, but it appears to be gone now.
CGamesPlay 7y ago
Shameless plug: I built an app for that. It’s not super polished but it is at a point where I use it to store technical documentation that doesn’t ship with the software (aka docs for every Javascript package I use). It works for moderately complex JavaScript apps as well. https://github.com/CGamesPlay/chronicler https://github.com/CGamesPlay/chronicler
grincho 7y ago
Check out https://blog.webmemex.org/ https://blog.webmemex.org/. It serializes pages, inlining their external resources, full-text indexes them, and saves them locally. Browser extensions are available for Firefox and Chrome.
thecabinet 7y ago
I’ve been telling people this for years. Bookmarking is for dynamic data. You bookmark the weather website. You don’t bookmark recipes. Personally I use Evernote and its web clipper. Unfortunately it doesn’t download images, but I wrote a little program using their API to replace the img tags with embedded resources.
Torwald 7y ago
Maybe I don't understand what you are saying, but I safe webpages all the time. Safari the creates a .webarchive file with everything in it. Isn't that what you want?
- kwhitefoot 7y ago
  No. Because that's extra work. It should just be automatically done whenever I bookmark a page. In fact with disks so big now I could just dedicate a terabyte to just caching everything I visit. Make that indexed and searchable and a lot of what I use would be forever available offline. Pages could expire automatically if unvisited for some period of time, perhaps weighted by the number of times they were visited. Perhaps I should try to resurrect my rusty and incomplete Firefox add-on skills.
thescriptkiddie 7y ago
You're describing IPFS pinning. https://docs.ipfs.io/guides/concepts/pinning/ https://docs.ipfs.io/guides/concepts/pinning/
tobias2014 7y ago
Use the wayback machine browser plugin allowing you to conveniently use the "save now" function which immediately saves the website to archive.org https://archive.org/web/ https://archive.org/web/
eitland 7y ago
There used to be a Firefox extension called Scrapbook. I think it was one of the winners of one of the two first extension competitions. It could easily download entire sites or just a subtree, including resources. It would the rewrite URLs as necessary to point to the downloaded documents instead of pointing to absolute URLs on tje Internet. Still hoping that Mozilla will improve its extension API. I mean I can see the reason for not allowing everyone to pone around everywhere under the hood, but the current API is rather anemic it seems.
- teknico 7y ago
  Been using it for so long. Currently it's Scrapbook X under Pale Moon (that I keep around just to run that). Looking for an alternative that would import my thousands of saved pages.

fooker 7y ago

wget -r ?

gambler 7y ago
No. There are some cases where it is useful to download many pages in a batch, but what I am talking about is, effectively, partial local replication. Bookmarking I describe should create a tiny (but useful) version of the web on your computer. It needs to be seamless. It needs to be searchable. It would be incredibly useful if it would capture relationships between pages (links) in addition to pages themselves to navigate offline. I can see many use cases for a kind of "super bookmark mode" where the browser automatically stores all the pages you visit within a certain domain.
- deleted 7y ago
  [deleted]
- kevin_thibedeau 7y ago
  You can't seamlessly replicate a REST server.
- idle_zealot 7y ago
  HTTrack does most of that, minus "seamless". You could probably write an extension that automatically sent bookmarked urls to HTTrack for mirroring/archival, then they would all be browsable offline.
- asdff 7y ago
  I would loooove this. Open one of your bookmarks and you can't tell it's the local copy unless you looked at the local path sitting in the URL bar. Maybe make an option to fetch a live copy though for sites like the front page of HN that are in constant turnover, but it would be great. Storage has never been cheaper, let me hoard! You can even take it a step further and create a viewable timeline of all the pages you've visited in case you have something at the tip of your tongue, but need to retrace your steps and logic to get there. Browsing history is kinda lackluster for this. My last ten entries on firefox are three hackernews articles with "Add Comment" dispersed half a dozen times in that list. If I had something along the lines of zoteros timeline for papers I could probably find stuff way easier.
mwest 7y ago
Don't forget --warc! :-) https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive

donmcronald 7y ago

https://historio.us https://historio.us AFAIK there’s no self hosted equivalent. Ive been using it for years. Whenever I put a link in my docs / notes I use that site to cache a copy as well. When I first signed up I wrote a script to convert my bookmarks to a text list so I could import them. Around 25% of the pages I’d bookmarked were gone.

fibonachos 7y ago

Tangentially related, but the reader view of a page seems ideally suited for archival bookmarking. It's unfortunate that so many sites seem to be intentionally breaking the reader view functionality in order to maximize newsletter subscriptions and eyeball monetization. Recipe sites are the worst offenders IMO.

asdff 7y ago
Pocket seems to have luck where other reader views and outlining services fail, and you can set up IFTTT to download pocket articles as html files to dropbox, but that's certainly not automatic and completely out of your way. I would have preferred it to archive as a plain text document but I couldn't figure it out through IFTTT.
- dredmorbius 7y ago
  Are the downloaded pages in original or Pocket/Reader view? You can export (with some pain) your Pocket list. Fetching sources from there is ... somewhat painful. I've done it a couple of times, but it's not something I relish doing.
- teknico 7y ago
  Pocket Premium stores the web pages on a server. If there was a way to batch export and access them locally, I'd gladly pay for it.

fwip 7y ago

Check out Beaker Browser. Solves this problem on the protocol level and bookmarking automatically stores a copy.

basch 7y ago

Best use of time and money is probably http://pinboard.in/ http://pinboard.in/ ? Seems like someone I can trust not to sell out and shut down.

ianlevesque 7y ago
But if they die?
- nine_k 7y ago
  ...or just run out of money? Storage must be local, physicslly user-controlled. All online stuff is a backup only.
  - rjf72 7y ago
    On top of this there is currently a push for a mix of censorship and 'compulsory removal' on the internet. For instance the Canadian Supreme Court ruling Canada can force Google to remove worldwide results for something, the UK censoring increasingly arbitrary things, or various 'right to be forgotten' type laws. And of course there are also intellectual property lobbyists and special interests pushing for ever more draconian rules on that front. Any centralized online site that aggregates other data has an, at best, unclear future ahead of it. Really though, centralized is the key word. The internet so desperately needs to migrate to decentralization but we're collectively about as inertial as an 882 foot long, 46,328 ton tanker, built in 1909.
  - basch 7y ago
    why cant backup be local, user controlled? revenue looks like hes doing just fine. https://blog.pinboard.in/2017/07/eight_years_of_victory/ https://blog.pinboard.in/2017/07/eight_years_of_victory/ https://news.ycombinator.com/user?id=idlewords https://news.ycombinator.com/user?id=idlewords

papln 7y ago

Saving data locally is not a protocol problem. Chromium is almost there. Chrome had “chrome://flags/#show-saved-copy” which would do you want if you could make Chrome's cache arbitrarily large and persistent, but it appears to be gone now.

CGamesPlay 7y ago

Shameless plug: I built an app for that. It’s not super polished but it is at a point where I use it to store technical documentation that doesn’t ship with the software (aka docs for every Javascript package I use). It works for moderately complex JavaScript apps as well. https://github.com/CGamesPlay/chronicler https://github.com/CGamesPlay/chronicler

grincho 7y ago

Check out https://blog.webmemex.org/ https://blog.webmemex.org/. It serializes pages, inlining their external resources, full-text indexes them, and saves them locally. Browser extensions are available for Firefox and Chrome.

thecabinet 7y ago

I’ve been telling people this for years. Bookmarking is for dynamic data. You bookmark the weather website. You don’t bookmark recipes. Personally I use Evernote and its web clipper. Unfortunately it doesn’t download images, but I wrote a little program using their API to replace the img tags with embedded resources.

Torwald 7y ago

Maybe I don't understand what you are saying, but I safe webpages all the time. Safari the creates a .webarchive file with everything in it. Isn't that what you want?

kwhitefoot 7y ago
No. Because that's extra work. It should just be automatically done whenever I bookmark a page. In fact with disks so big now I could just dedicate a terabyte to just caching everything I visit. Make that indexed and searchable and a lot of what I use would be forever available offline. Pages could expire automatically if unvisited for some period of time, perhaps weighted by the number of times they were visited. Perhaps I should try to resurrect my rusty and incomplete Firefox add-on skills.

thescriptkiddie 7y ago

You're describing IPFS pinning. https://docs.ipfs.io/guides/concepts/pinning/ https://docs.ipfs.io/guides/concepts/pinning/

tobias2014 7y ago

Use the wayback machine browser plugin allowing you to conveniently use the "save now" function which immediately saves the website to archive.org https://archive.org/web/ https://archive.org/web/

eitland 7y ago

There used to be a Firefox extension called Scrapbook. I think it was one of the winners of one of the two first extension competitions. It could easily download entire sites or just a subtree, including resources. It would the rewrite URLs as necessary to point to the downloaded documents instead of pointing to absolute URLs on tje Internet. Still hoping that Mozilla will improve its extension API. I mean I can see the reason for not allowing everyone to pone around everywhere under the hood, but the current API is rather anemic it seems.

teknico 7y ago
Been using it for so long. Currently it's Scrapbook X under Pale Moon (that I keep around just to run that). Looking for an alternative that would import my thousands of saved pages.