7 ms·

Show HN: Pocket Stream Archive – A personal Way-Back Machine

217 points by nikisweeting 10y ago

toomuchtodo 10y ago
Highly recommend switching to wpull (https://github.com/chfoo/wpull https://github.com/chfoo/wpull), which was built as a wget replacement. It's what grab-site uses, which is a successor to ArchiveTeam's ArchiveBot. "grab-site is made possible only because of wpull, written by Christopher Foo who spent a year making something much better than wget. ArchiveTeam's most pressing issue with wget at the time was that it kept the entire URL queue in memory instead of on disk. wpull has many other advantages over wget, including better link extraction and Python hooks."
- nikisweeting 9y ago
  This looks awesome, thanks for the suggestion! It'll help with WARC support as well, looks like it can output WARCs with just a cli flag.
anotheryou 10y ago
yes! thank you so much. Needed this badly.
- nikisweeting 10y ago
  You're welcome! I've been wanting to build this for ages but headless chrome finally inspired me to actually finish it.
anc84 10y ago
Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!
- motdiem 10y ago
  Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)
  - nikisweeting 10y ago
    Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).
    - rcarmo 10y ago
      Well, no, but on a Mac .webarchives are Spotlight indexable and make for a nice single-file archiving approach, and I might actually have some old code that tries to convert between the two...
      - nikisweeting 10y ago
        Oh sweet, that's pretty nice.
        rcarmo 10y ago
        Also, I've been hacking away at http://github.com/rcarmo/newsfeed-corpus http://github.com/rcarmo/newsfeed-corpus for a bit. Hadn't thought of doing archival on everything, but having this tied in seems like a logical step ;)
    - rcarmo 10y ago
      Oh, and kudos for the awesome GitHub username :)
    - MrRadar 10y ago
      > Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong). I've had good luck with WebArchivePlayer: https://github.com/ikreymer/webarchiveplayer https://github.com/ikreymer/webarchiveplayer
- rcarmo 10y ago
  I've been thinking along those very same lines for a long time (this project makes me wish for more free time). I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox https://github.com/internetarchive/warcprox, or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)
  - throwaway7767 10y ago
    I've been thinking I'd love to have a WARC archive of all my browsing. So many times sites I remember seeing have gone offline, and didn't get archived by the big services. Ideally this has to happen with browser cooperation, so it can save resources from complex dynamic pages, including responses to user action. This must happen either in the browser or in a proxy like the linked warcprox, in order to catch everything. But the proxy solution is getting less practical every day with key pinning and HSTS. Maybe a future firefox will have an option to export everything to WARC?
  - nikisweeting 10y ago
    I would be very on-board with adding a warc exporting option. I also hate how Chrome tosses all history older than 3 months. Running an archiving proxy hooked up to archive.py would kill both birds with one stone.
- frik 10y ago
  Or EML/MHT. It's the format the email programs use to store the HTML mail incl all pictures, JS, CSS, ... in one plain text file. IE 9-11 also supports that format (file -> save as...) but calls it MHT?
avian 10y ago
What version of Google Chrome do you need for the PDF export to work? I tried it on 58.0.3029.96 (Linux) and this does nothing (no error messages, it just quits without writing any files): $ google-chrome --headless --disable-gpu --print-to-pdf 'http://example.com' http://example.com' Edit: I'm completely baffled that such widely used software as Google Chrome can have this written in the man page: "Google Chrome has hundreds of undocumented command-line flags that are added and removed at the whim of the developers."
- nikisweeting 10y ago
  59 or later, --headless is a brand new feature. apt-get install google-chrome-canary. https://developers.google.com/web/updates/2017/04/headless-chrome https://developers.google.com/web/updates/2017/04/headless-c...
- edibleEnergy 10y ago
  This is the only place I've found them parsed and documented: http://peter.sh/experiments/chromium-command-line-switches/ http://peter.sh/experiments/chromium-command-line-switches/
ents 10y ago
Would be cool to see this for Instapaper or Pinboard
- nikisweeting 10y ago
  My script should work with very minimal tweaking if you can get a list of urls + titles from those services. Just one line of regex changes probably: https://github.com/pirate/pocket-archive-stream/blob/master/archive.py#L31 https://github.com/pirate/pocket-archive-stream/blob/master/...
  - no_news_is 10y ago
    Just about what I was hoping to see in the comments, was the expected format of the input. May I suggest including in the readme.md, a sample line of the Pocket export format/your input format? Thanks for publishing!
    - nikisweeting 10y ago
      I included a sample of the expected pocket list format in the repo: https://github.com/pirate/pocket-archive-stream/blob/master/example_ril_export.html https://github.com/pirate/pocket-archive-stream/blob/master/... And a comment next to the regex for parsing it: https://github.com/pirate/pocket-archive-stream/blob/master/archive.py#L31 https://github.com/pirate/pocket-archive-stream/blob/master/...
djhworld 10y ago
This is cool. Could you add an option to either add tagging, or separate the tagged items into folders? e.g. "programming/", "docker/" etc, I often find myself digging through my Pocket archive trying to find that one article I found 6 months ago and it gets incredibly annoying
- nikisweeting 10y ago
  I like having the sites by timestamp because they're guaranteed to be unique, and it makes traversing them easy. I'd be happy to add a tag column to the index though, which you could use with Ctrl+F to find articles. https://github.com/pirate/pocket-archive-stream/issues/1 https://github.com/pirate/pocket-archive-stream/issues/1
rcarmo 10y ago
This is great. All it needs is a Docker container and I'd be running it now (need to take some time aside this weekend to do that).
bicubic 10y ago
This seems neat, curious what are the use cases for this?
- nikisweeting 10y ago
  Slowing down the inevitable tide of https://en.wikipedia.org/wiki/Link_rot https://en.wikipedia.org/wiki/Link_rot. When I cite blog posts or want to share sites that have gone down, I can swap out the links for my archived versions.
  - bicubic 10y ago
    Why is archive.org or one of the other centralized web archives not suitable for that? They don't index the content you want to retain?
    - nikisweeting 10y ago
      Archive.org is a single-point-of-failure, also they don't take PDFs and screenshots of fully-rendered sites with their JS-loaded content.
    - anc84 10y ago
      Never rely on others to care about things you need.
jl6 10y ago
Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.
- nikisweeting 10y ago
  I couldn't agree more. I wish more sites would load without needing multiple seconds of JS execution and AJAX. One of my TODOs is to get full-page screenshots working as well.
  - AznHisoka 10y ago
    in some cases it leads to worse horrible experiences like my Linkedin feed. I have to wait a few seconss everytime before content loads.
  - zulln 10y ago
    > One of my TODOs is to get full-page screenshots working as well. https://medium.com/@dschnr/using-headless-chrome-as-an-automated-screenshot-tool-4b07dffba79a https://medium.com/@dschnr/using-headless-chrome-as-an-autom...
  - tomcooks 10y ago
    You might find this useful if you use Firefox: - shift+f2 - screenshot urcodiaz.png --fullpage
    - Drdrdrq 10y ago
      Thank you for this tip! I have always installed Abduction plugin if I needed this, but the less plugins the better.
- jccalhoun 10y ago
  Agreed. I research new media and archive.org is invaluable to me. I worry that current web sites won't be able to be preserved. (much like many of the flash sites and real audio of the past are largely gone.)
antman 10y ago
The demo does not have images. Maybe try wget -nc -np -E -H -k -K -p -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' -e robots=off
- nikisweeting 10y ago
  I opted not to download images using wget. I figured if I needed in-article images the PDF+screenshot would be enough.
fiatjaf 10y ago
You see something is flawed in Redux at the point you have to pass strings (uppercase constants defined somewhere) around, import them in every file, pass them as identifiers of what you should do with each data. Strings!
- nikisweeting 10y ago
  Did you comment on the wrong article by accident? https://news.ycombinator.com/item?id=14273549 https://news.ycombinator.com/item?id=14273549
  - fiatjaf 10y ago
    Yes! Thank you.
unicornporn 10y ago
If it would be connected to Pinboard.in as an alternative to Pocket I would be screaming with joy. :-)
- joshstrange 10y ago
  Shouldn't be too terribly difficult to modify since all Pocket does is provide a list of URL's same as pinboard.
- nikisweeting 10y ago
  If you can get me a sample pinboard export to look at, I'll whip up a regex that makes it work.
  - unicornporn 10y ago
    Oh, thanks! There's three backup formats: XML (same as Delicious v1 API), HTML (legacy Netscape format almost everyone can read) and JSON. I can get back with an XML file later. There's also an API that might be of interest. Here's someone who blogged about that: http://behindcompanies.com/2011/12/a-guide-to-backing-up-pinboard/ http://behindcompanies.com/2011/12/a-guide-to-backing-up-pin... The question was asked if you can use your API endpoint URL (https://[pinboardusername]:[pinboardpassword]@api.pinboard.in/v1/posts/all?format=json https://[pinboardusername]:[pinboardpassword]@api.pinboard.i...) straight into ifttt. Maciej confirmed that it should work, the problem is that you’re essentially storing your login credentials in a 3rd party service, and you don’t know if they’re storing and transmitting it securely.
  - unicornporn 10y ago
    Here's what the XML files look like: https://gist.github.com/anonymous/316b9fd1108b026e116e91f13447bc8a https://gist.github.com/anonymous/316b9fd1108b026e116e91f134...
- 8ig8 10y ago
  Pinboard offers an archive feature: https://pinboard.in/upgrade/ https://pinboard.in/upgrade/
- 10y ago
vijucat 10y ago
I still use Firefox ScrapBook for this: https://addons.mozilla.org/en-US/firefox/addon/scrapbook/ https://addons.mozilla.org/en-US/firefox/addon/scrapbook/ Just for articles, mind you, not entire websites.
- cJ0th 10y ago
  Ditto. I just hope the transition to web extension will be smooth or at least happening at all.
- deleted 10y ago
  [deleted]
burnbabyburn 10y ago
this is really cool! I always had in mind a project where you save every page you visit, and somehow expose them in the future to know what you visit and maybe remembering you important stuff based on some heuristic.
throw98987 10y ago
Use zotero and you have your own personal Pocket with snapshots. In addition, you can add tags, organize stuff into folders, etc. https://www.zotero.org/ https://www.zotero.org/
- nikisweeting 10y ago
  Zotero is awesome! It doesn't provide a publishable stream of recently added articles though afaik.
  - bovine3dom 10y ago
    You can, if you faff a little bit [0]. Unless you mean something else by publishable? [0] https://www.zotero.org/support/groups#public_open_membership https://www.zotero.org/support/groups#public_open_membership
ticoombs 10y ago
I've been running wallabag[1] for my own pocket instance. It's been running perfectly for a couple years. Also has a pocket import feature. [1] https://wallabag.org/en https://wallabag.org/en
arkenflame 10y ago
I wrote a Chrome extension that similarly saves copies of pages you bookmark: https://chrome.google.com/webstore/detail/backmark-back-up-the-page/cmbflafdbcidlkkdhbmechbcpmnbcfjf https://chrome.google.com/webstore/detail/backmark-back-up-t...

217 points by nikisweeting 10y ago

Highly recommend switching to wpull (https://github.com/chfoo/wpull https://github.com/chfoo/wpull), which was built as a wget replacement. It's what grab-site uses, which is a successor to ArchiveTeam's ArchiveBot. "grab-site is made possible only because of wpull, written by Christopher Foo who spent a year making something much better than wget. ArchiveTeam's most pressing issue with wget at the time was that it kept the entire URL queue in memory instead of on disk. wpull has many other advantages over wget, including better link extraction and Python hooks."

nikisweeting 9y ago
This looks awesome, thanks for the suggestion! It'll help with WARC support as well, looks like it can output WARCs with just a cli flag.

anotheryou 10y ago

yes! thank you so much. Needed this badly.

nikisweeting 10y ago
You're welcome! I've been wanting to build this for ages but headless chrome finally inspired me to actually finish it.

anc84 10y ago

Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!

motdiem 10y ago
Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)
- nikisweeting 10y ago
  Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).
  - rcarmo 10y ago
    Well, no, but on a Mac .webarchives are Spotlight indexable and make for a nice single-file archiving approach, and I might actually have some old code that tries to convert between the two...
    - nikisweeting 10y ago
      Oh sweet, that's pretty nice.
      - rcarmo 10y ago
        Also, I've been hacking away at http://github.com/rcarmo/newsfeed-corpus http://github.com/rcarmo/newsfeed-corpus for a bit. Hadn't thought of doing archival on everything, but having this tied in seems like a logical step ;)
  - rcarmo 10y ago
    Oh, and kudos for the awesome GitHub username :)
  - MrRadar 10y ago
    > Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong). I've had good luck with WebArchivePlayer: https://github.com/ikreymer/webarchiveplayer https://github.com/ikreymer/webarchiveplayer
rcarmo 10y ago
I've been thinking along those very same lines for a long time (this project makes me wish for more free time). I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox https://github.com/internetarchive/warcprox, or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)
- throwaway7767 10y ago
  I've been thinking I'd love to have a WARC archive of all my browsing. So many times sites I remember seeing have gone offline, and didn't get archived by the big services. Ideally this has to happen with browser cooperation, so it can save resources from complex dynamic pages, including responses to user action. This must happen either in the browser or in a proxy like the linked warcprox, in order to catch everything. But the proxy solution is getting less practical every day with key pinning and HSTS. Maybe a future firefox will have an option to export everything to WARC?
- nikisweeting 10y ago
  I would be very on-board with adding a warc exporting option. I also hate how Chrome tosses all history older than 3 months. Running an archiving proxy hooked up to archive.py would kill both birds with one stone.
frik 10y ago
Or EML/MHT. It's the format the email programs use to store the HTML mail incl all pictures, JS, CSS, ... in one plain text file. IE 9-11 also supports that format (file -> save as...) but calls it MHT?

avian 10y ago

What version of Google Chrome do you need for the PDF export to work? I tried it on 58.0.3029.96 (Linux) and this does nothing (no error messages, it just quits without writing any files): $ google-chrome --headless --disable-gpu --print-to-pdf 'http://example.com' http://example.com' Edit: I'm completely baffled that such widely used software as Google Chrome can have this written in the man page: "Google Chrome has hundreds of undocumented command-line flags that are added and removed at the whim of the developers."

nikisweeting 10y ago
59 or later, --headless is a brand new feature. apt-get install google-chrome-canary. https://developers.google.com/web/updates/2017/04/headless-chrome https://developers.google.com/web/updates/2017/04/headless-c...
edibleEnergy 10y ago
This is the only place I've found them parsed and documented: http://peter.sh/experiments/chromium-command-line-switches/ http://peter.sh/experiments/chromium-command-line-switches/

ents 10y ago

Would be cool to see this for Instapaper or Pinboard

nikisweeting 10y ago
My script should work with very minimal tweaking if you can get a list of urls + titles from those services. Just one line of regex changes probably: https://github.com/pirate/pocket-archive-stream/blob/master/archive.py#L31 https://github.com/pirate/pocket-archive-stream/blob/master/...
- no_news_is 10y ago
  Just about what I was hoping to see in the comments, was the expected format of the input. May I suggest including in the readme.md, a sample line of the Pocket export format/your input format? Thanks for publishing!
  - nikisweeting 10y ago
    I included a sample of the expected pocket list format in the repo: https://github.com/pirate/pocket-archive-stream/blob/master/example_ril_export.html https://github.com/pirate/pocket-archive-stream/blob/master/... And a comment next to the regex for parsing it: https://github.com/pirate/pocket-archive-stream/blob/master/archive.py#L31 https://github.com/pirate/pocket-archive-stream/blob/master/...

djhworld 10y ago

This is cool. Could you add an option to either add tagging, or separate the tagged items into folders? e.g. "programming/", "docker/" etc, I often find myself digging through my Pocket archive trying to find that one article I found 6 months ago and it gets incredibly annoying

nikisweeting 10y ago
I like having the sites by timestamp because they're guaranteed to be unique, and it makes traversing them easy. I'd be happy to add a tag column to the index though, which you could use with Ctrl+F to find articles. https://github.com/pirate/pocket-archive-stream/issues/1 https://github.com/pirate/pocket-archive-stream/issues/1

rcarmo 10y ago

This is great. All it needs is a Docker container and I'd be running it now (need to take some time aside this weekend to do that).

bicubic 10y ago

This seems neat, curious what are the use cases for this?

nikisweeting 10y ago
Slowing down the inevitable tide of https://en.wikipedia.org/wiki/Link_rot https://en.wikipedia.org/wiki/Link_rot. When I cite blog posts or want to share sites that have gone down, I can swap out the links for my archived versions.
- bicubic 10y ago
  Why is archive.org or one of the other centralized web archives not suitable for that? They don't index the content you want to retain?
  - nikisweeting 10y ago
    Archive.org is a single-point-of-failure, also they don't take PDFs and screenshots of fully-rendered sites with their JS-loaded content.
  - anc84 10y ago
    Never rely on others to care about things you need.

jl6 10y ago

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

nikisweeting 10y ago
I couldn't agree more. I wish more sites would load without needing multiple seconds of JS execution and AJAX. One of my TODOs is to get full-page screenshots working as well.
- AznHisoka 10y ago
  in some cases it leads to worse horrible experiences like my Linkedin feed. I have to wait a few seconss everytime before content loads.
- zulln 10y ago
  > One of my TODOs is to get full-page screenshots working as well. https://medium.com/@dschnr/using-headless-chrome-as-an-automated-screenshot-tool-4b07dffba79a https://medium.com/@dschnr/using-headless-chrome-as-an-autom...
- tomcooks 10y ago
  You might find this useful if you use Firefox: - shift+f2 - screenshot urcodiaz.png --fullpage
  - Drdrdrq 10y ago
    Thank you for this tip! I have always installed Abduction plugin if I needed this, but the less plugins the better.
jccalhoun 10y ago
Agreed. I research new media and archive.org is invaluable to me. I worry that current web sites won't be able to be preserved. (much like many of the flash sites and real audio of the past are largely gone.)

antman 10y ago

The demo does not have images. Maybe try wget -nc -np -E -H -k -K -p -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' -e robots=off

nikisweeting 10y ago
I opted not to download images using wget. I figured if I needed in-article images the PDF+screenshot would be enough.

fiatjaf 10y ago

You see something is flawed in Redux at the point you have to pass strings (uppercase constants defined somewhere) around, import them in every file, pass them as identifiers of what you should do with each data. Strings!

nikisweeting 10y ago
Did you comment on the wrong article by accident? https://news.ycombinator.com/item?id=14273549 https://news.ycombinator.com/item?id=14273549
- fiatjaf 10y ago
  Yes! Thank you.

unicornporn 10y ago

If it would be connected to Pinboard.in as an alternative to Pocket I would be screaming with joy. :-)

joshstrange 10y ago
Shouldn't be too terribly difficult to modify since all Pocket does is provide a list of URL's same as pinboard.
nikisweeting 10y ago
If you can get me a sample pinboard export to look at, I'll whip up a regex that makes it work.
- unicornporn 10y ago
  Oh, thanks! There's three backup formats: XML (same as Delicious v1 API), HTML (legacy Netscape format almost everyone can read) and JSON. I can get back with an XML file later. There's also an API that might be of interest. Here's someone who blogged about that: http://behindcompanies.com/2011/12/a-guide-to-backing-up-pinboard/ http://behindcompanies.com/2011/12/a-guide-to-backing-up-pin... The question was asked if you can use your API endpoint URL (https://[pinboardusername]:[pinboardpassword]@api.pinboard.in/v1/posts/all?format=json https://[pinboardusername]:[pinboardpassword]@api.pinboard.i...) straight into ifttt. Maciej confirmed that it should work, the problem is that you’re essentially storing your login credentials in a 3rd party service, and you don’t know if they’re storing and transmitting it securely.
- unicornporn 10y ago
  Here's what the XML files look like: https://gist.github.com/anonymous/316b9fd1108b026e116e91f13447bc8a https://gist.github.com/anonymous/316b9fd1108b026e116e91f134...
8ig8 10y ago
Pinboard offers an archive feature: https://pinboard.in/upgrade/ https://pinboard.in/upgrade/
10y ago

vijucat 10y ago

I still use Firefox ScrapBook for this: https://addons.mozilla.org/en-US/firefox/addon/scrapbook/ https://addons.mozilla.org/en-US/firefox/addon/scrapbook/ Just for articles, mind you, not entire websites.

cJ0th 10y ago
Ditto. I just hope the transition to web extension will be smooth or at least happening at all.
deleted 10y ago
[deleted]

burnbabyburn 10y ago

this is really cool! I always had in mind a project where you save every page you visit, and somehow expose them in the future to know what you visit and maybe remembering you important stuff based on some heuristic.

throw98987 10y ago

Use zotero and you have your own personal Pocket with snapshots. In addition, you can add tags, organize stuff into folders, etc. https://www.zotero.org/ https://www.zotero.org/

nikisweeting 10y ago
Zotero is awesome! It doesn't provide a publishable stream of recently added articles though afaik.
- bovine3dom 10y ago
  You can, if you faff a little bit [0]. Unless you mean something else by publishable? [0] https://www.zotero.org/support/groups#public_open_membership https://www.zotero.org/support/groups#public_open_membership

ticoombs 10y ago

I've been running wallabag[1] for my own pocket instance. It's been running perfectly for a couple years. Also has a pocket import feature. [1] https://wallabag.org/en https://wallabag.org/en

arkenflame 10y ago

I wrote a Chrome extension that similarly saves copies of pages you bookmark: https://chrome.google.com/webstore/detail/backmark-back-up-the-page/cmbflafdbcidlkkdhbmechbcpmnbcfjf https://chrome.google.com/webstore/detail/backmark-back-up-t...