5 ms·

Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!

by anc84 9y ago

motdiem 9y ago
Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)
- nikisweeting 9y ago
  Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).
  - rcarmo 9y ago
    Well, no, but on a Mac .webarchives are Spotlight indexable and make for a nice single-file archiving approach, and I might actually have some old code that tries to convert between the two...
    - nikisweeting 9y ago
      Oh sweet, that's pretty nice.
      - rcarmo 9y ago
        Also, I've been hacking away at http://github.com/rcarmo/newsfeed-corpus http://github.com/rcarmo/newsfeed-corpus for a bit. Hadn't thought of doing archival on everything, but having this tied in seems like a logical step ;)
  - rcarmo 9y ago
    Oh, and kudos for the awesome GitHub username :)
  - MrRadar 9y ago
    > Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong). I've had good luck with WebArchivePlayer: https://github.com/ikreymer/webarchiveplayer https://github.com/ikreymer/webarchiveplayer
  - zmix 9y ago
    I didn't try myself, but I did a quick search and found this: http://www.archiveteam.org/index.php?title=The_WARC_Ecosystem#WARC_viewer http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...
rcarmo 9y ago
I've been thinking along those very same lines for a long time (this project makes me wish for more free time). I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox https://github.com/internetarchive/warcprox, or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)
- throwaway7767 9y ago
  I've been thinking I'd love to have a WARC archive of all my browsing. So many times sites I remember seeing have gone offline, and didn't get archived by the big services. Ideally this has to happen with browser cooperation, so it can save resources from complex dynamic pages, including responses to user action. This must happen either in the browser or in a proxy like the linked warcprox, in order to catch everything. But the proxy solution is getting less practical every day with key pinning and HSTS. Maybe a future firefox will have an option to export everything to WARC?
- nikisweeting 9y ago
  I would be very on-board with adding a warc exporting option. I also hate how Chrome tosses all history older than 3 months. Running an archiving proxy hooked up to archive.py would kill both birds with one stone.
frik 9y ago
Or EML/MHT. It's the format the email programs use to store the HTML mail incl all pictures, JS, CSS, ... in one plain text file. IE 9-11 also supports that format (file -> save as...) but calls it MHT?

Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!

by anc84 9y ago

Now if only Chromium could learn to write WARC archives, then it would be on par! :) Great project!

Can one automate extensions through headless chrome ? then you might be able to trigger WarCreate instead (It will be more efficient to run the pocket export urls through WAIL though - this should give you the warcs you want)

nikisweeting 9y ago
Yeah, you can use the remote debugger protocol to make JS calls in the context of the page. https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong).
- rcarmo 9y ago
  Well, no, but on a Mac .webarchives are Spotlight indexable and make for a nice single-file archiving approach, and I might actually have some old code that tries to convert between the two...
  - nikisweeting 9y ago
    Oh sweet, that's pretty nice.
    - rcarmo 9y ago
      Also, I've been hacking away at http://github.com/rcarmo/newsfeed-corpus http://github.com/rcarmo/newsfeed-corpus for a bit. Hadn't thought of doing archival on everything, but having this tied in seems like a logical step ;)
- rcarmo 9y ago
  Oh, and kudos for the awesome GitHub username :)
- MrRadar 9y ago
  > Not sure if it's worth including in my script though, since WARCs aren't easily browseable (correct me if I'm wrong). I've had good luck with WebArchivePlayer: https://github.com/ikreymer/webarchiveplayer https://github.com/ikreymer/webarchiveplayer
- zmix 9y ago
  I didn't try myself, but I did a quick search and found this: http://www.archiveteam.org/index.php?title=The_WARC_Ecosystem#WARC_viewer http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...

rcarmo 9y ago

I've been thinking along those very same lines for a long time (this project makes me wish for more free time). I have half a mind to fork this and add something like https://github.com/internetarchive/warcprox https://github.com/internetarchive/warcprox, or at the very least walk through the generated HTML and brute-force inline all assets as a first pass :)

throwaway7767 9y ago
I've been thinking I'd love to have a WARC archive of all my browsing. So many times sites I remember seeing have gone offline, and didn't get archived by the big services. Ideally this has to happen with browser cooperation, so it can save resources from complex dynamic pages, including responses to user action. This must happen either in the browser or in a proxy like the linked warcprox, in order to catch everything. But the proxy solution is getting less practical every day with key pinning and HSTS. Maybe a future firefox will have an option to export everything to WARC?
nikisweeting 9y ago
I would be very on-board with adding a warc exporting option. I also hate how Chrome tosses all history older than 3 months. Running an archiving proxy hooked up to archive.py would kill both birds with one stone.

frik 9y ago

Or EML/MHT. It's the format the email programs use to store the HTML mail incl all pictures, JS, CSS, ... in one plain text file. IE 9-11 also supports that format (file -> save as...) but calls it MHT?