4 ms·
I have spent the last hours reading up on everything-WARC that I could find but I still haven't been able to answer my main question: why only as external crawl
by aloer 8y ago
I have spent the last hours reading up on everything-WARC that I could find but I still haven't been able to answer my main question: why only as external crawlers?
There does not seem to be a tool to actually capture a warc directly in your own browser session. webrecorder (http://webrecorder.io/ http://webrecorder.io/) is the only example I could find that comes close in terms of user experience but it still requires a third party and different browsing habits
- are there browser extensions that can save a warc while you browse?
- are there API limitations that require external browser control? something browser extensions can't be used for?
- or is it simply a question of use case. And crawlers are more popular (for archiving) than locally recorded browser history (for search/analytics)?
edit:
I have now found https://github.com/machawk1/warcreate https://github.com/machawk1/warcreate, related discussions in issues #111 and #112 are quite interesting. Looks like there are some serious limitations for browser extensions. I will look deeper into how webrecorder works and how this could be combined
- EamonnMR 8y agoDefinitely report back if you find anything, I would be very interested.
- machawk1 8y agoWhen I initially coded up WARCreate, the webRequest API was still experimental. I believe there are more mature APIs that can be used from the extension context but some require DevTools to be visually open, which is not a common usage pattern of a typical web user. Per the ticket, we have worked on a few other JavaScript-driven web preservation project like https://github.com/N0taN3rd/node-warc https://github.com/N0taN3rd/node-warc and https://github.com/N0taN3rd/Squidwarc https://github.com/N0taN3rd/Squidwarc, among others. Web archiving from a browser extension is difficult but can be improved. I don't know of any other approaches at trying to do this via a browser extension beyond submitting a URI to an archive.
- nikisweeting 8y agoWoah both these tools are awesome, thanks for sharing them! I'm adding links to both from https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm...