3 ms·
The claims are impressive: that the network requests are saved, not the document after the page is loaded. I will certainly give it a try! One downside: only w
by danuker 5y ago
The claims are impressive: that the network requests are saved, not the document after the page is loaded. I will certainly give it a try!
One downside: only works with Chrome-based browsers. Perhaps a way to implement this more generally is as a proxy server.
- 12907835202 5y agoHow does it work with a JS heavy app? With an SPA the article might not appear on initial request, but later requests would be JSON which presumably you don't want to return in a search
- graderjs 5y agoIt doesn't index each resource. Only the page text of the URLs you actually navigate to in a tab. So this is fine. It won't index a JSON response in an SPA, as search result (tho it does save it), but it will index (and re index) the actual page content, as you browse it and as it updates (eg dynamically loads in an SPA). Try it out, you'll have an easier time understanding how it works (as long as the current tag release works, heh :) if not try a previous tag).
- mananaysiempre 5y agoThe problem with a proxy server implementation is modern browsers (AFAIK) are unwilling to submit HTTPS requests in the clear to a proxy server rather than use CONNECT. There’s nothing about the protocol that would make that impossible or even inconvenient, browsers are just unwilling to do it. And I can see the reasoning, but if you actually want this to happen, like here, you’re stuck (or have to MITM yourself, which is its own can of worms).
- mdaniel 5y agoThat's the whole purpose behind ZAP and I use it for archiving pages all the time (they use hsqldb as the file format); it works fantastic for that purpose, but does -- as you correctly pointed out -- require MITM-ing the browser to trust their locally generated CA: https://github.com/zaproxy/zaproxy#readme https://github.com/zaproxy/zaproxy#readme
- mananaysiempre 5y agoThanks for the reference! I never investigated ZAP closely, for some reason it never occured to me it might be able to be used like that (if anything I’d have turned to mitmproxy, but that would require building a substantial amount of stuff to handle the actual archiving). The problem with MITMing your own browser is (apart from the fact that it is an ugly hack in a security-critical portion of your setup) I don’t think any tool for doing that (including the one you referenced, from what I can find quickly) applies the complex set of important stuff browsers do on top of just verifying chains against a root store. The bare minimum for me would be HSTS and the HSTS preload list, but I’d also like to see CT and Must-Staple enforcement, OneCRL support, TLD and validity term restrictions for some roots, and so on. (This is more or less what Chrome does from what I know, though I think they have their own equivalent to Mozilla’s OneCRL.)
- mdaniel 5y agoGiven what ZAP is designed to do, I'd bet $1 it will actively strip off any such headers before returning the response to the browser, since I think by definition injecting a MITM cert is the very case such stapling is designed to prevent against :-) But, corporate proxies must face similar problems since they, too, MITM things, but I'm deeply thankful that I don't work in such an environment in order to know what the behavior is in that circumstance I hope this doesn't come across as glib, but ZAP is Apache licensed, so if you are able to come up with the security behavior you want, I'd bet they'd welcome any patches to help implement it